 And it should be live welcome to the homelab show episode. What is the episode number again? We have to restore from backup that's Yeah, and I think my I think my brain is suspended and is slowly waking up in the morning as the caffeine takes effect But you know we have a podcast and we're on episode 9. This is the homelab show I'm Jay LeCroy and this is Tom Lawrence and Raid is not a backup. That is what we titled this show to be Raid is resiliency. I like to get that out there right away Catastrophic failures if you're in tech long enough you will not only witness one But you'll witness many and epic and catastrophic might even be more descriptive We've seen power surges fry arrays and boards and everything else So you can't just say hey, it's not a bunch of discs. Isn't it fine? And I always feel terrible when people lose data So I like to do as much as I can because it's the back end of the conversation is so hard to have You know not that it's as serious as an actual medical condition, but you know that that moment when the doctor walks out and say I have bad news No one really wants to deliver it like that, but I have bad news sometimes all those drives are melted The controller boards are destroyed. This these are gonna be super expensive or maybe even impossible To recover so the goal of this is to once you've built out you put all that effort into your home lab How to get all that data somewhere else whether you know It's your photos and photos are a big piece of this a lot of people You know fair and rightly so because of privacy concerns start saving all their own photos and this puts you at risk You know to make sure you okay cool. I backed up all the kids photos, right? Well, no, I took them on my phone or my camera moved them to a raid array and now it's backed up And that is where the misconceptions begin. I think this is one of those situations where There's no one right way to handle this There's plenty of wrong ways though like we could clearly define the wrong ways of handling this But how you handle it if you follow certain criteria Whatever works best for you within your budget and your capabilities is fine if it offers the You know true actual redundancy. I don't really care what service you use and we're gonna talk about Several aspects of this in this episode, but I don't want this to be like these are the only things that you should ever consider Thing here is that well There's all kinds of stuff out there to try of varying levels of quality and there's pros and cons to everything so ultimately You know, I don't care if people agree with me because there's you know, there's pros and cons I've mentioned good things about things. There's more than one right way to back up one thing That's important though an untested backup is just wishful thinking At the end of all this test backups test doing the restore So apply that to any product we talk about any process we talk about here always think about that piece of it Many I need to underscore that because I have seen plenty of times where someone has a really good backup strategy Where I'm looking at it like that's impressive you really thought about everything you have a clear system here You know, that's awesome, but they didn't test it and then it can't restore I mean you could back up garbage you could back up good data. It is what it is I mean sure we'll talk about it later like, you know, it can become garbage But what you have data that's backed up. It's either going to be useful back up or not and what as an aside one of the aspects of this is that This is a topic that Tom and I felt we definitely need to discuss and we're getting very close to Running out of high-level things to talk about Where we can start deep diving into individual technologies, which I'm Especially looking forward to but we kind of feel like it's really great If someone can just watch this or listen to this podcast from the you know episode zero on up Everything builds on the on the next the topics that we cover A lot of this stuff doesn't change but we add new stuff as we go along So I think backups are a very important foundational topic and then we can start deep diving which is going to be especially fun for sure that is a Definitely when we start deep diving into it We are going to do a lot of those singular topics where we dive into a very specific product and go deep We just wanted to get these first it looks like the you know This many episodes covers the broad topics But then again in the future there may be more broad topic episodes But don't worry we're not running out of things or ideas matter of fact The hardest part is landing them because we have such a large idea board going on between me and Jay is figuring out Which ones we want to cover Like every time we do an episode I think of like five things Additional that I'd like to talk about but then it's like well Yeah, we kind of need to cover some foundational things before we can talk about that Like I really wanted to talk about Ansible and I will and full detail in an episode I don't know which one But we covered it and we talked about it from a high level and I think it's time when we circle back around to it Let's talk about it in more detail So someone can understand is this a good fit for me Is this something that I want to do is we'll talk about the pros and cons And that'll enable our audience to make the decision on if this technology we're covering in that particular episode Resonates with them and their use case and if it does they'll have all the details they need to get started with it and One thing I think we should really kind of talk about Before we get too far into the topic is why raid is not a backup in particular because there are people Out there that do feel that way and I could come up with a lot of reasons. I'm going to give you some of my More important ones in my opinion I think one of the reasons that I think it's very important to not consider raid backup is The fact that well, I guess actually we should probably talk about what raid is I know everyone is you know probably knows that are the majority Basically, you have more than one hard drive and you could have them, you know mirrored So whatever is on the first drive is automatically on the other so the first drive fails, you know You still have the second you could do and there's other levels of raid, you know raid zero Zero redundancy. That's how I remember that one. Yeah, I'd never use it I don't see in that use case anymore by combining drives that if one dies, you know, you lose what's on both Raid one is a mirror, you know, what's on disc one is on disc two So if your main hard drive dies, you could just boot from the second you have you have raid five for example You know three or more drives one can die because there's one disc worth of parity meaning if you have three 500 gig drives, you don't have you know, 1500 gigabytes. You have a terabyte You have a thousand gigabytes essentially, um, I know I'm a little off because of you know the 1024 thing but but you can lose a drive and then with raid six you can lose two drives But you know, you also have two discs of parity So you have a lot less space to work with so it sounds great because you could lose a disc and there's a I don't want to say a decent chance. You won't lose data, but there's a possibility that you won't lose data Which is great, but Where I think that breaks down is that the The environment that causes a drive to fail There's a decent chance it could cause other things to fail too So if you get a power surge like Tom mentioned earlier that power surge isn't thinking at all But especially not thinking I hate that drive in particular I'm going to wipe out that one drive because it just rose me the wrong way No, it's gonna zap multiple things your motherboard could fry your drive controller to try all your hard drives can fry And then if you have even a USB hard drive plugged in to your server or computer Maybe the power surge will hit that too Or maybe you get a crypto Or not a yeah like one of those crypto locker things on your server and you don't have versioned backups You just you know, you have your external hard drive plugged in and It gets your data on your rate it gets your data on the external hard drive as well because it's attached Um Ultimately, I think the the biggest takeaway though is the environment Like I mentioned that caused a drive to fail can cause other drives to fail and in addition to that I've also seen situations where someone buys, you know, every drive in their rate at the same time Which means it's possible it came from the same assembly line Maybe there was something going on there and one drive fails the other drive fails right after it because something in the way In the manufacturing process Just wasn't quite up to speed or something and then you know every drive fails all at once it can happen So rate is great When it does make sense if you have the ability to benefit from it I don't see a reason why not but definitely have backups and that's going to be the nature of today's show Yes, and the secondary problem with that is if you have all of your data available and online That also presents a problem if mass corruption occurs because you know This is frequently in the business market where you see a lot of ransomware attacks on these companies They have a pool of data. It's active and online. Even if they have a backup to it wiping out that data because it may not have the ability to quickly roll back such as a ZFS snapshot would be able to do because they can't roll it back immediately now You've corrupted all of the data at once and that becomes another problem where rate doesn't save you in any manner And you go back to having to restore from backups So these are you know couple those factors you really have to think about and you know The resiliency part is really important and the part that Jay mentioned is also relatively important as well If you can and I've done this when we've bought large arrays Sometimes we buy half the drive from one vendor and have to drive from another vendor hoping the serial numbers aren't one off because if a drive Has a bad run this can actually contribute and you've watched Rater-rays unfortunately, especially if you look at the way some people don't build them properly They will be too much taxing going on the other drives to rebuild the one fail drive and one fail drive may have failed because It was you know a mismanufactured had a few bad sectors. Well, so did its neighbors So now the extra pressure coming from the neighbors to go Okay, we got to rebuild our friend here that we just replaced now They go into a high load mode and that can cause the problems or the failures I've kind of see like a cascading failure that's occurred It's always if you have something that only how offers one disk of redundancy Then you get nervous when you have to replace it because if it doesn't it's either going to crash the radar array or Rebuild it and it's a cross your fingers. I hope it happens now statistically. Yes. We're winning the game I've replaced plenty of single Drives that were single redundancy and went really well, but I can't guarantee that and how important is your data to you? Completely agree. I've seen that situation and I feel so anxious when I'm waiting for a radar to rebuild I'm like, well, I think it will it's probably a decent chance. It will but I don't know Maybe I'll have another disk fail before it finishes rebuilding then it's a lost cause at that point That's that's definitely happened and it just makes you feel really nervous because you're like, uh I don't know what's gonna happen here. So I hope it works But that's not a feeling that you want to feel at all when your data is very important to you, right? absolutely, so Sorry, go ahead. Now. I would say let's start with where to put it and I think I mean Granted don't get me wrong. It's awesome if you have a second location where you can store data You have two houses. You have a friend's house You have a family member who's willing to house your data. That's cool and awesome But realistically not everybody has that as an option So where should we what's a good cloud service to put our data in that's a good and I see good for Like the homelab market and the people that we're trying to help here because it's gonna have to be very reasonably priced. It's gonna have to Be something we've used and have experience with and by the way, they're not a sponsor of this show And anyway, this is our opinion and not their opinion Yeah, I like back plays for for my use case and I'll walk the audience through my system in a moment, but What I like about back plays is how you know how cheap it is and it's so affordable that it kind of made me nervous at first I'm like, okay Well back plays is a reputable company and they do a lot of hard drive testing like they they they actually have reports about Like the quality of drives and failure rates and things like that So even if you decide not to use back plays reading their reports is still a very useful thing to check out So they have, you know, good reputation in the industry and it is just affordable and I think that's a very important part obviously Stability is more important, right because if it's cheap and you you know, you can't rely on it And it could the whole service can fall over tomorrow. Okay, not a good idea but I feel like back plays has been stable and I back up so much that I think it cost me about 50 us dollars a month now The thing is you have to understand how hard it is to reach 50 dollars a month A lot it is a lot of money, you know, some people are probably listening to this like well 50 dollars a month Yeah, that's outrageous, but understand that I have like 4k video Going up there right now since I've started up, you know updating my channel to 4k We're talking about a serious amount of data I want to say it's somewhere between 8 and 10 terabytes or more that I have up there But I don't have it in front of me. There's a lot of data So then if you think about it like that 50 dollars isn't all that much when I have that much stuff up there um, that's pretty affordable now um The thing about back plays and some of these other services is that they give you a place to put it to put Your data, but they don't necessarily provide you with the the method to get it there Right because specifically we are talking about bucket storage I just want to make sure no one's uh too confused because they do offer some pc backup stuff That's separate. We're talking about b2 bucket storage too. That's yeah, I'm glad you brought that up That is specifically what we're talking about. So they give you a place to put it But they don't necessarily like I mentioned provide you with a mean So it's up to you to find a utility to get the data there. Um, thankfully most, um, network attached storage Solutions like Synology and I think probably all of them I don't want to say all of them, but I think that's true They support that as an endpoint so you could send your you could have your nas send your data there Yeah, um, I did a tutorial with true nas. It's on my youtube channel for this Um true nas is well integrated with not just back plays but many others. I seen someone mentioned s3 in the chat um, you do have options for Of course the amazon s3 azure blob storage Um, I'll I'll give a shout out to wasabi. We did a little bit of testing wasabi. I was confused About how the permissions work, but it does look like a reasonable competitor to back place Not enough to make me want to switch from back plays, but there's plenty of them out there but Like I said, Synology true nas and many of the companies Offer that as a target they offer b2 bucket storage as a target And if you have a backup system that is looking for an s3 compatible target That is something they offer with back plays. So b2 is their bucket standard is the protocol they use to talk to the bucket S3 is the made popular by amazon, but that's actually an open protocol That other companies can use so back plays does offer s3 connectivity. They have some Specific configuration parameters you need to use so you can substitute the two back and forth Just fyi for people wondering on that. So if you have a device that has it built in There's a lot of flexibility you can have Yep, and when I think of This object storage technology Um, I think it's important to differentiate it from like a hard You know a hard disk right because with a hard drive you have a file system You can store your data on there. You have metadata and I have permissions You have like date the date of you know when it was last modified and you know all kinds of different things Um with block store, excuse me object storage for the most part It's a it's almost like a key value pair. You have a name and an object So you have a file it has a name you have a bucket to put it in The object has a name so the object Could be a picture you're putting a picture up there or all of your pictures So you're not going to retain your permissions or anything like that But you do retain the file and it's you know true form So that's the whole point we want backups to have all of our stuff We we don't want anything to flip and then the file get corrupted We want it to be there and be in its pure form Name and object. That's what it is. Um, there are ways and this is why it gets confusing There are ways to mount it as a disk. It's not a disk You can force it to kind of act like that with aws and the s3 technology. There's a way to actually Make it look like a linux file system even though it isn't Which confuses people because they might see that and be like well, it looks like a file system of any other kind Why can't I change the permissions on it? Well, it's not quite that simple So you think of this as a bucket you can have a bucket for videos a bucket for music or pictures Documents whatever and categorize that way and then in each one you just well you have your data So how do you get the data there? Well, we already mentioned network attached storage That's an obvious answer if you have that already you could probably hook right into that It's usually pretty easy to do But you know, maybe you don't even have a nas i'm pretty sure our clone supports back plays if i'm not mistaken It's been a long time Our clone is more of a manual system So if you had like a a linux server that had your files Maybe you're not using a nas and you just kind of maybe rolling your own on debian or something You have a nfs share you want to get that data backed up You could use our clone to send that data to a service like back plays Yeah, the scanline you could put it in like a bash script You can have it in cron you could maybe build some email notification around it So you don't really have to use a specific platform in our clone Last i checked it supports a ridiculous number of services. So yes, our clone can talk b2 And back plays b2 provides their own tooling so you can integrate into your own scripts and things like that So if you want to do it all from the command line and linux, I believe they have windows clients as well So there's a lot of automation that can be built around that And our clone is also the back end for true nas so true nas is putting a web UI essentially on our clone to be able to You know manage that and they add the options for encryption because And I know people one of the reasons they go for a lot of self-hosted options and rightfully So is because they're like well I don't want to just take all my family photos and throw them in some cloud provider that next week has a security incident That makes them a public cloud provider Yeah, but you're not the way you're not the right way to use that term But yes that very public provider I'm going to buy this stock photo for this project. And why is my family there? That'd be pretty embarrassing Right. I think we got a we have to talk about encryption. I think that there's an important mindset here because There always seems to be this Mindset. I'm not saying it's an incorrect mindset. We want things to be encrypted But the question is like whose responsibility is it now? It's usually um It's often helpful to push the liability to a service because if we pay for a service. We're kind of like Shifting liability in a way because we're paying for a service. That's going to handle something for us It's going to handle the backups. We trust this service We're going to use this service and I don't I could feel a little bit more comfortable because I trust it Um, and maybe we want to trust that service with encryption. Maybe we don't maybe they have strong encryption Maybe they don't But if you encrypt it yourself Before you send it up to the you know a cloud service or any other service Now what that means is that you are in control obviously you don't want to lose your encryption key because that would be very bad Nobody can help you get that back But you know you can encrypt it yourself and send it up there And then you don't really care at that point if the target service even supports encryption at all It's okay if they don't because you can take care of that you can encrypt it yourself before you send it up there And even if the service supports encryption you might still want to encrypt it yourself because maybe You've researched the um, you know the the type of encryption that it is You want to use and the strength of that and you feel comfortable with a specific type of encryption Then you can make sure you're using that you have the key the key never I mean obviously don't put the key in the backup when it gets sent up Um, it'd be still it's encrypted. It's hard to get that even to see what that key is But still if something in transmission gets seen or there's a man in the middle or something You may as well just not encrypt it But as long as you don't do that and you keep the key somewhere safe multiple different places Because again you lose the key your data is useless You can send it up there that way and then you have full control So if back place supports encryption or they don't you don't care So that's something to keep in mind that you might want to take control of that You may not depending on the the service that you're using you have to Understand what they're using. Are you comfortable with it? Do you trust it and make the decision? That's right for you and your data on who should be responsible for that And to that note, you know, one of the more tragic incidents was an accounting firm that called us because They had had a full loss a fire happened in the building Good news is they had the backup They had the username and password to log into the backup account Then they also properly follow procedures and had an encryption key an aes type key That went with the backup So when you downloaded it and they did this for compliance It was encrypted at the end point prior to sending full compliance all done properly by this other it person Who couldn't recover it the reason they couldn't recover it and is what they were hoping we could help with Was trying to get the key off the Melted well partially melted computer what they had done and this is actually something I've seen more than once when we've taken over for clients They stored because the key being so long and high entropy. They stored it on the desktop of the server Thinking well, someone has access to server, you know, then they'd have the backup So we didn't think it's a big deal to save it on server until you're trying to do a full image Restore the server and you can't get the image for the server because the password to decrypt the image for the server was only on the server and This is actually a common scenario where you know, you have all your backup keys and you Backing up your own computer and you're like, this is where I also keep the keys because I don't trust putting them somewhere And this can be a real challenge. So I can't tell you how important it is and how many times this is overlooked In the out in the field when we've seen this like Jay said back up those encryption keys use the encryption But then make sure you are Very aware of where those keys are as a matter of fact when you walk through your restore plan As I said an untested backup is just wishful thinking. So as you walk through your restore plan You know, think about it turn your computer off and try to restore something without The computer you set up to encrypt it walk through the process Even if you don't go through the full restore the first time when you're kind of table talking this out You know, all right. Here's step one log into the backup account. Make sure we got credentials step two Where's this data going to land? All right Great step three decrypt data if that's where you fail then stop and look at your process before you have a failure I yep, absolutely. We can we could probably make an entire episode about Honestly, like I've heard it all and I've seen it all and I look at the reaction and In the the anxiety that the assistant man feels when they they see The fact that this isn't going to work like like this is not going to be readable I cannot restore this and and that is in that is not something you want to feel that is not something you want to experience You definitely want to make sure you test it. I understand everyone's busy Especially nowadays. There's all kinds of craziness going on, but test your backups. Absolutely. That's your backups Um, one thing I want to talk about is sync thing, but I want to be careful It's not a backup service and I don't mean to imply that it is it's absolutely not but Sync thing is something that could be part of your overall system now sync thing You could use it for whatever you want, but its main purpose is to sync more than one computer or server You know two different sources from one to the other. So if you change A file on one it gets synced to the other and vice versa that that's great Um, now that isn't backup necessarily because you know in my case I have syncing on all of my laptops and my desktop And that's great because if the hard drive fails in any one of my computers I'm not really so worried about it because I have all the same data on everything else However, if a tornado obliterates my house or a fire flood or something just You know hits all of my computers all my laptops then sync thing isn't really going to be all that helpful It can help in that situation, but it shouldn't be Relyed on but it is a great way to get data from one point to another and that's its main use case And I definitely wanted to uh mention that well, and it's part of the strategy like even what we use here with sync thing and Real time backups in you know the short Synopsis so speak of what sync thing is is real time file synchronization picture something like dropbox or microsoft's one drive but more stable The nice thing with sync thing is there's a lot of times when data isn't just important I need those changes not backed up tonight I'd like them backed up in real time or then synchronized in real time And part of the strategy that is common where you have a file system like this is also to synchronize it off site at the same time So I have my on-site servers and my off-site servers and as data gets dropped in certain folders It's immediately synchronized and for you know, those you that follow some other youtube videos You know, I use a lot of unify. Well, we make changes constantly with unify. I don't Have I don't want to wait till tonight to get those changes fixed I want those changes done right away. So it's constantly exporting the data. It's constantly synchronizing it so on a regular schedule sync thing can then Go ahead and Take over and move that data where it needs to be now someone pointed out uh in the chat and rightfully So you want to have some type of a mutable offline source This is where you have two options with sync thing being part of those options where you have Revisioning it does so as the files gets changed or let's say someone Deletes a lot of files or modifies a lot of files, you know, crypto ransomware type scenario There is the option to have revisions. The second thing is wherever it's landing should have something and I use shurnas as an example But it's also supported in other platforms Uh snapshots that give you points in time by which I can restore So if I know all the files got corrupted at 10 a.m On a Wednesday and I have a snapshot from 8 a.m I can just roll back to 8 a.m. Or however, you know part of your snapshot strategy This is all part of your real-time strategy You're thinking about when you're doing this because it's not just like set the backups And we should all set them at 1 a.m And that's it so much occurs during today and at so many of these businesses we service There's a constant flow of data and the cost is Predicted based on you know, how much is it going to cost you to lose a days with the data? Well, I have 50 employees in the office So that means 50 employees worth of work can be lost if I don't have a strategy that has me backing up faster That's why these real-time tools such as think thing kind of come into play and become a very important Component in there to make sure you're synchronizing the data or even Snapshotting it every half hour or whatever that strategy is it's it's figuring out where your risk tolerance is For your data, you know, and if it's at home, how often are you uploading? Movies or data or anything to your data sets that's important to you. Maybe you kick off a backup Right away when you're done downloading it, but you know, you don't go on vacation every week So you don't have that many to add every week to the photo set You know, you can think about different strategies like that The bigger thing is automation is one of the things that syncing offers because I put it in a folder I don't think about it. It just is now backed up. So when it's important to me I save it to syncing I save it to the folder I should say is syncing is synchronizing and then I just know it's everywhere I need it to be Confidently without having to remember to do anything else And a couple things about syncing I want to mention Implementation is key And that's a very clever way of implementing it and I use a very similar You know style here actually with with snapshots and everything Um now one thing to keep in mind. I don't even care what synchronization utility you use and there's others Is right syncing is not the only one Make darn sure ntp works Make sure all of your computers your laptops desktops your servers whatever you're syncing with syncing ntp the network time protocol must be working because if the if different computers have like, um non-synchronized clocks Whatever your syncing solution happens to be they will all have problems every single one of them because they're going by the time stamp Most of the time at least as far as I know all of them go by that And if your time is not correct on your systems You could literally have a situation where you have duplicated files because it doesn't really know which version is the latest So make sure before you install something like syncing NTP works just check the time on all your computers and servers. That's a good idea to do Now when it comes to syncing You could be very creative with it Someone in the live stream mentioned that syncing can be a backup If the you know destination is off site So in that situation, you know, syncing isn't necessarily meant to be a backup solution is a syncing solution but It's up to you how you implement it you can do that you you can Um responsibly, you know, you don't want anything just wide open or anything like that but you can sync it to a um cloud server or you know a friends server or something whatever the endpoint happens to be And yeah, you're making it into that. Um now we can have arguments about like, you know, is that the best way? It's fine. Um, but but you have to have multiple versions of things But creativity. I mean you could usually have full creativity here Like for example, I've mentioned this I think in a previous episode my retro pies you sync things So if I'm playing super mario world on one tv Um, and I save my game and I just beat a certain level and then I go to a different tv Power on that retro pie and resume super mario world I have the same save file because sync thing is making sure that every single emulation console has that and yeah, I even have a data set in um In true nas for this so if I lose a save file for one of my games I could get it back so you could use it for whatever you want some people even Install it on their phone. So anytime they take a picture. It's automatically um on their computer and on their server Which is pretty cool Especially if you want to edit that photo before you send it to someone There's all kinds of creative things that you can do with something like sync thing Um for myself personally the way that I do it is I have um almost kind of like the star topology methodology with this where All of my laptops desktops or whatever are all syncing to one thing So I have my true nas server in the center And I can only use and this is why this works. I can only use one computer at a time So I could save a file on my laptop and I could switch over to my desktop So what's going to happen is that file from my laptop will sync to the true nas server the central sync thing service And then that file will be sent over to my desktop Now you can sync with syncing everything to everything I could have my desktop and my laptop sync directly to each other I could totally do that But it just works for me where I make true nas the single source of truth for everything Everything syncs to that the data sets are there the snapshots are there There's that one single source like I mentioned that works well for me and that's how I use it but also Since my true nas server and this is where it comes full circle is backing up to back plays off site Then my server has everything And I do check this by the way because you do you do have to check to make sure like the sync solution didn't quietly fail but So everything goes to the central source that central source also has a schedule where it just starts Uploading that to back plays so then back plays also has a copy of that So sync thing just handles making sure the data is there to be sent And then as long as you know, there's no failures in the syncing process no failures in the upload process It works pretty well for me and kind of related to this You know another thing that's often important is synchronizing all the backups for your firewalls or any of the other devices you have because PF sense is a great example You know you just get an xml file when you're done a lot of firewalls have this So there's a config backup that has all the settings because it's easy to grab the ISO and reload untangle reload pf sense or open sense or whatever you're using And then once you have reloaded you just restore your config file and put it back Same thing's great because you can have a folder say this is the folder where these files are kept By the way, please encrypt these files because they have the keys to the kingdom inside of them Especially the pf sense xml file, you know if someone else got a hold of that They would have your vpn information if you had a vpn set up for example But you'd be able to take that Grab those config files and sync things is a great way to easily replicate them to the different Secure locations that you've configured to set up. So it's another you know part of the strategy So we're going to get into the imaging side of the strategy as well because there's a place for this as well There really is and clone zilla is just one of my favorite solutions I remember very early in my career And this isn't like a knock on a norton ghost. It had its time. It's it had its place. It was fine It had some quirks. It was a little rough, but it worked But when I worked for no, I switched to another company and I was introduced to clone zilla And this is still very early in my career. I was just blown away by how great it was So clone zilla is basically something you can download as a You know basically an iso image you can write it to a flash drive you can boot your computer from it And that that's called clone zilla live And it's it's exactly what it sounds like. It's a linux distribution That you can run live that has utilities you can use to taking it Basically an entire image of your hard disk So you could restore that image and I've used it just experimenting with it on x86 You know max works fine Windows machines works fine. In fact, if you are backing up a windows machine There's even an option in there to remove the I think it's the hibernate file essentially It'll regenerate that anyway So, um, I mean why include that in your image is just going to waste space So there's all kinds of different options You can tweak that are pretty powerful and it creates an image And you could store that image via clone zilla live on another flash drive An external hard drive a samba share An nfs share you can even go disk to disk like if you have Let's just say you have a 500 gigabyte hard drive and you want to go to a one terabyte You could literally have it copy that entire drive your 500 gig drive to the new drive That's you know one terabyte essentially and you could then resize the file system After the fact to take advantage of that extra space Another thing that I have used clone zilla for Several times is for recovery. I had um, and this is probably like a very special case But it's happened a few times where a computer doesn't boot and I just can't get it to boot I actually had a server that didn't boot once because the company didn't even back up anything They're kind of in a panic. So I took an image of the hard drive I mean, it wasn't like the hard drive failed so hard that the bios wouldn't see it It's just one of those situations where it has like bad sectors. It can't boot but you could still read the data And I took an image of it and I restored it on to a brand new drive And there's an option in clone zilla where you could tell it to If you encounter an error just just skip that error and keep keep cloning the drive It's very important to check that and then luckily for me with this windows server. I you know booted it Of course, it's not going to boot because um, well It I mean had a bad sector there's stuff that it can't read And then it did I did a file system check and it was actually able to repair itself and then boot fine And it was saved now. Obviously that company should have had a backup They should have had an image like before this happened. I'm not going to go there But there's all kinds of use cases for clone zilla live. I really like it quite a bit another thing you can do Is you could create an all-in-one recovery usv key that has clone zilla, you know built in With the image of a you know computer built in where you can literally boot from it And it'll ask you are you sure you want to continue press y and then enter it re-images the entire thing Um, so you literally have your own like recovery disk for a computer or server that's self contained now The other use case for clone zilla is not the live version. There's a server version I don't know if they're still calling it this. It's been a long time d rbl I think they want you to pronounce it durable, which is kind of weird But essentially it's setting up a clone zilla server So your computers and servers can pixie boot And you could actually you know without You know booting from a usb flash drive have it boot from the network grab the image similar to norton ghost how that used to be And a lot of people like that now I i'll say this i don't really care for that use case so much and i'll tell you why I found that in my experience Maintaining a clone zilla server just seemed to be more work than the value that you get out of it And that's not to say that there's not some good features there I mean multicasting your images and pixie booting is you know, it's cool But to set that up and maintain it especially in my experience where I I found like it kept changing I kept having to redo things. It just became a chore to keep up on I had like videos on my youtube channel that were about setting up a clone zilla server very early in my channel's life and um It just became a chore to maintain and keep in mind I create videos on arch linux a rolling distribution and I found that easier than making videos about clone zilla server So what I have found works well in my opinion is that you have a Central file storage a samba server an nfs server And then you just create um one or more clone zilla live flash drives just boot from it tell it where It can find the image. It's going to pull it from the network. Um I think that's great And not another thing you could do is you could load the clone zilla Flash drive in memory to where after you boot you could remove it And it's just running off a ram and then I I could take it from computer to computer Booting it or booting them into clone zilla live off of one flash drive and they're all pulling the image down all at once And tom, I think you have some experience with blasting an image onto a bunch of computers Yeah, so when you know, this goes back quite a number of years We would often help with um, I don't know if anyone on here would be familiar with penguin con But it's a big sci-fi linux event that happens here in michigan and greater detroit area Up until well in the before times it happened annually for the last number of years but the um We helped set up a lot of the linux laptops They would basically have one loaded with all the different functionality they wanted and then we would use clone zilla which has got some cool features for mass cloning So you can set one as a source and then use multicasting to clone to many many at a time It's definitely a been a great tool that we've been using for quite a quite a number of years for doing things The one hang up that some people if you start using this in the windows world Is clone zilla does fall short of some of the commercial options that offer dissimilar hardware restores Which is becoming less of an issue with windows 10 because it handles dealing with dissimilar hardware much better than previous versions of windows But it's not like a native feature where it can do the prepping for that so it's definitely It may not be the absolute best for tool for every situation when it comes to windows But it's solid on linux and as I typed in the comments But want to repeat here that yes, it's great for and I have a video on this when you're moving hypervisors You're on a hypervisor and you want to switch to a different type of hypervisor But the two won't talk to each other natively as in I can't export the vm And natively import it into the next type of hypervisor because they're dissimilar That's where clone zilla is actually really we've used it to move from vmware to xcp and g We've moved people from hyper v to xcp and g and just popped in the clone zilla booted it up and migration happens It's it's been pretty solid for those use cases especially for you know Just I need to clone a linux workload or linux server and get it over there The two talk to each other quite well, and it's been a really saving tool for that Yeah, I I want to yeah, I'll mention a few things about the windows side of things because Early in my career, that's what I was using it for actually was deploying windows installations And I did Find a workflow that allows me to get around that those problems with clone zilla actually But it's a it's a workflow you have to think is is you know It's it's a workflow if there's effort to put into this but so the issue That I and I think you might be referring to this is that If it's dissimilar hardware it could blue screen when you restore an image So you think about you have an old computer. It's time to buy a new one. Yay great new computer But you would really rather not set up everything all over again So with linux, you know just clone the hard drive or even just move the hard drive honestly to the new computer And it usually works as long as you don't have anything that's expecting a very specific You know gpu and monitor configuration with a custom x-word config file Most people don't go that far. It's probably going to work just fine Windows on the other hand will often blue screen because if anything is different Like if it feels like you're ripping the carpet out from underneath it And expecting it to still walk it's going to stumble and it's getting better But what you can do is sys prep the instance before you move it which You could use what's called a generalization. I think it's just called generalized and sys prep And that kind of removes the things that are specific to that installation to make it so that it can't work on dissimilar hardware The problem is you there's two issues with this actually one is that Okay, so i'm losing my train of thought because it's windows and not my fourth day. Okay, anyway So the issue is going to be that um, you only have a limited number of times you could sys prep I believe it was three. I don't know if it still is which means there's this registry key You had to flip after that too. So yeah, and I ran into that very frustrated. I'm and I had this awesome image It had all the company's defaults in there and I made several revisions because my thought process was I'll just keep Improving it and I came up with a versioning scheme. Okay. This is version one now. I'm up to version two You know, it was cool hit version three. This is awesome I want to make a version four Oh, I can't because there's a limit and they don't want you to sys prep more than three times the other issue that I ran into is um You know, I at the time I don't know if it's like 60 gig or 80 gig You know, that's the total hard drive size of the image And the image size is only going to be like what's used So if you use like four gigs of that 60 gig, um hard drive you your image is going to be closer to what you used But if that hard drive you're restoring it on is a 40 and your image was taken on a 60 It's not going to work because it's not going to go down. You could resize up So what I ended up doing was setting up a virtual box instance And I gave it like a 20 gigabyte virtual disk That's all and I installed windows on it and all the company's defaults I put in there And then I did a virtual box snapshot before I sys prepped it and I had infinite sys preps at that point I could I could sit I could use sys prep a hundred times And then every time I would just roll back the snapshot in virtual box that totally got around that problem And it worked just great and the fact that it was only a 20 gig virtual disk It didn't really matter What size the target hard disk was because they're all at least at that time going to be higher than 20 gigabytes So that's what I did and then I would just expand the file system I would take an image via virtual box by just booting the clone's alive Um, you know sys prep after taking the snapshot use clonezilla live grab the image And then I could just do however many deployments I want on dissimilar hardware work just fine But it took work get get it there and is clonezilla the best solution for windows Probably not there's going to be other solutions that are better And a windows expert would probably tell you what those solutions are But the point is you you could do all kinds of things with clonezilla And I think it's it's something to look at if nothing else Create snapshots or images of your servers and computers. So at least you have that starting point maybe There's a bunch of packages you install or config files. You really want You know everything to have you can have this base image and then just deploy it at any time you need the server and it's great for that now Something that wasn't in our original notes, but I want to bring up briefly here is Backing up the hypervisor systems and the VMs within there. This is one of the reasons I've always liked xcp and g because combined with zen orchestra You have an extremely complete option to do full backups Delta backups and all the incremental that you want within the same software for the hypervisor This is something I know proxmox added more recently. I don't know if you've used the proxmox backup server to back up the System but you've been able to script it from the command line from proxmox prior So there's ways to proxmox had methodologies to do it Zen has a very complete with full notification like proper backup system This is a shortcoming of hyper v and esxi and I've seen a few people in the comments mentioning it It's worth noting a beam is kind of made their Product popular by filling in that gap for those so it's something worth looking at I don't use it because i'm not a big esi esxi or hyper v person I just don't really use a vmware or the microsoft hypervisor So I don't really get much into vmuse, but it's out there. It's kind of it's right away There's plenty of discussion. I see in the comments going on about that Being a popular service. I just can't speak to it real intelligently But it's actually one of the plus one feather in my hand for a while like xcp and g It natively has all that built in with zen orchestra So you get a complete ecosystem where the people that wrote the software also wrote the full backups and everything else So yeah, I kind of feel like you know speaking personally that Sometimes I wish I had like a month-long vacation from you know, my to-dos and you know All the other things that i'm up to just one month and and during that month like every day during working hours Just try things like vm esxi random stuff that I normally don't dive into Um and just try them all right because I want to try them all unfortunately reality is that um the problem I face and tom I think you face this too is that There's only there's a finite amount of time we could try certain things Um and some things you know are just lower on the list doesn't mean that they're done Solution they're probably a great solution. It's just you know Hard to get to everything. How do I clone zilla myself? I need two of me Figure that out um and I could clone myself that would uh, that'd be awesome It'd be great for me, but probably um horrifying for um people in my circle. Yeah Um But yeah, I kind of do feel like you know if I could try more things like I I um at one point Tom and I are talking about open sense and and honestly I I like the idea of it I I do feel like if someone wants to fork something that's why things are open source You can work it you can create your own spin of that and everyone has the right to do that And I want to try it out. I just unfortunately haven't finding the time to test every one of these is tough As such you have backups you need a high level of confidence that the product works that the product can be restored And when I speak about things like xcp ng I can absolutely tell you I've done Bare metal disaster recovery of reload it point it back at the place where all the data was stored and bring it back Um, so to you know to from a full You know full bare metal restore It's important that you test these it's important that you have confidence And that's one of the reasons me and j we've both you know restored data from back plays We've restored data from snapshots. We've used sync thing to use the revisions to uh fix a save game gone wrong Yeah Or a boss level not done right so there is definitely because we're all speaking You know from things we actually use that's one of the things I want to be important We're not just kind of talking about the product lightly We do this commercially. We do this business wise and we do this in our Labs as well where we build things But I say all that then I do want to bring up duplicati It's a cool tool and I've used it for some testing. I've never put this in production But a lot of people have asked me about it. I think it's worth mentioning duplicati is a cross platform open source backup system. It's a little weird because it runs off of a web interface and So everything, you know, it loads a little back end server that you run locally on the computer Then you access it via local host in a port, but it's just pretty cool um Playing around with it. It has a very large broad support. I mean, we've got mac debby and red hat windows Even native synology support on there But being able to land it in a lot of different places whether there are places that you host yourself or outside of places you host yourself, let's say Places like back plays It's kind of a neat tool if you're looking for something to create recurring backups on a schedule The things I don't know whether or not it has as a robust notification system to let you know that it's Maintaining and running and everything else. I don't know about that production But I thought it's something I'll throw on people's radar there because one it's open source In to its cross platform. So if you're looking for something Um for your home network and things like that and you want, you know Just to back up a folder on a recurring basis that you're saving data It's a smaller tool for that, but it's pretty cool. I think it's something worth mentioning Yeah, I want to check out duplicati and it's um, it's it's on my list I mean on my list since just yesterday, but it's on my list And I set it up this morning to refresh because I I never did a video. I don't recall at least maybe I did I have 1100 videos sometimes I forget but I don't recall doing a video on it But I know I've used it because I needed to do some testing for another project and duplicati was just kind of a cool tool to land all the data somewhere And I'd set it up just on a demo box and thought this is a really novel thing and it came from a lot of the comments I've gotten on my videos for people saying hey, tom, please check out duplicati Same thing in the more larger robust scale that I have is more what I use in a production environments But I just think this one's really worth mentioning and check it out Yeah, absolutely. And I think that the notification thing really kind of does need to be underscored here because And even someone in the live note or the live chat here mentioned that duplicati doesn't really seem to have a great notification system I've heard other people say that too. So the more people that complain about it the more I believe it and other people do complain about that You know disclaimer. I have never used duplicati. It's on my radar. I do plan on using it So I don't have an opinion of it, but um, I think that We really do need to underscore this again. Yes, we mentioned test your backups, but Um, don't just assume everything is working Things quietly failing is a big problem And sometimes you just have to randomly check things one time I deleted accidentally my entire ebook collection now The good thing about that is the publisher. I bought all my ebooks from I could redownload them from That site, but that's a chore because I had like I think I have like three or four hundred of them I'm and even if I did like take the time to redownload all of them I'm not going to I mean, I'm going to accidentally like not click on one of them And that's going to just be a major chore But I did accidentally delete it and the thing is I'm not reading my ebooks every week So it was like maybe a couple months ago. I deleted it so right now I'm at risk because like Things can just go away if I'm not checking. Thankfully I had snapshots and it was a sync thing basically, but um True nas then free nas I had this um snapshot recurring going back. I think I don't know four or five months or more And I was able to mount that snapshot grab that folder with all my ebooks and just put it back on one of my computers where it's supposed to be And then sync thing who hasn't seen that in a long time says, oh You have a new folder fully ebooks. I'm going to synchronize that to uh, your you know sync thing main central point And then next thing you know every single computer has that restored But things can quietly fail too like monitoring systems backup systems like Um, oh, you know, especially if there's an option to basically alert you only on air That's a dangerous one Because what if the alerting system isn't working and there is an error and you and you assume it's fine because well I'm not getting any alerts. So I think everything is good. Eh Might not be good So I'm a big fan of just creating reminders for yourself either in your calendar reminder app A post-it note stuck to your monitor as long as it doesn't fall off And and you will actually check on it to not only just test your backups But just kind of look at the logs make sure things are actually happening the way you think they are and Some kind of recurring thing to where you could just every now and then check this and that's extremely important Do you know and this just goes back to Untested backups are wishful thinking and go ahead and walk through and test that make sure the files You know log into the remote system make sure the files are landed there and make sure they're the same in latest version since the last backup and then Test them see if you can decrypt them, you know, all data at rest should always be encrypted Just a reminder on that that is one of the best ways to think about it If the data is at rest it's not being accessed. It should also be encrypted But go ahead and remote into things try to decrypt it try to do it or restore and and also I'm surprised I haven't mentioned this earlier But the the three two one backup strategy like I probably should have opened the podcast Because we I mean that is good methodology The three two one backup strategy where you have to have at least three copies of all the things that are important And it should be on at least two different services one of which must be off-site Now I don't care if you have like ten different Backups on ten different services or however crazy you want to go with it But at least have three right in two of which on two different services But that's like more to me like a minimum But you really should look at that because um and also be honest because You know some people and I've done this They'll have like an external hard drive and they'll and maybe they're really good about this Backing up to it. Then it's encrypted. They take it to work where they have a standby drive They take the standby drive home with them and then they start syncing to that one And then they swap it every day or every week or something. I've seen a lot of people do this however You know that if you get really busy and you have a lot going on What if you forget and you know, it's now a month or two months And you've only been backing up to the one drive and then it dies Um, well, you're out a few months of data there. So, um, if you are like me and I'll fully admit I I don't procrastinate on purpose, but I will um if it's not like something I hyper focus on Um, I have to set reminders for myself to do various things And if you don't feel like you're going to keep up on it because you know life happens Maybe it's time to consider one of those um cloud backup services that you may not have considered before To take some of that liability off of you But you still have to test it every now and then and you still have to check it to make sure your automated process is actually Backing up things you think it's backing up Yeah, so it's just about set those reminders do those test restores Verify that all your data is backed up verify that it's being Uh, you know constantly where you think it should be, you know, the latest versions and uh, yeah, just try it out Like I said, there's I just can't hit home enough how many times we've come in and not been able to restore because the backups weren't processed as well I literally included checks in my nagio server that I use. I know nagios is old, right? But I still use it it works for me and it's literally checking the syncing process on all my stuff so um, and obviously I have to You know watch nagios to make sure that that isn't quietly failing But it'll it'll send me an alert that hey uh sync thing's not running on this machine So I don't you know edit files on that machine thinking it's being synchronized And it's not I get an alert. Uh, yeah sync thing isn't running So I'm I really need to get that running because it's not synchronizing anything off that machine So there's there's different avenues here And you have to think about what system works well for you what services work well for you What's going to play to your weaknesses if you're like me and you know You don't really feel fully confident in yourself to remember to do something set a reminder for it Whatever you have to do, but uh, definitely make sure you cover your bases three two one backup Definitely important test the backups like we've mentioned on 9999 times now You're mentioning that that many times Make sure things aren't quietly failing just Validate that things are are running the way you think they should be running And you know, you'll you'll be in a good situation Simple as that All right, I think we we reached the end of this Yeah, any other details we should add. I think we covered the top to bottom here Yeah, I mean there's there's probably a bunch more things that we can add because there's so many other services out there That we're not even checking or talking about because we haven't used them So I fully expect there to be a lot of comments like well, what about this and what about this and I like that because That's what's going to alert us to the existence of other things That we might want to look at, you know later on Well, and maybe we'll do a deep dive on some of them because maybe even people want us to a deep dive on What object storage is in these buckets that we talk about that's a topic into itself Matter of fact, I don't know if you you know, you can run your own object storage server with min i o Which emulates s3 and is natively built into true nas so you can actually create your own s3 targets as Backups that you own in colo centers. So it actually can work as s3. You put a certificate on it Boy, that could be a fun video, right? That's all a fun video and and that just underscores what I love about the homelab community because we we love running Servers and in the infrastructure. We're very passionate about the services that we personally like That get the job done for us. We like to tell other people about it I like to hear what other people are doing what they're using because that alerts me to things that I wish I would have known about earlier and sharing the ideas and strategies being creative and how you connect the different services like um, you know pf sends isn't made or in any association with unify, but they work well together Just so happens to be the case and sync thing works very well with backup solutions and how you connect these things is part of the fun Yep All right, so you've listened to another episode of the homelab show You can find all of the previous episodes at the homelab.show We have all the podcasts there and this podcast is uh able to be downloaded directly from the site or anywhere else Good podcasts are hosted. There's a lot of services out there now We trying to get to all of them if we're not on some platform Message us and let us know why leave a comment. I think we got them all covered here Yeah, we we also want to mention that we're going to try for a cadence of 10 a.m Eastern standard time on wednesdays for new episodes. Um, the podcast is still early So obviously, you know, there's some things where you know, tom and or I just have to go deal with We work in it obviously, but that is what we're You know at this current point in time until further notice for thinking wednesdays at 10 a.m Eastern standard time is when you could generally expect to to uh find new episodes on the live Um and the live stream and then later on obviously via all the networks. So yep It's having guests on that caused the times to be a little bit more erratic All right. Well, thank you everyone for watching and uh talk to you next week. Thanks You