 Welcome to the homelab show episode 118 backup strategies. How are you doing, Jay? Doing awesome. How are you? I'm doing great. You know, people over wonder what happens when we're a couple minutes late Sometimes it's technical difficulties. Sometimes me and Jay literally discuss how we're gonna look at each other over a podcast Do we want So it's like and then I have to remember to look at the right camera now in the studio because I've recorded segments I'm looking at the wrong camera while I'm talking and I'm like, okay, I got to re-record that again. Yeah, so kind of fun We're diving into the topic of Backups because they're important. They should be done at the frequency that you are risk tolerant for and There's a lot of nuances and we'll talk about some of the places to backup and some ideas around it The first thing I do is catch up on a few things Standardization on standard notes. So you you found a winner. I did. Yeah after all this time the runner-up was obsidian but Standard eye or standard notes one. It's open source. So that's automatically gonna put it at a higher Rank for me nothing against obsidian. It's but they're kind of different, right? Because it's a note-picking app Obsidian is more, you know, you have a directory you have markdown files in there And that's fine. The standard notes is more like I don't want to say it's like Evernote But it's more like Joplin, but a little simpler and I like the simplicity of it and it works really well I was kind of surprised that it took me this long to find it after all the note apps that came up And as good as this one is it's just it's like, okay Well, this could have came up sooner, but I tried everything else But I suppose I mean I'm now I know about note apps But standard notes is a hundred percent open source. It has Cha-cha 20 encryption Yeah, that's such a funny name I was wondering if I could say that the straight face but but but in all seriousness It's encrypted which is important to a lot of people as it should be so it's open source. It's encrypted It's easy to use. I I guess I can ask for more than that So it it is more like the Joplin style more more than the obsidian style I like the Joplin style better, but that's just a preference for individuals, I guess Yeah, and in for myself and this is the for those of you that are new Me and Jay have had this debate on numerous episodes of what is the best tool to use for keeping all the notes We do on all the technology the one thing we have in common is we really like markdown. We like simplicity We love open source outside of that we me and Jay don't even agree on which note apps to use But maybe none of you do either More of a journaling app than a note app though. So I think technically it's like a different workflow. I think yeah Yeah, I'm using log seek. I think that's how it said L o g s e q. I love it. I'm thrilled with it I've been using it now for long enough that I've got over a month's worth of daily journals in there So yes, it's definitely a more of a journal style. It is a different workflow It is a journal that has notes versus a note app, but it's fine. It works for my workflow I might do a video on it if there's enough demand leave that in the comments and say if me and Jay should do videos on Our respective channels about that as a topic. We definitely don't mind sharing it I don't feel as though I'm an extra actually a lot of good There's actually some good log seek channels that I've been learning from myself But at least maybe I can direct people as to here's the tool and here's the channels that I learned from maybe that's helpful If we're a bit of background in case someone's new to the podcast I was using Joplin for a while and Joplin is basically my favorite Functionally, but then the number of bugs was just So egregious that I couldn't even believe that it's released as a stable product. I mean it has one job You know take your notes. You write your notes. It saves notes. I mean simple, right? But even that That's a problem notes getting lost or they're not syncing properly Formatting is lost after you save the notes, which doesn't make sense You have syncing where it's not quite the same on both ends. We have to keep micro I mean there's just so many problems with it Which is a shame because functionally it's amazing, but at first I was pretty forgiving as I normally am Okay, I'll just wait for an update or something. They'll probably fix it But after update after update after update and nothing's getting fixed. I'm like, well, I just can't use something I can't rely on so that started the whole Spill about finding a note app I think we're both looking for a note app at the same time Yes, we landed on different solutions But I think the reason we started our our look at alternatives was probably very similar And I think the reason you you took a long time to find standard notes Is because if you google standard notes, you may not land where you expect So I put a link in the show notes because think about it. It's called standard notes. Okay. What notes did you standardize on j Yeah, that's true because I mean Well, yeah, it's like the discoverability of things is very important and I make this joke often It's true and it's it's not really a joke, but it's funny The band churches is famous for replacing the u in the band name with a v and when they're asked Why did you do that and they're like, well, when you google churches, you're not going to find us But if we change the u to a v you'll find us then and I'm like, yeah That's pretty clever because standard notes Is you know to your point it I guess it It would be hard to find and that does explain why it's such a basic name Maybe they should rename it with something catchy to try to get more traction And I might say log seek is easy to find But you have no idea log seek was a journaling your note taking app You're thinking oh, it's going to be some type of log indexing tool, right? I'm going to be able to seek out my logs That is the first thing that came to mind when people started talking about I'm like I already did a video on gray log Yeah, and he might somebody might search for that, you know seek s e e k and not come up with it not s e q Sometimes I feel like it's weird when you have like really great applications, but the name Just either is an obvious or it's too obvious or something like that And it becomes hard for the product to gain traction Just on account of that before you get to its functionality. So Um, I guess it's just a google problem It well, you know, it it's even worse than that because the tech the tech industry is littered with things We're going to come up with the new crowing language. What do you call it rust? Oh, I'm going to google rust. Great. What is the other one? We're going to use go Oh, I'm going to look for go like geo Well, they had a very effective strategy for making those names famous though. I mean They they started a debate that ended up as a holy war. Okay. I'm joking about that part But but seriously there's so much debate around programming languages I swear you could call a programming language cat and it would probably get traction So yeah because we debate about these things so much and and everyone has an opinion rightfully so because opinions are awesome Just like open source But when you have people aggressively expressing their opinion like rust being in the linux kernel that was in the news It's kind of hard to not be famous at this point But um, we're trying to get people to realize they're especially windows users There's life beyond notepad plus plus right Absolutely. Yep. Now A phrase that I always like that my dentist told me as a kid is you don't have to brush all your teeth Just the ones you want to keep and I'll lead that into our backup discussion because what should I back up? Not all your data. Just the data that you care about so It's a It's one of those things you don't realize what you've lost until it's gone so you should really think about the whole ecosystem of what you back up and What I mentioned before is your risk tolerance for backing it up. Uh, for example Dovetailing off the notes talk here all my notes are tied together with syncing because the moment I save a note I actually would like it backed up. I don't want to backed up tonight later at some scheduled time Uh, some things are to me best done with real time and you can say sync things on a backup But there is versioning in it therefore. Yes, even if someone gets a hold of them destroys it and even Syncing thing has a backup strategy. So where should we start in this three two one backup strategy? Yeah, we could get into that in a moment. I just wanted to to mention some of the problems here because you know It's backing up is not easy We we think it is and it can be easy We could simplify it down to throw everything on a flash drive as long as you remember to do it You never forget to do it and you have like multiple flash drives and you keep one off-site But we'll get into that backup strategy in a moment Some of the questions that could come up with the homelab is what to backup now an easy answer is backup everything But you might not care about everything, right? So maybe Since you're not a business you don't have data retention laws So it's not like you have to keep your logs for a certain amount of time So then you might be thinking well, I'll just keep a week's worth of logs Can I uh, you're not gonna you're probably not going to back up logs, but it's still data retention But if you think about You know videos like your plex movies, for example You could have terabytes of movies and you obviously don't want to lose that but You could argue you could get those back the movies that you have on your server You don't have the only copy of those movies. Let's be honest I mean, you can even rip your own copy again if you have to you could reproduce those files But there's other files like family photos and things like that. You can't reproduce so those might be More important, but but you might argue. I don't want to lose my plex movies It took me forever to rip all of those But then when you back them up to the cloud And you know in my case you have like my image looking garbled on the live version because I have so many cloud sinks running And backing up that I can't even get the camera across the wire But you know and then your bill goes up because you're backing up plex movies So then you're you're tasked with do I just not back them up save the money in worst-case scenario? I have to re-rip them or is the amount of work you spent ripping the movies It's like so egregious that you just can't bear to do it again And you have these types of things to figure out And I think that's an important mindset to be in when we start talking about this because Yeah, I'd like to tell you to back up everything The first thing is to back up the things that you can't get back that there's no way to reproduce like I produce a youtube video I'm going to back up the master copy because I could reproduce it again by Repeating everything I say redoing the b-roll and spending a bunch of work on it But I want to keep that backed up But have that mindset as far as like what you can easily recreate re-rip or reproduce And what you just can't get back and family movies And photos are easily a first consideration here because a lot of us have that kind of thing Even if you just have pictures of your cat and you're by yourself I mean those pictures are still important. So you want to back them up So with that in mind we could start talking about the backup strategies But as we do just think about the things that are important and the things that you can't be without Yeah, there is a cost to it Even if you're just creating multiple copies on site or maybe storing one off site When you start looking at if you have a large plex library of things You've ripped an entire dvd collection granted You know going through and re-ripping all of those dvds that I have tons of that I bought that was a big movie collector for a while that is still I'm not willing to spend too much money I'm buying a lot of hard drives to have an extra copy of it It's like look it's on a rate array right now And I I'm the first one to say raise on a backup and I do have extra copies here I just don't keep one extra copy off site of that right now that's spend my limit I keep two copies on site off two separate NAS boxes That is where my risk tolerance is for my plex movies Yeah, and another thing to keep in mind too As we talk about this is to encrypt or not encrypt now An answer here could be encrypt everything right but but maybe not for example I keep pretty much everything I download whether it's you know an application installer or an ISO image These are things that you could freely download online. There's nothing specific to me here If you have if you download a boon to 20 204 an ISO image the one you have is the same as the one I have And if I lose it, I can go get it. So I'm not going to encrypt my Dataset that's full of you know generally available files that anybody has access to That's just a waste of cpu resources in my mind But if there's any personally identifiable information, then yeah, you should be encrypting it Yeah, I don't encrypt my youtube videos either My the same reason, you know, I have a lot of storage dedicated to Years of being on youtube and creating content and because that content is actually available outside of the you know I know obviously not 100 of the b-roll goes into the video. So if someone were ever to Somehow get access to it and exfiltrate those which I would probably notice just because the bandwidth I was like, hey, why is everything going slow? But if you would just get a bunch of extra b-roll maybe a video of me picking my nose that wasn't publicly available But you know, I'm willing to risk it For the f-bombs I say in between Different scenes with me don't work out. Well It's pretty funny the things that that don't get uploaded. But you know in that case Yeah, you might not want to encrypt those because you could watch them for free on youtube anyway The only risk might be somebody uploading those videos But I think the the funny thing about this is you have enterprise Companies that are in the news because I didn't notice that a large amount of data was exfiltrated But the but people in a homelab will always notice that like I'm I'm sure of this like the minute that your significant Other is complaining about plex being slow You're immediately going to find that there's a bunch of things being, you know Ripped out of your um being ripped out of your data center there So you publicly shared your plex for too many of your friends and now you don't have any bandwidth left Or you get a like a cease and desist letter from your isp or something So we generally know about these things pretty easily So I don't really think we have to talk too much about that But the three two one backup strategy is probably the last or one of the last I have a couple more foundational bullet points to get through before we get to the main topic So three two one Um, I got that from the twit network. That's the first place. I heard it. I don't it's been around for a minute Yeah, I don't know. I don't know if they um coined the term I mean leo the port for example has been around for ages in the industry. So it's hard to say I don't know if you like him and steve gibson because I think steve gibson was one that coins spyware for the first time If i'm not mistaken. So I believe so So the three two one backup strategy came from somewhere, but that's just where I heard of it first So it's basically three copies of everything on at least two different forms of media one of which should be off-site So, um, I'm surprised I didn't have to rehearse that whole skill, but that's basically what it means. So Uh, they to break that down You don't want just one copy of anything that that's a risk, you know of and by itself So three copies of everything that's important, you know, two different backup media could be an external hard drive You know a nas or whatever you have and then one of one of those things should be off-site So you should always have the off-site backup in case, you know, you get a flood or something that You know just hoses everything and one time I came close to this so you think you you think it won't happen to you right but I At one point had water drain over My server rack for my home lab like a lot of water and this was years ago Like when some ice was melting outside it seeped inside and then drained Right over top of my servers. Thankfully. I just had something sitting on top of the rack that doesn't belong there That was just dispersing the water around the server rack and I was this close to having like everything Um, oh actually it was the cable modem the cable modem saturated all the water and caused it to spread over So I lost the cable modem and thankfully I didn't lose my data But if I did, you know, I would have wished I followed the three two one backup strategy You never know if something could happen a crazy lightning strike or something right and on topic of off-site backups So ideally it, you know, you can back up at a friend's house or something like that That's actually kind of a cool I thought about talking about that as a project, but it's pretty simple There's different tools you can use with true nas and even tail scale I've actually done a video on how to set up true nas and tail scale where this doesn't Doesn't need any vpn set up outside of tail scale where you can have two true nas servers talking to each other One of them remote you compare them locally So they get everything done fast and then do the incremental changes and take it off-site I mean dive deeper into it as a topic Maybe because j moodle bit closer to me me and j will set up a reciprocal thing where we can back up each other's data Um, that sounds like a good idea. Yeah, so, you know that we have off-site copies But the reality is if you don't have a friend's house, you can put it at And you maybe don't want to spend the money to put it in your own Private data center that you have in your back pocket. You may go. What about places like back plays And those are actually some really good solutions if you're using snell g snell g has their c2 cloud back plays has their system and I say this because We have come a long way and we don't worry as much about these cloud vendors like we used to because We have the ability to encrypt before send now with this Wonderful encryption you have prior to sending. Please. Please. Please make sure you have backed up thoroughly I don't care if you've printed it out on paper whatever you use for that encryption password Because think about how you recover from a loss because this has absolutely happened where people have lost the nas And realized oh the only copy of the password I created a great password and I saved it in that nas to upload that data And the only way to get that data back it'll come back from the cloud They will hold encrypted data for you and they'll give it back to you But there's no way to decipher it without that password But because you're encrypting prior to sending there's not a worry of these cloud providers because well Security's hard and eventually mistakes may be made and that risk is there. They may get these encrypted blobs There any threat actors ability to Break this encryption is completely dependent on how good of a password you use So hey, it's easy to be able to do this. It's easy to be able to do Encrypted backup so it kind of removes that ability or problem you have Where I don't know if I trust these cloud providers I've been using back plays for years just because they've been around a while to me I feel like I'm supporting two things one. They're affordable and reasonably priced to As long as back plays exists as a company and they keep doing this I love their drive reports. They give you a really cool His that I like the transparency back plays has about things and their commitment to open source has been good I feel they've been they've been committed to giving back to me That's like it's kind of a win-win It's a company that actually is doing something that I need back in up data making it affordable And then actually giving me some information back about, you know, what what happens all the drives that my data's on It's just kind of a win-win for me Yeah, it's a good idea to uh, take a look at their reports before buying a hard drive because uh, you know If there's a failure rate in a certain model That's higher, you know right now than normal You might want to avoid that one because there might be I don't know a bad batch or something came out Who knows at least to know which ones are more reliable, you know They since they use a lot of hard drives, you know that they know what they're talking about And we are going to talk about back plays a little bit more But um, we should talk about why versioned backups are important. So Yes And that the thing is about this there's many reasons why this is important that the you know at the lowest level Sometimes we make mistakes. We accidentally save over something we didn't intend to and maybe we wanted to just go back and You know get an earlier copy of that to undo the mistake. I mean, I find myself sometimes going in Um snapshots and true nas to restore something that I thought I wouldn't need but ended up needing it happens But that that's one run reason But then when you have ransomware and all this all these other forms of malware out there You know, they're they're trying to get into your file shares and they're trying to encrypt things and often ransomware will have things encrypted before it announces itself It's not going to say your files are encrypted before it actually encrypts your files Unless it's a beginner at creating these kinds of things. I didn't think that through very well But versioned backups are really important And I I tell this story sometimes to kind of tell why this is important This was earlier in my career. I had a computer repair business I stopped doing that a long time ago But I had a few clients even after I stopped that just insisted that I helped them out and one of them Came to me in tears because all of his family photos were encrypted all of them No version of backups of any kind So I remember at the time I I gave him a copy of all of his files And I'm like, you know, just hang on to these files because someday we might be able to break that encryption can't promise anything But you know, we we could keep checking back and and seeing but you know, he didn't think about this kind of thing and Maybe some people don't until it's too late But if you had versioned backups, perhaps he could just go back before This happened and restore the files back to their original unencrypted state Assuming of course that the malware doesn't go after the versions as well, which they kind of do now And that leads us to my next point Why you should not have your external backup drives connected all the time because if you had versioned backups And it wasn't within reach of that malware then it's a lot easier to recover from that And um, also you could have a power surge. I could take out external hard drives as well So there's a lot of reasons why you shouldn't always have them connected security and stability being two of the biggest ones And with that said we're moving right through because we mentioned this already the offsite backup at a family member's house or a friend's house I mean, it's it's so cool that people, you know, when people do this It's kind of funny to me to think about the reaction on the other end Where there if it's a non-technical person like well, why do you want to keep your data at my house? Like isn't your your crazy amount of servers at home? Good enough And then you explain well if anything happens, it's it's great to have an offsite copy and um, you know That's why some people do that We'll talk about back plays being an option for that here in a bit But that's that could be one thing that that helps out a lot of people Just especially if it's a family member and they're they don't want to bill you for this then it's even better The only problem being you could kill their bandwidth if they have a bandwidth limit with their isp You should always check that first before you start doing this because you you could be doing them a disservice if they start getting Out of bandwidth errors or a bandwidth bill or something like that So always be mindful of exactly how much data you're sending there because that's important too Yes, and someone asked a question in in the chat and that's what I said I'm using when I see back plays I am talking about back plays and their b2 service their bucket service that they offer Um, it's actually they have their b2 service, but it also is s3 compatible So if you have a ability to back up to s3 that is another way you can back up So there's native b2 which is built into a lot of things or there's s3 s3 is an open standard by amazon And obviously most popular by amazon, but other places use it as well for backups that does include back plays Yep, absolutely. We should probably just get right into back plays since it's something that it's hard to talk about Home labs and backing up without at least mentioning them and they've already come up So just like you said b2 it's like a bucket service. It's a it's object storage Not block storage block storage being hard drives and things like that the things that you've always been saving things The things you're the kind of thing your movies are on but we'll we all know What we're talking about here with home labs since we have one, but um Object storage may not be as obvious to some people. That's essentially a key value pair With the key value pair being name of object and the actual payload or the object itself It's not the same as a linux file system where you have You know actual permissions It's there's really no metadata not not so much It's just a bucket to store files into and that's just how that works So it's simple and there's ways of mounting these like they're an external storage to make them act that way But initially object storage by itself natively is not like that It's essentially something where that wants to receive files And then send you back files almost like it's a new age version of ftp in a way Yeah, and it works Over more like a standard tls protocol The other advantage you get is you can set your life cycle and revisioning inside of back plays as well When you set up the object storage So even though it's encrypted you can still say hey keep this many versions that way if your Data and we talked about this like we're we're visions you can still roll back to a previous revision of it And one recommendation that I have if you use object storage is to just make sure things aren't being duplicated It it's probably unlikely to happen And I think this bug was fixed by ran into a bug with true nas where it's supposed to overwrite something on the other end On back plays but it doesn't because it doesn't it didn't know at the time that it was the same object And that it thought every time is just backing up the same file over and over again I'm pretty sure I haven't seen that lately. So I have to assume that it's fixed I have a video where I cover what has to be set where to make that work properly I must have been inadvertently did it right then at some point But just make sure it's not duplicating things and that it's it's sending what you think it's sending You just check your bill every now and then it's the thing about back plays is that it It seems to be a lot cheaper than I thought it should be for as much data that I have up there I've had terabytes of data up there. I don't even know how many and it's I think my bill is like 40 dollars with terabytes Up there. I mean that's pretty cheap compared to how much I think like four or five Maybe even six terabytes or something like that. I have up there. It's crazy Yeah All right So next we should probably talk about sync thing, which is another thing that we brought up a few times and You know, it's not backup, but it can be now imagine if you have a Let's say a sync thing server in a cloud instance And you have a firewall to where only you can access it in your home network to get to it It's it is off-site. I mean, let's be honest. I mean, that's not even your data center That's in the cloud. So you could use it for that But it's not in and of itself a backup solution It's a syncing solution And it's something that's really important to me because I don't know what computer I'm going to be on from one minute to the next I have a few and I don't want to think about Which computer I've saved which file on I just want my documents directory to be my documents directory the same on everything Kind of like we had roaming profiles. That was pretty popular But yeah, and the windows don't really ask about that as much now Although it does come up sometimes roaming profiles being where you have, you know, a network Profile server that serves your home directory or your windows user files Syncing thing is just making sure both ends have the same file So it's kind of the same thing, but whether or not it's a backup solution depends on how it's implemented You do have versions in there. So you can do versioned You know versioned files and go back to previous files They recently added that to the ui, which I thought was pretty cool because you had to go You know into a secret folder. It's a hidden folder not really secret But yeah, your previous versions will be in there now You can just click on something right in your browser and it just brings up a list You just click restore right then and there it makes it pretty easy Um, we have videos on sync thing But the way I use it that's always worked the best for me Is to use the star topology if you could apply topology to a syncing solution Instead of having like my laptop synced to my desktop that syncs to another laptop I have everything syncing to my true nas So every device syncs to only one thing the true nas device So if my laptop updates a file, then it gets updated at true nas and then another computer We'll see that it was updated on the true nas and then pull that over So that way I don't I feel like you have fewer sync conflicts that way if everything is only thinking the one thing And that just makes the most sense to me and that's just the way that I do it And I've never had a problem using it that way So that's how I recommend doing it and an added bonus With true nas is true nas has snapshots So if your syncing files are there and you have a snapshot schedule You have syncing given you version backups, but you also have true nas having a snapshot of the entirety of syncing So either way you have two different places You could go to to pull a earlier version of a file or to restore a file. So it's it's like a double benefit Yeah, I do almost the same thing It's not exactly star Maybe we'll call it double pulsar because I have two true nas servers one at my studio and one at my office those two true nas studio the true nas studio and the true nas at my Office do talk to each other via sync thing and then my desktop computer talks to both of those So in case either one of them is down it always has two connections So it's still kind of starter typology because anything connects to those This allows instant syncing of all my files But one more thing someone had asked about does sync things support encryption Now the transport layer between any two syncing nodes is encrypted But there's actually a step further. They added a few years ago and I thought this was really cool if you have A syncing node that we let's say Stick in a cloud somewhere we put it in a linoad instance Now obviously I don't have control over the back end of linoad. So who knows what can happen It's in the cloud. It's publicly accessible But me and jay would like to share files and we don't want to have a vpm between each other But we both can access that node you can set up what they call a Why I want to refer to it as a blind node they call it an encrypted node But it's blind to the data that traverses it and it's blind to the data that it stores So me and jay can synchronize files talking to this intermediary that's posted in the cloud But if you were to get a hold of it, you'll find that you can't even understand the directory and structure It doesn't just encrypt the files it encrypts their names and it obfuscates even how the directories are structured To help destroy evidence of metadata if you will or create enough entropy So you can't defer easily what the metadata even was as in the file names or the structure that we're using So me and jay can have an entire anything syncs between us a bunch of files projects We're working on and if that blind intermediary gets taken over It doesn't have any information because the password that we use to decrypt it is only stored in our local sync things The one on the end there the one in the cloud just handles the encryption back and forth It sees the changes now It does lose the ability to revision because it doesn't understand the revisioning because it doesn't because it's lost it You can turn it on actually it just is very you don't know what files are which so you could say Hey, let's roll back to another version. It goes. Okay Take this long string and roll it back to the previous version of that long string We don't know what file me and jay wouldn't even know what file that is unless we know exactly which one we last did But we can still do the versioning on the local instances that do have encrypted so you can see what they are I think this is such a neat feature of sync thing And then you combine that like you said with true nas snapshots where you're snapshotting on a regular basis This allows us to roll back to previous versions So something terrible happens I've overwrote a bunch of files or you know worst case scenario Someone has gotten a hold of my computer to overwrote my files and it syncs between my two true nases I just roll back to a snapshot of which I have 30 days of Yeah, exactly. Yeah, that's exactly how I use it too. So I think that You know, I've never looked at other solutions since sync thing because it just works And I another thing I like about it is it doesn't change much now What's interesting is they they're constantly releasing new versions like I can't count how often this happens But it doesn't change much even though it's there's new versions I think the feature you mentioned and the one that I just mentioned are two of them the ability to restore right from the Web interface I discovered that on accident But you know the reason why that's surprising is because they don't really add all that much It it looks almost identical to the way that it did years ago So if you want something that's stable, I guess this is more like almost like the Debian of sync utilities at this point Um, it's a very stable product doesn't change. Yeah, it's stable Yeah, I need to do an updated video on it because my the best thing about it being stable is you can go a video I did probably I think six years ago on sync thing. It's still relevant It just doesn't mention all the features that have been added But if you looked at the way the ui looks like oh wait, that's the way it looks today Six years ago, which I'm 100 fine with they didn't reinvent the they added dark mode. I will give them that thank you for dark mode So dark mode. I don't think was an option when I did my first syncing video So I don't know when that came. Yeah, it's another thing. I just discovered poking around at the settings. Oh, what does this do? Oh, I don't look at the ui very often I'm happy it's there that they have dark mode but honestly I spent once sync thing is set up and much like yourself it's kind of uh This is the the settings done and I want to mention you didn't mention this one of the cool use cases jay has for syncing is jay's big into the retro emulators And it syncs all of his save games so he can set down one retro system pick up another and he's got his save game synced And I'm like, yes Yeah, exactly and the and the roms too because yeah on all the rounds I'm a big retro gaming collector. So I have the physical games and the physical systems But sometimes It's easier to use an emulator than it is to pull a sega cd out of the closet and have all the cables I mean that's still novel But I like the you know the emulation side of things because it preserves these things But what I don't like to do is use one retro gaming system and then use another and forget Which one I currently had my my game on that I was playing if I want to play final fantasy 6 I want that synced through every single device I have so I have syncing on the retro systems syncing the roms and the save files and that leads to another feature The roms share I call it a share, but it's a sync is one way it's server to the end units It doesn't sync from the end units back So if I make a change to one of the rom files, it's going to be undone because it's a one way sync I don't make changes on the You know game systems I just put the files on the file server and then it sends files So you could do a one way sync with sync thing if you want it to be Receive only and only receive changes not send any changes back And and depending on your use case you might have a use for that too. It's definitely a granular system Small reasons me and Jay love it so much because it's supported on many platforms. It's very granular It just makes that easy As a matter of fact, if you're wondering how I take care of log seek sync thing is what handles log seek even on my phone So I use log seek on my phone and sync thing syncs all the files for it One last use case for sync thing that I think that might be clever for some people If you use app images, which are standalone applications for linux, you don't have to install them You don't have to install libraries. You download the blob the application double click on it in the run So app images are pretty cool But one thing that's pretty novel a sync thing is to have an applications Sync and throw all of your app images in there and then put that on all of your systems and set your path Variable to include that directory And now when you update an app image on one system or even install a new one by just dropping it in the folder That application is now on every single system instantly like just like that You just drop an application and all of your systems have it just and that that's Just another example of the fact that with homelab you could find some clever use cases for things that Might not have occurred to everyone at first. Yeah, you can get really innovative with it I'm like I said, it's kind of the heart of a lot of anytime. I'm going to sync anything new I just dropped sync thing. I modify the config. So it just syncs everywhere. It's supposed to be I even synced joplin plugins back when I use that but let's move off a sync thing though because this will end up becoming the syncing episode We'll just never videos on that as a topic We should talk about next cloud So I'm going to give this a quick mention because it's especially not a backup server for a solution But it like sync thing it can be and it offers syncing as well And at first next cloud the syncing was kind of problematic and not reliable But it's reliable now the I was using it for a while And I switched to sync thing not because I had a problem with next cloud syncing I just like the way that sync thing does it better. It's a personal preference That's the only reason but so the sync syncing is solid in next cloud But it gives you an additional benefit You can access and edit your files in your browser with the office Or the word or excuse me the what office suite is it again? I forget the name of it, but you can install an office suite right there and edit your files inside your browser And that might be a benefit that some people might want to have especially if you have non technical people That want to use your home lab. It might be easier to give them a bookmark and say your files are here You can edit them right in your browser. They don't even have to know anything Um But if your next cloud is off-site and is syncing you could argue that it's technically off-site backup because it's pulling your files in that direction I have videos on next cloud that Goes into way more detail about it But I just want to throw a mention there because next cloud is one of those things that We talk about a lot at its core. It's it's essentially competing with Google workspace or whatever google's calling their business solution. Um this particular week whatever it's called this week I think it's Google workspace still so it's essentially the same thing It you could have a email service right there inside your browser Just like you would have gmail as part of that you could have online document editing inside your browser A calendar contact syncing Next cloud could really become a central hub for a large number of things that the average person does in their home lab So it's definitely worth checking out. So I wanted to put a mention out there for it. It's a great piece of software Now another thing i'll mention Also quickly Is synology which is something that we don't talk about as much as true nas But it it's also a good solution And you could install sync thing in in other applications as a nas But it's something you can also install your backup utilities on it And I think you're probably going to mention their backup or their enterprise backup So they have a few different backup options and the active backup You have active backup in case you wanted to back up your office 365 if you have a business account You can back up your gmail your google workspace as we joked whatever it's called And you can back up your computers. You can do your windows system as a matter of fact I use the active backup to keep an image the only windows computer that's in my regular usage is my studio computer that ingests all the Media and it works great with active backup because it will do images of that system And I've had things break on my studio computer once you get there's a lot of things because I got all the capture cards Everything in there and occasionally something will bad. I've used it to roll back things that have gone wrong And it's it definitely gives me the competence I have it actually backing up on action that action is every time that system gets rebooted It automatically does the backup and I turn it off anytime. I'm not using it So it gets backed up all the time. But the active backup is just a Really great feature of synology and it's kind of a good selling point now for those of you that want to integrate Or have like I do a synology and true nas because you can't decide which one you want because they're not exactly feature parity There's advantages and disadvantages both You can synchronize the data between them using rsync and I do have a video You'll find if you type in synology rsync or even true nas rsync You'll land on the video where I talk about how to synchronize the two of them So you can get the data from one to the other using rsync Yeah, I need to look into that because that's something I need to set up my myself I figured that'd be something good to do and I was going to ask if there's a newer Solution because I could never get the built-in gooey version of rsync to work on true nas with synology I always had to go on the command line and write a bash script for it not that I mine because I'm I like that Kind of yeah, I just feel like there should be like an easier way to send data from finally I've made it easier. No, you know with with true nas scale If you use the rsync plugin in the true nas scale app, that's made it a lot easier and you can combine that with Hyper backup and rsync. So now you don't have to do the command line to make these things talk to each other The downside is of course There's not a lot of security between them when you do it this way They're just passing open rsync back and forth. So SSH if you encapsulate an SSH you're going to have better security So that's still an option to do but if you have a storage network where these two devices are talking You're fine with that. That's the way you can handle it I think another issue for me is whether or not You know, they're the turnkey version Would be as extensive as my bash script because I always go into detail. I don't probably don't need to But what the way I have my rsync backup script set up is that they any time they run they create a new folder That's named the date. It's just you know 2024 dash 01 dash whatever the date is And every time it runs it creates a new folder for that day because it runs every day And any file that gets replaced when it syncs the most recent snapshot of everything Any replaced files go into the date folder and because they're in a folder named by the date Then tomorrow, I don't have to worry about overwritten files Overwriting the previous overwritten files because it's going to have a different folder with a different date So if I want to pull a different file, I just go to the dated folder that I think has that file around the time I replaced it and then I'll go grab a backup copy of it And I don't technically need to do that because I have true nas that has snapshots of things I have sync thing that's you know has versions of things But I have rsync doing the same thing, but it's kind of like I have the script I may as well use it so I might check that out to see how it compares nowadays I think it's been at least a year since I last looked into the turnkey method to that Yeah, um But we don't jump the next topic. There's two questions. I do want to answer one tape backups Are they a good backup medium? Are they too expensive for the home lab? I think that's kind of a personal preference. I'm not a big fan of tape backups unless they're absolutely necessary I prefer to keep all my data available and spinning With the exception of I do have a extra synology Tube a I think the ds 218 with a couple large stripes just in a single mirror I plug it in. I synchronize things. I physically unplug it like power it off But I I just prefer things on disc because I feel like it's easier to validate the integrity and things like that versus tape I teams are pretty reliable. They're not bad But I always fear like if I had tapes I'd have to do two tapes for each one because what if I'm pulling a tape and it says Cannot read this tape. I mean years ago. I've had this we used to do tape backups a long time I was so happy When I didn't have literally a giant pile of tapes anymore So, um, I feel like it's going backwards But I get it if you can't afford the spinning drives and you're like, hey the the price and data density of these Seems reasonable. It may be an option. The second one is going to be The myth of a mutable. We'll call it the myth Immutable backups that is something people want, but it's only as immutable as accurate to that control plane is So if you have access to that control plane You can say the backups are immutable when I have a share that is shared out via smb to Some windows machine and I've stored data on there and then that windows user There's snapshots on there But the windows user decides to delete them The windows user cannot delete those snapshots because they're immutable But they're not if someone gets a hold of the true nas itself There's no way to make data not deleteable It only is not deleteable up until someone gets access to whatever the control plane that that data is on So if you have the username and password to your true nas server That is going to be the accessible thing. This actually is something we worked with a client for that had a ransomware event And they did not get their true nas servers the first time they got ransom. They got answered by second time and uh Because they didn't listen to our advice But what they had not done was roll their passwords off their true nas servers. They were able to delete things They were to recover from the snapshots and they thought it was great. I said roll those passwords They just they were assuring me. They had got the threat actor out on the consulting. I said, I don't think you have I would roll the passwords and Yeah, that um, they ended up Getting that so your immutability is only as good as the access to the control plane can be restricted So that's why there's not really There's not really a big talk about immutable homeland backups But you can set retention policies and something like back plays and say don't allow this to be deleted for so many days But then if I get the master password account to your back plays It can be deleted and obviously someone at back plays Could delete it something could happen at back plays to cause it to be deleted So nothing's really immutable in the tech world as much as we'd like it to be I hate that term when I see it. Yeah, that's a requirement on insurance forms This is this is a discussion. I've had too much Yeah, it's hard to explain these things to the people that don't know about them one comment. I'll make about tape is that Um, I think I'm conditioned to hate it or they have bias at no fault of tape. Okay. I want to be clear I'm not about to find actual fault with tape But as a kid I would have situations where A vhs movie would get eaten by the vcr and when that happens that sucked I I sit down for a movie with popcorn and now the tape's ruined and that happens so many times It's the first thing I think of when I think of a tape, you know, like I'm not saying that's It's likely to happen or anything, but you know It does kind of worry me a bit But one good thing I'll say about tape is it has a long shelf life or data like everything you store data on And I'm not going to go down this rabbit hole in this episode because it's an episode of and by itself When you talk about data retention and shelf life archival media Exactly because you know, I remember a long time ago like years ago when burning dvds was popular somebody would say Oh, I'm good. I backed up my hard drive on dvds. I'm like, well, how long ago did you back it up? Oh, it's just uh, they'll give me the time period. I'm like, you do understand that on average Uh, it start, you know cdrs and dvdr start losing data after like big 18 months Well, but people didn't realize media they're sort on. Yes Well, even but see all the interesting thing about optical medias all of it is starting to show fault now We have we thought factory CDs would be like permanently protected from disc rot, which is what this is called But we're starting to see early generation You know cd games like sega cd like back when cd games were first coming out first We thought that wouldn't happen. We're starting to see disc rot on those old discs So everything has disc rot even like spinning rust hard drives lose magnetism after about a year of being on your shelf So my so i'll stop this rabbit hole right now and I I say all of that to say this Just look into it don't go down a rabbit hole But understand the basics and and look at this about the shelf life of your data because it might surprise you Just how little time you might have with something's just sitting on the shelf So that's you're always in your mind and you're always fighting against entropy with these This is the the reality of it the cd burnables that you have They actually use a die versus the stamping methods that are used on like he says sega cd But there are some life expectancy issues on any media the charge you put into a little ssd Starts fading the moment you're done charging it if it does not get done again It does fade over time just like the magnetism is not some absolute permanent It will fade over time. This is why there's so much air correcting on any of these drives It's a fun rabbit hole to go down. It'll make you nervous about it. This is why when I say I keep my data spinning it is On zfs it is on these constantly being integrity checks and there and just completely reminds me and has me paranoid As I glance over I was thinking about a dvd I was like I still have in a drawer a 20 year old dvd that've been meaning to copy just I think I have a copy of I wanted to double check because I want to start getting rid of some my old dvd media It makes but I want before I do so I want to make sure I've got a copy of all of it somewhere And I that sounds like a lot until you realize how small files were back then This is literally why I have personally made ISO images of all of my DOS games Like mortal combat 3 sim city 2000 You name it. I have an ISO image of all of these because I know at some point I'm not going to be able to play those games directly off the cd anymore But you know, that's not really an issue. Uh, don't copy that floppy, but apparently I am so here we are I wanted to just have a couple quick mentions just honorable mentions real quick So the episode we have a lot to talk about today I wanted to mention lvm snapshots mainly just to make sure everybody knows that this exists because I found this out late in my career. No one talked about lvm snapshots. I mean we talked about lvm constantly Like why it's a good thing to have You know, you could online resize things like that. You could add another drive to it And you know expand over that drive. There's a lot of good benefits there But no one really talked about lvm snapshots in the career that I've seen And I want to make sure people know that it exists You can literally take a snapshot if you run on lvm Make a change maybe install a piece of software try it out and maybe you hate it You want to go back you can uninstall it or you just revert the snapshot And it's like you never installed it and that'll make sure you don't have cruft all over your hard drive too But lvm snapshots are not the easiest method. I know that doesn't scare us. We're homeland people if I say it's difficult Someone's like that sounds great. Like that sounds fun. I'm going to learn that Um and get mastery over it But lvm snapshots are not something you can just let go like if you take a snapshot It will eventually fill up your hard drive. It takes a long time But the idea is you have the snapshot for as long as you need it then you remove it Always remove your lvm snapshots and use them only as long as you need them It sure it could take months where it fills up your hard drive But if you leave a snap snapshot dangling there and you're not You know watching it then you could find out you have no hard drive space left So just keep an eye on that and I just mention it just so people know that it does exist You can look into it and see if this is a solution that might be a part of your backup scheme Yeah, I'll note. I believe jay has an entire video on lvm. Do you not? Or maybe a bit series on it head over to have a one big massive video that covers All of lvm. You know pretty much. Yeah, all of lvm and even my new arch Linux video that I just put out Which I've refreshed just just like this week Tells you how to install arch Linux on lvm both the easy way and the manual way So if you want to get up and running in lvm, I have a coverage for sure Um, the last honorable mention that I'll mention is the proxmox backup server Which is another option and yes, I do have a video on that too. If you want to learn how that works There's also a video for that but that's definitely something to consider Especially if you have a spare server lying around and you want to find a purpose for it Maybe setting up proxmox backup server could be a good use case Yep Absolutely, you don't have to just back up proxmox with it by the way You could back up your files too. You could back up your proxmox vms But also your files. I just want to make that clear in case someone might think it's proxmox only not necessarily I think it's what we learned before when we talked about that was the proxmox backup server supporting more than just proxmox Yep, which I thought was cool Yep, so it's all in the video and and I guess now I've given everyone several weekend projects So you could tell your significant other in advance. I'm very sorry. Just blame me and it'll be fine Just get your dive into this One last thing untested backups are just wishful thinking So if anything else remember that, you know, because nobody cares about a backup that works Everybody cares about a restore that works This is what you really want to focus on is test any of the solutions we talked about or any scenarios Validate them. That is how you know it's working. So absolutely. That's true All right, and with that we're out of here. Thank you for joining us We have an upcoming interview with the folks at net data. We're getting that scheduled That look for that that's coming out. That's I'm excited about that And oh, you have another series on your channel. You release. Just make sure you get that Right. Yeah, I think I was referring to the arch Linux video that I just Okay, you have an arch Linux video check out the notes Yeah, it's like a it's it's I think nearly an hour long like it goes into in depth Both methods of installing arch Linux and it's not brand new because I've had this video on the channel before but since it's a rolling release Every now and then I just So if you want a deep dive into arch Jay's got you covered on that check that video out I I was the one you were in a shirt and didn't your shirt say something on it something like I run our by the way. Yeah I haven't that shirt in the video because I thought I almost thought about rerecording the on camera moments But I didn't want to just uh flaunt that or make that meme too much because I wanted people to really realize This is a serious tutorial, but I figured I would have fun with a thumbnail I don't normally do this, but I figured, you know, why not I'll just throw a thumbnail out there and see If it makes people look at it cross I hear something. What is he doing? Yeah, yep I run by the way Run arch by the way Thank you everyone for joining this live stream and listen to our podcast and we'll see you next week later