 Hello, everyone. Welcome to my talk. My name is Tomasz Fronczak. I'm sure all of you knew straight away how to pronounce that just by looking at it. I'm a local here based in Berlin. By day, I am a lead SRE working as one of the local fintech companies. By night, I play with homelabs and self-hosting, mostly in order to achieve more data privacy for me and my family. Want to find out more? Here's your link. Today, we're going to talk about backups and restores and stuff like this in the context of self-hosting and running things in some more bare-metal situations. So, I'll start with a quick round of questions for you lovely people. I'd like a raise of hands for anyone who actually hosts anything or manages or hosts a service, be it at work or at home. That is less than I expected somehow. Interesting. So, did any of you ever have to actually restore from a backup? Like something broke and then you had to restore? Fantastic. Awesome. And then those that didn't raise a hand, when is the last time you tested that you can actually restore from a backup? Nice. But yeah, from the show, Fronczak can already see you're the right audience for this talk. So, let's start with a cool anecdote, GitLab. It's a small company. We probably haven't heard about it. They do code-hosting and they generally use PostgreSQL clusters to store the databases. And they are really good about their backups, free-to-won, everything. They have LVM snapshots on the disks where the database runs. They have that copied over to another series, an offset backup using PGDump, whatever the command is. And they even have, like, an auxiliary, another backup that syncs as well in S3. Or at least they thought they have, into one day, they had to restore from a backup because one of the engineers accidentally did a whoopsie and deleted one of the light database instances. Turns out he went to S3, his leg out just rolled it from S3, and then it turns out the S3 bucket was empty. Then he found out the PG backup command was silently failing for weeks, so they had, like, two weeks old data. Oh, and the LVM snapshots were enabled but not on the one set of disks that was running the database just for the other stuff. So that was a fantastic ride. I highly recommend reading a blog post, two takeaways from this. Don't be like GitLab in terms of not checking your backups, but also be like GitLab and write fantastic post mortems. There's a link to it later in this slide. So with this in mind, if we look at how some of you might be running the next cloud instance, this is more or less how I run it. I have a MariaDB for the database, just a single instance. Then you have the configuration files that have to live somewhere, and then you have the actual data for the files. And how you may backup these really depends. In a lot of scenarios, you would say, okay, so my configuration is actually somewhere in a Git repo, and then it's also on my local hard drive because I'm working on it or changing this stuff. So with MariaDB, you can also have a backup drive running to a disk somewhere or a snapshot or an FS, and then use the data. It's just like a mix. So now imagine something breaks, and you use all of these built-in mechanisms for each part or each data type. It's a bit of a hindrance to figure out what to restore from where if it's not uniform. You don't really have a good idea of, or good control over where the snapshot of each data was taken. So I had this problem, and I completely borked my instance of Next Cloud and then couldn't recover it. You live, you learn, right? So I thought, okay, let's start over, and I'm going to try and rewind and take a closer look at how I'm going to do it. I need it to be simple. I'm doing this in my own time. This is not my job. I need it to be simple. I need ideally all of the backup and restore operations for each data type, and that's not just for Next Cloud but all the other services I'm running to be ideally the same, right? I don't want to have to learn five different types of backup recovery mechanisms and then juggle this. And because that's what I work with at work, I wanted to be all running in Kubernetes because that's what I'm familiar with. You do you. You use whatever you're familiar with, but I recommend sticking with what you know. Then I stumbled across this beauty of a system, both open source, I think both from Ranchelabs. Three years is a lightweight distribution of Kubernetes, and Longhorn is basically a layer of storage. It exposes the local storage that you have on your Kubernetes nodes as Kubernetes volumes, and then uses iSCSI to mount these into your nodes in the background, so it doesn't matter where your volumes live. The coolest thing about it though is for backup reasons is it can automatically backup to NFS targets, and it does a multi-tenant backup, so you can have multiple copies of this, or multiple clusters backup to a single NFS pool, and then they can all see it, and you can basically use that as you see here as a mechanism to transmit your backup. So again, three to one. Live data lives in my Kubernetes cluster in my homelab. Then I use Trunas Scale, another open source tool that you can freely use, and then I ship all of the backup volumes via WireGuard to a friend's place who also happens to be running Trunas Scale. All of this is done basically without any code. This is all already ready of the shelf open source tooling. You can just configure it mostly for the UI. So how do you say, okay, what does that have to do with restoring a disaster recovery? Well, the cool thing is all you need to restore your service is spawn another Kubernetes cluster, and ideally, you have all your config data to do that, or you have it already done in the first place, and then you can just load all the volumes from this copy, remote copy, and then to, you know, slam dunk this baby. You just point your HA proxy to the new copy, and now you're basically restored and you're back alive. And it is super simple. I recommend playing with it. Key takeaways here. Start testing your backups and restores today. Don't be like GitLab. And also make it all simple and make your backups, or design your backups with recovery in mind in the first place, just to not have a headache when you actually need to use them. With that, thank you very much. So for me...