 So I figured we'd talk about the backup and recovery process. So I did a video the other day and it was all about, you know, the oops, I accidentally said yes to a production server and updated it and people, you know, like I'd said in a video, I'm not panicking because we have a solid, clear backup process the way we can handle all the data. So I figured I'd cover what our disaster recovery process is here in March of 2018. So it is a Sunday, I happen to be here, we just finished recording this Sunday morning Linux review, and I usually take that as an opportunity and I can do this remotely but I do like doing it here, backing up all of our servers. Now for those you don't know, or didn't watch any previous videos, you're new here today, I'm still running Citrix Zen server, and I know someone's screaming, but Tom, Proxmox, Proxmox, and let's cover that real quick. Here is the procedure to import like an OVA over into Proxmox. It's not real in depth, but obviously it's, there's some steps to it, the same with exporting it back out, there's a lot of steps to it, versus in my playlist here, and you can find this in my Citrix Zen server, Citrix Zen server scripts for backing up your virtual machines and how to restore them in OVA format. This is something really nice, well, XVA OVA, their standard formats, and that's what I, that's what we use, and this is why we use Zen server over some of the other ones, and I know the Zen servers made some changes, I'm aware that they want you to license for some of the Zen motion stuff now, though, that closes that out. This is more or less Zen server has very simple ways of backing up and importing and moving virtual machines over. So close that window there. This is Zen Orcasha, and I've talked about this before, Zen Orcasha is a great web interface for Citrix Zen server. This is mostly how we manage it, because I know the tool is in Windows, I know they have a tool that is in Linux, kind of a port of the Zen server tool, one, it's, it hasn't been updated in a while, two, it just don't really use it. I don't need an application. I like things being a web interface. When it comes to most of the backups, we don't really need to backup any of the workstations at our office. Pretty much every tool we use from screen connecting and everything else or invoice center for invoicing, it's all a tool that is web based. So there's not like a ton of data locally that we really have to worry about for the limited data that we do have to worry about. We just use syncing to keep it in sync. And I've covered that in some other videos. You can find them in my channel. So let's cover specifically. Here's like our production machines that we run. This is the Zen Orcasha appliance, wiki for documentation, unify video, screen connect, old point of sale, which we've migrated to the new one, but we keep it running because occasionally just in case any mismatch eventually we're going to kill it. So it's kind of, I don't, it's backed up, but eventually it won't even need to be anymore. This is not, this is a mail server that we run, but the mail server is mostly for notifications for internal servers and let me know things. It's not my primary domain. LearnSystems.com is a G Suite address, but we still have a mail server here for playing around in a couple other random domains and personal stuff that I still test with. Invoicing, gentlemen, we've talked about this before. It's what we're using for invoicing, once again, web facing. And OSEC, if you're not familiar with OSEC and someone asked me if I had done a video on it, I haven't, but OSEC is a host intrusion detection system and this is the host system that keeps an eye on things. Sometimes I'll do a video on that, but look it up. It's a pretty cool project. It's fairly in-depth, so the video will probably be pretty long. Anyways, it's not going to do all on that. So here's all the main virtual machines that we have in-house. And I use Ansible, I've got a whole Ansible tutorial video. I'm not as good at it as Phil, but I've got the structure set up. So I can use Ansible to control all these servers with one command. So that one command is often just shutting them all down. So we turn it off all the servers and then we run the backup scripts that we talked about there. Now that backup script then creates these. And I'll just show you what it looks like. I only need two revisions. So there is the current status of running, then there are two backups created. The backups are dated and the oldest one is automatically deleted. This is part of that script that I run. Let me show you just how big the backup server is real quick. So this was today's backup of InvoiceNinja 9.4 gigs. So let's show you what those files look like. Whoops. And there's just invoiceNinja.xva. No big deal. If I want to import that into another machine, I would just grab that xva file and pull it back into any Zen server, not just this one, any Zen server. If I wanted to right now copy of it, I can just hit this and export it back out of here. And it will just download no VA file. So really no big deal there at all. The backups aren't very big. So there's a 9 gig backup times two. So there's 18 gigs of InvoiceNinja. And overall, actually let's do it this way, 138 gigs to back up all of my virtual machines and there's two copies of each of those machines in here. So those are all the virtual machines. Now the first thing is obviously I care greatly about the security of these virtual machines. This is all mounted onto our FreeNAS box. So there is a SAN setup, and I've talked about this in my lab setup, but it's the same thing. It's the main head end of the Zen server, then tied through a backplane at 10 gigabit to the FreeNAS box. This is how we provision the Zen server, so FreeNAS access storage, and then the Zen server is the head end that actually runs the virtual machines. I know I could run them all directly on the Zen server, but hey, I like running them on FreeNAS. FreeNAS gives me plenty of flexibility with ZFS and everything else. Now I keep, this is where the backup's coming from, I keep in second Zen server, sitting there. I turn it on, I make sure it works, I make sure there's any updates or patches as they need to be loaded, then I will randomly grab one of these virtual machines as part of my testing process, make sure I can restore to that other Zen server, then I destroy it off of that Zen server, then I shut it back off. It sits there, does nothing, doesn't use any watts when it's not plugged in. So it's physically disconnected from the network, that way any high voltage spike that may come through, it's just a extra spare server. It's not quite as fast as our main server, but it's plenty fast enough and has 40 gigs of RAM in it, so I can run all the virtual machines, which as you can tell, even running invoice engine, it just doesn't take much, a couple gigs of RAM, so all these virtual machines will completely fit in there. So if the FreeNAS server dies, if the main Zen server dies, either one of those scenarios, no big deal, we can import all these files over to the other server. Now they sit here on the FreeNAS server, which is encrypted if you power it off. So there's no shares directly to this, we copy all the files back over to my computer and I use full disk encryption and if you're not familiar with LUX, go ahead and look it up, LUKS, I use LUX encryption for the drives. Then to get the files out of the building, we then synchronize them after the backups are run with an external hard drive that's also LUX encrypted, which goes and sits on the shelf in a safe at my house. So there is a complete offset copy. So if something ever happened to the building to give you an idea here, the entirety of all of this is completely restoreable from the external hard drives that we keep an ICID to because I make more than one copy as I rotate which drive I bring in and synchronize. So really simple backup process. It keeps the data offsite. I could synchronize it offsite via encryption and get it off here, but I come here anyways, just about every day or at least once a week. So taking them home, not a big deal. I could have a staff member run this, drop it off at my house if they wanna do as well. So this is kind of just our disaster recovery process. It's pretty simple and it allows us to have a couple different versions of it. Now, in some of you may have seen, I also run a Zen server at my house. That's not by accident either. Zen because it's backups, exports, imports and standard XVA OVA formats, I can import it into my Zen server at home, just grab those files right off that hard drive, pull them right in and have, actually run my company from my home if something happened to the building. So we still have access to all of our data, all of our client data. We'd have to make some DNS centuries to point everything at my house versus pointed everything here into our own server stack. But you kind of get the idea that we keep it all completely separate. Now, a side note too, the spare Zen server, as I mentioned, does not connect to FreeNAS, does have local storage on it. And the reason why, obviously, because if the FreeNAS goes bad, I still have another server but I don't have any data storage for it. So I would still migrate to the other server. So it's about thinking ahead when you put these things in there going, okay, what if this, what if this, what if this? What if the building, something minor happens to server? What if something major happens to the building? What if there's a power outage for a long time? Whatever those scenarios are, we have a plan for it. Now, of course, you may have noticed there's a weaker part on these backups. There has to be things happening between. Yes. Once again, back to the encryption of everything. Everything has to be encryption. You always, always encrypt data at rest. So I don't really need to go over it with you, but they're pretty straightforward. These all run MySQL databases, MariaDB on the backend. So I just run a script that grabs the SQL file before, this is the important part, before it leaves the server, it's completely encrypted. That backup is encrypted, boom, sent over then to SyncThing, which is a, there's a private, SyncThing allows you to have, basically set up different containers. It has a private lockdown container that's not shared with anything other than one other server within encrypted hard drive off site. So it lands on that encrypted hard drive. So it's encrypted before it leaves. The only place that password is kept is within the virtual machine itself. There's not another way to decode it. I don't keep the password anywhere else. So that happens hourly during business hours and during off business hours, it happens every, there's just a couple other schedules, like a three AM schedule, in case something got created afterwards, but from seven AM, and I, business hours are broad for us. So it's from seven AM till nine PM every hour, all the databases are backed up. One of the nice things is even though having thousands and thousands of customers and invoices and everything else, the backups for things like the invoice ninja, for example, which is the bigger one or a documentation, they're not, they're less than a hundred bags once you compress them. So you encrypt, compress, then it leaves the box and hits over there and synchronizes. So it's not really, it's an inconvenience if anything breaks. Obviously it's really inconvenient when things breaks is you still have to, you know, some action has to be taken. But with these steps, we have a copy of every VM. We have a copy of every piece of the pie as it goes. We synchronize it. And even in, I can show you here when we are, let's see if I can actually have it pulled up already. Yeah, here's the script that pulls it over. Let me see if I can view zoom in here. Even in keeping with to make sure nothing just passes across when we've been using SMB share, this is a lockdown spot on the Zen server. I'm sorry, on the free NAS server to get the files. We're just using our sync with an SSH key to pull it across. And that's what copies the data there. And obviously I have cleverly named Vbox back up to local. So this is what pulls all the data back over and we keep a local copy that lands on an encrypted drive on my system. Like I said, I maybe I'll do a video if someone really wants me to for, if someone really wants me to do it for the Lux encryption, you can find a few people who have covered it, it's pretty straight forward to set up. It's built in to Linux distros. So it's not, not anything. Most of the modern distros come with it already enabled. So it's a great way to keep everything locked up. But that's really it. That's all that backups run. The other things that we do, maybe little side notes might be things like we, for any time I changed the free NAS, we download a copy and keep a copy of the free NAS backup configs. It contains the encrypted GEL keys. I've covered the drive encryption on that. So that gets backed up and it just gets dropped into a folder within the Zen folder. So it's once again replicated and backed up and locked down on these Lux encrypted drives. But hopefully this is a little bit insightful for how we do the backups. It's really simple. There's not a lot to it. It's just being able, and this is that, my coverage I have, and I'll put that video right in there for how I backup the Zen servers and that tool. This is all free open source to create full snapshot backups. And someone's asking, well, how do you do it for your clients? If they have a Zen server net, we've actually set this up for some of them and they can take the drives off site. Or a lot of times when they're paying, they use a solar winds system. And I've covered that one as well that will do full image backups of every computer. So if you like to count here, like, subscribe. Just like I said, give you an idea of how these are done. Pretty straightforward, pretty simple. If you got questions or think I'm a moron, leave the question below. By the way, if you think I'm an idiot, I don't mind. Someone said I do things insecure, that's fine. But can you elaborate? Because I like to learn and get better and not be insecure. So if I'm doing something really wrong, tell me and tell me what I should be doing right. Someone commented on one of the videos that the unifies are insecure, but of course, dead air when I said how. Anyways, if you like to count here, like subscribe and tell me what you think of this. Like I said, I'm open to updates or other methodologies, but maybe you're better. Maybe I missed something. Let me know. Thanks.