 Tom here from Lawrence systems and we're going to talk about disaster recovery planning and backups with Zen Orgisho So the goal of this video is to show you how one to properly back up the VMs and The different ways you can back them up and why one way is different than the other and why that matters and which one would be better But that does depend on your scenario. So we're going to play out those scenarios Second how to recover a Zen server. This has come up a few times Unfortunately, when it's come up is because people said I thought I could just Restore this file that I pulled and backed up over here and it turns out all the metadata is gone I'm hoping you are not in that scenario You're watching this before you get to that scenario and that's some of the goals here We have is to make sure you understand how these backups should be done Also, we're going to be doing this with the fully open source version as in there are no paid utilities in here or anything like That I bring it up because I like to make sure this is accessible for people wanting to test this out Try it with their home lab. So I will leave a link to this how to build the Zen orchestra from sources So we're actually going to be doing the demos with Zen orchestra built from source, which means don't use it in production This is for people who want to test it set up your home lab You're an enthusiast and you don't not need support from the corporate That's something I wanted to cover real quick those to make sure that yes, this is all open source Yes, you can get all of this and yes I'll leave the instructions which I have in this video right here of exactly how to set all this up And of course, I'll also leave a link to my entire getting started with Zen server video if you're unfamiliar with Zen server And you want to get it loaded and XC PNG specifically I have an entire video I'll link to below for that before we get further into this video if you can click that like button and first If you'd like to learn more about me or my company head over to Lawrence systems comm if you'd like to hire Sure project there's a hires button right at the top if you'd like to help keep this channel sponsor free and thank you to everyone Who already has there is a join button here for YouTube and a patreon page your support is greatly appreciated If you're looking for deals or discounts on products and services we offer on this channel check out the affiliate links down below They're in the description of all of our videos including a link to our shirt store We have a wide variety of shirts that we sell and new designs come out well randomly So check back frequently and finally our forums forums dot Lawrence systems comm is where you can have a more in-depth discussion about this video And other tech topics you've seen on this channel now back to our content Now I have another video that I'll leave a link to where I go in depth about the structures inside of Zen But in short so we are on the same nomenclature here We have a series of Zen server hosts which make up a pool and even if there's only one host It is a lonely pool of one if you have many hosts That's great and where all these hosts are doing is Sharing all the metadata for all the VMs whether the VM is stored on local storage whether it's stored on a shared storage pool All that information is part of a metadata and this is one of the first problems I want to address right away that I've seen a lot of people come to me with They have files they backed up the VHD file because they saved it on a share on an external device They go hey, I just grabbed those VHD files. I backed them up right kind of you now have files that have UU IDs That's fine except for I don't know what they go to and this is a challenge because if you don't know what VM They belong to you know You don't really have a great backup because they're not something I can easily and quickly restore That's all stored in this metadata information now as I said it's spread across the first the master and then every Subsequent host added to the pool and if it's a host of one then it only resides there So if there's a catastrophic failure of that one single host all the metadata goes out the door So even if you had amazing resilient redundant backups of all these VHD files with some external shared storage Systems such as a sand a really high-end failover. You still don't have the metadata related to them So that's why I want to cover the proper ways to back up zen because this is seems to be the way a lot of people That at least we've done some consulting with have come and if you look at the forums There's a few posts about this where people talk about well, I just grabbed the files I can't just restore them. No, it's a little more complicated now, but we're going to dive into all those complexities Zen offers a lot of different ways to do backups We've got Delta continuous replication disaster recovery and rolling snapshots not all these are technically backup So I made a little chart here to kind of show you which one is so metadata is like it sounds That's the metadata is on the pool itself You can back that up and save it to a font as a file on a storage server that way you can do whatever with it Take it offline move it somewhere back it up get it off site However, you want we'll cover that shortly here Rolling snapshots aren't exactly snapshots, but they're facilitated by the same automation that facilitates backups in Zen Orchestra and they just create snapshots on the same server But do some reading before you decide to create a whole lot of snapshots one snapshots aren't leaving the server They're on that server, so they're not really backups, but they are points in time at which you can restore But if you create too many snapshots you end up with a problem and you'll run into the coalescing problem Which is the system trying to track all those different Snapshots, which are essentially you're splicing virtual hard drives and creating a series of them that can be tricky to For storage servers, especially if they're under heavy load to manage that and they'll slow down your machine So keeping more than just a few snapshots not a great idea. I've seen people Keep like 10 and 15 of these they can create some problems for you and once again They're on the same servers that are not exactly Incredibly useful. They're not really backups, but they are facilitated because at least they do allow you to restore to a point In time these are the ones that I think are the most important We have full backups and delta backups full backups. Well, they're full backups They create an offline file to a storage location Delta backups create a full backup to a storage location But they also create the deltas in between what the deltas are as we create one full backup And then each subsequent backup we run in between is only the delta between the original And the next iteration of it What that means is if there's any incremental change, so I back it up on monday There's a hundred megs worth of changes now My delta is a hundred megs worth of changes. I only have to back up a hundred megs Not the entirety of whatever the original was so delta backups are a much more efficient than doing full backup Now back over here These are the other two that are very similar to full backups disaster recovery It's a full copy that can go on the same pool or a different pool So we have this pool here and is for example, we'll call this one our lab pool What if I had another lab you could actually replicate it to a completely other lab That is not even on the same site as long as they can communicate zen orchestra can communicate with both of these servers It can facilitate copying the data including the metadata around a vm and Restore that vm all the way on another system or create copies of it on another system That's great except for Kind of like our other example here. It's doing a full copy each time It's going to take a while continuous replication same thing. We can do a full copy on the same pool or different pool And if it's on the same pool once again kind of still a backup, but it's still on the same pool It's just probably located at the same site, but maybe you do have the pools separate in the building somewhere, but I'm not going to get into that level of design But once again, it'll do it but the difference is it's doing continuous replication Which means doing the deltas so it's going to do a full copy takes a long time However long it takes to get the vm from where it is to where you want it to be Whether it's in the same pool or different pool and then it does deltas afterwards Those aren't bad because now you can create a lot of these incremental versions and go back to them But now you're also creating a whole lot of little vhd files or however they're being stored You know if it's on lvm or whatever your storage methods are all on another server Plus this requires you have another pool or the same pool with another xcpng host Not the most ideal way to backup the ones we're going to focus on because these work and they work the same way No matter which one we choose. It's just a matter of where they land We're going to talk mostly here about the delta backups because they're probably the most ideal way to do your disaster recovery Or even the full backups, but once again the interface is the same So it's your choice of which one you want to do the first thing we need to do is have a place to put them I said on a storage location For our example here and true nas is not the only thing that can start but yes, I have a true nas server I have this true nas mini x Plus and it seems like a great device to do this demo on So we go over my pool here and I've created this xcpng backup youtube demo. That is a File place. We're going to store these just a regular file and we've have it set up as an nfs share And we've actually pulled this up and you can see there's just nothing in it So we're actually going to populate it and show you what the files look like So this is the me looking at the share on this particular system Then we've exported it via nfs and like I said anything that can support nfs is going to be ideal. Yes, you can do it smb nfs is much more ideal for speed And for usability, I just find nfs to work a lot better on here, but it will work either way So here is our exported nfs path and We go here. We copy it and if our charity set it up when you go to settings remotes And here is the remote. We'll just click at it. All I do is copy paste like this But get rid of that treating trailing slash at the beginning and now we have The whole thing set up with an nfs here really straightforward. You can choose smb It does warn you smb remotes are meant to work with windows server for other systems such as linuxamba Which means almost all nas please use nfs. It just works more efficiently. That's why you want to do that And then you hit save configuration and it's added A little button here says test your remote And it's basically telling us how much we have free How fast it was able to talk to it which is reasonably fast in this case So it's able to talk to it so as long as zen orcs can talk to it and it's able to store these files the way we go It works now that we have the remote set up. Let's go create a backup You go to your new Let's back up the metadata first because let's assume we only have one server So we want to at least have the metadata backed up in case it gets corrupted in case I have to reload my xcpng server I should probably have that data Actually, we'll just call it lab backup because we know what it is. It's metadata. So it's the lab server backup lab server Backup and xo. So click the metadata click the xo config now all the configuration settings that are inside a zen orchestra Don't live inside of xcpng. They live inside of zen orchestra And if I back up the zen orchestra vm Yes, I do back them up But also the option is here to back up the xo config. We choose a remote that youtube demo one Report settings on failure Select the pool that we want to back up. We only have one pool But zen orchestra can actually coordinate backing up multiple pools But for this lab demo, we just have the one and we'll go ahead hit create Oh, we gotta have a schedule And is wanting to know the retention so um Keep three or keep five daily What I mean is I want to keep five of this and five versions of this and I want to do it daily And you can pick the hour it works out. You could do it hourly, but as long as you at least have some metadata back up You're good. So we'll go ahead and do this a create by default this backup job is Disabled you can run it manually, but it's disabled to be scheduled. We'll just run it manually So we don't really need to do anything more than this click that it's gonna back it up and away we go success over here to restore Go to metadata and I have the ability to push this back to the pool if there's more than one version So actually we'll go back with the overview again run it again Give it a few seconds It's backed up go to restore metadata I can restore Two different versions here based on date pretty shape forward for the metadata now. Let's talk about backing up the vm's So the vm's we have in the system. I got a couple of them here. We got this true nas one the windows one My axle lab and one's on a local storage one's on a External storage. It really doesn't matter where they're stored What it does matter is how we want to back them up and what we may want to tag them with I'm actually going to add another tag. So they're tagged as test machine, but let's call them debian as well So we'll call this one debian So this one's been tagged with debian and let's go ahead and filter none Go to this one. I just want to add that same tag All right now my reason for adding these tags now. We're going to go make a new backup Back up virtual machines and we just want to back up the debian servers. There's two options over here smart mode really cool vm status is I can just say all running halted That's kind of neat select everything in this pool or not resident if I had certain pools you can you know filter down here or We can do this tags Just back up the debian servers just like that and now I can just filter by tag. How am I going to back them up? Well, we talked about the different options. We'll talk about just doing a delta on these We'll call it xcpng youtube backup and There's not much else I need to do here the other options if you do code advanced full backup interval Let's bring up what that is real quick Now the challenge with deltas and we'll actually scroll to the top here Full backups backup boom boom boom all the files. They're going to be huge every one of them Delta backups one full and a series of incrementals But on that off chance our storage server has a goof And it corrupted this I can't go grab fridays backup and bring it all the way back to sunday because the way you have deltas If I want to restore Friday's backup I need the incremental backups in between the chain of them in order to work So while delta backups are certainly cost savings in terms of storage They do potentially open up yourself to risk if there's too many from now. It has integrity checking It does a series of backups, but that's what this option is for is maybe you want 15 20 of them you want a long amount of these for some reason And you may want a backup one to be full in between. That's what this allows you to do. So if you click on this Look down here delta backup initial seed It allows you to have multiple versions of the full backup interval This advanced setting defines the number of backups after which a full backup is triggered the maximum length Of a delta chain for example the value of two the first two backups will be a full And a delta and then a third will start a new chain with a full backup. So basically it allows you to Add some chains in between so that's a pretty neat feature that's able to do that Now let's turn off smart mode, which is cool because maybe you want to tag everything production or whatever Let's show you how to do this manually We can just select the VMs really easy. So we have this one right here And we select the VMs put the schedule. Um, I don't know we'll say Hourly because we want an hourly version of this And I usually tag this for like keep five or keep however many on there And then you change the backup retention to five and you'll see why when I did this if we say every hour Hit okay Now we know it's hourly. We'll turn it to enabled even hourly backup That's spelled back upright. There we go And snapshot mode underneath it just so you know, it does have the option to with memory or offline Actually has the option to start and stop the server if you need to the last little thing I'll mention is they have the proxy option If you are using zen orchestra to manage a remote system goes behind the scope of this video But you obviously couldn't use zen orchestra and manage your remotely because it's handling all the backups And if it's remote, it's probably not high bandwidth. You can actually specify proxies onsite Maybe one day I'll do a more advanced video about that But it's kind of a neat option to where you can say Coordinate that through this other proxy server that's very specialized for doing that But that's a feature and there's a write up they have on how to do that We'll just do this pretty simple. It's enabled hourly backup And I don't really got time for an hour. So we're going to run it right now and just run it manually So you can show you how it looks All right, the backup is successful We have the full export here. It was 13 gigs exported It says merge, but there's not really anything to merge. We'll get what that means in a second And we skipped ahead all of four minutes to make this happen reasonable time to back up Now let's see what happened as far as looking at the VM here itself. It has a snapshot. Don't touch this snapshot This is the xo backup hourly db and how as in brackets there This is attached to the backup for this particular system Now what that means is when we run the next one, it will back up again And we'll go ahead and run this real quick When it backs up again this time It's going to look at the differential first We create a second snapshot because we never destroy the old until we start with the news So it's going to create a second second snapshot and now it deleted that snapshot Now we can look at the backup. It's going to say started successful turns yellow while it's doing this And of course we can see it from two different views We can go back over to the overview and see it here actually turn off the air release. I don't need it And we only transferred 34 megabytes. That's it So it was successful. We transferred 34 megs and it only still took a few seconds this time to back up. So It's the snapshot starts from here to here. So 39 to 42. So three seconds for the snapshot Six seconds for the transfer Then it goes to the merge once there's a merge and it's actually merge and purge Would be a good way to think about it when we were talking about creating these deltas Once it exceeds the number of deltas, which we said keep five of them. It's got to start trashing Once there's more than five So it'll have a merge and purge essentially as a part of the process So pretty straightforward and each time we run this it is one more iteration So this one was 34 megs. But before we do that, let's create some noise So go over here go to the council And uh, I've run this before and it basically we're just going to run this little speed test All that's going to do is dump a big file here move it around So we've made changes to the vm because I don't really have anything going on this vm So we just created a file and deleted a 1.9 gig file now I was going to compress some so it won't be quite 1.9 gigs of transfer I'm gonna suspect here. Uh, let's go ahead and run that again now That we've changed a few things on this vm Takes a little bit longer slowing down here because it's moving pretty fast Let's see go back over to the backups overview. We'll see just how much is transferring Gives you the status yellow just means it's running. We got through the snapshot part. We're doing the transfer part Successful. Okay, and I was wrong. It is pretty much the full amount of data that I changed It said, hey, there's 1.8 gigs worth of changes that we could see So the vm difference is now backed up 1.8 gigs over here. So when there's almost no changes, it's 34 and we can run it again Because we're not changing much and see what happens around this time Oh skipped it was hoping it would do this This is something that is important if you have too many backups and not enough storage speed And all trying to run at the same time. It has to coalesce these changes Now i'm not going to dive deep into zen server coalescence But this is a problem a lot of people run into they start setting a not really fast storage pool with a whole lot of backups And each backup as it takes a snapshot once it purges out the old snapshot That has to coalesce back into one file while you're waiting for that to happen if there's a heavy io load It will take longer. So i'm actually glad it did that and this is important that it does this So let's go back over here And run this one more time Storage should be fast enough. It's probably coalesced by now If not, we'll just get the same skipped error But no damage is done when you're seeing this and i like that they've added all the context Like it takes you right to the explainer of why that happens. All right, it's still skipped Not a big deal We'll just wait a few minutes and if you're ever wondering while something's coalescing we can go look at the Machine that's going to coalescing go to the disk So you can click where the storage is and just click on advanced And right there are those coalescing files. So while we're waiting for that to coalesce it only takes a few minutes But what does that look like on our free nas storage machine? So we go over here We'll go over to the pool storage pool And we've got about 4.7 gig used even though we technically have three iterations of That on there we're still not using too much and the reason why is because well, they're all stored inside of here So this is what it looks like when we look in here There is the metadata backups which once again, these are just files matter of fact Specifically there are json files that we can just open To dig and pull data out of same thing over here. There's each one of them And scroll back up a little bit and go to the vm backups same thing They've got a uu id attached to them and a series of json files three backups three json files Now don't worry. You don't need to understand any of what these are So if you have a total meltdown you have a complete failure of your Zen server But you have this share and you all you have to do is back this share up However, you want you can back it up to another in my case. I'm using true nas I can replicate it to a true nas. I can just replicate everything in this folder Over to the cloud for example, I just back this folder up send it to the cloud and all of my data is in there And let me show you how that works. So we just have all these files And there are vhd's inside here And these are the incremental ones matter of fact, we'll go ahead and create one more real quick Should have coalesced by now. Let's find out All right started And this one should transfer fast because we didn't create much of a file Successful only 16 meg of changes down here And we look over here sort by modified time. Hey look only 16 meg of changes. There's the 2 gig one. There's the 25 megs from the other one and you get the idea and there's the initial these are all those different deltas on there But because all that data is stored in these let me go back up a couple this vm backups folder Let's go to a completely different zen orchestra And only thing we're going to do is point that zen orchestra at this share So let's assume this entire system has failed catastrophically. It's gone. Whatever happened is unrecoverable And we have to rebuild from the files we just grabbed from the cloud and threw back into a folder So we'll start from scratch over here Now i've actually already got a couple things in here, but this is like our production system For doing this and youtube demo is what we'll call this And i'm not going to set up a new system, but we'll say all right We still have the file server somehow it survived whatever catastrophe happened more We've rebuilt the file server put at the same location And then path the backup. Well, we go over here in tunas. We look at our nfs export like before so the sharing nfs Copy as if I type that out. I'll make mistakes paste get rid of that little slash up front and uh save config All right, make sure it works test your remote. Yep. Your remote appears to work completely Notice how it has the same amount of storage in there. So let's go Look because we can't at least so we're on the same page here Go over here settings backup overview. No, this is our one that we're pretending died And we see this tunas mini x plus file over here that's stored with four deltas We go over here to our new system. I know it's not new, but you know bear with me here use imagination Go to restore. I know I got everything on here Hey, there's those same files. I filtered forum found it and now I can go here And restore them to my working zen server So this is the advantage you have With this type of backup now we could have done this with a full backup and it would have worked fine But I like to deltas a little bit better that just comes down to choice I'll leave links to where you can read all the in depth on there But this is a really convenient way to back things up to a file location This allows you to have the vm all the metadata everything with it backed up to a file You can throw that on you know, for example a tunas server And on that tunas server now you just back that thing up to the cloud and away you go Just grab that entire folder right here And as long as go back to storage pool As long as you're backing up this folder wherever it is you back it up to If there's a disaster at that site, hopefully you have that data kept off site This is what you need to restore your zen server now The other four mention backup types if you were to do this It does require that you have either another pool to copy it to may not be ideal There are certainly use cases for it. One of them I'll bring up They did a recent write up on how they use continuous replication to migrate data centers That's a blog post I'll leave a link to Over at xcpng maybe I'll do a video on that too is I think it was clever the way they use this This is great for replicating sites where you have a zen server at this site I want to replicate all my data to another zen server That is up and running that is a very valid way to do it as well Not everybody has that level of convenience more likely and something we've set for more of our clients Is delta backups where they have a single zen server or even if they have a couple in a pool for ha reasons And we want to have static offline files that we can do something with and handle as a file level thing And all you have to do to get these back up and running get zen orchestra up and running Well, I guess load zen server first get zen orchestra up and running point zen orchestra up at wherever that Storage location was for these full or delta backups and start restoring things. That's it We don't even need the metadata because we're building out a new zen server And like I said, the metadata is important if you keep all those nfs files and you want to keep a backup of it I mean keep that but having the deltas is the quickest and fastest way to restore your vms It's also convenient whenever vms get somehow broken or you've done something weird Which a lot of people especially when you're building your home lab I know you do things that you're thought something might have been a fun idea and you're not sure if it'll work It's also convenient to have these because if you break your zen server and have to Completely reload it being able just to go back in there and say, hey, here's a delta. Here's my vms I'm just going to go and restore them No problem because that is something that is uh, well really easy to do You can go here restore vms latest and I can I chose one but actually let's go over here where we have more stuff You want to restore the latest backup of every vm you did inside of there? You could do that it does mass restore it'll do mass backup mass restore So it's actually kind of a fun way to do it so you can actually rebuild your array This is actually also what we use That's why our labs are always loaded on different systems that come in for demo and review when I do these videos We're using this same methodology. We have all our vms that we use for testing pf sense and everything all Stood up right here as a option to be restored actually go over here So here's like everything we have and we can quickly Throw it grab a list of them and dump it to a freshly loaded zen server with no thought and have all the vms stood up In a manner of well hover on it takes a transfer. We mostly waited on the transfer But the functional time management time spent on it's very little because you can just mass select vms If you want filter them for a specific Value of you know lab or whatever grab them just throw them over there check the restore box Let the system go when it's done. I start doing more videos. So hopefully this give you a better idea I'll leave links to all this they've got it all broke down for all the different methodologies and details They may have left out or all in here, but it's a really solid backup It's really one of the things I love about when you use zen orchestra with xcp and g It's a very complete system for being able to do these full backups And all these demos were done as I say at the beginning on the open source version That you can use at home in your lab build this out Learn how it all works. It's the same as you get when you pay for it, but you don't get any support with it So great for tinkering great for messing around if you move it in production You can make the people happy at the top who you have to sell this to in management Yes, they offer full support packages and everything if you want to buy them from zen orchestra They got it over on their website on there. All right. Thanks And thank you for making it to the end of the video If you like this video, please give it a thumbs up If you'd like to see more content from the channel hit the subscribe button and hit the bell icon If you like youtube to notify you when new videos come out If you'd like to hire us head over to laurancesystems.com fill out our contact page And let us know what we can help you with and what projects you'd like us to work together on If you want to carry on the discussion head over to forums.laurancesystems.com Or we can carry on the discussion about this video other videos or other tech topics in general Even suggestions for new videos. They're accepted right there on our forums, which are free Also, if you like to help the channel in other ways head over to our affiliate page We have a lot of great tech offers for you. And once again, thanks for watching and see you next time