 Time here from Orange Systems and nobody really cares about a backup that works. Everybody really cares about a restore that works. And this is an important distinction here because untested backups are just wishful thinking. I'm going to show you today how to use Zen Orchestra to automate your backup testing. And I don't mean just verifying the integrity, but actually booting up the VM in an automated way and having the system notify you of the pass or fail of the success of that VM booting all the way up. It's some clever ways that this is implemented and let's get started. It is October of 2023. And I'm using the latest XO community self-compiled version. I do these demos to show that the people who are in the home lab that would like to try this out can absolutely compile this yourself and do all of the same things that I'm doing here. There's no licenses required for this. But if you like to use it in production environment, they do have a simpler to install version over at zen-orchestra.com that comes with licensing and support and would be more ideal for a production environment because, well, licensing and support, plus they have a 30-day trial if you just want to trial all of it. Back to this version here, I have a tutorial you'll find linked down below how to compile this yourself. Now that we've covered the version, let's get started with the demo. Now I have two virtual machines here, one running Linux, one running Windows 10, and I'll show you how these demos work. But it really doesn't matter about the version of the VM in terms of Linux version or Windows version. It matters the most that these management engines are able to be detected. This is a prerequisite because this is what it uses to determine whether or not the VM actually booted. Now, before I show you how to set one up, I want to show you an existing demo of one that's been running. This one right here failed. The reason it failed is purposely I did not have the management engine installed as I wanted to show you what a timeout looks like. So it did the backup, it actually backs things up, but it fails on the health check. That's why the health check is read. It didn't fail on the transfer of restoring it just on the booting it up. So technically this is a failed backup, but we do have a backup that occurred because the snapshot occurred and the transfer of that snapshot occurred. So the backup part worked, the restore had failed. That's what this is telling you. Now I installed the management engine and I ran it twice. And this one right here is what it looks like when the backup is successful, pretty straightforward. We're going to go ahead and close this one more successful one. And you can see that these happen relatively quick, but that's going to be completely dependent on the speed of the equipment you're using and how big the VMs are that you're backing up. So let's go ahead and create a new one based on our Linux VM. Now to build a new backup, we're going to go over here to new VM backup. And yes, you can put emojis in the name. I think it looks cooler if you do. Then we're going to select our VM and we're going to type in health and we're going to grab our Debian 12 health check demo. You can if you have many VMs do this to multiple VMs at a time. That's actually completely an option if you want to do it for simplicity. We're only doing one. I'm going to choose Delta backup as the type. I would like it to go to my lab backups because this is a demo. And then over here on schedules, we want to choose and we'll give it a name. We'll call it the same name. We're going to keep two copies of this and I want to perform a health check. This is where you get to ask the question, where do you want to do the health check from whence it came or somewhere else? So I can choose the pool or individual storage that it was on or a different one. So this actually lives this particular VM on the pool called new rice and labyrinth. We actually want to test it on pools and that way we're actually restoring it to a completely different system and a completely different pool. You can do it to the same, but hey, why not try something like your entire environment? This is actually a little backwards because this lives in my lab and we're going to restore it to a production environment, but why not test if this works and say, okay, now I'm not going to enable this in terms of the pattern. This is what allows you to choose what day, month, minute that you have this running. If you want it to run more frequently, obviously you don't want it running more frequently than the time it takes to actually do the backup, but you can choose this. Now the reason it's like this is because you can actually set more than one. So the health check demo and let's give a little bit more realistic scenario here. Maybe we want the health check demo to only run on Sundays or Saturdays. Let's say Saturday, maybe there's less load on your system on a Saturday and you want it to run on Saturday. So we'll say, okay, this is going to run four o'clock on a Saturday is the time you want this to run. We'll hit okay, but we want regular backups to occur at the same time. We don't want to do separate jobs. You can actually build a new schedule that doesn't do a health check and we can just call this one delta backup and we can even say this is our hourly backup because every hour we'd like to keep a delta. Maybe we want five copies of that delta. Well, let's just go all out. Maybe we have a lot of storage and we can do 12 deltas and well, that's the advantage of deltas is they don't need as much. And we want to say every hour, do a delta backup and don't do a health check. So this will be a second pattern that we have and you just enable each of these and they will run on those schedules and all that's going to be saved to this particular backup that we chose. And then down here we can choose some advanced settings, which is the way you'd like to do this snapshot mode normal with memory or offline offline would be shutting the system down on normal just means grab a snapshot and let it go. We'll just do that choice right there. I don't want to stop the VM or maybe you do want to stop the VM that this is something important that you have to decide. But this is all we'd have to do to get this set up. Now, I don't really care about this delta hourly. So I'm just going to delete it. We're just going to leave in the one health check demo that we have right here because we're just going to run this manually. So go ahead and hit create. And we'll go back over to our VMs, click on health check demo. And we can see the backup is now attached to this Debian 12. There's our health check demo. And let's go ahead and just run this and watch it go through the process. Are you sure you want to run health check demo? Absolutely. Now I'm going to go here because you're going to watch it export. That's what this task is doing right now. And then we're going to watch it do the import. So it's exporting from Ryzen Labyrinth and it will actually come back over here and import over to our production environment. Now you can see with ones in orchestra, both environments simultaneously. And here it is importing for the health check demo. And if we click on it, we'll watch this in a moment boot after it's done importing. But you'll notice there's not going to be a network adapter. It's going to delete it before the boot. It shows up now. But before the council becomes active and this starts, it'll actually do a script that deletes this network adapter. If we watch the council boot up here, now it's going to boot. And as soon as it detects the management engine, it'll actually destroy this virtual machine. It detects the management engine and it's gone and it drops us back to the screen. Now we'll go back over here to our backup. We can see that this was successful. We can see it successful here. And the whole process only took two minutes because well, this is a small VM with only 5.5 gigs. We click on successful. And we can walk through the process here where the snapshot occurred start and where the repository is that it dropped all this, how much time the transfer took, the backup took that it successfully VM started the success and it destroys it. And it's done simple as that. Now I love how simple this is to get up and running, but I want to make a couple of notes here for people who want to manage it at scale. And we have clients doing this to have hundreds of VMs. No one really wants to go through and set up hundreds of VMs in that list. So that's where tagging works. And yes, health check works with tags. You can put, for example, whatever tag you define, but health check to be simple. And you'd put that on a VM and then you can choose that tag to choose which VMs get health checked. And maybe you want to choose like health check for lab one, health check for lab two, because you want to do them separately, you want one to roll back to your lab servers and one to roll back to your other lab two servers, for example, this allows you to then build out VMs at scale without having to edit the backup job to every new VM that gets added. Just make sure when you add a new VM, you add the tag to it and that will determine where it lands in terms of backups and where it lands in terms of whether or not it needs a health check done. Now this is still no replacement for a full DR test. That is something you should still do because this isn't checking your application. This looks for the management agent and make sure it starts and communicates with the XCPNG server. Once that's been noted, it says success. That does not tell you whether or not your database or any application specific thing that you're running inside, they're fully started and works fine on another system. That could be a problem you run into. So there's still some testing you should do. But I think this gets you most of the way there because once you know that your system will boot even if it's in another environment, you can usually start solving application problems a little bit easier once you know the virtual machine is booted. Love hearing from all of you. Leave your thoughts and comments down below on this topic. Check out my playlist of Zen and it is just a lot of things I've been covering with Zen Orchestra. I'm doing this new series on it. It's been a lot of fun and I love hearing all the comments and feedback from people. Like and subscribe to see more content or connect with me over on the forums. Thanks.