 One of the really critical things about any of your servers and data is making sure you have good backups, understanding how they work, verifying them and testing them. This has saved so many people because even the best antivirus has missed things occasionally, hacks happen, security flaws are found, and if your data gets destroyed or crypto-lockered or things like that, being able to restore from a solid backup is so hugely important. And I've even encouraged people to spend more time thinking about backups, especially when you're talking about, do I get a better antivirus or a backup to a practical crypto-locker? Get a better backup first, because the best programs can fail. Your backup should not. If your data is important to you at all, make sure you get a backup. So that being said, let's talk about how to backup the virtual machines inside of Citrix Sense Server. There's a really cool script here, let me move my head all the way, and it's called VM Backup. So first I'll start with the basics of how Citrix handles backups and how you can do it from the command line. And then this fancy, really great open source script that you can get on here that does an amazing job of automating your backups and making them very easy to understand. I like that it doesn't create its own files, it doesn't create anything strange. It uses the same tools that Zen has built in to import and export the backups. That is something I see with some backup programs where they come up with their own versions of backup. That's fine, but that also means you only can restore things from their software, so you always have to get their software set back up. That can be good, that can be bad, that can be a pain in the butt. This being open source and you can easily have this tiny little program on your computer. It's just written in Python. It's a very simple way to backup and it doesn't really have any requirements outside of Zen. One developer tool that you copy from Zen, which is another the Zen API Python script. So let's start with how do you install this? You can clone or download it, pretty shape forward there. And there's an example config included, the VM backup to UI, any readme file, which is actually right here. The other thing you need is the software development kit, 7.2.0, which I'll leave a link here. Part of the download for Zen, you get the development tool kit, download the file. And you need this one file called Zen API. Now all you have to do is copy this into the same folder as the VM backup. Really straightforward, not a big deal here. So there's not really any install, there's not programs installed, there's not any changes to make to the system. Load Zen, copy the 10.2 kilobyte Zen.API file into the same folder as your VM backup folder. So let's get these out of the way here, and I'll leave links to all this in there. So here are all the different virtual servers I have. I have the PSense one, Mail server, Wiki server, things I'm testing and running on here. And this is actually the Zen Orchestra. I just made it simple and put this on here to view things. So here's the basic DB and nine load I've been using. It's small, so it's easy to back up and fast to back up for the demonstration purposes, but it'll work with any of these. So let's take a look at the code, the settings and what we actually have to do. To run the script, it does require a couple of parameters. So you would run VM backup and then you put in your Zen server password here and the name of the server you want to back up. That's kind of the basics to get it started. You can specify all the parameters, but the better way to do this right away, I'm going to tell you, is we're going to edit the backup config and I'll show you my backup config. The other thing this supports is because you have to pass the password, you don't really want to put the password in a script. So you create a password hash and the way you do that is actually really simple. So it's VM backup, PY, your Zen server root password, whatever that might be, and create file, test dot pass. I'll show you what that does, Ron. So now if we look at test dot pass, there is a hash of the password, which the password I passed it was your Zen root server password. Now let's say we did a different password here, some password, and we can see it's a different hash. So it only needs the hash of the password to do that. That way, it's not as easy to reverse engineer. So when you do this, you can create the password file and we'll show you how that works. One side note that caused me confusion, I use lots of special characters. So let's say we wanted to do something like this. We had a bunch of exclamation points or other characters that bashed me and interpret it. The way you have to enter those is you have to escape them like this or it will not enter them properly. So if you were using repeated or anything that bashed may interpret from the command line as special characters, add the slash before that and create password file or you can't create the password file or use a password that doesn't use special characters, which seems like less entropy and probably a bad idea. I have a collection of everything from special password, special characters and words in my password. So I had to use that and it caused some confusion at first because I thought you could just put quotes around it and that failed to create the file. So let's clear this and we'll show you how this runs. First, let's edit the example config. I've already edited it and I've got it called LTSbackup.comfig. So we run down here. How many backups do you want to keep? Max backups, two. The script defaults to four is what it says. I actually changed it to two. You can change this to how many backups you think you want to keep. Where to back them up? Well, I created a mount called ZenBackupZen that actually is going to back to my FreeNAS server. You can make these go wherever you want. However you want to mount another spot and you can add to save them locally and then copy them off the server directly so you can specify a local spot. So this is really wherever you want and because I have a mount called ZenBackupZen, that is the place that I want all the backup stored. Format raw or VDI, you can actually use this. I leave the rest of these at defaults. Not really an issue here. Now this is where it gets kind of cool. This is the actual VMs you're going to back up. Now I was mistaken in thinking that it would support the UUIDs, but instead it actually wants the plain text names, friendly names of the servers to back up. Now this actually supports wildcarding as well. So you can actually see back up everything and then you can add exclusionary. So you can say, OK, exclude these, exclude these. And I have all this commented out for things to exclude, things to include. And you can do the wildcard, for example, VM export equals. And let's say I had all of them named production dot. I could just put a wildcard after that. Now this is also a way you can override things. This is all in documentation. I'm not going to read all of it through. But you can actually add the override to the parameters you set earlier, for example, the max backups by putting a colon two. It only keeps two backups of that because that's the parameter you pass, but maybe more backups of the other ones because you have a different setting in there. So I'm only, for now, because it's fast and easy for the demonstration of the test, VM export, DB9 test. Now let's go into, and there's my DB9 test because I've already run this. And I'll show you what it looks like. So here are two backups I ran this morning. And there's only two because I said max two backups. It'll auto purge the other backups because once I create the third, it'll delete the older of the two backups. And what's inside of here? Whoops. There's the XVA, some metadata, a success file that you know it's us, and a VM config. So let's actually show you and get into actually how it runs. I'm just going to go VM backup, LTS.pass. That's my hashed password I have. And LTS config. So these are parameters. This is what you're going to want to put, for example, if you wanted to automate this into a cron job, you would pass this as the command you're going to run the cron job. So LTS pass as a back config, the config file we edited. And it's only set to back up the devian nine. Now, first, the DB9 right now is not running. So here it is, basic DB9 loads and server two. And it's not running. Nothing going on here. No snapshots, no nothing. I just want to do a backup of that. So press Center. Now, let's read through what it's doing. Now it's kicked off and it's backing up. So VM backup running, snapshots, it gives you the max backups too all the parameters it's reading through. If any of the parameters are wrong, if you have a typo, it will fail. Like if you can't find the name of the VM you said or doesn't find other things, it fails right away and doesn't start the backup and lets you know what failed. This is really nice. It does a space check on that. So is there enough? Yep, we got 4.7 terabytes, just 242 gigs used. That means there's enough for this on there. So it will fail out if there's no space on it. So it'll give you an error for that. Then it's using the VM export. Now this is actually a standard command VM export that you run. It's xc-vm-export and then uuid and then the file name. That's, you can do that from the command line in Zen, however you want. So xc-vm-export or xc-vm-import are the commands you can use right from the command line. So this is like I said, using all the built in tools right inside of Zen and nothing all that special and creating a backup file right now. I'll fast forward so it'll show you this complete. It only takes about two minutes to run, but who wants to sit here for two minutes? All right, now the command is complete and you notice it says deleting oldest object and at oldest object is the other backup that we had here. So then it lets you know that this was success. It does a space check again and VM backup ended and the parameter is here. So this is pretty slick for doing that and the other thing you're gonna ask next, what if I don't wanna shut down my VMs because they're too critical and things like that. Well, obviously we know there's problems when you do snapshots of running VMs and one of the problems is data in flight. So if the data's in flight while you take the snapshot you could have things that are broken read, especially with databases and things like that. So that's still an issue, but you can do this and it does it in a very clever way. So let's actually show you, let's fire up this VM. Get it started here. Okay, it's booting up. So here is my basic W9 up and running and let's run the same backup command again. Just up arrow, same backup, but this time it's doing it on a live VM. So what it's doing now is it created a snapshot of the VM and because one of the things you can do is export directly from a snapshot the entire VM. So you're basically forking it, backing it up off of that snapshot and having that exact instance copied. And that's really clever because what's appeared right here is there's a snapshot called restore underscore DB9 test and it creates a snapshot on a fly. And you can even see this is the Zen Orchestra and you can see even here I can export the snapshot, download it as a file. It's not downloading this to differential changes, it's downloading the entirety of it when you do the export of a snapshot. So it's not just some differential, it is I've worked it at 8.25 a.m. That is a copy of it, of everything up until 8.25 a.m. of that particular virtual machine. So we have the snapshot created, it's doing the restore DB9 test. Now the nice thing about the way the script works is it's going to auto delete this when it's done. So it only creates a snapshot, downloads the backup from the snapshot, creates the folders, puts it all in there and then goes back and deletes off the snapshot so you don't clutter things up. I believe there might be an option to leave the snapshots intact, but for the most part it seems pretty convenient to me just to create one long enough to do the VM export and get this done. So we'll wait a few more minutes and watch this one complete. So, and if you notice a couple of things changing here, you know the VM nice, power state is running, VM is running, check for pre-VM snapshot. It's looking to see if there's a previous one that didn't get deleted and then runs the command. Now the reason I was surprised it didn't use UUID on the config files because when you're running the XE VM export, it always wants to use UUID or it's definitely a better way to do it. And if you select the UUID assigned to a snapshot versus the UUID assigned to VM is how you distinguish those when you're doing an export and the XE VM export tool does not work on live running VMs but their workaround is to export from a snapshot which is this software takes advantage of. So we'll give this a few minutes to complete. All right, and the same thing, VM backup success and we have now the latest backups. And once again, after it was completed, it deletes the oldest backup because we only said keep two of them. So let's go and take a look in that folder. DB9, there's a two different backups, one at 8.19 and one at 8.25. So the next question you're probably asking me is how do I restore these backups? How do I verify them? Well, hopefully, you can trust that these are good and you've copied them somewhere else. So you have an offline backup. Please tell me you're encrypting them because all data at rest should be encrypted which is actually, this is mounted on an encrypted volume. So let's actually get to the details of how do I put it back? Something goes wrong. You need to restore one of your backups or you just want to copy it to another Zen server that you have loaded up. So something catastrophic happens and we gotta get to the restoring part. That's actually really easy. So from the command line, because you don't use this software only as the backups, I don't think it has any import restore options. I didn't see them in the documentation but it's not really a big deal because, whoops, a VM-import, file name and DB9 test. Simple, that's it, I'm gonna press center and it's going to restore that. Now, please note, when it did the backup you may have noticed up here, it called itself restore underscore and the name right here. So it's restore underscore DB9 test is actually what it's naming the files and it also was the name of the snapshot and that's an auto naming scheme it adds to there and the reason why that way if I want to restore DB9 test, I'm not overriding DB9 test. Now you could also import these through the GUI by going to file and import VM like I had showed you before on the main demo. So you can do it from the web interface. Well, you can do it from the interface on the Zen Center interface in Windows or just from the command line which is really fast and easy to do because it's XEVM import file name dbn underscore nine test which is the name of the file, press enter and let's show you what's going on in the interface. So right away and I didn't have to refresh this while it's importing, it shows up right here and restore DB9 test. So it doesn't overwrite or have any conflicts it's gonna get a different UUID So if I'm looking here, the UUIDs are gonna be different. So as you can see, there's a UUID for this one, UUID for this one, this one's still running. This is restoring off the backup that was taken from a snapshot. So it's restoring. Now this happens to be just a side note here. The storage, this was created and it's on local storage. The restore because my default is the remote storage it defaults the restore to go there but all the other settings are there. Also it's now complete. It doesn't really, it does the restore even faster it does the backups. That also by the way, shows up in here. So 54 seconds to restore, they show up in the message even though I did it from the command line the Zen Center software inside of Windows shows all the messages or if you're using ZenOrchestructs that's an API interface to this, it shows them on there. So the exporting shows up the importing so you can actually see this in your Zen Center software as well. So here's the restored one and you'll see it was attached to this network. It's still attached to the same network. It restores everything the same. The only changes are the fact that it defaulted to the storage. Now you can change that if you do look up the help documentation for the XCVM import or export command line files you'll see that you can actually specify specifics about them. You could say go ahead and start it on run assign it to this storage pool, change each parameters. What you can actually do is part of the restore and the same thing here when you're doing a importing or exporting a VM if I import or export those I'm able to do the same thing. So I'm able to come in here and add a VM and attach it and like I said before with the other one. So when I'm importing something I can change those parameters around from here from this interface or you can do it from the command line whichever you're comfortable with. I kind of like the command line once it's easy the file's already there and ideally you want to be able to restore fast and if you have a whole lot of VMs to restore you could just simply script it know all the paths of the VMs put a script together to import them all at once and away you go. So the other question might be what happens because we did this as a snapshot of a running VM how does that work? So let's take a look and show you that it starts right back up like normal. There it is and it's back up and running. So that quickly I duplicated the VM now I could have put this on more than one server if your backup repository is shared you could have imported, exported to another VM or another Zen server or as I do when I do these backups of my servers I take them and put them on a external hard drive that I use Lux encryption on. So I have Lux encrypted drives I keep one here at the office and I keep one in a safe elsewhere and that's my backup plan for my virtual machines generally I back them up once a week but the data on them is backed up more frequently is backed up hourly and I think I've talked talked about that for my own business backup schema because the databases we have are small and are easy to encrypt and send off site. So anyways that was the backup and restore using the VM backup tool I'll leave the links to all these different things below in the description and if you have questions or if I got something wrong let me know if you like the content here like and subscribe and these are still part of a series so maybe I'll make a whole playlist of all my Zen server videos and all the different things inside of Zen working on getting to some of the high availability and some of the other stuff too we got another server that we're gonna be putting in so we can tie some more stuff together and do some more demos. So thanks again for watching if you like the content here like and subscribe.