 Alright, so we're going to talk today about the ultimate Zen server XC PNG backup solution as called by the Zen Orchestra people. Now we're going to be using the open source version, which you can get for free. I've got videos on this of how to set that up. And if you want to start testing it in your home lab, it's awesome. It's great. I've been playing around with it. And that's what this demo is about. If you want the free or go beyond, they have a free version and then they go into paid support versions. So if you are running this in the enterprise and you want a really high end with really good support, these guys are great at doing all this and backing up large environments. They do have pricing for all that. Easy enough to find. Go to zen-orchestra.com. I'll leave links below. Alright, they support a lot of backup solutions here, which is really cool. It's very diverse. One of the things that's really cool is, and I like this about the way they do it, there's no vendor lock-in, which means you can bring free NAS, you can bring whichever storage solution you think works for you and your preference. It also exports all of these out to standard file systems, which is really nice. So it just creates an XVA file if you want to do just a standard one, which is very portable. You can just import it to another Zen server. They also have, and this is really cool too for the disaster recovery, some continuous replication and rolling snapshots even between remote Zen servers. So you can do an initial backup, like the full backup because the VM is large, and then do continuous syncing or delta syncing, which means only sync the changes between those servers, which of course are much smaller depending on how much read and write activity is going on, and at particular VM, if there's not a lot, you can keep a lower bandwidth connection between the two, maybe two remote sites, and continuously replicate. There's some advanced things you can do with this, which is pretty amazing and pretty awesome. So close all this. I want to talk real briefly about Zen server coalesce detection and what coalesce is. Coalesce is, well, it can create a backup problem. And let me give you a quick why it's a backup problem. Snapshot change is too long. SR backend failure insufficient space. When you're doing these snapshots, and this is a lot of the way backups work in Zen server, is it creates a snapshot and then starts backing up that snapshot that way the VM can continue working because it's hard to backup something that's continually changing. So the snapshot essentially takes the, all right, this is when the backup starts. We're going to parse it over to a backup. Now we're going to back up this data. And when we're done, we coalesce the data back in because we don't need it. So it creates a snapshot in time. They have the details of how that works and why does this matter faster than coalescing speed. What about creating a snapshot, call backup jobs faster than Zen server can coalesce while the chain will continue to grow? There is the ability to create multiple snapshots, even though you may not see them, so to speak, because you've done a backup and you want to do another backup, but the other backup stuff hasn't merged back and purged the backend operation that occurs and you kick off another backup before that happens and another backup on that VM before it happens. This system is smart and will stop you from doing that because you could actually run out of space doing this. If you don't have enough provisioned space for your server, I just wanted to cover this and they actually have some in depth links here. If you want to get into the engineering behind how Zen server and the coalescing system works, they actually link to a Citrix document in here. You got to sign up for a free account with Citrix to get the document, but if you want to know the high level engineering stuff, it's great read. It's pretty cool. I like it, but you don't have to know all that. You just have to know that if you click back up a bunch of times and you don't have a fast server, it will give you this error. So that's the reason I covered it because sometimes when you're doing testing, you want to play around with it so you test quickly as opposed to putting on a normal schedule and you end up overrunning the server. That's part of the reason they wrote the article as well. So continuous replication, Delta backups, and just the different options. So let's cover what are all the options in here? You have full backups, rolling snapshots, Delta backups, disaster recovery, continuous replication and file level restores. Now, because we're using the open source version, these features are all enabled, but if you're using the free version, the appliance version that they have that you just download, it requires paid subscriptions for some of those that are on there. All backups rely on snapshots. Like I said, this is an internal design because it's the best way to do this. You take the server that's running, you quickly snapshot it, and then you kick off the backup on it. Now, how does this work? Important features, especially on your Windows machines, make sure you have the Zen tools installed. That way it creates the proper shadow copies, kind of the unlocked file copies of the server. For you may give an example here of backing up like a Windows server. So when you're doing that, that's one of the prerequisites. If you have it and you want it to make sure, if not, you're going to basically get a snapshot of a server in a running state. And what happens is when you go to restore it, it'll have an unclean file system boot and everything else. It may work fine. It may not. You don't know because you grabbed a snapshot while it was in some type of open file state, and all those open files weren't properly talked to with the Zen tools. That's the purpose of the Zen tools, be able to do that. So this is a prerequisite that you should have those installed. Now, the really cool ones that are in here, the backup obviously works fine for creating XVA files, which are the Zen files. And I'll show you real quick here. So this is in our system. And this is actually what it looks like from the command line file. What does it create? Here is the JSON file, which contains all the details of that virtual machine. And here's the XVA file. You can actually just import the XVA file into any Zen server and it restores. It doesn't even have to be the one it came from. So you can just take this file, load up a Zen server from scratch, import XVA. All the features of that VM are tied into that XVA file. That's the export file. And I've covered this in how you do command line exports in Citrix Zen server. I have another video on that topic. So these are the same files. Now, they don't get named anything that is easily human readable. They're stored in this directory. They're given a UUID because the way you do it, and we'll cover that in a second, is you want to use the Zen Orchestra NG tool for doing the backup. So all right, let's close this. Now, this is the continuous delta backups. I'm going to show you this in action because it's pretty cool. Full backup. Here's a full backup works. Here's the full backup incremental backups. Now, this is what's really cool. So obviously these files, and if you look here, this is just full backups, 3.8 gig, 3.8 gig. Between the week that these were backed up, there was very little change in the file. So the file size is essentially the same. Even when you look at it, it's very fine. There's very few changes between the two of them. So, so to speak, I'm wasting a little bit of space here. Even though it's only 4 gigs, a delta backup would have only been the difference between these two VMs. So it would not have been as much of a waste of space. So whatever the difference is between June 30th and July 7th, which a few hundred, such a tiny amount of data, that's all the delta backup. So they're really efficient. If you want to have a lot of, let's say, two weeks of snapshots or you want to do a daily snapshot so you can easily roll things back, well, this is your, you do a full backup first. This is when you set up the delta replications, and then it syncs on the deltas, which the delta is the difference between each day of the week here for this example. So each backup, it is only that difference. Now that differential also means it's very fast. So doing the delta replication is very fast, but very important feature of the delta replication is that this full backup always be intact because these are all partials that have to go back to this incremental. So that's an important little thing in there. Now, it also does merging the data blocks. This is also, like they said, they are called continuous. They never export at the first one. You will merge the oldest delta into the full. So it handles all this behind the scenes. There's nothing you really have to do. So as you get rid of these, it'll take care of all that. Now continuous replication is another feature. This is a really cool one. And this feature continuous replication system for your sensor without any storage vendor lock-in. This is one of my favorite ones for doing a disaster recovery scenario because there's no storage vendor lock-in. There's no configuration. It's agentless. You just have to have a couple of Zen servers and Zen Orchestra talking to those Zen servers. Not like any special things that have to be set up. Low recovery point, which means it can be done really, really fast. No intermediate storage. It goes directly to the other system. So here's Zen Orchestra. Here's site A, site B. And I say site because they don't even have to be in the same site. So once, and this depends on your bandwidth between there, once you have the initial one done, you can then just send these incrementals, which of course, like we showed a week of that particular server running was only kilobytes of data. So the continuous replication, because once the initial seed is done, it only has that little bit of data changes to do. And then you can create an entirely off site with not ton of bandwidth in there. Unless you have a system that has an absolute ton of writing, it is based on how much that VM is in use. So let's actually look at all this in action. So here's the one I just had all production. And I'll show you how I created that. So let's first edit it. So name, name I gave it was all production. VM statuses, you can say running, not running, select a pool. I can say resides on pool, not resides on pools. We have all kinds of differential exclusions or inclusive ones, production. Now, what I did was actually just use the tag, because we have all of our VMs tagged. So you can see the matching VMs. So tag type production, here's everything labeled with the production tag, which you just add a tag right here and it tags it. You can do this also with the, you can add tags with the Zen Center software as well. So it's showing me tags production VM. So pretty straightforward to see which ones you're matching. And anytime I add one to production, I don't have to change the backup. The backup just goes, you added another production VM, it's automatically in the backup. Then you have the schedule. Currently have the schedule disabled because I, whatever reasons I have it set to manually do, but we set something up for a business. We generally want things to be automated and it does have logging and features and everything else. But I do have it disabled as I run them in a manual basis. Let's go back to the overview. Here is the continuous replication demo for the Delta changes. And I've already run one here, continuous replication. So it took four minutes the first time it ran in a few seconds, the second time. By the way, if you're wondering, so my last backup of all production took about an hour to back up all those production VMs and create the files. So let's roll back a little bit is where's all this data going? Okay, let's show you real quick how the remotes are set up. Now, this is called FreeNAS because I named it that. And this is a, you see it says Mount FreeNAS Backup Zen. That is on, it's referred to as a local one. So if we actually go to click on it here, I'm actually giving it a local path. So you can do local NFS or SMB. So if you have an NFS share, you can put that information in here. If you're doing it local, this is looking at the local Zen Orchestra server itself. That's what we're on here. So this is 2.24. And if you didn't notice up here, 2.24. So it's the center server. But, and when you look at it here, right here is the Mount. It's actually an SMB Mount. So we actually have the Zen Orchestra tool mounting the FreeNAS. So this is a Debian, as it says, Debian XO for Zen Orchestra. It's a Debian server and it's mounted to the FreeNAS backup. Therefore, even though we say local in terms of here, it's actually not storing it within there. That wouldn't be a good idea because if you're storing the backups of VM inside another VM, first, there's a space issue. Second, you can just think of the problems that can occur with doing it that way. So the Zen is doing local as far as Zen Orchestra concern, but it's actually going to this mounted remote right here, this SMB remote to FreeNAS backup. And then you can see the Mount path right here that we're in. So just so you're aware of how that's doing, but I did find, and they do warn about this, you can do it to an SMB. SMB remotes are meant to work on Windows Server for other systems, Linux, Samba, which means almost all NAS, please use NFS. I've had this problem. I haven't done a lot of testing with it since they updated the version. I don't know if they fixed it, but they still have the warning in here. So I'll assume they haven't. It seems not to like Samba very well. If you put this in, it just keeps giving a failure when you try to connect to Samba, but it does connect. We have a, I spun up a Windows Server and it does connect fine to Windows Server. So that's just strange to me, but whatever. If you choose NFS, that works perfectly fine too. I just don't like doing the NFS mounts. Maybe I will in the future. I don't know. I have it as I like Samba mounts with passwords on them. You can configure the NFS mounts, but the NFS mounts, the security's not as good or maybe I'm not as aware of how to set up NFS4. I don't know. I don't really mess much with NFS anymore. It feels kind of dated. So using SMB with the username password seems like pretty ideal. There's someone that's going to call me out on this and say it's better to do it this way. It works perfectly fine this way and it's very fast. So we're just going to leave it at that. So this is where you set up your remotes and you can have multiple storage repositories in here. So if you have a series of places you want to dump this to, you go ahead. I just happen to like handling the mounts with Debian. So actually you can mount different backups, storage solutions inside of Debian, including if I wanted like a USB drive and then you could put it to this. So I'd mount it in here via pass through and then you would just have another local mount in here. So let's go back over to the backups. Let's create a new one real quick. So name of the backup. U, tube, demo. What VMs do we want to backup? Now we can just hand select the VMs here if we want and say, okay, I just want to backup this particular VM. What kind of backup do we want? I think we have a U2 demo one in here. We have a U2 cool. So there's that backup there. We want just rolling snapshots done. Do we want backups? Do you want all of them done? Now it does gray out because if you're doing one you can't do all of them but you can do more than one backup through a command. So I can set it up for disaster recovery and backup. So disaster recovery means copy it to another Zen server and then choose the remote. So I can, one file goes to a backup. The other one's going disaster recovery. It can run as a single job. I like the way they do this so you can have multiple things going on. You can do continuous certification and delta backup. So you're continuously replicating and creating delta backups locally or you can just choose to do one at a time. Now if you turn on smart mode that's this little button up here then you get even more where you can select the tags. For example, the production tag that protects all machines. So you have your more advanced options that come in here. You can even do it based on pools. So if you have a series of pools set up I have a Xenoper and Xenoper 2. You can see the backup everything on these pools and send them over here. So you can see if you're managing this at scale and you have, my friend works at a place that has tons of these and they use this. It's great. We have 400 VMs backup. So this allows us to go, hey, let's just do this to snapshot 400 VMs on a schedule. It's actually kind of cool. So this system really works well at scale and that's part of the purpose of Xenorchestra. It allows you to orchestrate massive amounts of Xen servers. But we're just gonna choose this one and you know, backup, select the remote, free NAS. Now that's actually another feature in here. If you had those multiple remotes and you're just doing your standard full backup you can choose your multiple remotes in here. So you can set up, you know, we only have and pull up remotes here. So we only have this one connected and available this free NAS one. But if we add another remote you can also back up to that too. So it doesn't have to land in one place. You can actually have this landing in a series of places for the backups which is also really nice. Once again, one backup, multiple locations and makes it really simple. Now, something I noticed if you click the create I believe it fails, yeah. And you're like, why is it failing? Why is it my stuck on? And you just gotta look up here, this is missing schedule. You have to put, even if you're not going to schedule it you have to at least choose a schedule and hit save. You don't have to enable it. Just a little side note that if you're missing the schedules cause you're down here, you're clicking you're not sure what's going on. That's what's going on. Now export retention comes up in a schedule that's probably the reason you need this. How many versions you wanna keep? And it auto cleans up all the old versions. So if we go, I want three versions of this it'll retain three versions and after fourth. Even if you're running it manually it still only retains as many versions are in here. So we'll go ahead and hit save on this and then hit create and away you go. So here's our YouTube demo and even though it's disabled the same thing goes here. So if we look here export retention two when we look at all my restore options here there's all the two retentions cause I've been running this as my regular backup. Now it does go to the free dance replicated to another computer for so it's not on the same computer to VM's design and it's replicated again in case you're wondering I think I've talked about that that I take a copy of everything off site once a week after I run all these backups. So if you're wondering where all the data ends up eventually for people have that question all the time off site in a series of encrypted hard drives. So this is the YouTube demo one we just created we're gonna delete it cause I don't really want this one and I'm gonna show you cause I have it running here. So here's this machine and we'll exit out of this one. Actually I didn't see what the IP address is. All right, see if there's any updates for it. Okay a couple of little changes here so we download a very small amount 2000 kilobits it's changing right now. So let that run in a background. Now here's that continuous replication demo this is one here we ran it here it took four minutes it only took a few seconds it says 10 seconds to run when I ran it again we're gonna run it again manually. We could be having this on a schedule but we don't and we're gonna go after it runs I'll show you where it's all landing. So now it's running it started and it's gonna finish rather quickly. If we go to the tasks there it is syncing it no pending tasks go back over here. Now this is important XO backup continuous record patient for demo this is important it is taking care of this on its own for doing these it actually only keeps one snapshot because it's part of the process for this. So you go very back up in G overview look at our continuous replication demo and we edit it. So we have our DB9 YouTube backup demo continuous replication target is the free Nash storage repository on Xenoper 2. So that's it retention copy one go over here to VMs and we type in YouTube. So here's Xenoper 2 where this resides here's Xenoper where the backup data is going and it created this automatically DB9 YouTube demo da da da da. You can script this you can control because this is all very scripted from the command line as well if you say hey look for this running machine and start this one automatically based on this because this one's not responding there's ways to set that up I'm not gonna get into a high availability and automatic failover of VMs so they start. I just wanna show you that it immediately is able to do these backups in only a few seconds. So this one took 14 seconds versus this one 10 based on the changes we made but I will show you how you restart them because the ideal way when you're doing these backups so let's say this machine fails and actually let's just fail it let's actually go here and we're just going to for shutdown don't do anything just boom done. So this machine whatever reason on Xenoper 2 dies. So you go over here and how do we get this to start? The best way to get it to start is actually going to be you can clone it you can migrate it back to the server and this is what I was talking about how is there so you start a copy of it the other way to start a copy would be to just fast clone it. So we'll go ahead and just they start a copy and I'll show you how fast that says so current status clone and VMs was to show you how quick it's doing this. There's that clone and note this is all real time it's in boot mode now and it's booted and replicated based on those updates. Matter of fact, it should get the same IP address. So this is the session I had that was broken. Oops. 172.16.69.165. Oh, I guess it did when I I'm sorry my bad when it did the clone it did changes so that will change the IP address that it gets. You can always swap the IP the MAC addresses back if that's an issue when you're doing this. 167 is the address for this. We'll run an update real quick cause we did the application after it and it should see no upgrade. So it's absolutely the same state from which we did that continuous replication and this is kind of a neat feature of the way these work is being able to just quickly okay I've been cloning it it's been going over to another server so it was going from Zennefer to Zennefer two on a continuous replication. I could have had it scheduled and it immediately works perfectly fine and backed up. What about restoring other VMs? That's actually really easy to do I'm not going to restore any right now so I don't really have the need to but you could just go any of these and go all right I just want to restore it. Choose which date you want to restore where do you want to restore it to? You know the Zennefer the free NAS which is the ISCSI storage the local storage that's on Zennefer the local storage on Zennefer two cause it doesn't have any ISCSI connections and whether or not you want to start the VM after restoring just hit okay and it restores it it's very fast it's very it's as fast as your system is unable to do that. Now a couple other things there is a health over here if you're learning what that is it wants to know if any of these got broken or orphaned if you play around with the file system you'll break things. If you go into those JSON files I had at the beginning you'll end up with like some issues of you broke some of the backups cause you played with them always use this interface for the backups. Also if you're case you're wondering so let's exit out of this. So this is mounted right here for free NAS backup if we unmount this go over here to backup NG they're gone. You're like oh no where did they go? Refresh backup list. It's actually kind of simple the way this works. So we're going to go ahead and I think I got a command already you already got scripted so now it's back. This is back and mounted. Refresh, they're back. The way that works just so you know if you ever have to do a restore is you just take this file so we're going to go to mount free NAS backup. I have it under Zen then it's under XLVM backups. As long as it finds this file all the data is kept here. None of it's kept actually in Zen Orchestra. So as long as Zen Orchestra knows where this file is pointed at you're able to see all of your backups pretty straightforward. That's important because let's say it's the absolute worst case scenario you lose your Zen Orchestra install you lose everything but that external hard drive that you copied all of these files onto. No problem. Just point Zen Orchestra back at these files however way you get them to a storage place that they can be seen and mounted through one of the remotes you can then go back to restoring and put it all back together again. All that data is not at all retained here in Zen Orchestra. It's all retained in these series of JSON files. So when we look at the JSON file itself this is all the data for that JSON file. This is everything it needs to understand how to restore any of these VMs and if you look this one in particular it happens to be free PBX I just know because I noticed that in here so all the data is retained here and oops don't edit this file I'm making sure I don't write to it but that's how all of the data is retained so that way when you do these backups they're completely portable they can be copied to external hard drive taken into somewhere else but this is also the reason I encrypt everything including my external hard drives because if you were to take my hard drives you would then have access to all of this if you wanted to this is why encryption is so important so all these backups land on encrypted drives when they leave the building they're on encrypted drives when they're stored and replicated in the building they're on encrypted drives as soon as power is lost to any of these drives they all require a password and I've talked about that with FreeNAS I've talked about that maybe a little bit with how you encrypt using dncrypt and linux I don't know if I've done a video on that maybe one day I will but it's everything's about encryption because all this data is really important being able to restore my entire company is right here and easily backed up to external hard drive now obviously this could easily backed up to the cloud this can also work within any type of data center scenario or maybe you have a rack and a colo this is also an easy way to manage all that so always gives you a good idea of how the backups work how simple they are and it's just a it's an easy system to use to set all this up and create all the schedules on here especially doing it on a schedule is probably one of the best ways to do it I'm not practicing when I'm preaching specifically with our systems but I spend so much time in here the only time I probably set it to schedule is when I know I won't be here which is pretty rare because I'm like religious about backups for our clients or anything where we build this out for someone we always set it to scheduled backups because if you leave it up to the humans they will forget that is generally how most average people are like well I don't play with it every day versus me I don't think there's a day that I'm not logged in here so I'm producing YouTube videos or scenarios or building things out for people but this is the whole overview of the backup NG system it's impressive it works really, really well all you need is a couple Zen servers to do the continuous replication no special agents like I showed it that quickly we set up that continuous replication and you just choose the server you wanna land on or even multiple that's actually I love this feature if I had a couple more Zen servers I'd show you you just continuously disaster recovery sync this to all the Zen servers if you wanted to and multiple storage repositories and everything else alright hopefully this was helpful kinda gives you an idea get downloaded and play with it man have fun with this this is really cool and if you're looking to put this in the air prize they have full support packages over at Zen Orchestra that you can buy where they can really scale this and if you're you know if you're running something at large scale you may want a little bit of support especially if it sounds like it was large scale it's probably mission critical but that's it for Zen Orchestra and backup NG thanks for watching if you liked this video go ahead and click the thumbs up leave us some feedback below to let us know any details what you like and didn't like as well because we love hearing the feedback or if you just wanna say thanks leave a comment if you wanted to be notified of new videos as they come out go ahead and subscribe and the bell icon that lets YouTube know that you're interested in notifications hopefully they send them as we've learned with YouTube anyways if you wanna contract us for consulting services you go ahead and hit launch systems.com and you can reach out to us for all the projects that we can do and help you we work with a lot of small businesses, IT companies even some large companies and you can farm different work out to us or just hire us as a consultant to help design your network also if you wanna help the channel in other ways we have a Patreon we have affiliate links you'll find them in the description you'll also find recommendations to other affiliate links and things you can sign up for on laurencesystems.com once again thanks for watching and I'll see you in the next video