 Tom here from Lawrence Systems. We're going to talk about Zen Orchestra, Zen Server, XCPNG, and how backups work. I've actually done this video before a while ago, but there's been some changes and they've, you know, made a lot of enhancements. So I figured it's time for an update. If you want to learn more about me and my company, head over to LawrenceSystems.com. If you need a hires for a project, there's a hires button up at the top. If you'd like to support the channel up in other ways, there are affiliate links down below for products and services we talk about on this channel. And we're going to be really diving into this because I wanted to cover this topic because it creates a little bit of confusion. But once you kind of get the concepts down, this is an excellent way to back up all of your systems and have a disaster recovery plan, including one that's very manageable. And this is one of the advantages I really see with the XCPNG server. And one of the reasons we like it so much is the same group of people, the same team is writing XCPNG and the Zen Orchestra software. So there's a very tight integration. I almost look at them as one product, but they are in fact two separate pieces. Zen Orchestra runs on top of XCPNG and provides all of the orchestration for it, the Web Interstate face, if you will. And much, much more, including backups. So you don't need a third party software to do complete machine disaster recovery backups. Now, that being said, one other concept, the reason you have it as a separate product is, and this works out well for us, you can manage many systems from one Zen Orchestra install. So I can have Zen Orchestra installed here at my office and manage different groups, different pools, different resource pools, different servers, and all done with one Zen Orchestra install. And all those backups can be managed. So I have my lab and I have my production environments, and they're all managed to one if you'd like. And it doesn't have to live on XCPNG, but generally speaking, if you have the server and a virtualization server, you would probably run it in there. All right. Now, when it comes to the backups, the reason I bring all that up, and this is an important feature, is how the backups actually work. So you have two things we're going to talk about. So I wanted to get some of the nomenclature out of the way. We have the remotes. The remotes are defined storages within not XCPNG, but in Zen Orchestra. And by defining a storage, and specifically, we're going to look at this. I have this true NAS box and we have this folder, sorry, data set called ZenLab backup. And then we go over here to sharing Unix shares NFS, and I have it shared out as big tanks and lab backup. So we've created a separate outside of Zen Orchestra and outside of XCPNG, which is where we place to land these particular backups. Now, this can really be not just free NAS, true NAS, this works fine with anything that can do an NFS share. You could have a Windows server landing them, for example, you can create an SMB share for Windows. So there's another option. But wherever you want to land it, that's where the remote is. And the remotes have a other option. We'll cover that real quick right here. So when you're adding the remotes, you can do them with NFS SMB or local. Now, local may not make the most sense unless you've done something inside of Zen Orchestra, because it actually will take the backups and put them inside of the Zen Orchestra. And if Zen Orchestra is also running on that server, you're not really backing up, you're just taking data and sticking it inside of Zen Orchestra that's also running on that server. So there's limited cases to use that. But an example might be people ask about how do I plug in a USB drive? Well, you could plug in a USB drive, map it to Zen Orchestra and map it there. We're not covering that. But just so you know, that's an option. Ideally, you use an NFS share or an SMB share. And the pretty straightforward, and I'm gonna hit edit on this one with NFS shares type NFS. We'll talk about the proxy in a second, put in an IP address. If you notice that's the same IP address as our TrueNAS here. And the path, save configuration and away you go. It runs a little test right here. Hit the test button, your remote appears to be working properly. That's an important feature to make sure it tests. And actually, if it wasn't working, it'll give an error and won't let you attach to it. But we'll kind of cover how that works in setting it up. So there's the first part. The second part is the storage repositories. And what those are is these. These are the places. So DozierLab, NFS, i97, local NVMe, specifically this one right here for the demo we're going to be doing. So here is the Debian YouTube backup demo. And it's on the i7 local NVMe drive. And I name these i7 is the name of the server that's running and will there host that this is running? I'm running xdpng. This is my lab setup. And we go back over here to the VM and we see the disk resides here. So Zen Orchestra can talk to because it's talking to the xdpng system. It can talk to all the storage devices that are on there as well. So here's all the list of storages. So we have this one, this one, this one, this one, etc. You kind of get the idea. So we're going to take and build some copies on here of the system for backups. Now if you have more than one host, this gets even better, of course, we're going to do it with one host, but it doesn't matter. As long as Zen Orchestra can talk to the host, it can back up to that host. So we're going to create a continuous replication to the host in this demo. And we're going to also create a snapshot, so to speak, that gets into the backup folder on those remotes. And when they're on the remotes means they're just standard files that you can back up with whatever backup software you want to get. And the example of running it on FreeNAS or TrueNAS like we do, you can set up a cloud account and encrypt it and send it up to the cloud so you have an offsite backup. So you landed on something that has NFS, that can be FreeNAS, TrueNAS, or Synology would be another example. You set it up to share the Synology and then however you want to deal with the backups from there to get them offsite for better safety reasons, you go ahead and set up whatever backup, they're just files. And if you ever have to storm, you just bring those files back and you can restore those files. Now the proxy part is pretty cool. This is a new feature and we're not going to cover it in a demo as I haven't really spent the time to learn it, but it's pretty neat. Normally when we're deploying this or we have XEPNG, we have an XEPNG server running local at a client. And what this does is we have XEP, Zen server, XEPNG, we have the Zen Orchestra and we have the remote. And that's because Zen Orchestra talks to it, but all data must traverse through Zen Orchestra to get to the different destinations in terms of those remotes. So obviously backing it up over a VPN, I can manage over a VPN because you know, just changing and moving VMs around no big deal because it coordinates that. But if the data has to flow through in the case of backing things up to a remote, it flows through Zen Orchestra. Well, VPN is not a real effective way. And this is a cool feature up and coming in at some point, like I said, I'll do a future video on it, but I want to let you know that this exists and it's kind of in testing right now as I understand it is XO proxy. And this allows me to be at Site A and coordinate over at Site B like this. So Site A, talk to the Zen server, define a proxy and I bring this up because you will see the proxy option where we're not going to turn it on, but it's an option there, remote. So what you do is you have essentially a custom Zen Orchestra installed proxy right here. So Zen server talks to it and then whatever the remote is. So if you've got some other remote storage defined there, so now you don't have to even have an XZen server, XOA server running over here, you can keep it running at your site VPN in and to specify where you want the backups to land. So really cool feature that they're working on. I just figured I'd cover it. Now we are also going to cover further the continuous backups, delta backups, full backups are cool, but full backups if you're doing full virtual machine backups, well, they're big. That's just the nature of it. So if you want to keep seven iterations, you would have seven times the size of the VM, even though you compressed it, that can be a lot of data. And each time you do the backup, it takes a lot longer because well, you're doing just like it says a full backup with the incremental backup system. They have it just synchronizes on the delta. So there's only the differential changes between from when you backed up the last time. So you can actually have this backup running quite frequently, even we do it on our production systems hourly, hourly backup scenes. I'll never lose more than an hour of an entire virtual machine, which is a pretty cool feature because I can restore very quickly. If something goes bad and main or rate array all falls apart, everything goes to hell. So to speak, it's quick to describe these files and I'm only an hour old on any virtual machine that we backed up running an hourly during business hours. So let's start it and take a look at what that means. Talk about the more practical uses and do the example demo. So I hear this Debbie and YouTube demo backup. Now one thing of note and we'll go over here to the host and look at the networking on it. Everything's connected at 10 gigs. So I bring that up because like I said, the data is going to be passing through the Zen server to Zen orchestra server. So the Zen server and the Zen orchestra server are both all talking at 10 gig and they're talking to 10 gig to this true NAS system. That makes a big difference. And when you're designing your storage, that's something to think about that your storage and your activity to these machines. It can't backup any faster than the network. Of course, you can't backup faster than the hard drive speed either. But because this is right now, this particular demo is living on an MVME storage, that's really fast. So you'll see these backups go pretty reasonably fast when they do it. Debbie and YouTube backup demo actually upgraded Debbie and Debbie and 10. So just for BI, we'll just rename that and BI and 10 Tube demo. There we go. Just so it's, I updated this to 10 from 9. So I didn't change the name there. Basic Debbie and 10 load. This is just a standard load. It's connected right now at 192.168.342. But you know, it said just I wanted a VM that's reasonably small to get this project done. So not real special about it, but we're going to leave it running for the first part of the demo here. Backup overview. Then we go here to new. So the added this option since the last hit it back, you can back up the pool metadata. And this is a nice feature that this is built in or you see has to be done kind of a more manual process. This can be included from the backups just so you know, but we're going to focus on the VM backups. The backing up the metadata is arbitrarily easy. Set one backup job for it. Figure out the frequency and it backs up like all the general pool information. So nice that they added it. It's not much to really dive in and cover. One, you Tube demo. They're alphabetically ordered. So if I put a one in front, it's going to show up at the top. And here's the traditional full backups. That's not the way we want to do this. We want to do a delta backup and we're going to do continuous replication at the same time. Now the difference is delta hands off to a remote. So when we only show the delta, we only see the remote and I have this true Nash remote to find. Actually, let's go back over to the remote settings. I'm going to make sure. Yeah, this one's off. I just wanted to do that. So there's less clutter in there. You can turn these on and off kind of as needed. When you turn them off, the backups won't run. Obviously, if they if they're selected because you'll get an error. But so when you select delta chooses the remote. So we have these two remotes to choose from. We're going to choose the true Nash remote. You have the option to delete first. What does delete first mean? Delete the extras before you do the backup. Probably not a great idea, but if for some space constraints you have, it does have that option to delete the files, then do the backup. I usually would I'd prefer to do it if you have the space to do this. Of course, when you're keeping all the backups, you wanted to do a full verification, do the backup. And once you're all done creating it, delete the oldest backup. That way it has a purge method, but you probably want to delete the oldest backup after you verified the latest one work. So you keep all the iterations that you're looking for. So there's a proxy option. No proxy on there. And report on failures does have reporting options and concurrency. How many times do you want it? How many do you want to run at once? We can say one if we want, but we'll let them run simultaneously. And it's it's a matter of if you have multiple VMs there, do you want them all running at once? Or do you want them one at a time? Now, this is a load issue. And if you overload the system, you're going to have problems because you're backing up running VMs. And they're obviously using the storage and you're trying to pull from the storage to back it all up. So if you set the concurrency to be 10, because I have 10 VMs, I want to back up simultaneously. Well, you could have a problem because your storage may start, you know, really slowing down for the people working on it. When you're selecting the VMs over here, we're going to do the YouTube demo one, but you can select multiple VMs and this can be done on multiples at a time. So that's really not a big deal. You could select each one of these and drag them in here. This is one of the reasons it has the concurrency options. So the final option here is offline snapshot. What this does will actually send the signal to shut down the VM, do the snapshot in the backup, start it back up, and then it'll finish the backup. It's a nice feature if you want to make sure the machine's in a non running state to do it. But that obviously can be very disruptive. But if you don't have any users on it to end of morning, you can just tell it to shut down, back up, start back up, might make sense to do it that way. So you know that it's always in a non running state and there's nothing to worry about. Now continuous replication. This is where we select a storage repository attached to it. And what this allows us to do is we have this single NVMe drive in here, but maybe that single NVMe drive is not very robust because it's not even rated. It's just one drive. So if that drive fails, so does any VMs running on it. So what this does is we chose a continuous replication going to another system. I'm sorry, another storage. And that way we always are sending two copies simultaneously. So one goes to the file level backup that we'll take off site later. The second one just goes to another storage repository we have attached. And that means we can fire it up at any time and get that going. Scheduling. Tons of options here. I'm not going to get too deep into this. But I like to name these by their time and date and how many I'm keeping. So, you know, keep two daily and backup retention too. Well, actually let's say keep four. So we'll name it keep four and we want four replications. So it does allow you to choose the separate. Now this also more is better. Of course, it's great to have more iterations of your VMs, but you may or may not have the storage. Even though they're incremental, when you're replicating them to the storage system versus the backup system, you have more constraints because, well, there may not be as much storage. It really comes down to your system, but it does allow you to keep those two in separate. You choose the minute, the hour, every minute, if you want. If you want a minute backups, that might be really taxing on the system or every hour, hourly backups. Or in the case that seems to make more sense to me, you usually pick business hours to backup when you're producing data. Why waste the time backing up when you're not doing it? But you're going to get the idea that you can do this. Go down here, hit save. By default, even when you create it, it just says disabled. You can also create multiple schedules. You can create different types of schedules, monthly schedules, weekly schedules, et cetera, et cetera. It does allow for that, which is really nice. So we've went through all this. We've now created the delta backup. We've created the continuous replication. And we're going to go ahead and create. And this will run for the first time now, which means it won't run very fast. The very first time it does, it's got to get all the data from the YouTube VM that we have kicked off here. We'll go ahead and kick it off right now. And it's got to get all the data from there over to these two locations. So it's going here to Storages, TrueNAS Ice-Cuzzy. And we see it importing Debbie and YouTube backup demo. So one of the copies is going there. And the other one is going over to the TrueNAS box right now. So that's heading over to the TrueNAS over in FS. Let's go ahead and take a look at what the files look like there. We're going to SSH into it. So we look at this XOVM backups. And as you can see from the command line, it gives each one of them a UUID. So it should be in this one. There we go. 2.9 gigs, 3 gigs. Put this back at the top. 3.2, it's so climbing. You can see it's almost done in the background here. We'll let this complete. Fast forward real quick. OK, the backup's completed. So let's see how much space it took. We've got about 4.2 gigs worth of data in here. So which the VM itself, because there's some compression going on here, is actually has a 16 gig drive. So it's only using that much. So there's that. And then we go back over to Backups Overview, look at the process, and it breaks it down. So we have 11.2 gigs, data moved, speed, 75, 74. So it didn't bolt simultaneously. You see duration three minutes. So about three minutes to do this full backup of the VM, because it was the first time. So type is full. Hence the first time you run it, it's going to do that. And then we look over here at the storages, TrueNAS Ice Guzzi, disk. And here is this. Now this is the W and U2 backup demo. And it's got a time date stamp in here. And it's actually a complete system that we can start. So let's say our main storage repository diet, how do you do your restore with these? Well, actually you don't have to. It's living in the same pool. And I can hit Start. And what it's saying is forbidden operation. And what it says is, wait a minute, you're trying to start an exact copy of a VM over on a different storage repository. And it's doing that because the only time you should do this is if something happened to the main one. So they would instantly conflict with each other because they have, well, all the same features, same MAC address and everything else. So you would end up with a conflict on a network. But that's the simple warning. So let's say the main storage system dies. Now, if you have multiples and servers, you would back these up to separate servers, physically, host, even though they're in the same pool. And then that would allow you to easily start it on the one. So if an entire host failed, local storage and all, you would just start it on the other host because this is the advantage of that. Granted, this is a manual process. This is different than HA. I've done a whole separate video on how HA works. But this is a nice way to create snapshots of everything on there. Now, let's log into that particular VM, which is this one here. The IP address is 192.1683.142. And everyone asked what I'm doing. I'm using Tmux here. Yeah, 192. Oops. 3.142. This generates a little bit of noise. So I have this little file that's called speed test. It's not really a true speed test, but it kind of gives you a quick idea on what it just did was, and just so people always ask, what does it actually do? We just did ddifdev0 and write a file out that's about two gigs of data, two gigabytes of data written out. Now we have two gigs worth of changes that we made to the servers to simulate changes. So we'll go over here, go to backups, and let's just go ahead and kick this off and run it again. So it's going to take a second. We'll actually see the transfers here. And it's going much faster, because now we don't have anything more than two gigs worth of data to move. So yeah, a whole lot faster. Fast forward through this. Now we're going to go to backups. It's finishing up. So we go to the backup, overview, and we see it's yellow because it's still merging. So continuous replications. And after it's done, it does a verification to make sure the data actually did what it's supposed to do. So you can see where it's at the transfer stage, transfer stage, and then it'll pause and do the merge. And after it does the merge, all the data will be backed up. It just takes a minute. And now we're at success. Started at 753, only took a few seconds here. So 754, 907 ended here. So less than a minute to get that done. How much 2.7 gigs worth of changes were discovered and merged, duration, type, delta. So you can see. Now it did this to both simultaneously. So we moved two gigs of data, one to the backups that are defile level backups and one to that. So let's go over here. And I want to see, go storage, and we're going to look at the NVMe Storage Advanced Coalescence. So right now, merge hasn't coalesced out of the actual storage. And so if I try to run the backups again, I'm bringing this up because this will happen sometimes. Should give me an error that says going to protect the VDI chain. Unless it coalesced right there. This is what the error message actually looks like. Job canceled to protect VDI chain. And I like that they link right to the more information on this particular topic. What this is, is when you try to do backups in a rapid succession, your backups haven't coalesced. Now this is very determined by the speed at which your storage repository works. Each one of these snapshots happens real time to you and can be rolled back in real time. But all of them do create points in time that they have to track changes in the VDI chain. So by doing that, you end up with a problem of if you do this really fast, you'll end up with they haven't had a chance to actually coalesce in the back end. So technically the snapshots may not back up properly with proper verification, so it just pauses. So as a matter of fact, we didn't get that error message because this is an MVME drive so it went really fast. But you can also see, because we didn't do those changes, we only moved 38 megabytes worth of data between these two locations. Now that we've done a few backups, and let's go ahead and do one more, because that would coalesce fast because it's also less data, so we'll go ahead and kick it off again. And you could say a few seconds, this time we get that error skipped because they haven't had a chance to coalesce, so this backup was skipped. Normally you're not running them in rapid succession like that, but if you have a really heavily in use storage repository that takes longer to coalesce because you just have really high read writes, high IOPS on there, this may be something you run into, but there's an explainer as to why it does it, so you can look at that and troubleshoot it based on that information. Now let's go back over here and we have 4.2 gigs last time we looked. Now we're at 4.7. So let's actually look at what the file structure looks in here. So the VMs themselves, like I said, this is all just files and this is what we would back up to get off-site. They're pretty simple. These JSON files reference the points in time, so if we actually 20242 we actually took a look at these JSON files. They're completely, so to speak, human readable just while they're in JSON format. So you can look through and see the job ID. You can see everything that was in here, current operations, etc, etc. They're just files. There's nothing really special about these. Then here the pieces that are in here. So we have this, we'll go into here and there's the VHD files that build this up. The JSON files are references and have the details in there. So what does it do with backing up files? So if you go to restore these, let's say there's a complete site level disaster where all the on-site backups and servers and everything were destroyed. The only thing you really need is go back a little bit more this folder right here that says XOVM backups. So as long as you backup that XOVM backups you're able to then point this at the system. So you point it just like we did before and then you would load Zen server and point and be able to restore all those files right from the backup. So you set the load Zen server and load Zen architecture again, but pretty easy to go back through into a whole system level restore when you pull all that back. So pretty straightforward there. Now last couple things we'll show is what happens when we actually have to restore or do some of that. So let's go ahead and run this again because by now it should have coalesced. Still hasn't coalesced yet. No problem. Go back over here. Let's just go ahead and destroy the VM because I can do the restore right now. So we'll talk about ways to get rid of it. So we'll go here and kill it. Done. This is removed from the system. We have lost that VM. Couple ways we can deal with getting it back right over here. Matter of fact, if we look at the true Nassai, it's because we have the disks over here. I can actually go to VMs. There's more than one way to get to this. So here's all the iterations we have each one time stamped. Which one would you like to start? Well, we can start the most recent one. So we go over here and just hit start. Yep. We're going to force it to start. Now we can start at a copy. We can take it off line. Let's say we needed something out of one of them that was seven iterations back. However, your timing is, those are all complete possibilities. So there's definitely more than one way to start it. In case you don't want to start it to destroy the other one, you want to do that. But quickly we're right back up and running. And actually this logged me out down here. Yep. Broken pipe. SSH back in. All right, we're back in. Same IP address and everything. Really straightforward. A VM is completely restored. So we went from IA completely destroyed it to booted back up right back where we left. No problem that fast. So we can now take this one. We would just rename it. So instead of YouTube based on the name there, just delete that cool. And one thing to note about the backups, because we destroyed the VMs and it is done by UUID. We get this error here. So we would find the DBN YouTube backup demo, the new one we created and now we didn't even have to do anything else. Hit save. And now this one's part of the backups. Now the first time it runs it, it's going to do a full backup again, because it recognizes a completely different machine. So your incrementals do start over in your the whole delta thing and everything has to start back over. Now what about restoring it the other way? So let's go ahead and do the site level destroy. So go ahead and remove, which will shut it down and make it all go away. And we'll just destroy both of these two. So remove. All right. Go to backup. We've completely removed our YouTube demo. It's gone but we still have these. So we point it back at a remote. We have these delta backups and we can choose from them. So we have the 902, 907, 910. Let's use the 910-1. Where do we want to land it? Do we want to put it back on that here, but we can land it back over here on the ISKIS. Let's put it back on the MVME. I can say start right after restore. Hit OK. And now we see an import task. Take off. And it's pulling it from the file repository and dropping it right back in with all the settings ready to restore. No big deal. So let's fast forward through this. All right. Go back here. It started. It's booting. And we're right back up and running from that incremental one. And it broke of course because I was logged into it. SSH back in. Nothing changed because it's a full same IP address and everything because it's a clone. And then back to the renaming and away we go. Rename it back to its original name so we know this is now our main system again instead of that. And also it does do this so restore from backup gets tagged on there and we can just take that off for the tag and it's back to just being a lab test machine back up and running. So it's pretty fast to do this. Now the last thing I will mention though is if you go into backups and we want to restore again one of the other things you can do is let's say you needed something not out of the latest one like oh I needed this one from three versions ago. You can drop that and restore it but then not start it and then start it up on a separate network for example change the settings on that particular one so they will let you do this restored again without any error messages but of course you can remember if you look at machines so if you start them they have the same MAC address and everything else so that may create a problem for you so you have to think about that if you do want to restore because you wanted to get something out of it something to consider on there. Now the final and last little thing I'll talk about is the Zen Orchestra system itself just to mention this so they do have a whole and lots of details on how their backups work etc which is really nice lots of reading you can do here and I'll leave links to this stuff but the paid version there's the free version and paid version I just bring this up because someone may ask if you are a business and want to use with a fully supported version absolutely they offer those services both on XTP NG server and on Zen Orchestra so the free edition does not come with all the backup features you don't start getting backup features until you do the starter enterprise. Now for business this is very reasonably priced and things like that but what I did here and this is for purposes to so to speak prove it all works you can also compile this yourself it is a 100% open source product if you wanted to test it out in your lab they do give you a 15 day trial but obviously for people running a lab maybe you want to test it long term absolutely you can just compile this yourself you're going to receive no support it's just community forum support but the product works as if you have an emergency run production I highly recommend getting paid tickets unless you're super familiar with how all this works in case you run into problems but if you want to fully use this in your lab you can build this out absolutely you can build all this from source and there's no licensees for any of the XTP NG it's all unlimited with all these features and everything that I showed you here that's always why I do my demos in this to show people want to get started with it in home lab but for businesses when we sell them we usually work with our customers and tell them go ahead and buy the support contracts for very reasonably priced at $77 a month because you know these are larger customers running quite a few VMs on them so they do have a whole system for this and of course this pricing is relevant for February of 2020 I like the fact that they don't hide any of this so if you ever do need to buy support it's very clear they've always kept it publicly available on their website so I always take that call for pricing and let our sales rep thrash you to death and if you go to XTP dot com they do offer support for this as well so same thing they have $600 per host per year for some incident level support on there or some enterprise 1200 per host per year but there's no license fees with XTP and G it is a fully open source project so that comes up a lot when people ask questions to take my videos so figure out leave this on here for people asking but it is the turnkey open source hypervisor and all everything I did here was open source everything I did here was free minus the exception someone may point out what you talked about true nas yes true nas is the commercially supported version of free nas and I've got some videos on this and some of the differences as well so if you want to dive into that topic I can leave a link to my true nas videos but whatever you want to use as your remote storage feel free to use it our eight and thanks and thank you for making it to the end of the video if you like this video please give it a thumbs up if you'd like to see more content from the channel hit the subscribe button and hit the bell icon if you like YouTube to notify you when new videos come out if you'd like to hire us head over to LawrenceSystems.com fill out our contact page and let us know what we can help you with projects you'd like us to work together on if you want to carry on the discussion head over to forums.laurancesystems.com where we can carry on the discussion about this video other videos or other tech topics in general even suggestions for new videos they're accepted right there on our forums which are free also if you like to help the channel in other ways head over to our affiliate page we have a lot of great tech offers for you and once again thanks for watching and see you next time