 Time here for Lauren systems and we're going to talk about Zen Orchestra and XC PNG and how the backups are facilitated one of the great things about the way XC PNG works and of course with Zen Orchestra the company behind it Vates being support for both of these has created a very tightly integrated product when it comes to backups restores and Just makes it really simple to do now for those of you that want to keep your cloud at home. Yes, this is all open source Yes, this is available for you to compile yourself And you can find a link down below where I talk about that in a separate video for those of you that are in the business world Yes, this is fully supported changes your total cost of ownership If you're looking at switching away from other hypervisors over tech CPNG because often you pay for a third party backup utility Zen Orchestra has this integrated has subscriptions comes with support So it checks all the business boxes But can also be completely used in a home lab or your testing environments as needed for free I just recommend if you want to use it in business go ahead and buy that support support the project and you know It's a great product overall what we're gonna do though is not death by slideshow There's only a couple slides I want to show and the reason why is I want to make sure we're all on the same page when it comes to the language and The terms used for how they describe it then we'll jump into hands-on if you don't want to watch the slideshow Everything's time indexed down below so you can jump to the part that's most interesting to you So let's start and get the language down because I want to make sure people understand how the remotes work What a remote is what a pool is and what a host is and how these things integrate and the different backup types that are Supported alright, let's start with hosts and pools a host of the individual system It runs on a pool is a pool of other hosts or even single hosts that the host is in So even if you only have a singular XC PNG host it technically belongs to a pool It's just lonely and by itself as you increase the number of hosts that are in a resource pool So you can have three you can have well well more than three you can have a single instance though of Zen orchestra talking to all of these Simultaneously generally speaking Zen orchestra is gonna run as a virtual machine inside of one of your hosts It doesn't have to run on all of them or any of them I bring it up because you can actually run Zen orchestra completely separate on its own hardware if you so wanted to That is an option that maybe people don't always think about but usually it's running as a virtual machine on One of your hosts and you ever need a redundant copies Yes, you could always use one of these backup methods to keep an extra copy on another host Or just have your own copy somewhere else not on your host to be able to manage them a few thoughts on that But there's important aspects that affect where you put it one of the ways It works is it's pulling the data from the host any one of these hosts or all of them going through Zen orchestra and then to the NAS storage There's no dotted line here between this host and this NAS storage when it comes to backups It always passes through Zen orchestra So you have to think about that from a communication method and then we have the NAS storage options You can have many of them. You can even have S3 which is in beta right now, but there's S3 options as well To further things if you're wondering if it'll work remotely. Yes, this is where the XO proxy comes in I'm not gonna dive a lot into it But it is and documented feature and a really cool feature where you can have a Zen orchestra at your main site And we have clients that are doing this it talks to your local NAS storage It talks to all the onsite Zen pools that are set up and all the hosts on there, but then over a VPN You can then have it talking to these other hosts Now this is the one exception where it does not when you do a backup talk to the NAS that is on the remote end You could do that but obviously doing it over a VPN may not be as effective So should we talk about a lot of backups and a lot of data flowing so they have a tool called XO proxy It's basically a mini version of Zen orchestra that is running onsite So Zen orchestra can then talk to XO proxy and then that flow goes from these hosts to XO proxy To the NAS storage technically it's flowing through XO proxy each time So the same rule applies as I mentioned where the data doesn't go directly from these to the NAS storage But it's basically I could have made this a little clear I just want to be very clear on this part though It is going from XO proxy to the NAS storage flowing out of these So that level of control is still there and it's a great way if you have many remote sites They have one Zen orchestra managing keeping all your data in one place Managing all the different facets that are going on even on your remote sites including your backups all the reporting and everything else But just facilitated this way great feature Remotes now this is something jump back over here. Make sure we're clear Remotes are all the different storage Destinations that you set up for your backups to land on because they're called remotes Sometimes it caused a little confusion But the remote options are local file on the XO server itself This is a weird option because I don't necessarily recommend it unless you know what you're doing But you could create some types of shares inside of yours and orchestra Server and then point it at that local file if not, you're backing up within it But if you have an external and you've created some storage spots on that XO server It is a feature you can use next is going to be SMB So it'll facilitate connecting over SMB to I use a lot of true NAS But Synology works just as well. I've done a lot of testing with both same thing goes with NFS Synology and true NAS or two that I've tested I'm sure it works with most any NAS or Server that you have set up that'll support NFS and SMB And Amazon or S3 compatible it's a beta feature But yes, you can tie it to things like Minio and other S3 compatible or natively to Amazon S3 Details about the remotes Encryption is supported but in early release next you can store the backups as multiple data blocks instead of a whole VHD file This creates 500 to a thousand files per backed up terabyte But allows faster merging requires a file system I can support it such as CFS or VTR FS butter FS I really make sure and they give you a warning when you check this box It's a great way to have faster merges But but you have to make sure you're not going to run out of the file systems ability to handle thousands of files It's probably better also Most of time I find the NFS to be a little bit more effective while doing that many files But this is all comes down to the performance of the target and the NAS that you have set up But keep that in mind You can't just use a standard file system because if you have a lot of backups you could run out of The file systems ability to handle that quantity of files. It's not a storage size at that point It's a quantity of file problem. And you don't want that backup types And where they land this is the important part full backups land on one of the remotes NFS SMB, etc Delta backups land on the remote Now I put the delta symbol next to delta and continuous replication because the equivalent is called continuous replication But that lands on host storage. So it doesn't even have to be in the same pool This is actually a really cool disaster recovery feature where you're continuously replicating the VM onto another host That host can be in another pool kind of a neat feature And by doing so you created this ability to if the main host dies You can instantly start it up based on the last time you ran that and it's only doing a delta Each time so from the first seed the first one's going to be big but that is incremental afterwards to do continuous replication Because it's only sinking on the deltas between the last time that was ramp Disaster recovery is more like a full backup once again lands on host storage But it's doing a full copy of the virtual machine to the other host storage, whichever that storage destination may be Rolling snapshots. It's under backup technically. It's not a backup It's just a rolling snapshot, but this is where you would control that You can have a rolling snapshot On a regular basis running whatever schedule you want so you can have consistent snapshots of it But technically I know and I just want to make sure i'm clear on this. It's not a backup It's a state in time stayed on the same host storage Finally pool metadata and xo config to the remote I really like this feature because the way zen works is it's like a database of all the VMs that are status and the way you back it up is really simple You can just use zen orchestra and it grabs all that pool metadata all the settings all the networks You created all those functions and you can easily restore it right through zen orchestra So if you somehow deleted or wanted to redo a bunch of networks, but you messed something up You actually can roll back in time fairly easily and restore the pool metadata They've actually had some nice restore features for it because before if you'd watched my past videos The restore features were all not all integrated you kind of restore it from the command line So be careful restoring because you don't want to break anything next is the xo config itself So all the settings you put into zen orchestra all the backups you Build and everything else the configuration of that can be backed up as well And that can all be schedules part of the backup backup features smart backups based on VM tags This is used a lot when you have large environments and you want to add a VM without updating the backup job You can use tagging extensively throughout zen orchestra to really understand where all your VMs are and for example Let's say we're going to use the tag production or critical And by doing that you can have a backup that just looks for the tags that way as you have developers may be putting Systems together you can go up. I need to add this one to the backup So I'm just going to change a tag on it and then the smart backup will just pick up on the tags and apply Whatever backup job is looking for those tags. It's really nice because you can build Simple workflows this way without having to modify the backup jobs Automatic VM restore health check. This is just really cool This does so not by setting a screenshot like you may have seen some other tools use But instead by making sure the zen tools start by doing this are able to Do the automatic vm restore and by the way it does not have to be on the same host You can actually have it even on a separate pool doing it So you can have a test environment that you send the backups to to test and this is all automated And what it does is boots up the machine with no network interface Therefore will not interfere with current running VMs It looks for the zen server tools to start and once the tools start It knows the machine must have booted up to get that far really clever and very simple way to do your backup automation testing Backup vm in a running state with memory. This is actually cool not just For backing up to have an exact running version of it But it's also neat for forensics if you ever wanted to have a forensic snapshot and actually pull through it There's some advanced use cases you can dig into and read about on that Per job backup transfer rate control any specific job can be controlled and rate limited if needed For a certain bandwidth requirements. Maybe you want it just to go a little slower. That's an option The default of course is going to be full speed File level restores only with delta backups. This is actually an interesting feature I don't use often, but hey want to make sure you knew it's in here backup reports email slack webhooks xmpp And of course a rest api you can easily get all your backup data Pretty integrated to whatever workflow you want because between the normal reports having that rest api gives you In more features for doing some of the backup monitoring I think it's really cool if they've integrated that and the rest api supports more than backups This is just a function within the rest api. They have that also supports that Finally documentation rtfm. There's a lot of it. They've done a good job of documenting it I've got a link to the documents down below and uh, I really think they've just Made it easy to Understand and they've got plenty of graphics and things like that in or two Let's jump into my lab and show you some of the backups and kind of give me an idea what it looks like Now I want to start with showing the version i'm running right here and also of note You want to use in production? Yes, this is the free open source compiled version I did I did this so people using a home lab can you know get the question answered definitively Can I use this in my home lab and test with it with all the features you talked about tom? Yes, you can if you're a business Yes, head over to other Zenorcasture.com if you're interested in buying a license for it Now the first thing you do is set up your remotes before you get backups going Here's how the remotes look so it's under settings remotes and remotes are the different destinations You can see I have this one right here and we're going to go ahead and test my remote I like the fact that they have a test button because once you set one up you can say all right Can I read and write to that remote important thing to Make sure When you're setting them up you can choose nfs local as I said if you have a local spot you want to do this nfs or smb in amazon s3 Of note and I don't have a proxy setup But you would select the proxy when you have it set up in a way as I mentioned at slide 2 Where you're using it remotely over vpn you can have the proxy workers and it would list the remotes for that So that's another option when you're setting these up Now once the remote is set up you can always go back and edit it and sometimes it's a little confusing if you click the edit button It's actually bringing all the data back down here So you can go okay. This is where the edit button is Just clarify on that if you need any special options Here's where you would click to add any special parameters or options as needed for example in the nfs shares Or in the smb shares Something else worth noting is the subfolder options that have been added I like this because this can help organize backups. You can actually build a secondary remote but have a second Folder in there and this is some edge use cases that we found helpful for clients when They needed things broke out differently because on the back end where they were landing them They're able to send them to different places based on the subfolder for different retention policies Now once you've got all that configured and we'll go appear to the vms I have set up and I have this backup testing demo just one vm We're going to do some testing with here. I've actually already gotten some snapshots That are part of a delta backup. We can just run another delta backup real quick I really like the way it attaches the backup jobs that are related to this vm right here so we'll go ahead and Make things easy. I'm going to copy that and we'll go over here to Create a new backup and while that's running we'll back up another one and we'll create a new one back backup testing demo Another one select the vms. This is How you can select all the vms that are on here running not running You can turn into smart mode where we only do it based on a tag of vm has And then you would see which pools so they reside on etc etc But we're just going to leave it here and choose this one. We can choose the schedule now The schedule is the one to many relationships. So if we said daily And we wanted this to be a daily backup at this time or whatever time we wanted it to be So let's set it to maybe we want to back up there. All right, great We can do that and we're going to hit the retention policy daily cool Hit okay, I got to choose the backup type here Maybe you want to do a full backup once a day now Let me go back and edit this because now it wants to know what's the retention policy We'll keep three of them pretty simple and then we can just hit okay I'm not going to enable it because I don't want it to run at 11 today and choose the remotes I'll choose the only remote I have but you can actually choose multiple remotes This can go to multiple destinations simultaneously now because this is a full backup Disaster recovery is similar to a full backup. So that means I can click on the disaster recovery and then choose a target storage repository So this chooses the remote this chooses the repository where you want this to go So disaster recovery type as I said goes to another host And so I want one copy to go to a host I want one copy to go to my remote for a backup And then go down here to like your advanced settings And you can say do you want these run concurrently or how many you want to run concurrently email the reports only on failure always Select the proxy once again if there was a proxy that needed to be done is compression It does support uh ztsd xcp and g only but good compression on there offline backup means go ahead and Make sure you shut this vm down And you can also say offline with memory or normal when you do these So there's different options That you break down pretty simple back to the scheduling over here Once you chose the backup type you can still have multiple schedules for that particular backup type But you notice it grays out delta and continuous replication So let's go ahead and rework this if we start with delta that allows continuous replication Because I said both of these are delta types. So they're only doing differential But same rules apply for the backup and everything else So you can't get the idea. They're pretty easy to set up when you're Building these and we'll go ahead and hit save here or create And by doing so we can go back over to our vms And now that other backup job has become attached because he knows it's associated with this If you associate a backup with multiple vms, each one of those vms under the backup tab Will show this right here really simple the way they time together So you can start from the backup page or the vm And then for any one of these you can see click to see matching vms And it would give you a list of all the vms that match on there Now finally, let's go over to the backup jobs and overview and let's talk about well the one we just did Here's what it looks like So transferred using nbd. This is a new feature for network block device. I believe it stands for Where it went. Here's the backup report what it looks like looks the same in an email when you send it But full backup demo with health check. Let's look at that one. This is the cooler one So we did the snapshot we sent it here. So here's our start and end for the backup Now here's the health check part, which of course is successful and you can tell it to notify and failure It transferred it restored this vm duration was two minutes It gives you the speed it started the vm on this time here start end and then it destroys the vm when it's done It's just really a simple flow. You can download this as a log You can copy it to the clipboard. You can report a bug right from here So if you want to drive into it further and by the way backup job failed I love that when you do this it took you right over to Start the post over on github and it copied all the data in here No, of course make sure Because I have not submitted this issue because it's not an issue Make sure when you're doing this that there's nothing personal you didn't want in there It's not submitting any of your data outside of the backup log But hey, they've made it easy to Follow this flow all the way through to even reporting if there's a problem So you can engage with the team and then give them all the accurate information or relevant information that they have To start the conversation. Uh, I think they just did this at such a smooth workflow. It's really impressive Now I want to talk about my production environment and doing restores because I have a lot more backups in here So you can see the size of the different backups I have the different deltas Different logs for the systems that we're running and I'll point out how to do a restore So this is actually our backup testing demo because my lab is in its own world But my production system can see both lab and production at the same time And this is where we would do the restores. So we can do a restore health check by checking this box We could choose which version so we'll say maybe this last one we did and we can choose not only my Lab art pool, which is our lab pool pool of zen is our main one But you get all the different options You can do this cross pool So you don't have to do dr testing or restores In the pool that they came from or the host they came from they can all be sent somewhere else, of course And let's actually just go ahead and do a restore. So if we go here to restore Select the backup we'll do that one we did today on april 7th and we'll say, what are we going to land this? Actually, let's bring this over here. We're going to throw it in my system that is Destined for production. So this is the scuzzy production setup I have We can generate a new MAC address. We can start the vm after restore. Yeah, why not? We got it running still So I don't necessarily want to have same MAC address because that'll cause a conflict that would be a bad thing We just do that and it's going to run over here and we'll see a task start up for the restore And it's creating Running through all the processes and this can be restored. It looks like in about two minutes here I can actually go over to the overview of the backups Go to actually restore and the process has started. You can follow along the process here As it progresses to so let's uh do the magic of video editing Okay, it's going to take Probably a total of four minutes to get this done. So through the magic video any we're going to jump ahead All right, the restore has completed in four minutes or more technically three minutes and 48 seconds And then from here we can click on it. We can see the storage repository went to but let's go ahead and go to the vm Loading and it says restored from backup. It gives it the same name But it does add this because technically now there are two of them. So if I click on youtube, there's the one that was Running here. There's this one here and because we created new MAC address This won't have a conflict with the other one running So if you need to do that check it if not, yes, it will just restore with the same MAC address But as I noted here restores are relatively easy to do And one more last thing I'll comment on because it's the last have in there is the health checks And that's because you can have jobs that you have removed, but you still have the backup itself It'll give you a reason of missing jobs This is showing up because like this backup testing as I delete the jobs the health becomes Hey, the job is missing but you have a backup So it lets you know you can keep the backups even if you get rid of the job But it'll just make a note in here that those things exist And maybe you want to delete those backups that actually can be done right from here and detach vm snapshots Kind of the same answer you can have like your deltas where they're going to create snapshots But then if the job is missing it goes I don't know why this snapshots here Real referencing a backup job that no longer exists So I like to give you some of these maintenance things that you can do in here Now go back over to restore. This is where you can delete backups as well So if we hit the trash one here, we can choose all of them or individual ones that we may want to delete So you just click on them to select them So if you wanted to purge any of those backups for anything and the metadata is just over here Here's the metadata if we wanted to do a restore of that. So here's the zen orchestras and lab one Zen xcp ng new rise and labor You can see the dates for any of these some of these are older because they're older backups that just haven't really purged I probably should because I don't think I need anything here for my old april 2020 lab And that's all I have on these zen backups a few final thoughts though I really like the validation testing through the health check I love that they have that feature on there because it doesn't just do the health check It does it automatically so you can constantly have some process to validate your vms and the ability to validate them on maybe a Secondary server means we don't just know they work. We know they will work on our backup server if the main server fails So it kind of comes into that whole continuity and something I've said a lot is Untested backups are just wishful thinking so having some automation to the testing taking some of the human factor up You should still do a full dr plan and walk through every now and then it's really important to go through But I think they've made it really easy to integrate this into your full continuity plans that you put together Nonetheless, leave your thoughts and comments down below Head over to my forums for more in-depth discussion or connect with me on this and other topics head over to the xcp and g forums if you want to Discuss things with the developers and kind of look behind the scenes They're very public about what they're doing their roadmap going forward They do all this on github to keep it very Transparent so just one of the beautiful things about open source And it's just a really cool project and a whole team over at vates that's behind it is pretty awesome Thanks, and see you next time