 In this video, I'm going to walk you through getting started with XCP and G and Zen Orchestra. We're going to start with loading the hypervisor and take you all the way through creating your first virtual machine. I have several videos I'll be referencing along the way that offer deeper tutorials on certain topics, and you will find those right in the description down below. I have timestamps on this video for those of you that just want to skip ahead. So let's get started. Are you an individual or forward thinking company looking for expert assistance with network engineering, storage, or virtualization projects? Perhaps you're an internal IT team seeking help to proactively manage, monitor, or secure your systems. We offer comprehensive consulting services tailored to meet your specific project needs. Whether you require fully managed or co-managed IT services, our experienced team is ready to step in and help. We specialize in supporting businesses that need IT administration or IT team seeking and extra layer of support to enhance their operations. To learn more about any of our services, head over to our website and fill out the higher S form at laurancesystems.com. Let us start crafting the perfect IT solution for you. If you want to show some extra love for our channel, check out our swag store and affiliate links down below that will lead you to discounts and deals for products and services we've discussed on this channel. With the ad read out of the way, let's get you back to the content that you really came here for. Now we're going to start here by talking about the hardware because the first question is what do I load this on? What does it support? And XCPNG, if you would have followed some older posts or maybe you found something from three or four years ago when they first released it and said, hey, it doesn't have support for like Intel two and a half gig NICs or it doesn't work well with certain Ryzen or Epic processors. Those are issues that have been fixed. They have updated the kernel packed in the new drivers. The production release here in February of 2024 is eight point two point one. That does have the extra drivers in there. We're going to be using the eight point three, which is in beta two. So I'm going to say it's pretty close to being released. That's we're going to use for our demo here, which also has updated drivers. And by the time you're watching this will probably be in full release. But if you just want to stick with the eight point two one, yes, it does have that Intel two and a half gig support and better support for Ryzen and Epic or overall AMD processors. I have a video of a Ryzen system I built. I'll leave a link down below that has been running for the last few years. My XC PNG production systems and lab systems, kind of both that we have at my office now as far as doing things like rate arrays that comes into question quite a bit. And that's why I pointed out that this does have the ability to put two drives in it. And we can set this up as a rate array. But as of right now in February of 2024, there is no UI to manage that rate array. It's really a Linux RAID managed system. And that includes if you choose ZFS, yes, ZFS is supported. Yes, you can choose that as your storage device on an XC PNG system, but it is manually set up and manually managed. You manage the storage repositories through the system in terms of once you've created them, but the underlying like looking at snapshots and things for the ZFS specifically, not snapshots for the M's that is not handled within the UI that is all handled from the command line. And the same goes if you have any issues with your rate array, you would be rebuilding it from the command line. I just want to put that here for some clarification. Now if you're choosing some enterprise hardware to run this on frequently, let's say Dell has a built-in controller that presents all the drives as a single drive because Dell is handling the rate rate, well, then this won't be an issue at all. And of course, I've always talked a lot about on this channel using an external storage server as your target for your system, such as either a Synology or a TrueNAS, so you can also use those external storages, but once again, they're presenting the storage and just bringing it to XC PNG. But now let's get into actually loading XC PNG, because that part's pretty simple. And yes, we're just going to use local storage to keep this whole demo really easy to follow. Now if you're looking to do a deep dive into the storage systems that drive XC PNG, how migrations work, how coalescence works, you'll find that video linked down to below. It's a deep dive into that topic and all the different features around it. Now once you've determined which hardware you want this to run on, the next step is to download XC PNG from XC PNG.org. They give you some pretty basic instructions of how to copy it to a USB drive. There's also an option on Windows that you can use Rufus. This does work with Ventoy as well. Now we're going to do the 8.3 and I'll leave a link to this blog post to talk about the beta two that was released in February 15th of 2024. And by the time you're watching this might be out of beta and in full release, there is a link down here to download it. The boot installer is pretty straightforward. The first thing is going to ask is which keyboard layout. We're going to choose US for mine. It checks for any existing products if you're going to be doing an upgrade. Finding none, it'll just go through and let you accept the license agreement and go forward. Now as I mentioned earlier, you can do software raids. So we'll go ahead and do that for this particular sub to show you how it works. Simply choose the drives. Go ahead and hit create. Let's me know it's going to race and it's going to create the ray for me and it's going to be a simple mirror. With that being created, we're going to click OK. Now from here it's asking where you would like to do the virtual machine storage. It did not need all of that space for the install of XCPNG. So the leftover space, we're going to set up as an EXT volume and that will allow thin provisioning. It does give you the option for LVM if you'd like to use that. I recommend EXT first standard thin provisioning and we're going to head and click OK. Please select a type of media. The media is fine unless you have some extras. I like to live dangerously. So we're going to go ahead and skip the verification. And from here we want to set the password. You can change this later. So I'm just going to set it to something simple. We're going to head and IPv4. That's fine. This is where you can choose automatic via DCP or set your static IP. I'm fine with DCP. So I'm going to go ahead and let it do that. If you have a management VLAN that you'd prefer this to be on, there's an option here to choose it. All these things can be configured later. We'll give it a name and we'll go ahead and let it automatically configure the DHCP servers. Now we're going to choose a time zone. Use DHCP for NTP servers. Use default NTP servers. I'll provide them manually. These are all great options. But make sure if you choose DHCP that you do have it set up to provide NTP servers in your DHCP. If not, you can use the ones that are the defaults. And we're going to go ahead and click install. If the install is complete, we're going to compress OK and let it reboot. Now once the system boots up, it gives us the IP address of the system, the network management interface. And if we needed to make changes, this is how we would do this. We can configure the management interface again. Root, go ahead and put that password you set. And then from there, we could go through and change these, disabling it or choosing a different one. We're going to head back out of this because we don't need to change any of that. But this gives us just basic functionality of the host. It does not have a lot of features in here. It will start and stop any VMs running on this host. You can show them currently because we just loaded it. There's none. And if you went to this IP address, you would not be greeted with a normal management interface like you might expect. That is all handled through Zen Orchestra. So it's actually the next step is we're going to load Zen Orchestra on this. Now, if you do go to the IP address of the system after you first load it and type in the password, you will be greeted with XO Lite. Now XO Lite is still very much under construction. It is a work in progress that there'll be a lot more progress made later in 2024. But for now, it's kind of, well, under construction, like it says. They're making a lot of headway you can follow along in the forums on this. It will do basic tasks in terms of starting and stopping virtual machines, but it doesn't have a lot of other control beyond that. In the future though, this will be a default load when you load XCPNG, you'll be able to get to XO Lite connected to your system and perform quite a few of the functions in it. By the way, this new UI design that you see for XO Lite is going to be similar to the new UI that is upcoming in the newer versions, version six of Zen Orchestra. We're going to keep with the current version, the five series, but in the future, this is also going to be a common theme that both of these will be following. They have an entire dev blog that really explains their concept for UI. They're doing this very carefully. So this is why it's not being done with rapid deployment from the time they announced, but they're doing it in a very concise way. So there'll be a very common theme between them and a very good UI that they're coming up with, but for now we're going to keep using the Zen Orchestra five series and start from there. Now let's talk about getting Zen Orchestra deployed. The challenge you may find, if you watched my build from sources video, which you'll link down below, is that if you want to build from the sources and you want to run it on the VM, how do you get that initial VM set up on there? Well, there's an interim way to do this, which would be load Zen Orchestra from sources using something like virtual box, something I mentioned in that particular video, that you can load it on a separate system. Zen Orchestra does not have to run on the host itself. It can run on a separate system or even another host you already have. This is actually kind of an interesting relationship Zen Orchestra has with XCPNG, the hypervisor host, is it's a one to many relationship. One instance of Zen Orchestra can run many, many instances of the different XCPNG hosts. I have a whole explainer video where I talk about how the architecture of the Zen API works. And like I said, you'll find that link down below. Common theme you might be hearing here. That being said, how do you get it on the host itself? And was there a easy way to do this or what if you're a business that would like to use the full version completely supported from the team over at Bates? Well, they have a really simple deploy message for the Zen Orchestra appliance. So XO refers to Zen Orchestra, XOA refers to Zen Orchestra appliance. The appliance is the completely auto updating service delivery model that you can get a license for. They have all of the features, but you get quite a bit of functionality without a license. Matter of fact, you can even get a free 30 day trial. As of right now, they've been offering that for a number of years that will give you all of the features if you want to test them out. But even with the basic deployment of Zen Orchestra, it's pretty easy to do. Now there's two ways to get the Zen Orchestra appliance deployed you can do from the command line. SSH into the Zen host, run this one line right here. You'll find a link to that in the description. And from there, it will download the Zen Orchestra after you've answered a couple of questions about whether or not you want it DCP, the IP address, et cetera, and you'll be able to log in. Option two is doing it through their web interface. You can actually go to the VATES site and they have the option where you can actually connect from their website to your local server and it'll build a connection that allows the Zen Orchestra appliance VM to be imported directly in there and get going. No matter which option you choose, whether it was the XOA appliance or building from sources, the next step I wanna do is connecting your Zen Orchestra instance to your host because even if you auto deploy it and won't auto connect to that host, you do have to put the credentials in. And then from there, I will show you how to set up the networking and then we're gonna build our first VM. Now this is the Zen Orchestra built from sources that I'm gonna be using for this. If we click on no support, it lets me know that one, it's up to date. It'll see the commit version I'm running at. This is the latest version available as of February 22nd, 2024. But when we click on home, we'll notice it says add server no servers in the list here. You can click on the add server or go to settings and servers. Now as I noted, you can have multiple servers in here. We're only gonna need to connect one for this demo but we're gonna type in XEP and G and give it whatever name we want. So we'll call this one my YouTube demo. The address. Now when I installed, yes, for those of you that were sharp-eyed and watching, 172.16.16.195 was the address but I actually moved that to a reserved address of 49. Put the username in, put the password in. We're going to check the unauthorized certificate and we're gonna hit connect. And you can see it's connected. It lets me know cause it's green here and we have a pool called XEP and G test. Now we can rename the pool right now if we want but I wanna point out that the name of the pool is the same as the name that it was assigned when we set it up. So if we go here to the home and then we choose host so we can see the host itself. There's only one host in here. This is called test. We'll talk about the patches later and we can rename the server as well. So we can call this our test server that's perfectly fine but let's give the pool a better name. Pools are the resource pools is what I'm referring to. I have a whole video where I dive deep into the functionality of Zen as I mentioned earlier and it explains what resource pools are but they're essentially pools that are of one or more machines all grouped together. So what you set in the pool applies to all the machines. This pool we're gonna call our YouTube demo. And if we wanna make some notes here we can just click and type them. This is a common theme you'll find throughout Zen Orchestra that there's a lot of spots that you can just simply click and rename things. Now while I'm under the pools here I wanna go to the network and change the network settings to make this a little bit more clear. I've only got one ethernet plugged in for this and we're just gonna go ahead and rename it to East Zero but by default it is called pool wide network associated with East Zero. I think East Zero is easier. I'm not using this one at all so I'll just call this one not in use. That way when I assign this to a VM it's pretty obvious this is not one I'm plugged in and this one is East Zero. I could give it a description if I wanted to to what network it's connected to. It's kind of a personal preference just for whatever makes sense to you. Now while we're here on the networking I'm gonna show you how to add a VLAN. So we're gonna go here to add network and I want you to remember you always add them from the pool side and then all the hosts within this pool will have the same uniform network configuration and this is just gonna be VLAN 10 for our demo here. Give it whatever name I want. With the VLAN in and hit create network. Now we have VLAN 10 created just so we have one more option when we're setting up for our VMs. Now the next step is to add a new repository to store ISOs so we can eventually get to loading our virtual machine. So we wanna do a new storage and select the host. Well there's only one host in here to select storage name. We'll call this one ISO. I'll give it the same description. We want to choose that it is a local ISO storage and we're just gonna use slash media. This will create the folder on the system directly for us to upload it to and we're gonna head and hit create. Now that we have the ISO created we want to upload an ISO to this. So we're gonna go over here to import and we want to import a disk to import an ISO into an ISA porty and ISO repository is required. We've already done that. This actually just links to the instructions to do it. And from here we're going to select our ISO and we can drag and drop one in here or just click it and choose to upload or they has the option to download it from a URL and give it a name so we can pull it in that way. I already have one downloaded so I'm gonna choose to grab it from here. I'll choose my Debbie and 12 ISO and I just click import and it's going to upload it right into the repository. All right now that we have that loaded I can add a description if I want that's completely optional and I can delete it if I want right from here or even rename it and give it some other name. If I wanted I can just maybe call it Debbie and 12 but I like leaving the full name cause I do know it's a Debbie and 12 and it is the net install version so I'll leave it at that. Now from here let's go ahead and create a new virtual machine. We're gonna go to new we'll select the pool that we're connected to we'll select a template. Now these templates are just to set some general parameters. It is not required that I have a Debbie and 12 template so I'll choose a Debbie and 11 one but it can be renamed like Debbie and 12 just by typing that. Debbie and 12 have any CPUs. We're gonna go ahead and go more CPU sounds reasonable we'll give it two gigs of RAM. PXC boots an option if you have that set up but we're gonna go here and we're gonna choose our Debbie and 12 ISO. Once you have a VM loaded you can switch it out for the guest tools to load the different tools but for now we'll just choose this. We'll leave this fine. This is our eight zero and as you notice there's our VLAN 10 and not in use. Now it's nice and clear from labeling it and I'm just gonna go ahead and let it auto generate the Mac address but you can specify it and you if you want to now or this can be done later add more than one interface it will let you do that right here as well. Go ahead and leave it at one interface then we can choose the storage. We'll just call this Deb 12 we can leave it at the local storage repository so only when we have and we'll just let it say created by XO and then maybe give this 60 gigs of space. If we wanted to add more disks we can simply do that as well and we'll go and close that and close that. If you go to advanced settings here whether or not you wanted to boot VM after creation auto power on or use cloud and net it has some more advanced options including repeating so you can actually build these if you wanted to build quite a few of them at a time and you can come up with a naming scheme up but we're only gonna create one for simplicity here and we also can choose the default BIOS or choose UEFI I'm not worried about either of those we're just gonna leave it at BIOS and go ahead and hit create. Now from here it started up the VM and we're gonna go to our console and run through a Debian install. Make this a little bit more viewable I'm clicking this to show it brings us to the screen. We'll go ahead and set up DB and really basic here if you leave the root blank it will automatically allow you to create another user and create sudo so let's go ahead and enter that. Now the system's resolved it'll automatically reboot and now we have a working Debian 12 VM installed. One thing I will note here is the drive is still installed we can just click this eject and it will take out the disk and if we go under here for advanced and we can see the boot order is hard drive DVD network and this is as it should be but if ever we needed to move it you just drag this up and hit save if we wanted to boot from DVD again to do a reinstall it's pretty simple to do it just may not be obvious that you just drag and drop these and click save. Now if we go over the network tab it's not showing the IP address of the system and as because we don't have any of the guest tools installed and if we go over to general you'll see management agent not detected. So let's go here and to the console and log in and there's two options we can do from here we can select the guest tools and attach them to this one mount them run the guest utilities on there that would show the IP address and load the management tools or they do have some more updated tools that you can run that are in currently beta and I'll leave a link to those down below it is their new tools written in Rust but for simplicity we'll just use the guest tools to load the guest utilities in Debian we're gonna go sudo mount dev CD-ROM 2 slash mount CD-ROM I already have a directory called that put the sudo password in go to my this mount CD-ROM Linux and go ahead and just run the install SH don't forget to add sudo if you're not running as root say yes now the utilities have been selected and run it's installed and we can go over here go to the networking and we see the IP address that's been assigned to this and if we go over here to general it says management agent 7.3 detected this also allows for the dynamic changing of memory so you can see that the memory is now being detected in the stats here now while we're on this stats page let's go ahead and just stop the virtual machine hit okay and I want to point out that each of these are clickable and so we can change this to eight cores for example and we can change this to four gigs of RAM and we can change this right here if we need a bigger drive and just click the drive and maybe we want to make this 80 gigs now making this 80 gigs only changes it here you still have to go into the OS itself and expand the partition unless you have something set up to auto expand it but this does easily set the size difference to now 80 gigs and if we go ahead and restart this VM we are going to get a no hosts available and I wanted to point that out because if we go over here to our host and we go and look that we only have four cores available on this particular host that's why it will not let me start this sometimes is where people can get a little bit confused so we'll go back over to our VM and we'll go to advanced if you read the logs it's telling you there's no hosts with that many processors available and I can't just downsize it to the four cores well I can but it's still going to give the same error because once you've upsized it to a certain size you have to go here because you have the limits set we have four now but we want a maximum available of four so let me explain how this works if we set the first one to single core this means we're going to start with one core but even live can dynamically upgrade it to four cores I want to go ahead and start it let's say with two cores but laborer I want to expand it but I don't want to reboot it and the same thing goes for the memory we have the minimum and the maximum that you can set for the memory it's four and four but we can also set it to four and eight so it's still going to have if we go over here to general two and eight gigs of RAM let's go ahead and start this VM up and once it boots in a management engine is detected we can now go over here to general and change how many cores it has so we can up it to four per center and now without rebooting this VM we've changed it to update the number of cores and then we can also shrink or grow the amount of memory depending on how much is available so now we're going to shrink this VM to only four gigs of memory I've got a whole video about provisioning memory and doing this dynamically and some of the challenges that can come with applications running while it happens some applications are fine with it others may not be but the OS itself, Debian and Windows both can actually have their memory changed while it's running without crashing now if we want to change the network settings and we did create that other VLAN and we can go here to networking and we can choose VLAN 10 the other one in there and once you have the management tools you can dynamically change these on the fly without restarting the VM matter of fact, while the VM is running we can go back over to our pool we can go to the demo here and we can create another network if we wanted to change this network even while it's attached so we can just call this one NSFW and then we go back over to our VM and we'll see that the name is now changed as well these changed dynamically and it's quite convenient because now I didn't have to do anything to actually swap the network around to make that work now the next thing I want to point out are snapshots before I do that let's go over here to the console and log in I'm going to install the package Btop so I have something running on the screen here so I'll leave this running and we're going to go over here to snapshots and instead of just doing a new snapshot which is easy I could simply do this this is a snapshot in time of this particular system but let's go ahead and do a new snapshot with memory and when we do this it takes a little bit longer but I want to show you the state that it brings it into because you can do more than just snapshots in XEPNG you can do freeze frames of where the memory was and all running processes so this pauses a little bit here and now we have a snapshot with memory you can see how it says memory saved it's got a little green versus the normal snapshot here we can go to the console here and see everything running and let's go ahead and quit and just run regular top so I'm still doing something on the machine I've got it running top right now so I should be able to go back to the snapshot here and hit restore so we want to revert VM to the snapshot I don't want a snapshot before of it currently running let's just hit okay and it's going to take a second and it's going to restore the VM go back to the console here it's taking a second to push the memory this machine is not particularly performance oriented that we're running this on this is just a little mini PC and there we go it's exactly where we left off from a frozen state essentially of it with the memory restored to all the applications running as we left it before including I had ejected the disk and it brought the disk back the other thing to note about snapshots in Zen so go ahead and purge these ones here purge the snapshot when you do the snapshots it's more than just the drive itself your snapshotting so right now we have this with four gigs and four cores and we want to do a snapshot of it I'm not going to bother with the memory one again but now let's change it let's go ahead and reduce this to a two core system and then let's go ahead and reduce the memory down to three gigs so we've now made two separate changes on here and we can even make some notes test hit okay and we'll call this WN12 with some more words here now let's go ahead and revert this back now we didn't do a memory one so when we do this reversion it's going to go here to revert to snapshots and then we're not going to snapshot before and it's going to reboot this because it's not a memory snapshot so it's going to bring it all the way back and revert it and start it up but you notice it fixed all the things we did the notes are gone the changes here back to four and four everything is as it was when we did the snapshot these are great so when you make these modifications it's bringing not just the settings of the drive or certain settings it's bringing all settings related to this VM within that snapshot back to the point at which you made the snapshot and from here if we want to go ahead and delete the snapshot I don't really need it I can just go ahead and remove this snapshot hit okay and now it's back to having those snapshots but I want to point out a few more things we can do a snapshot of this virtual machine and then from here we can create a VM from this snapshot by clicking this so if we do this name copy we'll just call it Debian from snapshot bold copy or fast clone I'll go ahead and hit fast clone fast clone means base it off of the other one which is going to be pretty much instantaneously because they share a common base but not really a big deal when they share a common base in XEP and G because this did not tie this particular VM to the other VM it is forked from it but if I wanted to move this virtual machine to another server another repository it will automatically do the work in the background to split it from it so we can start this one up here and I can show that even though we have this one up and running that we fast clone from that snapshot when we get back over to our VMs I can now take this VM I can go ahead and purge the snapshot I can matter of fact stop this VM it's now off and we can completely remove it and in the background it's automatically doing the work that needs to be done to take that fast clone and all merge it into this particular clone and we can see that by going to disk clicking on local storage and you'll see where it's coalescing we have two pieces to coalesce it is the two pieces from the original VM and the snapshot now they've coalesced into the new VM that we created XEP and G handles this seamlessly on the back end it's actually a really nice feature that the system has we can go back over here to the VM and let's rename it just call it WN12 again just not really from a snapshot anymore and once again we can start the process over create more snapshots or we're gonna go ahead and stop this VM we can also just fast clone a VM once it's in a stop state so we can click this a few times go back over to home and now we have several clones that we can then group startup which I will probably get an out of memory I don't think there's enough RAM in this to start all of them at once and it will give you a warning are you sure you wanna start by virtual machines and it was only able to start three cause it ran out of memory for the other two we'll also just go ahead and remove these other two go ahead and hit remove you do have to type the words out delete two VMs and now they're gone when you're in this view right here you can also expand these out so you can see statistics on each of them this is just another view that you have and yes it's going to start showing the IP addresses as these boot up and then we can click on each one of these and look at the actual running virtual machine the next thing I wanna mention for management is tagging so if we go here and we'll call this one Tom maybe wanna call this one production as well cause it's a production VM and now we can put these tags on here these, this is the only VM that has a tag so we can see the tags once again by expanding it out like this we can see the production or Tom we can also go in and add it to other VMs we can see let's go ahead and add Tom to this one this allows me to then type in VMs tagged with Tom that's these two here but not that one the clone ones I added that tag as you can see here but I did not have that tag on that one same with clicking production only shows this one and we can have our tag production this is a really nice feature that is throughout the entire system through Zen Orca Show that makes managing things a lot easier you can add tags to each of the hosts in here we only have one but we'll call this our production host and this helps us distinguish when we have more than one host and for example let's go ahead and go in here to our servers label another host, hit connect and now we have two host connects to this matter of fact if we go here and we go to our hosts you can see we have the test and one called XCPNG server one I actually used this one in another demo that I had where I was showing how to set up networking and this one's called YouTube demo pool and if we look at the pools one is called XCPNG YouTube demo and this one's called YouTube demo pool this one's letting me know that there's a bunch of missing patches for hosts in this one this one's fully patched enough to date so there's no patches on this notice here we'll get to loading into patches in just a moment because I wanna cover just a couple more things related to the tag this is something that can be very confusing when you look at the storage because we have more than one thing called local storage local storage you can see on the end one belongs to XCPNG server one and one belongs to XCPNG test now you can rename the storage and this one belongs to local test storage we'll just go ahead and rename it but we can also tag them this way where we set things to be production then when we look at the storage go back over here to storage and expand these out we can see that this is a production storage and we can set filters here only show me production storage and filter them out so when you have a lot of hosts connected across multiple resource pools you can filter things rather quickly throughout the filter at the top here and as I said this applies to VMs, hosts, pools and so on and so forth now let's touch on loading the patches because there's more than one way to do it we can simply load the patches from here it'll do if there's multiple hosts in a pool a rolling pool update or you can install pool patches by clicking here something else you can do you can go to the host we're gonna go into our XCPNG server test we wanna go to the console in this one you could also just SSH in and from here you can just type yum update and it will grab all the packages and you can say yes and it will install all the latest packages on this system pretty simple to do so either way you wanna do it whether you SSH and do a yum update and say yes I like it cause you can watch them or you simply click the install pool updates the updates will get loaded and installed from here now this is loaded we'll go ahead and hit reboot and we're gonna hit okay here some VMs cannot be migrated without first rebooting host please try and force reboot it will not let you reboot until you stop those VMs I wanted to point that out so let's go over here and find the VMs related to that pool so we're gonna filter for that pool the XCPNG pool we're just gonna go ahead and select these we're gonna hit stop and we'll go ahead and shut these VMs down then go back over to the host XCPN test and let's go ahead and reboot this and now the host is fully updated we can go over to patches and it tells us our host is up to date now let's talk about how VM migration works we can go back over here and we have all these different VMs and because if we went over here to settings and servers we added two different servers our demo one and the another host that we added you can actually migrate VMs between different resource pools so if we look at the two different pools we have the XCPNG YouTube demo and we have the YouTube demo pool we're gonna go here to our VMs and we can even do this live it doesn't have to be a off migration we can fire this one up right here all right now that you can see that this is up and running live I'm gonna go ahead and log into it so it's got something to do now we're gonna go ahead and click migrate there's only one host so it's the only one in the option here but if you've had other hosts and you notice that this is on a different pool because it doesn't care it doesn't have to be part of the same resource pool to do a live migration and it will figure out that we called ETH0 the same on each of these but we could line up different networks if there were more available that these are the only ones it sees and then what is the network on the destination where I'm gonna use ETH0 because I have that set up on that second host as well so we'll just leave this all the same and we're gonna click okay and this will actually kick off a live migration between two completely different hosts off of two different resource pools this will take a few minutes because these are as I mentioned earlier not super high-performance systems but it will do live migrations or non-live migrations between any two hosts provided so an orchestra is able to talk to both of them and provided there's a network connection that both of these are able to talk over to negotiate and move the data across. All right and now that VM has been moved and now it's on the other server but it's called XTPNG server one and you can see it's still running and working perfectly fine. Now let's take the time to talk about resource pools. Now resource pools are called clusters in other hypervisor platforms and they're very similar. They're a way we can take multiple hosts and put them into a single pool to make things easier such as shared storage or as we define networks always by the pool even when there's only one host in there this will actually allow you to define the pool network and when you add a network it'll add it to all the hosts in that resource pool. Now resource pools are very similar to what other hypervisor platforms call clusters. They work in much the same way with Zen server kind of being an exception the fact that it has no problem migrating a VM between unlike resource pools provided as I said the process is the same you can even do it live but even if they're not the same hey if you shut down the VM you can migrate them without combining them which may ask do I even need a resource pool and in some cases you do not but if you plan to do things like high availability or want to use a shared storage between multiple hosts you're going to want to combine them in a resource pool. Now this represents how things are set up right now we have two hosts and each one even though they're only one they technically are in a pool and they're both being able to talk to this one instance of Zen Orchestra. The next step we're going to do is combine these hosts into a single resource pool. So by combining them into a pool they're still going to have their individual local storage but they're now going to have a common network setup and they're going to have a common setup when we add things like shared storage. We can add shared storage once and that one time shared storage add as a two-pole system provided we've set the networking up so each one of the hosts can talk to that shared storage and this also makes VM transfers really easy. Now if we want to go full HA that would require a minimum of three hosts we're not going to do that in this particular video but I just wanted to give you the idea of how you would add these. Now one thing that is really important here is we have to pick which host we're going to join to the other because they're always in a pool it's a matter of someone's going to have to lose their pool. In order to do that a prerequisite that has to be met is we have to take and make sure there's no virtual machines and no configuration on the incoming one. So we're going to pick which host is going to be the pool we bring into and then the other one we're going to delete all the virtual machines and make sure there's nothing on there. Generally when you're adding these to a resource pool you are going to have one that's already set up and you're going to then load bring it into that resource pool so usually deleting VMs isn't something that's needed. By the way though if you have two separate systems and you decide later you want to combine them as I noted earlier without them being in the same resource pool you can still migrate all those VMs you could just migrate them off of the pool into the pool you're planning on bringing them into from a planning standpoint that works perfectly fine. Now one more thing I'll comment on a resource pool is there's always has to be a master. Now the master is where the source of truth is for all the metadata that is related to all the VMs and network settings, et cetera but that data is replicated onto all of the other ones in that same resource pool. So you have one set to master but then all of them get a replicated copy of the data and in an HA setup the master if it were to fail it will automatically choose a new master within the pool. As I mentioned though that requires three or more. So what happens in a situation when you only have two or even 10 in there and there's no set HA quorum for them? Well, you can still force a host that was not master to become master because it has all the data it needs it just doesn't have the rights to be master and there is a way you can go into any of them in an emergency situation if you were to lose that master host you do this by going to the council and set new master and it will echo out to the other ones and you'll be able to still manage that particular resource pool. This is something else I wanna note about the way Zen Orchestra works and you'll see this when we do this we have two servers under the server settings once we combine these you're only gonna see one because the Zen API only ever takes its cues or talks to the master and that data is as I said echoed across all the other ones. Now there are some limitations to having a resource pool as I said you can have as few as one but the maximum really is 64. Once you go over 64 I believe you can set it to maybe work with more than 64 hosts but that's when things can get outside in the recommendations. So no more than 64 is the general rule here in February of 2024 for any resource pool. And we have our two hosts are set up under servers and we're gonna choose YouTube demo pool as the incoming pool and XCPNG demo that we've been working with as the one we're going to merge into this one here. First step is going to the hosts and I wanna make sure that I see that they're on the same version. So XCPNG server one, this is XCPNG test and we'll just rename it too. So we have some consistency. This is the one we're going to be bringing into the other system. I wanna make sure that all the patches are loaded that's around the same version. So this is version 8.3. We go to the other host. This is version 8.3. And as I noted both have all the patches loaded. The next thing we wanna do is make sure that there are no VMs. So if we choose the pool, let me look at the demo pool here. We don't see any VMs. I've already deleted all of them. And we should also note that if I look in the pool, look at the demo and look at the networks, I should have no extra VLANs or any other network settings other than the base config of eth1 eth0, just the two default networks. I'm not worried about the names that have been given that are gonna assume the names of the incoming pool. So let's go to the pool, go to the YouTube demo pool and let's go ahead and add a host. It's the only host in here. So if anyone shows up in a pull down and we're just gonna click okay. And now we have the pools combined. It lets you know right at the bottom here, which one is master. Changing which one's master is actually really easy. We can go here to advanced and you just hit the pull down. Before I do that, I wanna show something. If we go over here to our VMs, I have a VM running. If we can go to the console and you can see it running here. And we have this as the master server. As noted, when we go over to pools, go to the YouTube demo pool. It says it's master here. But if we wanted to switch which one's master, not a big deal. We can say server one or we can switch it to be this XCPNG two. Now let's look at something before we do this. We go to here to settings and then servers. You only see one server, even though we have two different systems in here. That is because Zen Orcashire only ever talks to whichever server is the master. If you were to try to add another server in here that was already part of a resource pool that was joined, it would give an error that it's already joined in part of a resource pool. So it's always gonna only have one server listed in here for each resource pool that you're attaching to. So when we bring in another resource pool, for example, we only need one of the hosts that are in that resource pool. Let's go back over here though and show the VM running that I have right here and show what happens when we do a switch. So if we go to the pool itself, go to advanced and switch from server one to server two. It says this operation may take several minutes. Do you wanna continue? This is fine, it will take several minutes, but we will lose access momentarily to Zen Orchestra. But the VMs themselves are going to continue in the state that they left off then. So it's preparing in the background to do this and all of a sudden it goes blank. Once it goes blank, you just have to sit patiently and wait. And after about 30 seconds, maybe 60 seconds, depending on the speed of the systems on there, you will see this automatically reconnect and you'll be exactly where you left off. All the virtual machines have continued to run. Everything in the background was functioning fine. You just lose access to make any changes because it locks them all while it resynchronizes and chooses the new master. All right, now the systems come back. And if we go and look at the YouTube demo pool, go to advanced, we can see that this server is now the master. Something of note, if we go over here to the hosts and just for consistency, this one's called XCPNG server one and this one's called XCMD two. Let's rename it. Renaming it will not break anything. So we can call this one server two, just so we have some consistencies because it's all being communicated by IP, changing the name of the servers, not going to break or change anything. It's dynamic and can easily be renamed on the fly like this. Now let's talk about the networking really quick here. We have our YouTube demo and we have the network. Whatever we define here, as I have this network defined, it will automatically propagate to all of the hosts in this pool. One of the advantages of the resource pool that we have this one called not in use, I can just change it to do not use. And now if we go to our hosts, we can see under networking, it says do not use. This is also going to be in the other one too. So if we go back, same thing says do not use. Also, each host still maintains their own IP address as they were set. I have them set via DHCP, but if you want to adjust the IP address of any particular host, you can even set a dysthetic right here or go into the XO console by SSH again and override and change which one is set to management. We'll cancel out of this and we can look at the other server network and we can see that this one right here, same thing I set it to DHCP, but if we want to set it to static or set a different IP address, you still set those individually. Also, as you add networks, like this one has this VLAN 10, you can assign static addresses if you need something for the host to be in that network. It's not always needed, but under certain use cases, for example, if you're adding a storage network, that would be where you would set it up, is you would add the network to the pool and then you would set the IP addresses to the servers. Now let's talk about storage. A lot of people have been asking about XoSAN, their hyperconverge, V2 will be out soon. Once it comes out, I will do a full review of it, but you can go into forums and check out a preview of it. Today, we're just going to focus on NFS. iSCSI, of course, is an option. So if you have a NAS that only supports iSCSI, it will work, but please note iSCSI is not thin provision, NFS is. NFS is generally my preferred storage when I'm tying either a single host or, in this case, two hosts in a resource pool to a NAS. This particular NAS we're tying to is TrueNAS, but it will work with others. NFS just works well because it is thin provision. It is file-based, it is easy to manage. So let's go ahead and set that up. To set up new storage, we go to new and then storage. Then we select the host. We can choose server one or server two. This doesn't matter if it's a shared storage. We'll call this one TrueNAS NFS. We're going to go ahead and select the NFS as an option. The server would matter if we were choosing, for example, a local one, because it would be looking for something local on that one. But because this is shared, either server we choose in this resource pool is going to connect to both servers, provided they have the networking setup so both servers can talk to it. And we're going to put in the IP address. We're going to query it. And that will let us then select the path. I have my NFS storage right here. Default NFS version is fine, unless you have custom reasons or need some custom pathing and custom options. But I'm not worried about any of that. We're just going to leave it all default and then go ahead and hit create. Now it's created that storage repository and we can even take and migrate a VM to it. So let's go ahead and take a VM and we don't even have to shut this down. This one's currently running on local storage here. We can go ahead and migrate this to the TrueNAS NFS. So hit okay and it'll kick off the migration process and a task. It will not shut down the VM while it does this. It's just takes some time to migrate from the local storage over to the NFS storage. And it's going to go as fast as well. Your server connections are. All right, now that the storage for our virtual machine has been moved, let's go back over to VMs. We can click on it here and we can see that it's showing the TrueNAS NFS is where it's living now. Let's go ahead and look at this particular storage. We only have the one disk in it, but we do have the Delta that was attached to that disk in here as well. So you see the disk, you see the Delta attached to it. And if we click on it, it brings us back to our VM and let's say we go ahead and create some more snapshots and create another one. Let me go back over to the disk. You'll see those as well. So each one of these is on here. Now they don't each take 80 gigs because of the thin provisioning. Hence the reason I said NFS is a good way to go. But let's go ahead and look at the advanced and talk about what some of these do. We have a reclaimed freed space, but that doesn't matter under NFS and it won't do anything because that's not needed for this particular resource type. The buttons I really want to talk about are remove SR which kind of as it may sound, we'll delete this disk, not something we want to do but that's how you remove a storage repository but enable maintenance mode. I really like this feature if you have a lot of VMs on this particular server and you'd want to, for example, update your TrueNAS you can take this and hit maintenance mode. And we're going to go ahead and hit okay with this, puts this in maintenance mode which is going to shut down all the VMs that are running on it. There's only one right now. So this VM was running and if we try to start it, it lets me know no host available because it is in maintenance mode. So let's go back over to our storage, our TrueNAS NFS and once I'm done with maintenance mode we can go back over to advanced and we can say disable maintenance mode. This may sound trivial but when you have as many as 100 VMs running on here this can be kind of a tedious problem to solve to say how many VMs are on here and I need them to stop so I can update my TrueNAS. Another option as I noted here though is it does not require you shut down the VM to move these to another storage. So if you have multiple NASs and you want to do maintenance you can just migrate them over to another one live and then bring them back after you've done maintenance. But I think maintenance mode works really well. It's a quick way to shut down all the VMs provide, do your updates on whatever your NAS is or whatever reason you had to put it maintenance mode and once you take it out of maintenance mode it will restart those VMs and you're back up and running. Now this VM is running on XCP and G server one. What if we wanted to migrate this to server two but we don't have to migrate the storage now because when we added this storage because it was a shared type storage it was automatically added to both of these. So we can go here and it's the only other option is send it to this server, server two and we just hit okay. Now when we do this I want to note that it's going to go substantially faster because it only has to take the memory that is running and move it over. It does not have to actually move the storage. So now this VM will move quite fast over to the other host. Now the VMs moved over, go back over to virtual machine and you can see it's now on XCP and G server two and the storage is exactly where it was still on the TrueNAS NFS storage. Now if you want to create a new VM it's pretty simple to create them on that storage when we go to create new we'll select the pool, select any template and then for the storage you can go down here and select where you want this to land and we can choose TrueNAS NFS. So it works much the same way if you want to create anything new on that particular storage. I'll also note if you want this to be the default storage for any new VMs that are created you can go over here to home then storage and we can go here and we choose the set as default. So now this is the default storage. So anything new automatically will be created here. If you want to change which the default is we could change it to any one of the other ones in here but we'll leave it right here as the TrueNAS NFS as the default storage repository. So anytime we go back and create a new VM now it's automatically gonna have this one selected right here. One more thing to note when we're looking at the storage and we can go here to the TrueNAS and you want to see which host is connected to it lets you know the path that these are connected to on each of the hosts. If there's ever a problem there is an option to go ahead and connect all hosts we can click that now and it's just going to in case there was a problem or one of the servers was down and you see that these are disconnected it will force reconnect these to all the hosts. Now let's talk about backups. We're gonna go back over to the home and I want to clean up some things I have on this VM I just don't need all these snapshots let's go ahead and purge them and you can see under backups there are no backups attached to this particular VM and if we go to backups and new we can go ahead and create one but before we go here I want to point out that if we go under settings and remotes, remote are the term for where you want to store your backups. I've got this attached to my TrueNAS already I've actually got a lot of other backups that are in here and we can verify that it's working by clicking on the little test to remote it'll tell me the speed at which it's able to talk to this particular remote. These as I said are not particularly fast machines but it'll get the job done for what we need. Now we're going to go ahead and go back to the backups and we want to create a new backup. There's a couple different options in here. First, we have the option for XoConfig pool metadata backup. This will actually backup as it says pool metadata and XoConfig. So let's go ahead and make one of those and then we're going to choose the target remote and I'm not worried if I wanted to I could set it up so it always sends me an email or only on failure or never notifies me but that is in the plugin settings if you set up the plugin for setting email and put a mail server in there we need to select the pool that we're going to do this to. This backup job and go ahead and set a schedule as well we'll say daily and we can pick some time that we want. So let's retain three copies of each of these and whatever time works for you we're just going to head and click okay gives you a preview that this will happen daily at 12 a.m. If we wanted to change that we can then say okay we want it to happen at 2 p.m. So go ahead and hit okay. You want to enable this if you want it to run automatically I'm not worried about that we're just going to head and hit create. Now the importance of a pool backup is if you lose the data in the pool for all the metadata for the network settings you have and all of those other configurations they're all stored right here including the settings for XO. Now you can actually export all the XO config right here as well so we go to settings XO config you have the ability to import a new config or export the old config right here but it's nice that they have a backup option that will let you do that and you can see how quickly this runs we'll run it again just to show you it runs really fast and now we have the ability to have a copy of the entire metadata and the backup. Now as far as restore goes for this we'll go ahead and hit restore and choose metadata and there is the restore option for this particular metadata we'd select which version we want and we could just push that back and it does break them out. This is the XO backup this is the pool data backup. Now let's go ahead and create a new backup again and via mirror backup I'll mention is the way you can take two backup repositories and mirror them they got some documentation on it this is a pretty cool feature we're not gonna spend much time on that it's kind of self-explanatory that if you have one backup repository you like to mirror to another you can do that this is just mirroring the backup repositories maybe you have one onsite and you like to mirror it offsite or just to a secondary location you would just select the different remotes but because I don't have two selected I'll cover that in a later video talking about how that feature works but as I said it's relatively simple. Let's go to new and let's do a virtual machine backup and we'll call this one demo deb 12 backup and we have a lot of different options here we've got a normal backup which is just a full backup of the VM or we can do a delta backup which is a full backup and then deltas afterwards essentially incremental both of these backup types land on one of these remotes as they're referred to but it's an external storage. These ones though such as disaster recovery you actually send them to another system so we can actually have it backing up to the local storage or even another pool if we had more than one pool in here to another host. This is nice if you wanna create a backup that is on another machine that you can just click the start button without restoring and run it at that moment when you wanna recover that VM. Same thing goes for doing continuous replication it will replicate this VM onto another host. These are kind of unique ones that you may or may not want depending on your needs. Most of the time people are looking for the full backup or a delta backup and the delta backup is going to as I said be a full first and an incremental afterwards. So let's go ahead and select that debbing 12 VM and you may have noticed here you have the ability to backup more than one system at the same time under one backup job. There's also what they refer to as smart modes the tags that we mentioned earlier you can select things based on their tag. So you can choose certain tags and then let those tags such as Tom drive the backup. So everything you tag is Tom would then be put into the backup. This is good for people who are building VMs and then they wanna put them into production you can set a production tag and then anytime you set that production tag it'll automatically without going back to the backup backup all of those VMs. But we're just gonna keep it simple here we're just gonna back up this one and we'll set a schedule and I'm gonna call the schedule just keep three daily copies backup retention of three and this is the health check option. This will actually automatically boot that VM and verify that it works. If you'd like it to do that I've got a whole video where I dive deep into exactly how that works. It's a wonderful feature to validate that your backups are working. We'll set these and we'll just say every day but maybe we wanna be weekday. So we'll just say every day like this or you can choose the days that you want the backups to run but frequently this may be more popular because people may want the incrementals every day or maybe they don't need them on Sundays because well no data is being transferred. Whichever works for you we'll just leave it at every day selected only one of the hours is selected but we could back it up every hour. Delta's actually happened really fast kind of depends on your risk tolerance for how much data you want on there. They also give you a little slider to make it easier if you wanna stagger them around for when the backups are going for ever so many hours but we'll just keep it simple here as I said and do a daily backup and we're gonna go ahead and hit okay. Now I'll turn this off enables I don't wanna accidentally run but it defaults to enable. This is also a one to many relationship you can have this schedule and then you can add a second schedule to this same backup or even a third and fourth. So if we do this, click okay, do this and click okay you can see that we can have multiple backup schedules with different parameters in them for how we want things backed up. So we'll go ahead and stop those we only need this one here for this demo. Select the remotes we've only selected one but you can actually send it to two remotes at the same time that is supported under advanced settings. We have a few more options. We have the report settings so we want it to notify always, failure or never and we can set specific email recipients. It does require that the email plug in be set up. Concurrency, how many do we want running at once? I'm gonna leave this blank but you can specify depending on how fast your server is maybe you don't want them all to run at once cause you have multiple jobs this will set the concurrency of them especially important when you're backing up a lot of VMs at once we're only backing them one so it doesn't matter. If this takes more than an hour I think it should fail and let me know because it must be stuck these generally go really fast. Full backup interval. This is how frequently you'd like it to run a full if not it'll continuously run deltas but you can set this to something like 20 this links to the documentation that suggests us be a good setting but the every 20 times it reinitializes your delta to start from the beginning and then you always have that full backup there as opposed to the chain of deltas. It's just an extra assurance I haven't run into a problem but it is nice that they added this in here. Use MBD protocol to transfer if available we're gonna turn this on and I'll show you where that has to be turned on number of MBD connections per disk we'll leave it at one you can rate limit these if you want and finally we have normal with memory or offline you can do these snapshots just like I showed earlier with memory but I like offline cause I wanna stop the VM I don't have to stop the VM but it is nice cause this will stop the VM as quick as possible go through the shutdown grab the snapshot and put the VM back up completely optional to do this we can leave it in either mode but we'll go ahead and leave it right here in offline mode to show you how it works now the MBD let's go over here to our pool and our network and we're gonna choose MBD connection on E0 I know E0 is where this is attached to so this is where the backup's going to occur because this is where Zen Orchestra talks to these servers this is for the network block device transfer this is an enhanced way to allow for faster speeds on deltas so we go ahead and turn it on let's go over our backup and we can run the backup from here this is our demo W12 backup I don't want to accidentally run so I'd turn that back off there's no successful here cause it's never done it lets me know it's three daily copies I can click this to see the matching VMs I can just run the backup job once here I can click to see the matching VMs we can also while we're looking at the VMs here because it's attached to this or any VM it's attached to it also now shows up under the backup tab let's go ahead and just kick this off and run it and click okay it's shutting down the VM which happens rather fast now it's booting back up now the snapshot is exporting over to the remote so it's doing the backup now now the backup's complete so we can go back over to our overview and we can see that it transferred 3.72 gigs which is the size of that VM took about two minutes to do if we click on it it gives us the details of the transferred data started using MBD it did the snapshot then did the transfer the duration was two minutes get the transfer tells you the speed and then the type and then full so we're all set and there's only one VM here so there's no other information available for this particular VM and we know the backup successful now we can go back over to this VM and look at it and we see a snapshot created when you do the deltas it leaves a snapshot there so this snapshot will be updated every time this delta backup is this is actually a nice snapshot point that we can revert the VM to the VM is currently up and running let's log into it let's do a quick apt-get update and apt-get upgrade there's nothing changed but we made a few changes on it so let's go ahead and run the backup again but I wanna show you the differential now because there's not much change so this will go even faster the second time so we're going ahead and run that same backup we go over here to tasks it's going through the shutdown process and now it's back up and running and transferring that data back over and completed I did this all in real time without speeding it up so if we go back over to our backup and overview there was only a 176 megabytes of change so the whole process took 16 seconds so from snapshot, shutdown, transfer, backup and system being started back up and it transferred that little bit of data only 16 seconds passed so the backups being natively integrated with ZenOrchishra are really nice because they work so well and so fast because this system knows exactly only what blocks have changed since we last booted that system of course the more data in the system the longer it's gonna take to run the Delta because there's a difference between the last backup but you can see the efficiency of it running this way now as far as restores, go to here to restore there is our Debian 12 system if we wanted to restore it we could just go here, select the version we want select where we want it to land I can leave the current VM there and we can even say let's send it to the local storage of server one do we wanna start the VM? Do we wanna generate a new MAC address? Do we wanna use a differential restore? Differential restore is interesting because we can actually restore it differential if it was in the same repository where it can take the blocks that are already there and just figure out what the differences are without having to do a full restore if you're restoring larger VMs into the same place this can be a great way to do it if you wanted one of the older versions but let's just walk through the restore process and we're gonna generate a new MAC address that way it'll get a new IP address because it'll technically be like a new VM although they will have the same name and we're gonna send it to local storage but we could just as easily send it to any one of these so hit okay and we can look at the tasks while it completes now the virtual machine's been restored so we can go over here to VMs and you'll see that it has the date that we did the restore in it and there's going to be a tag that says restored from backup so we know that this one restored but if we go ahead and start it it'll work perfectly fine and because I did tell to change the MAC address it won't have a conflict where it tries to get the same IP because it has a new MAC by the way if you ever want to change your MAC address just click and you can type a new one in now we can look at the console this one's booting up and if we're gonna look at the restore logs we can go to backup and then restore and it has the details right here that this was restored it took about three minutes and 26 seconds to complete the restore and if we wanted to see a raw log they have an option right here to give you more details now some of you may have noticed this little health button over here this lets you know when there are detached backups there are backups that don't have jobs listed to them it's pretty simple health report and it'll say well for some reason there's a detached VM snapshot that's located here that doesn't seem to have a job related to it you can remove the snapshot it will delete the snapshot from that particular VM so there's a snapshot here but there's no backup so we can go back over to the backup and health and if we wanna delete it we can cause there's not a job related to it but do note it is deleting that snapshot so we'll go ahead and get rid of that and now the health is back to no warnings now while this video is enough to take you from loading Zen to getting a VM setup I do have several videos as I mentioned that are all linked down below and also in my forum post of deeper dives into specific topics also take the time to read through the documentation they have quite a bit of it they have a lot of visuals and graphics there to help explain things and the documentation is constantly being updated as the project progresses there's always new features they do a monthly run of all the new features and you can check out their blog posts to see this over time they've been doing this for years it's actually really cool to see the development and stay up with the once a month announcements for all the new features that they add each month both to XCPNG and Zen Orchestra and of course I'll be doing an updated video when the Zen Orchestra 6 comes out along with XO Lite because that will be a facelift as well as I mentioned the XO Store when that comes into full production the Version 2 that will be their hyper-converged platform and I'll be talking about that as well and I'll keep those videos up to date in this particular video so you're always able to find them and I have a playlist of all of my XCPNG and Zen Orchestra videos that's also linked down there so if you have questions, comments or concerns leave them in the comments down below or head over to my forums for more in-depth discussion like and subscribe to see more content from my channel head over to the forums that are by XCPNG as well the VATES team is very active in the forums and I highly recommend participating with them and participating in development and engaging with them this is one of the things I really like about the whole team is they're very engaged with the community and building it out to work for people's different scenarios all right and thanks