 In this video, I'm going to cover some tips and tricks for Zen Orchestra using tags, smart reboots versus no more reboots, because there's a couple of different options there, but they're all there to make your life easier for adminning these systems. And a couple of emergency situations you may run into where you have to go to the command line. The Zen admin command line is actually really powerful. It'll let you admin the entire system, but I want to cover some very specific use cases and you'll find link down below all those exacting commands that I have, because what if you lost a host that was set to master in a pool and suddenly you can't communicate with the pool because, well, they all take their cues from the master, but don't worry. It's not that hard to do. It's even easier to do when you're not doing it under duress. So I suggest testing this out in your lab like I do to show you how to do it. It's not that bad. And the commands, as I said, are down below if you just want to be able to copy and paste them because you are in duress right now and you found this video as the answer to that question. So let's get started. Are you an individual or forward thinking company looking for expert assistance with network engineering storage or virtualization projects? Perhaps you're an internal IT team seeking help to practically manage, monitor or secure your systems. We offer comprehensive consulting services tailored to meet your specific project needs. Whether you require fully managed or co-managed IT services, our experienced team is ready to step in and help. We specialize in supporting businesses that need IT administration or IT team seeking an extra layer of support to enhance their operations. To learn more about any of our services, head over to our website and fill out the hire us form at lorenzsystems.com. Let us start crafting the perfect IT solution for you. If you want to show some extra love for our channel, check out our swag store and affiliate links down below that will lead you to discounts and deals for products and services we've discussed on this channel. With the ad read out of the way, let's get you back to the content that you really came here for. Now the first thing I'd like to point out when I start these videos using ZenOrchishra is what version I'm using. So you can see the commit I'm using, the ZenOrchishra DCDF2 commit, master commit 214B8. I am a few commits behind because, well, I didn't take the time to update it prior to this video. So yes, this is the self-compiled version, which means everything I'm showing you here is completely doable in your home lab by using the self-compiled version. If you're a business and I like to use this in production, I do, as it suggests right here, suggest that you use a production version of this and pay to support the project. And they have a link right here. If you're interested or you can go over to the VATES and they have some more options available. Now let's get started because the first thing I'm going to talk about is UUIDs. These are used throughout ZenServer and exposed within ZenOrchishra. This is what allows me to do things like change the name of this is a new name. And we left that in there, but all the backups and everything attached to it haven't changed at all. Matter of fact, if we go to edit one of these backups, you can see it says the new name is Bitwarden because it's doing everything by UUID. And we'll go back over here or VM. And right after you do something like accidentally type something, you can click that and it'll revert back that way. If you screw up while you're doing this, you can just quickly revert back to that last known good name. Now these UUIDs, you'll see this one right here for this particular virtual machine. It makes this so much more human readable and is what allows us to have all these different tags, names, and dynamic changes of them with the underlying OS completely understanding still what we mean. This is just a really handy admin tool to be able to do this and it persists throughout. For example, if we look at the host, every host is going to have its unique UUID when it's generated. Every VM, as I said, has a unique UUID when it was generated and even objects within here, such as storage, it's going to have a unique ID. This also allows us to rename things like storage while, and I'll point out these are all attached and running. But if we rename this and put something in front of it, it really doesn't matter because as long as the UUIDs are the same, this is what allows these to stay running and know exactly where these are. So there's not a risk of breaking something. But of course, we, when we have lots of systems and the idea of Zen Orchestra and the concept behind it is scalability. We want to be able to filter and be able to see these at scale, change them and possibly rename them to make them more human readable, which is of course, what's really important here. Now let's get on to tagging because this is an extension of that, making it more usable for people. If we go over here to our VMs, I don't have that many in this particular system. Some of the consulting clients we have have thousands of them. And these are ways you can then manage those servers is with tags. Let's talk about what those tags can be viewed. And this is going to be a little bit messy, but you'll get an idea and see some of the tags that are across these. You just click this little expand right here. But let's get more drilled down and show you what a tag does. So we can filter for tag production. And we'll just clear this out. And you can see up here we have the tag production. I've actually got a few of these saved. I've got Tom's projects, YouTube, none, just show me everything, or we can go through and go back over to like production VMs. And these are all tagged with production. And the tags can be changed dynamically. You may have noticed I labeled this one red, and this one blue. Blue is the default. They actually added an option more recently that allows you to change the colors of these. We can click here and then click advanced tag creation. And we can look at the existing tags and actually change the color of them. So if I clicked on this production one here, we can add a color, pick the color we'd like, hit okay. And now the production has been changed to purple. We go here, click this little advanced again, pull one of these tags up such as the production one, and we can simply remove the color and it goes back to fault. I like the blue default. And I do like that the critical is red, which was easy enough to set as I showed right there. And if I want to add another tag, now this can also have the time tag. If we go back over here to home filters, we filter the town's projects. Now suddenly this one has listed in my projects because I added that tag. And we'll go ahead and remove that tag. We go back home, switch to town's projects. And now that's not in the list. Now what if we wanted to do double tags? So we don't just see towns projects with me another tag, you can actually combine them together, maybe we'll put Tom in critical. And you can see how you can just have a regex up here, but you don't have to know regex, it will build it automatically by selecting these. So now we can have this one, and these series of tags that I've grabbed. Actually, let's remove a few of them. Because maybe we only want Tom and Eric's projects together in one tag. And we want to save it save it as Tom and Eric. So now we have one more filter save. So if I go here, there's my lab, there's my production, or there's my Tom and Eric. Now how do you set these as default is actually really easy to do as well. So instead of every time you come here, it goes to power state running, maybe you want one of those defaults to be such as Tom. Well, we go down here to the bottom it says edit settings for Tom, scroll down here. And we'll set that to Tom's projects to be the default. So now we can go back over here to home, and it defaults to Tom's projects, we still have all the other ones we saved. But we can switch to that one. If we wanted to delete that one, for example, I probably don't need the one that says Tom and Eric that we have right here. So we can just hit delete, and it'll take it out of the list. This also allows you to create default filters for the same hosts, templates, or storage filters, so on and so forth, when you're going through and creating these. This is actually a really nice feature to customize if you go to the same spot all the time. And this is why you'll frequently see it default to, for example, my YouTube demo that I have set up here for later in the video, I actually will go and set this, and I'll choose YouTube. That way, every time I click back on home, it's exactly where I want to be to continue with the demo. Now there's two more interesting places that you can use tags. One of them is in backup. So if we go over here, and we look at our production VMs, and we look at things like Bitwarden, and we have the critical tag attached to them. And if we go over here to our backups overview, and I go for this backup I called critical backup of systems, we go to edit, you'll see that I'm pulling by VM tag, and it will list the matching virtual machines, and I can click here to see all the machines that match. But this is also a really handy way to use tags, especially in large environments, where you build out the VMs, and then you add the tag, and then they automatically become part of a process or a backup job that you have running in the background. So you don't have to go and modify the backup job. The backup job will look for all VMs with a specific tag, and then automatically backup all of those VMs. I've covered this in my backup videos, but it's another use of tags tags are also available when you're creating new virtual machines. So if we were to go here, and we wanted to create some new virtual machines, such as a Debbie and Bookworm machine, let it be called Debbie and Bookworm, fill this out really basic here. But we get down here at the bottom, and we can auto tag these VMs, and maybe we want to call them Tom, so you want to make these part of a Tom project, or maybe I could also make these part of a YouTube video project. And then from here, we can go to our multiple VMs. And I will just have it called Tom, and we'll call it eight here. And now it will build these eight VMs pre tagged with this. So now I can then group them together. This is another way that you can build templates. Now I'm building this not off of a safe template, but there's ways you could do this off of safe templates that are already built from VMs, a little out of scope of this video, but the same rules apply where you can have this create and auto tag all of them automatically as they're being built. Now let's talk quickly about job scheduling. This is actually a really neat feature that you may or may not have a use case for, but it's actually really clever. You can actually go here and create a job, some job to do a thing. And there's a list of different commands that you can have that does include maybe you want a snapshot, or you start and stop that runs, for example, choose a host and choose a VM. And then say we want this VM to start on a specific schedule. So we'll say, so actually we'll call this start VM, go ahead and save. And why not also have one called stop lab VM. So if we said stop lab VM, we'll set VM stop. Be that same one save does want us to give some seconds, if it doesn't stop within let's say 20 seconds, we want to go ahead and force that shutdown. So now we have a VM start and stop job, we actually already had one here called shut down the wiki and we can play these and they will run the job now. But then we can actually go here to scheduling, select one of the jobs such as start the lab VM and create a schedule for it or we can stop it at a certain time. These are actually really handy to create automations that may start, stop or even if you looked in the list here, there's quite a few men there, things like migration can happen. So you can choose to migrate a VM at a certain time and migrate it back at another time. There's actually a lot of possibilities you can think of in here. One of the other interesting ones is the fact that you can revert a snapshot here. So if we VM revert that snapshot, and you choose a particular snapshot, and then you can take and have a VM that reverts automatically based on the schedule or just leave a job that will allow you to revert that VM, this VM, and we'll hit save. And we can now revert it anytime we run this job. Now there's even more automation where you can string several of these things together to create a whole workflow. But this is what the job system does. It's a great feature. And if you need to mass delete all the jobs, because I don't need any of them except for this shutdown wiki, we're just going to hit delete. And they go away. Now maintenance thing you may run into is over here under the hosts, we'll go to this rise and elaborate. And we see we have one, two, three, four virtual machines running on that host. These are the virtual machines running on there. But we want to, for example, restart it maybe because we loaded some updates or some patches. This is where smart reboot comes in. This is a feature instead of having to reboot this host, we can smart reboot it when all that means is suspend the resident VMs, reboot the host and resume the VMs automatically really simple. If you don't want to deal with actually stopping all those VMs and restarting it, it just puts them all in suspended state. So they'll come back after the system reboots back to that state. And you don't have to auto power them back on pretty handy feature. Now another maintenance function you may have to do is over here on storage. And right now I have filtered for production. Let's look at the lab storage and go ahead and actually do a demo of this. And we can go here's my Trinity NFS lab. And here's all the things that are running on it. And I can see the green ones are running the red ones are not. But what if I wanted to reboot this server and it happens to be a true NAS server and I need to update it or do some type of maintenance on it. Well, I don't really want to evacuate the hosts and move them over. I could just stop all the hosts from running on there and restart them or I can just click enable maintenance mode and it will do it for me. It will automatically find the running VMs that are on this particular storage repository. And if we hit okay, it's going to go ahead and bring those all to shut down. So we go here to task, it's shutting down all of those virtual machines. Now that they're all stopped, if I needed to perform any maintenance, I can go back over here to storage. I'll switch it back over to lab and you can see it lets me know that this is in maintenance mode. And if I were to start or try to start any of the VMs on there, if we go to this one here, for example, it lets me know no host available because it's in maintenance mode. If I look at the air log, but we can go back over here to the disk, go back over to advanced and we want to disable maintenance mode. Now it's going to reverse that task and those VMs are already starting back up and booting because we just took them out of maintenance mode. It'll bring all the ones that were in a running state back to that start running state as they were before we put in a maintenance mode. All right, now we're going to SSH into this particular server that's called Ryzen Labbert. And we're going to mess with this Debian 12 VM. I'm going to show you how to start and stop things from the command line. So as you can see, the system is in the off state. So let's go ahead and XE VM dash start and we can do it by its UID. So you give it the UID and we're going to say B6 and I'll just hit tab for autocomplete. You can use tab autocomplete to complete a lot of these VM commands all from the command line. Now if you notice this UID right here, this B68, et cetera, is the same as the UID up here. And of course, this makes it easy if I wanted to copy and paste these. Now we can see that this VM fully started up and is working. So let's actually go back and shut it back down. And for that XE VM dash shut down UID equals B6. And we'll just have autocomplete is the only one that begins with B6. And you can see in the background it shutting that one down. Worth noting XE VM start. And we can also do this by VM name. So we can say VM equals Dev 12 lab, it'll put slashes in for the escapes. And now we're starting that VM back up. This is the way, for example, if you had Zen Orcasha for some reason did not start, but you can get into the command line of the host, you can go in here and restart things like your Zen Orcasha. Matter of fact, if you put VM equals tab, tab, it will list out all the VMs available for you to put in there and you can fill them out and autocomplete your way through figuring out which one you want to start, or you can go XE VM dash list, like more, and it will list them all out one by one with their full name and just UID. So you can simply copy and paste and start the one that you want to start. All right, now let's talk how to deal with host failure when you have a pool of XCPNG hosts. I want to point out that these are all virtualized. And you do that by setting up nested virtualization under advanced. So yes, these are running XCPNG nested inside of XCPNG. This makes the demo really easy to do. And I want to encourage people to set this up for themselves so they can go through testing what a host failure may look like. Now, doing this, we're going to go back over here and talk about these different three demo servers. It's lab one, lab two and lab three. Those correspond completely to this Zen Orcasha instance connected to these three servers. First thing I want to note is one for two is the master server as noted here. One of the things that's really important to know when you go here and we look at the different servers is that Zen Orcasha wants to communicate with the master server in the instance and we'll go back over here to these posts. This server is down. Even though the VMs may be running on these other servers, you will lose your ability to see them. Everything works off the master in Zen Orcasha. I've got this covered in more depth in my video explaining how XCPNG and Zen Orcasha work together. But essentially the master always has the main copy of the status of everything about the systems. And there's a copy that is also available on these two servers as well. So this one is where all changes actually occur, even though they may be working, such as start and stopping of VMs, changing a network settings, etc. There's a working copy on each one of these. But all changes are starting at the master and replicate out. That means though, if you lose the master, then you lose your ability to communicate. And therefore, you don't know the status of the VMs running. The VMs will continue to run on the other host. If the master were to die, no problem. Now, if there's an HA setup between these hosts in a pool, so you have all the hosts and they're all part of this one pool, there's the three hosts on here, it will automatically in an HA situation figure out who the new master is that does require a minimum of three hosts. Many clients only have two hosts, one primary and one that's going to be secondary, maybe because they have it for backup reasons, just so they have something available in terms of hardware or not. Everyone needs a setup that requires full HA. So let's demonstrate what happens when you lose the master and how you promote one of these other systems to becoming the master. We're going to go ahead and kill this one right here, the lab one. So let's go ahead and switch over to this lab one. And I'm not going to do a graceful shutdown, we're just going to do a forced shutdown of the system. So it's going to go completely off. And if we go back over here, things look like they're working. So you go here to hosts, they look like they're there, but in a moment, Zenor actually is going to realize it can't communicate and it's going to fail and all these will simply disappear. We can make this faster by hitting enable and try and enable it and it's going to fail on communication. Even if we try changing the IP address and just simply pointing it at another host, that won't work. That will not allow Zen Orchestra to communicate with this because it's only expecting to talk to a master in a pool, not any of the non masters in that particular pool. Now I will point out if we simply bring the master back online, Zen Orchestra will start communicating with it and these servers will also start communicating with it as well and everything will go completely back to normal in a failed situation. That's ideal. Just get the main master server back up and running. And once it's up and running, everything will just be working again. Any VMs running on the other servers will continue to be in the state that they were in and everything will sync back up. Hopefully, unless you've had something more go wrong, such as a master that you cannot recover. And now you have to promote one of the other servers to being the master. Let's walk you through how to do that. Now this server two is the one I'm going to pick and we're going to look at the console just so I can show you the IP address of 192 1683.144. So that's where we're going to SSH into and promote this one to master. So you SSH into XC PNG lab two, we're going to do XE emergency pool transition to master. In 10 seconds, this one will become the master. And there's one more thing we need to do because we have at least one other slave host that was in here. And we want that one to know that we are the new master. So that command is XC pool recover slaves. And that will do the same thing. It's going to let any of the other ones in this pool know that this is the new master and this is where you need to take your cues. Now, because the old master was 142 and the new one is at 144 will just change the IP address now that we've promoted it. And we'll go ahead and hit enable. And there's our hosts. This one now is the master. What happens though, if, and we'll go ahead and filter none. What happens if this one comes back? Well, good news. It will actually figure out that it belongs to this pool because it has not been ejected from the pool and we can fix it. But if this one was to never be online again, this is where you would want to get rid of this one in the pool, but and forget it and make sure it's no longer in the list. But let's say we recover that one. And then we want it back to being part of this pool because it probably has some information on it that we want. We fixed whatever went wrong with it. Just going to go ahead and start that lab one back up. Let it go through its boot process. And now that it booted up, it figured out it was no longer the master. So now it just becomes another member of the pool with this one being the master. I didn't have to issue any new commands. Once it realizes the pool it was part of, it will automatically join that pool again with whatever parameters and settings were applied to it. Now, switching masters is actually really easy. We can go over here to pools and maybe want to make that one to be master again, go to advanced and you can simply go here and choose who the new master is. We'll put that one back at lab one. Operation will take several minutes. You want to continue when you switch who the pool master is, it does not affect the running VMs, but doing this will temporarily stop you from being able to control all the hosts while they reconfigure and decide who the new master is based on what you selected here. Now let's talk about the XCPNG console that you see on the display of booted hosts and you can also get into it via SSH is what we're going to do here. And by typing, like it says at the top here, access console, it gives us the status, let's us know what the IP address is in the gateway, some basic information. And if we press enter, it will give us the SSL fingerprints for this system. Now if we go here to network and management interface, and this is probably your most common use for this is to change which management interface is set to the system. This may happen if you've changed out the network card or replugged it in lost communication with it. I need to go over to the machine and go through and configure the management interface or even do a quick network test to try to ping like the gateway address or ping a custom address. Let's go here to configure management interface. It lets me know the connected status of these or the unconnected status of these ones here. And then you can choose a new management interface. If I wanted to choose this one here, for example, I could do that would choose DCP or static, etc. And let me walk through the process of setting this up. But I don't want to do it while I'm on SSH because well, that will break my connection, but it's pretty self explanatory. As is the virtual machines are running in here, I showed earlier how to do it from the command line. But you can also see which VMs are running on this host, get the IP address of them, provide they have these end management tools on there, you can suspend, reboot, shut down, migrate, or force reboot for shut down. If you have this shut down with those VMs, or you can go to all VMs and it will show them as well. And we can go ahead and just give a demo real quick and we'll go ahead and shut this down. Press F8. And it will stop that virtual machine. Go back over to all the VMs and I can start it back up, start on this host because the only host available press F8. And it will go through the process of booting that virtual machine back up and let me know if it was successful. Now not included in this particular demo, but will be linked in that forum post that has all the commands I did use is the XE emergency reset of the pool master IP address, because there are occasions and I recommend setting all the IPs prior to putting all the VMs and building at your infrastructure, get all of your stuff set. But what if you've done that and now you need to change where the master is in terms of what its IP address is and how do you tell all the other hosts in the pool where it went? Well, there's a way to do that and that command is listed in there as well. So yes, there is a way to recover from that situation and you don't have to just reload or rebuild the pool and that would be messy, but it is a recoverable situation. Just it's not the most ideal situation to have to run around a host and inform them that their new IP address is this, but I get it. Sometimes things change and you have to redo the network after you set it up. And well, there's at least solutions around there. There's actually a ton of other resources I'll link to for command line options, because if you're wondering, can you manage Zen server without Zen orchestra? Well, the backups are all built in the Zen orchestra, but as far as manipulating VMs, moving things around, starting, stopping, snapshotting, et cetera, and all kinds of other advanced parameters. Yep, you can do that all from the command line. Yes, you can build it even into a script. You can build out some pretty elaborate functions that are all just controlled via series of commands. So it's actually really cool the way that works being able to SSH in something I didn't demo in this video was something I've definitely used a few times, especially migrating people off of version six of Zen server, like the old Citrix stuff was the ability to SSH over and pull files as in like the whole VM and send it to another machine without even tying Zen orchestra to it. Actually, I've uploaded stuff remotely where I built a VM for someone, then used SSH copy to send it across the internet over to their server. There's all kinds of fun little tricks you can do by piping things together inside of Zen server. Maybe I'll do if there's some demand and leave the comments down below or head over to my forums on this, some more advanced videos for some unusual situations that are running to Yoshi Google around, you'll find all kinds of documentation because that's where I learned these tips and tricks and admin things of people that came up with clever ways to solve problems. Like when someone has that, as I mentioned, is on a really old server, you need to bring them up to a new one and you can't get the Zen orchestra to communicate because the other servers so old that there's some bug and, well, there's ways to solve those headaches. Nonetheless, love hearing from you. Leave your thoughts and comments down below or, as I said, head over to my forums, forums.loren systems.com to have a more in-depth discussion about this and other topics. Like and subscribe to see more content from the channel. And I just really want to hear some of your unusual command line use cases that you've done for Zen and some of the fun hacky things that are going on. They are really stuff that I find fun, as I like a challenge. I like to solve some of these challenges that are not created by xCPNG or Zen server itself, but mostly by the weird things people do when they set these up and then you inherit these messes, which is the consulting side. That's where I get these experiences from. Not from the systems I set up properly, but from the thought process that led to some of the interesting things we find as this admins. And yeah, that's always fun topics. Thanks.