 Welcome to vlog Thursday number 374 tune ass sexy P&G home lab and tech talk Tech talk. There we go. Hey Greetings everyone and I see we have a few people in here We even have the folks over at net bird joining in if you didn't see My friend Christian lemba released a video on net bird is still on my to-do list. So if net birds watching yes, it's still something I'm working on it's I've had a few other priorities with some things But I have the net bird stuff that I got running in the background actually the server still up and running The one I built so it's doing kind of something It just isn't doing much right now until I get around to doing the video on it And I'll update it and all that fun stuff, but I just got back from Chicago. I met a few people there that Watch my videos, which is kind of cool. I Gotta grab something here No, I don't I don't have There it is. I was looking for my I have a glove To grab the thing I want to grab I got this. There we go Yes, I got I wanted to use gloves because I now I the moment I touched it I realized I should put gloves on and then I realized why they put gloves in it. This is a Award I got from the folks at Camtia now I don't know. Maybe at some point I'll do a video like a more in-depth discussion about this as a topic But there's more than one side to Camtia the side the majority of people obviously know with Camtia is going to be like your certification side and You know, I see the bearded IT dad and he is certainly someone who was going to talk a lot about, you know Good certifications to get that may include network plus and things like that among other certifications along your career path There's another side of Camtia that is more the B2B side where Camtia Pools in a lot of the industry experts and Has us all in a room together helping with projects and things specifically I was helping with the Camtia cyber security information sharing and analysis organization Or ISAO as they call it for short. So that's where I go in and influence things if you will The reality of the way Camtia works is really interesting as an organization, there's Ben times when I will completely admit I've But unhappy as many of us were with Camtia stance on things years ago such as the right to repair if you may have noticed that has changed and the reason That changed is because people like myself and Camtia is a place that people like myself Learned we can all join and influence the decisions that are made and matter of fact We can work together with them as an organization to make things happen. So it's really interesting Working within the organization of Camtia along with many many people There's a lot of people that show up at these business events And you know, we talked about things like how do we move the bar forward for businesses? There's going to be more news you're going to hearing from Camtia on things like trust mark Basically what the the side of the Camtia. I'm on is not the you know Security plus or a plus or certification side, but we are working together Towards how do we? Validate as an industry like that a company such as mine, you know that it knows what it's doing So if you're a Consumer, how do we make sure that these companies are the right one to buy from so to speak, you know If you want to have an IT company out there You know that you know that the IT company may have individually certified people But what about the overall process that IT company has can we validate that they're using? proper security methods proper processes in order to secure their network to protect you and that's where some It's some bigger picture stuff and yeah, so The oh is Ray or Jason Ray and Jason are there on boards as well. So Ray OIT Ray is Many people if you're in the IT MSP space, you may know him as Ray or Cines his name, but also a lot of people has come OIT Ray That but he's there. So is let me share this screen. I was also there with Jason Slagle and Matt Lee Matt Lee is also on there as well. So yes, these are all people we participate in all these different forums and Have discussions going and things like that. We you know the goal is to really Figure out how we can raise awareness. I actually have some ideas You know, I don't know where these ideas are gonna go exactly yet But we want better reporting methods better information methods for people when it comes to cyber security So that was why I was on the ISO Team doing it. That's a little what a little plaque says here It's as I was showing but uh, I'm take the glove. I just wanted to move this out of the way I can take the glove off. I don't even fingerprints all glass. That's really it's really heavy too because it's glass It's uh, it's that's pretty neat. So I'm pretty happy. I got a little word there for it But like I said, these are some of the things that I work on when I'm not Doing some of the other things like making videos about your NASA stuff like that and if you didn't notice from the photos The the one that I use is a thumbnail, but these are this is a view out of my room in Chicago So this is that's where I was for the last few days participating in all the community stuff a lot of fun though Doing all that. So that's just a wanted to mention that something else I do something else Maybe you are or not interested in if you own a business but joining as a business as a IT company Comtea is a good thing. Oh, by the way, if you join as a company you get discounts on certifying your staff So there's something to think about Morning from the land down under been a hot minute since I cut a live stream. Yeah, I don't think I've seen you out here in a minute So cool. Happy to have you here Excellent timing just finished the two and a half admins podcast I will probably listen to that after I'm done with this. I do like the two and a half admins podcast Be there this weekend and celebrate paint backstage and the Unified conference. Yep. I was planning on being there for the Unified conference as well Let's That's on my list for what that's like next week I think I know it's like just got back to Chicago and I'm gonna go back to Chicago. I'll probably drive again I drove there Detroit to Chicago's. I don't know three and a half four hours. It's not that far of a drive By the time you go to the airport and deal with all the airport things It's not that bad. Of course. There's also the train is an option. I can take a train from Here to there and I thought about doing it except never taking the train. It just seems novel I've never really gone anywhere on a train. So maybe I'll take the train Not sure back over to the other fun things I'll be talking about what one of the things I want to do next with xcp and g Well, actually, let's talk about shooting us first for you Because I'm gonna go in the order now that they're they're on here. So take the train Jason's take Jason takes the train Jason took the train out there But I finished shuffling around all the things on my true nas I finished a video Where I showed how to replace a boot drive in a true nas So those things are complete It's a nine-hour drive to Chicago for Cody. So yeah, that's definitely That's a big difference there. This is the one thing that I was told as well is the The delays from the train can be a pain and That's a nice thing when I have a car. I just kind of wander around I don't I didn't go directly home after I was not the event. I went and made several stops That's one of the reasons I like driving is because I love Not being on the freeway and driving to random places So that's Part of my reasoning for wanting to I'll probably end up driving again is because I'll pick places along the way and then stop on there Oh, yeah Yeah, for Cody, you don't want to deal with the uh, the border crossing and things like that as well Cody's coming from Canada Back to the true nas system Uh, I rebuilt this is the 45 drive server that was running true nas core. I finally get rid of true nas core it's not that I Really had a reason to other than I just wanted to use scale So i'm using it in on a larger system So I went ahead and reloaded this one with scale because I needed to rebuild all the data sets on it This particular drive here drives. I should say these devices Is we have wait how much storage is this one? I think it's about 200 terabytes 217 terabytes Of storage on there We have 203 available left because I've only got six percent use on there But this was a system that I just had to rebuild now something interesting is I have a few drives in here That kind of give me some drama and I don't really understand why But they are weird because they work unless you scrub them then they stop working So that's been kind of a weird puzzle Maybe if I load them up enough they will but one of the things I've been doing to figure out which drives are which Was I take them in and out of service you can swap them in and out So if you look at your disc one of the things you can do here, let me zoom it in and make it easier to read Is you can add notes to drives if you click edit there's a note section This drive gives errors when scrubbing. It's a weird problem. I ran into The on the true nasty just kind of weird it was like it works I loaded the data, but then when it ran a scrub it just keeps giving random errors But when it's not scrubbing it doesn't give an error And then the error count goes to zero So when it does the scrub the drive like drops and comes back in and re-silvers and works Drops come back in it's kind of a weird puzzle because the smart on the drive is fine There's not another error message on the drive. That's what's got me so puzzled when I work on this because Uh smart true smart you can run this test on it and the smart test results are good on it. So we don't It it all successes on the smart, but it keeps giving an error when it re-silvers These are some of those weird quirky problems And I might do another video on it just to show that these are things that happen But I don't know if that video is valuable outside of what I'm just talking about here because I don't have a solution Other than don't use that drive Um, but before I do that I'm actually taking these drives out and putting them in another system to see if I can reproduce the same thing In another system. It's been a real head scratcher for me um because If you look on the if we go to the storage You see that I've got three unassigned disks. I have three spares right now Well, technically two of them are the ones that give errors. The other ones are not I've already shuffled them around in different locations. It's not a location problem I thought maybe I just had a problem with the slot they were in Nope, not at all you can shuffle these around when you hit scrub They don't They start giving errors So it's been it's just kind of a weird puzzle. So we're going to go in another machine I'm going to try as I have another 45 drive server. I'm going to put them in just to see if I can repeat the problem again um, definitely weird Uh, the smart is fine, but zfs is probably reacting to something there black playing issue cable issue Yeah, like I said, that's why that's my first thought was move them around And uh, see if I can repeat the problem. It doesn't seem to matter where they are. They always give the error Uh, what's the point of having a meta log data drive when using mvme storage in true nas The well, it's part of How the data is stored. So they each your data vdevs versus your metadata versus your log drive slog drive they stored data differently and Because of that, you're not going to get much benefit by the way if you have an mvme array Um, you have to have a drive faster than the array or it's not going to matter. So the If you all have mvme, you probably don't need this log drive on there if you're running all mvme This well, the log drive is not going to be able to write any faster than the other drives The idea of the log drive is you can Take the zil the intent the zil intent I have a video on this and write it to that drive so we can go to the others Wendell actually just did a Reasonably recent, maybe it's been a few months ago Maybe even longer of talking about using the octane drive to that because they have a really high q depth So you can get a series of writes on certain Intel octane drives the ones he talks about on his video So that does make sense to do it because you're then streaming the writes back out to the other drives But the metadata You probably won't get a big difference on there with them because once again The reason for a metadata drive is to be able to take those big look-up tables and store them in a place That is nice and fast But I think there still might be some advantages because depends on how many data vdebs. So if you had 100 data vdebs but stored all that metadata on a fast array with the meta Vdebs You still might have faster look-ups on there. Wendell also did a video on that It's not my to-do list to make some videos on it, but we've done a good job on there Weird stuff happens. Sometimes I had a motherboard that would delete nvr sectors on any drive. I tried to manipulate predictions weird Maybe the board on the drive is just flaking out that is probably the best answer It's also a hard answer because it's not anything you can easily put your finger on Uh, I've got a couple of the drives that do that and I don't know Like I said, it's just puzzles I have one of them that it's kind of in a failed state It actually does give I think one of them does give smart errors But these other two that I have do not give smart errors I'm reading about this drive scrubbing error into your NAS by several people. Yeah Yeah, it's uh, it's interesting because They're all the same drives. They're all installed at the same time So they're not like I have a variety of drives and this one's flaking out. It's like You know, it's it's a 20. How many drives are in this system again? Let me look What do we have like 30 20 26 discs in here? Yeah, there's um Like 26 drives. They're all the same drives, but 20 or 27 drives. They're all the same Two of them are unhappy and it doesn't matter where you put them Can you use different size drives in sure nests like an unrate now unrate and sure nests work Incredibly different from each other. So the answer is it depends I have a video called imbalanced vdevs You can set this up. There is a way to do it. You have to Group them in vdevs in order to have it work. That's the problem with zfs It's a calm. It's a beautiful system But that beautiful system the absolute incredible amount of redundancy you get and bit rot protection you get means That each one of these groupings and vdevs here that they have Work really well, but there's some rigidity to it for the way it has to be set up. So here's the grouping Here's the grouping You could have each one of the vdevs groupings be different sizes But the drives within them can't be uh hodgepodge together different sizes Uh, yeah, uh, wendell did a video on the flex 170. It's uh, it's the most recent channel over on level one text So yeah, if you want to check it out the flex 170 shows um SRIov with an intel flex gpu. So pretty cool Uh, what linux is os would I use for tail scale and home network any Debian is my answer for everything. So if it's a server, I'm pretty much with some exceptions I have a couple because there's some requirements For ubuntu, but ubuntu adebian are my go-to server os's night almost all the time. I prefer devian Uh, I had a scale issue where smb shares with deny access or seven days change permissions to groups Other than users fixed it weird I think about it even with flaky drives. You still have your data because using zfs. Yes I have no worries about zfs losing my data Doesn't mean I don't back it up because raid is not a backup But zfs is absolutely one of those file systems that I have done in the past and will do again The video i'm going to be doing first before I play with a bunch of vdev videos Over on my bench behind me There's no way to get the camera to point at it. So I'll gesture towards it is Uh, old system that i'm going to do my new zfs torture test on and that's Something I did a long time ago years ago and I realized I should probably do this for often We're just going to start yanking things out of the drive system while it's on and not just the drives but like pulling a controller card while it's on the goal is to see How resilient is zfs when you have someone Providing a level of chaos around it, you know, not just unplugging it But what happens if we pop a piece of memory out for example, you know, what are the extreme ways? We could try to get zfs default. What if we did this while it's doing a scrub? How Much resilience is there and uh, what does it take to recover and break zfs? So i'll be doing a series of videos on that as a topic That's going to be some fun fun times for sure Yeah, the power cable is not an issue with the 45 drives. They're all on a powered backplane um The way the backplane works in there the drives all get the same power and also moving the drives around Uh, yeah That's definitely not because I can move the drive to a different location and repeat the problem It's it can't be the back. It can't be a power cable issue How do you run sure nas always bare metal? I always prefer to your nas bare metals not that you can't Pass a controller through I just did a video on pass through you can pass it through an xcp A g and work it that way not my preferred way to do it though I generally all my sure nas servers right now are 100 on bare metal You have a pf sense with three vlands my laptop is on vlan one my server is on vlan two can't figure how to get them to communicate Firewall rules between the vlands. That's the way I've got a Like getting started with pf sense. I've got like a small business pf sense setup video I talk about vlands and rules between them in that video Uh, that some drives are more sensitive multiple changes than others It's like variation in load under full load some drives can play counter whatever is the theory. Yeah, that's I mean I and that actually lends itself to It would be the controller on the board having some flaky issue because why would the drive platters not have any errors that We can find but somehow when it's doing the scrub it has a problem It also now I I mean it's almost saturating when I was moving the data around on this I was hitting just about the peak of the 10 gig I was moving about you know Uh 8.9 8.8 gigs. It was really fast as far as all the data movement But that never seemed to trigger it. It didn't trigger until the scrub. That's just so weird about it Uh, what's the best way to do a do a true gas backup server does need to run 24 7 like the main server Is a way to have it run backup snapshots and then turn off? Well, there is no facility within true nas to power it up and down you would have to Come up with a script to power off your true gas and then come up with something like wake on land to power it out There's nothing built in a true nas for it Um, but if you are worried about the power that it consumes then Yeah, you would have to just build your own scripting for that. There's nothing inside a true nas to do that though Well, I mean true nas has the ability to run scripts So I can't say there's nothing there's just that I know of there's not an option to just turn off a true nas like Say hey turn off at uh, 9 p.m. But turn back on tomorrow at 9 a.m I have a new use drive that's reporting device seh uh five on record Uncorrectable sectors at true nas should I return it now ran bad blocks of five days same results Yeah, if you have bad sectors on it, I would replace it Um, that is I mean have I seen drives that only ever developed a couple and then seem to run forever with those couple? Yeah, but usually that's indicative of a drive in pre-failure if there's a few bad sectors You usually start seeing a few more pop up. That's an indicator that that drive may not be the best one to rely on You know, I don't know why people want to do this all the time with one gig networks jumbo frames have a Definite help once you get to 10 gig There's not as much of a difference 10 Moving to jumbo frames are usually how people end up in trouble trying to uh configure everything So maybe I'll do an updated video to show how minimal the difference is I mean, it's not like you're night and day You're like, oh look, I gained a few percent, but I had to spend a lot of time configuring switches Most of the time your saturation Problems to reaching 10 gig have nothing to do with mtu and everything to do with are the drives you have fast enough to saturate 10 gig Is the hardware you're using fast enough to saturate 10 gig. So the mtu 9 000 It's not needed unless you're trying to unless you're just running up to performance bottlenecks and you're like, okay I need that last couple percent squeezed out. It's not like it does not double the speed is the It's more like very incremental increase on there So I did the math on How that works and I I'll have to look it up. So I kind of want to do a video on it because there is a math equation to Why it works better at lower speeds, but as your speeds increase up to 10 gig and beyond the mtu Doesn't affect it as much with the modern hardware Followed your video and they have a rule for vlan one and the source of vlan two destination to pass But it still is saying network unreachable. Not sure what I'm messing up. I'm not sure either because that is um Having it having an allow rule is the way the data gets from point a to point b You're not supposed to route your storage. Why do you have storage in different vlan? Uh, don't do different vlands who router between each other The best way to wake up your true nas or any other nas is just to use the ipmi tool Oh, I mean, yeah If you have an eye-drag on it. Yes, uh wake on land will work as well If you don't have an eye-drag on there Is there a way to send cloud backup storage to true nas for backup? I cloud one drive to home server There's a few abilities I I don't we can look because why not i'm here Uh, we can look which are the ones that are in here so if we go to credentials you can these are the ones that are available and So amazon back plays box dropbox google cloud google drive google photos that's interesting out the play with that one http hubic mega azure blob Webdav I think they pulled out because microsoft changed something Um, there was another one in there for pulling some of the microsoft stuff. So this is what's built in You'll have to find third party, uh cloud connectors if you want more than what's built in Thank you very much for, uh The donation and yes untangle arista has Discontinued all home subs. I forgot to make a note of that Uh, I was I was going to log into my arista account because I still have an arista reseller account But we're not really reselling much arista Uh, untangle what we're untangled. They have the same portal. I was going to see what it looks like because I probably have a subscription in there somewhere because I I paid for a subscription to do videos The videos were not only not sponsored that I did for untangle. They were actually paid for by me because I bought the license Uh, I haven't looked at untangle in a long time The product hasn't changed because someone said asked if I want to do an updated video or said I should do an updated video And I'm like, well the product's the same the only thing that changed was it went from untangle to arista They haven't really I even maybe i'm wrong and maybe i'm just not keeping up with the updates But this is the biggest change they've made now is getting rid of the Home one which kind of sucks I got tagged in a few things for people to ask me what an alternative is And the problem is people want web filtering for a good price I don't know anyone who offers web filtering for a good price anymore Untangle was my answer to home users that wanted web filtering on the firewall level I don't have an answer now. It kind of sort of works in unify. It's not wonderful But that's at least the next runner up probably for web filtering I built an eight base sata ssd true nas with a max two and a half gig network Uh, it's a raid z2 sufficient to saturate. Uh, yeah Yeah, I would say raid z2 with eight sata ssd should saturate two and a half gig Greetings from seattle I want to host servers in the colos or any suggestion you have for psd firewall rules I may want to consider well psd blocks by default So you're not Worried about them being open anything open is going to be things you did to open it So that's one of the things I really like is that they lock things down by default my Consideration would be only open up. What's necessary and not any more That is where you can find yourself in trouble by opening up too many things So you start with that Everything's closed that you decide Exactly and only what you want to open and open that I also have a video on pf blocker If you want to do g oip blocking and blocking tour nodes, those are definitely good extras to do So if you're going to lock down your pf sense, that's definitely a good way to do it does Wireguard, do you know the obtained? I I don't understand the question wire guard is a vpn protocol So I can't depending on who or what firewall is using this Protocol they will decide if they want to log or not It is not a protocol question. It is a whoever's running The protocol question Chumble frames depends on a protocol as well. For example ice cozy versus invest ice because he will benefit even at 10 g higher speeds Because the pack of size uses I want to say that the I'll do the I'll do the video on this at some point And break down the math behind it. It's so insignificant even on ice cozy um Well, this is fuzzy ubiquity finally has acl for layer three switches. Let's put an asterisk on that How granular are these acls for the switches is the question having acls that are really basic or they have actual acls That give you advanced use I did see that that got announced today, but I didn't see if they were actually advanced acls This is something ubiquity Like they give you some basics. They don't give you Always a lot. They're like, hey, this works, but does it work? Well, does it fit my use case? Does it fit these advanced uses says that's okay So they give you You you well, let's make your correction. I like your point here. So ubiquity has acl They don't have acls. They have acl they support. I'm happy. They support it Don't get me wrong and if they support it that probably means they may be adding more later But so ubiquity has acl Have you heard of arachnids networking? If so, do you want a good laugh? Hmm um Hi, Tom, I plan to add a 10 gig dik to my analogy and ask how easy to move the ip all services and containers over the new Nick anything to watch out for it's been a while since I done it But it was pretty easy you add the new network you Um, I don't think you can set the same ip on two at the same time I believe it the the system will invalidate that so you'll have to change your main nick to another ip Change your 10 gig nick to the ip you wanted at and then it should work It's been a while since I've tested that but yeah I'm currently on a unified cloud via my home network using a self-signed certificate. I believe until again should be Should it be a priority to get this replaced with something from let's encrypt? I don't think so. I mean, I don't think as a home user I have a lot of things with self-signed certificates because I don't I'm not that worried about it. I have ha proxy that I put in front of things I do care about having a certificate, but I don't bother Running the um, I I don't especially there's a way to install certificates on unifying It's just it's more work and not worth it to me. So yeah, uh for the law the home user stuff I wouldn't even worry about that VPS and privacy thinking of running head scale to get multiple servers and users thoughts Um, take a look at net bird if you're looking for a self-hosted Overlay network. I mean head scale is fine. I don't know of any flaws with it But uh, check out the folks at net bird. They've built a tail scale alternative. I guess we could say and My friend christian lempa just did a video on it. So it's pretty easy to get started with they've got good documentation But it's fully open source and self-hostable. Uh, the control plane is self-hostable. So yes I'm a bit late today. Hi time and channel for everyone's doing well. I saw the mention of the chunas Oh, I got a mention in chunas. That's cool That um, I'll have to look at the newsletter The dragonfish video that's really cool Yes, and chris's point here increasing m2 doesn't increase the throughput so much as it reduces the uh, required compute resources to handle that thoroughput including endpoint routers So you when you're looking at the m2 thing and adjusting it Are you looking at an article from 12 years ago? And are you looking at article from today when we have different hardware that's running out of this stuff? So that's one of the things I that's probably why I should do like an updated video and write up on it It's on my to-do list, but that's kind of low on my to-do list It's seen someone maybe talking about the privacy I think maybe that person was asking about the privacy vpns and i'm trying to figure out A rackness. Oh is an av brand You know, I think linus tech tips did a really funny video a while back On making fun of all those av companies. So Yeah, there's a lot of I don't get it. Like I guess there's a market to overpay for things. That's how the av market works I don't know much about it. So that's um Yeah, that's probably Yeah, I don't know the the av world is just a head scratcher to me So the other thing I wanted to talk about You find the page for it. I think this page needs more documentation So let me pull up the pages here that I'm talking about Because I want to talk about this as a topic But you should have there we go my xcpng notes. Where did the notes go? Huh, maybe I lost that one or didn't bookmark it. I should say Come on. Where'd these go? There we go This is the one I want because I want to share it with all of you This is still over on the citric side of the house. So I'll answer a true nasty question before we jump in xcpng And sure now should I create a new data set versus create folder and existing data sets just seem like advanced folders Data sets are very much like advanced folders I have a whole video breaking down the advantages of data sets In many ways, they are advanced folders It's not that you need Every folder to be a data set, but if you have a different data policy Snapshotting etc You want to build it as a data set. So if you go, well, I like this folder To have a retention policy based on these snapshots. You can't say that with a folder you say that at a data set so Yeah, you can nest the data sets and then create rules I have a whole video where I think it's called like data sets explained or zfs data sets explained Where I break all that down of why you would or maybe wouldn't want to Have a bunch of data sets, but yeah, you can nest them in each other There's all kinds of things now far as like When you're presenting it as a share they look like folders, but that advanced feature means quite a bit Yep, all these rules here you can But not limited to snapshot retention acl's encryption others You can even have data sets that have pass phrases on them That way if you have something really protected in there and you don't want someone to physically take the true nest You could set it up So you have to type in a password every time the true nest gets rebooted to unlock that particular data set Where it has your most critical data I do that a lot I actually use a lot of pass phrases because it's the easiest way to Encrypt the important things that way if anyone were to physically take one of my true nases They'd have to have the passwords as people love to point out Tom you can't just reboot those true nases because they require someone to go in there and unlock them that is actually Absolutely true and that's exactly the point. I don't want them in any way to be easy So yes I recently migrated from vmware to xcbt discovered nested virtualization isn't supported for wsl 20 windows xcb just approximately other hybrids support this nested virtualization is supported You can on any particular vm You can choose to turn nested virtualization on right there. It's under the advanced tab for nested vms So I don't know unless there's something more specific that you're looking for I do know there's a There was a forum post about not working well with nesting hyper v inside of it But I yeah, that's hyper v I don't know why you'd want to nest hyper v inside of xcpng I'm setting up a 12 drive expandable 24 terabyte 18 pool. What beta parity A lot of configuration would you go with considering primary use is getting editing mini-share for your sampling and taking clients um, you're going to get better performance if you start splitting them up in groupings of six So if you did like z2 at six should get you good performance You might want to do some testing build it test it See if you can get the performance you're looking for if you did 12 12 is a good size for vdev But of course 12 is not going to get you any extra Uh, you may not have quite the performance you want if you do it 12 wide until you get the next 20 To get all the way to 24 and then split it. But if you did it in six now, you'll have Six and six feeding it and when you get more drives later, you'll have another six and six and that may perform a little bit better for you wsl, I mean, I don't use wsl so But I I mean should So it's some type of hyper v issue because w. Oh because the wsl uses a version of hyper v weird Huh interesting I never used wsl. I just run linux That's that's been my solution. I mean, I may I don't have maybe the use case you have or the challenge you have with it So Yeah window windows problems That's for sure But the other thing I want to talk about here the command line interface with xcp and g which yes, I'm going to share something from citrix. This is a You don't have to sign in or anything for this, but yes, this is on citrix There's actually a lot of things you can do so I thought about making a basic admin guide That might be my next video on xcp and g to show you how you can do a lot of things from the command line And it's also handy when you're doing troubleshooting From the command lines. You're able to go through Modify you can actually and this comes up and I think someone asked the last live stream about this talking about the ability to manage xcp and g completely from the command line and I'm like, yeah, there's a ton of commands There's all kinds of things you can do from the command line that May not be fully exposed under the zen orchestra So if you have some custom use case for example some of the past through things I showed how to use the command line to add past your parameters The nice thing about the way the command line works. It's really easy to add scripts to your system To go through and execute on all of these things So if you want to automate some of the things by doing it from the command line, absolutely you can Not only that one of the things I want to showcase is From the command line, you can take a vm start and stop it then send it To another server you can even pipe it over ssh You can even pipe it over ssh across the vpn to another server There's there's some really cool things you can do on there So I thought about making kind of an admin tips and tricks and command line Tricks that you can do for doing things all from the command line The more important ones though the admin ones that I would really cover is The emergency ones like what happens when this breaks or that breaks? What are the tasks that you have used Tom to solve a problem such as a downed Master server in a pool. So those are some of the ones I was going to Cover on there UUID is not a pain UI UUID is much better in my opinion. I don't consider UID a pain at all um Can you demo and nested xcbe with two xcbgs in the dedicated xo running inside of a single xcng? actually kind of on that Uh, we're hold on We find the right one. I think it's that 21 I can actually turn this one on Here I've already got one of them set up. So it's not hard to demo. You just you can load You can load zen server inside a zen server So And and you can keep load zen server inside a zen server inside a zen server Until you don't have resources to run any more zen servers Any idea if the kernel updates include support for p&e? I don't know It's a linux debbie and kernel. So look at the kernel version and does that linux debbie and kernel support it is my question If if the answer is yes, the current it's it's a linux debbie and kernel But the reason I like uuid so much. I mean one they're right here But the nice thing about uid is I can do this. It doesn't matter what I name the system I can be very implicit that no matter what this uuid is absolutely the uuid for this vm No matter what I want to call it. These are the reasons I think uuid is a very good way to do it because each network has a uuid each disk has a uuid This allows me to manipulate all the different things as a uuid As a matter of fact, if you notice when I mouse over You even get the uuid of this device It makes scripting a lot easier It makes your ability to set these things up way easier uuid is definitely to me It's like one of the huge advantages you have in zen server is being able to specify everything by the uuid and not Worried about someone's naming scheme This is also why and I as I shown before in my production systems you can put you know the The little icons and everything inside of here because none of this matters This is always the you based on the uuid and not anything else Uh, good subject order a bunch of vms from accidentally disconnecting storage driving xcp and g and reconnect the vdi's as a royal pain hmm You know, that's an interesting one because one I mean That's really creating your own disaster. It takes it takes some Bad you're you're doing something bad when that happens. So if I weren't here to my storage and I have is there something running on this one right now now. There's no vdi's in it. We Okay, yeah, I got some on here But if I actually delete this it it doesn't delete easy when you go and tell it to remove Or forget a storage it will ask you questions it will It will tell you delete this vm like you've gone through some steps to Absolutely be sure that you want to delete this system uh to In order to break all of these that's just not yeah Uh, so I don't know it's not a problem. I've run into very often at all I too enjoy uu id's no information about the resource behind it and identifiers should only be used as a reference to the resource. Yes Have a unique id uh that unique id be uid you need to type it by hand is not better But that's where you don't see that's the problem It definitely if you had to type by hand that would suck But if we go here And oh, I can't make this one bigger. Um, or can I? Maybe this will be big enough Let me look up one of the commands here to make it easier And so well, that's a good command to run here So if we did xe vm I'm sharing the right tab. Yeah vm list I can just tab twice and it will Give me the vm I can type two letters of it and tab again and it does it so it It's nice the fact that it works like that. You can Uh have it Auto complete so you actually don't have to type out these it will auto complete all the way through So there's not often you there's not often at all that I have run into any instances Zen has built in nothing, especially you have to do auto complete options for all your uid typing Uh, the problem is that vdi's reference to storage not the vm. So you suddenly have uh, yeah The warnings don't lend to the severity of the function It makes you type out delete storage like if you don't know what you're doing I can't I mean If I feel like when you are deleting an attached storage and it makes you type out remove this storage And it gives you I think it even has a warning. It tells you that there's still drives on the storage You you have now gone To the point of beyond Uh I mean at what point do we do we put bumpers around this thing? I Love to be able to upgrade the switches firmware without having a power off all the VMs that use storage across that switch Uh, you can just migrate them. That is probably the easy solution My whole week was me, but I just covered to have to have auto complete for those. Yes Yeah, the there's a there's a couple options here. So option a Go to the storage here, which I forgot which one is do I have it on this? Okay You can just do this I can go enable maintenance mode and it's going to stop these VMs. That's absolutely One of the options. So if I go in enable maintenance mode stops the virtual machines no big deal and then I restart them Uh, that then, you know, once it's in maintenance mode Any switches that need firmware updates I can update option two migrate the VMs somewhere else. That is Equally an option. You don't not have to shut down VMs to migrate them if you would like this VM. I have this, uh Debian 12 VM running here And if I want this disk migrated off of here, I just can choose another destination for it Uh local storage, for example and hit migrate it'll move it from where it is to where I want it to be The next option Is architectural because if you run redundant switches, you can actually take a switch out and update it Bring it back online and then update the other switch So if you haven't from an architectural standpoint you if you are running this in production, you'd probably want to You know set up redundant switches and link bonding So you can go back and forth between the two different switches When one of them is out getting a firmware update no big deal the data will traverse the other switch Just like vmware motion. Yes No redundancy slow devices Over fiber. Yeah, I mean, yeah, if you got a bunch of legacy hardware support, that's a whole different problem But let me go here setting servers um one in here But like this right here is one of my uh one of my vms and it Is it stored on one of these disks? There's the lab one Actually, I should have my home lab 15 in here No disks on it. So let me move one of my disks over to it So I'll migrate this to the HL 15 lab just I would walk people through the process, but yeah, this is um Got a 10 gig network. So it's reasonably fast. It's going to take less than two minutes to move this uh virtual machine It doesn't it moves really fast probably gonna take less than that 13 seconds to go We'll count it down Yes, the it's all validated these napi validates all of it while it's moving. So yeah, there's no problems with like I have never Never had corruption when it's uh moving. It's never been an incident that I've had. I'm not saying it can never happen Computers are how they are and things can happen and things can go wrong But it's not a problem. I have run into at all So yeah, 44 seconds remaining. I forget how big this vms It's got some snapshots too. It's got one snapshot. So Um 80 gigs 36 seconds remaining Is it possible to air gap within zen a vm that only opens to a usb? I mean, you're the vm itself you could just not have networking on it That's a that's an option So, I mean, you don't you could boot a vm up without a network attached to it If that's the air gap you're looking for There are ways to do it Go system go There we go. Now we're transferred over here I should look up. I don't even know what speed the hl 15 operates at right now I'm going to be doing some new videos on the homelab 15 server. So The the other challenge you run into though is Uh jeff from craft competing covered this you get the 100 gig network and then you realize You have another bottleneck. You have to have a raise that can actually perform at that level So then you're buying a bunch of mvme drives and things like that. So, yeah It just it becomes very expensive when you want to go It's it's not just the networking and the networking has a certain cost to it as well The switches have a cost to it But then all the infrastructure that supports it also needs to be able to Handle that level of speed There's there's so many trade-offs getting drives fast enough and everything else It's not that you can't buy them. You just got to pay a lot of money for You know the uh, the the really fast ones But i'm not like for me. I don't know two minutes to move vms or The other option as I wanted to point out here I go here. I go to advance and just hit maintenance mode. It says, hey, we're gonna turn this off. No problem. I hit okay We enter maintenance mode It shuts down those vms once the vms are shut down So if we try to This is on that drive. So if we try to start it, I'll get an error No host available because it's in maintenance mode Then we can go back over to it and we can disable maintenance mode It's gonna go ahead and restart all the vms that are on that particular system. It's really it's really kind of ingenious It's really simple. Um, I I like it. It's just easy but I also if you notice have multiple storage servers, so if I were to Move this disk around somewhere. I have several places. I can move it to I've got the hl 15 I've got my trinity lab. I've got local storage in this right now There was four I forgot I deleted the synology out of here, but I have a synology I could reattach to this really easily and migrate it over there So if I wanted to you know, migrate it to local storage, for example, you'd probably move faster to get over there Three years would take me to move in Uh or an overnight to get a vm anything. Let's create it took me like 15 20 minutes already hooked up tail scale Thank you nice Do you know what could be the reason that my pf sense kvm vps? I get max 200 megabit one stream per I first cpu cp is running sufficient hardware offloading Um, I don't know when you start virtualizing pf sense you run into a lot of those things So there is some chance that there is some incompatibility with the virtualizing of it Do you have any recommendations on how to speak to a cpu to run? I mean the question is what's your workload of this particular system here um This host I have a video on it, but this is just an amd 9500 x Works fine for what i'm doing here, but I also have I go setting servers I'm actually shocked at how good zen works on this one. So if we go back over to hosts This is an amd rise in 7 7 30 35 h with radion graphics works fine on this one too I'm actually running quite a few things on this one Maintenance mode not surviving reboots. I may not understand what you're talking about there Yeah, I mean are you saying that if I put a we can try this I I can try this in my lab So let me pull up the lab one here and we want to put something in maintenance mode and see if it works. So All right, if I enable maintenance mode it'll stop these VMs So let's go ahead and do that. I should make sure these VMs are working first. I may have rebooted all kinds of things Well, actually what we'll do is I'll take it out of maintenance modes and see if they boot back up so To save a maintenance mode, let's see if they boot up. I just want to make sure they're all working before we do this Okay, they look like they're all booting so I'll wait till they all boot back up And then we'll shut them down again. And then you see if I reboot the server, there's a problem VMs will auto start again. Well, if the VMs are set to auto start, they'll They'll try to start they'll give an error Um, so let's go ahead. Oh, I should have left that. Well, we'll just leave these ones on What is this one doing? Oh, it's probably the windows one taking so long Windows is probably doing something in the background here So maintenance mode reboot host got it this one stuck This this one might be broken It's it's looking for a very specific network that doesn't exist. I'm gonna force shut down. Yeah This one I don't want run that one a different but these are running so I got two of them So that's enough for an example here So if we go over here to the disk We have two VMs are running a maintenance mode it okay So we're in maintenance mode. These are in a stop state now go to hosts These are attached to I'll just show you these Are attached to the youtube demo pool which is going to be this host So you can see it says if you can see at the top it's probably kind of small but it says demo pool So if we restart this host We're gonna go ahead and reboot the host This will restart your host Yes Oh, it gives them that's an interesting error Wait is there hold on Hosts I got to shut this down It's not able to shut this one down because it doesn't have the tools installed. So now we're going to go back to the host youtube demo pool Go to those it's still in maintenance mode. We're still following the procedure here And we want to go ahead and reboot this server and see what happens. So reboot We'll watch it reboot. I actually have a kvm on it so we can watch that Tom's the chaos monkey. Absolutely Okay, so I need to set them to restart So we'll do it once without telling them to auto restart and then we'll do it again and tell them to auto restart Which they'll probably just give an error Um, I think there is a way you can export data from gray log. It's not something I've usually tried But I think there's a way to do it I think like you can export a csv in gray log I mean if you have someone that wants it in some completely different format It's just running elastic on the back end So query the elastic server and you can get out whatever you want. So you can just grab all the data from elastic from gray log Yes, this is the bligh kvm All right, so this is now booted back up and do this again Well, we gotta wait for the server to attach We'll just force it There we go There's those This is still in maintenance mode. So now the next uh test is to turn on auto boot So let's go ahead and look at the disk And we want to turn on we'll tell windows to auto start So tell windows auto power on I'm gonna go back to the host and we'll reboot it again Which I think it just is gonna give me an error. I don't I know it won't start if it's in maintenance mode But i'm curious what what race condition do we create by doing this? You know, does it break something? You know, I've never said I I already did a video on how to do pf sense auto backup I've never used any auto backup scripts for true nas because I never felt the need I don't change my nas enough When I set up a true nas or when I make changes to it I back it up when I'm like I go in as a project. I'm like here's a project I need to make these configuration changes to my true nas When I'm done with those configuration changes I do a backup So auto backup doesn't do much for me because I'm not in making configuration changes. Matter of fact, it was funny I looked I'm like, oh well that one had died the restore was from like a year maybe two years before No, I can't not a year But numerous months behind because no changes have been made to the system So yeah that um, but I have one on pf sense. So that that I do have pf sense got that has that feature built in for auto backup Um, I've used true church from time to time. I try to stick with just the true nas apps I don't run that many apps on my true nas. So I It seems okay I've definitely had some disagreements with the way they implement things because they keep talking about using the pve instead of the host storage But I've had people that lost all their data because they don't understand where their data is So I've definitely had some complaints about it. Um about the way they implemented things All right Oh, there we go Did windows actually start interesting Oh, that's That's a bug So if you reboot a host and it's set to turn on it will definitely do that. I will report that Here's the thing you knew about the bug. Did you report the bug? I didn't know about the bug because I never think to while it's in maintenance mode to reboot my host So I guess the the question is This this is the problem these conditions exist because no one reported them most of the time So my question is did you report it and if not, um Then Okay, I mean if you reported it awesome They'll get around to it It's probably not a high priority bug because of all the times I've used maintenance modes There are zero times while using maintenance mode. I also thought to reboot my host Um, I just never thought to do that and I I'm assuming that's probably how this bug became because no one ever thought to do it Um later this year. I don't know what they have any They didn't set any exact dates for it that I know of for doing that Hey, here here's the next question. That's can be fun. So this is running What happens when we disable maintenance mode while the drive is running? Ha no host available, but they're all still running I don't know it gives an error, but it works Which I think is funny windows is still working. Oh kind of a funny bug Yeah, windows is fine shutting down This one's working This is why I have all this stuff in the lab is so I can do all the fun stuff This one should get an IP address now I gotta figure out what network this should be attached to That feels like the right network Do not use feels like the wrong network Local address Yeah This thing has failed in some way If rebooting it fixes it I don't know that I would I mean it has granular permissions, but it really was a design for reselling like that like if you It's more designed for teams So you can give granular permissions to your teammates for how things might be set up But it's not really designed to publicly expose it and build users out for multi-tenancy like that That's not that's not really an architecture design for it You're having performances your iron wolf four terabytes in a mirror Um, you can't go over 150 on two and a half gig with a large file transfer. It varies a lot between 90 Yeah, there's that's uh, you need faster drives so You can't take it out of maintenance mode because the host is available even though the vm is running race condition. Yeah Like I said, it's uh Yeah, definitely It's an odd problem. Like I said, I I mean If you can and I forgot to put this up earlier But you can email things to blog. There's if you can email me where You posted it. Um, I will follow up. I'll bump it and in Suggest that auto power on get disabled Because that's the simple solution for this Is auto power on be disabled when Excuse me when it's in maintenance mode. I mean, it's an easy fix So, yeah So uses nvm to access churnass storage, but the churnass itself is running on the storage drives No, I don't I this is an attached storage So and this is an nfs on my churnass system Churnass is running as a physically separate server Uh, should I enable auto trim for an ssd pool? I think it enables it by default. If not, you can. Yes That's what that's for if you have an ssd pool Okay, so as a private ticket. Yeah, I try to post these publicly because that way One of the advantages of posting publicly on their forums when there's a problem Is more people get to chime in and go, hey, I have that problem too And then vates can go. Oh, look lots of people talking about the problem. I guess we should fix the problem so that's um That's one of the advantages I'd say to doing it that way So it might be a new uh related problem Yeah, I mean if there's not a public forum post that you're aware of I'll I'll take the time to write one I I go in there and we'll note these these issues I didn't even uh checking out a great kenean kebab 45 drives Lee any comments using 45 drives versus churnass for your Vm storage We use a lot of 45 drive servers and I actually talk with them a lot. They're Just a great bunch of people besides making a cool product. They're also good people but It kind of depends we have uh Two petabyte storage servers that we sold to a client that are 45 drives xl 60s that do run churnass We actually have quite a few churnass servers that are Built on 45 drives hardware because the client knows the hard knows that software very well. So there's no issues on doing it, but Uh, if you have certain needs, we also are a reseller for ix systems. So if you need A single system with a level of redundancy. We do resell the ix system hardware hardware for those circumstances Yeah, and they both sell the hardware and we've done the other side as well. We've run Uh 45 drives with the houston os for some projects as well So we have 45 drives hardware and 45 drives, which is open source the houston platform is open source And i've reviewed it before but you then have both of those we've got the houston system Along with the 45 drives hardware running at another client. So it just kind of Varies there's not like a one size fits all matter of fact That's one of the things that I think really makes us a little bit different as I meet other people at it Too many businesses are Very cookie cutter. They only sell a solution Which also means when you come to them Their solution is the only one they're willing to offer. They're not willing to try A different type. They're like we only sell this we've got blinders on to all those solutions And that sometimes doesn't lend itself well, which Literally One of the reasons is other client is spending so much money with us We're not even their primary it their primary it has such blinders on about what they do that we keep winning all the bids because they Further way the company operates will just say they are required to submit this up for bid, which means they're They're managed it provider bids on it and we've been on it We keep winning it because we actually have solutions that they want the other it company does not matter of fact The other it company does things like quote del servers for petabytes of storage Do you know who's not competitive at building a petabyte high-performance true nasty server? Dell compared to 45 drives. So we keep winning bids Is uh the houston 45 drives os wasn't aware they have their own they don't have their own operating system houston is a software that runs on top of linux a bunch to rocky debbie n and will Give you a really nice ui with an overlay for the 45 drive system. So yes Uh, they they do have people who run windows on 45 drive servers. That's a that's that's legit something that happens sometimes uh, we matter of fact sometimes when you have You need a storage server that has to run windows because you're running an application that has to run windows sometimes that happens Yeah, I I'm having really that's one of the reasons I moved so many things to true nasty scale. It just works Uh, does houston offer redundant ha zfs? There's not really an ha zfs. That's the challenge There's not if you're trying to talk about like zfs itself is not sef sef is A file system a distributed file system that would allow you for high availability between servers Zfs isn't designed that way. So there's not an ha zfs option Now what there is is a ha option Where and i've covered this before with the ix systems hardware where you have like dual motherboards Controlling the system that allows for A failure of a motherboard, but they have a backplane that is specially crafted. So all the drives are the same So it's a very different thing, but you can use like houston and sef and this is 45 drives like they're really really good at sef That is part of their engineering team. So 45 drives will offer you a you know multi system sef storage with Resonant redundancy and resiliency. I'd try to put those two words together. It didn't work redundancy and resiliency Well del servers are price competitive Used del servers not new ones The company we're selling these two is spending I mean the latest bid was Right around three quarters of a million So they are not buying used equipment from us. They don't want used equipment. This is um This is a pretty big series of projects we've been doing with them I know I can get used stuff for cheap. That's a whole different topic houston's more of a control plane. I would say that yeah You're trying to not use sef. I mean it's not sef as well as things. It has its place It's not the solution for everything like I just don't need it myself and not every client We have needs it, but we do have clients that do need it That's the thing once again We are not as a it service person myself and as a company the back end of the company being cnwr We do custom solutions a lot for clients. We have a variety of things um, there is no Horizontal scaling right now Uh of chunas scale it was designed to be a scale out solution Unfortunately, they chose cluster cluster did not scale out very well Gluster more or less got abandoned So abandoned is the best way to describe the current status of cluster. It does exist It's just not very active. There's still some community around it So I don't know what they're going to build in next I've always uh hard pressed to get convince myself to purchase use power edges the performance compared to consumer desktop parts This age doesn't cut it for me. Yeah, the comes on to whatever your use case is Uh, just curious how are you connecting storage your xab to your office? I assume fiber, but just curious um DAC cables the uh The systems matter of fact hold on. I have a video on this I can share with everyone. Um We will I will throw this video in there if you want to see it in more detail But yeah, these they're all just DAC cables the fiber goes across the building Uh or across to the other racks. So we do have fiber between the racks But inside the racks themselves um It's all DAC cables. I have a pair of DAC cables going to each of the xc This is the back of my xcp and g servers you're looking at But I I mean I detail it out in the video. Um that I post the video link. Hold on. We'll just Copy So there we go I did never mind. I I sent it twice. I didn't realize I sent it twice Uh, what's the price of your 45 drive solution and a del solution anyone know? Um, it's been a while since I looked up what del would charge for that a lot more I was told it's close to double. I didn't even look at the other bin I I didn't try to price out del someone told me it was close to double our solution But I don't know if that was true or not I we don't look at the other people's bids Um is a four core processor good for scale depends on how much performance you need out of it one of the things I've pointed out before is With true nas we find trendy miniar I remember what ip address is for this one There we go The one of my backup servers. I'm gonna replace this soon. I keep running it just because This is a just because right now Uh, this is a two core system. This is an atom 1.5 gigahertz. It only has I here's here's something fun. It has uh eight gigs of ram And it has almost three gigs freeing right now And it also has uh five terabytes of data on it And I still got 12 terabytes free. So I will be replacing this at some point, but I don't need performance. All this is is an extra copy of all my data I have an offsite copy of my data I have I have data in different places But I wanted one more copy because this is a really low wattage low power system And it works. So you can do this What do you like to use for endpoint backup for your clients? Ideally something local network storage of business-wise? We use vm where where a vm company at cnwr Are you going to keep your location matter? We just renewed the lease on that one We did not have to but we decided to renew the lease on the Lawrence systems office For now, we're going to keep it Until we don't really use it anymore A few of the people work out of that office. So we don't really have any reason to get rid of right now There's only a couple people working. There's three or four people that work out of the Lawrence systems Southgate office But I don't own that building. So it is a lease. We do own the building in Toledo Well, jason owns it The building in Toledo is one where it is and I've done a Brief tour of it. I need to do an updated tour of that building as we Uh Make some more enhancements to it. Uh, the Toledo building is much much bigger I think it's like 6,000 square feet. Maybe more. We it's a it's a three-story building. We have in Toledo right now Yeah, eight giga ram Laptop cpu. Yeah, really low wattage Uh, that it's all set at default Uh, when using snapshots Toff say backups do the pools have to be set up the same. No, they do not Matter of fact, that's something we're talking about is Let's close all these tabs. That's a Probably a good question. So if we were to look at the True nas server Here And we were to look at the data protection. So there's a lot of replication going on on this server And if you noticed and we'll open up these tasks and expand them out a little bit So they're All labeled pretty simple Is it can I not there we go zoom in a little so each one of these goes over to a server called apoc. So client data gray log Uh, stress relief synology We have a synology that backs up to here and then we back up the synology and over there tech tools, uh, nfs Lab and production backups. So these are lab backups and these are production backups That all these land on another server called apoc This server over here Is the apoc one. Matter of fact, that's the one that just died This is one of the places that all the data goes the day is actually backed up somewhere else again um but Here is that other server And if you look at the storage on this It's not the same layout. Matter of fact, this is, uh This is a raid z1 because the backup server so I don't mind it being a z1 So here's here's the z1 Of this drives, but if we go back over to the other server And we'll get the storage on it This is a raid z2 8 wide so there's actually way more devices in here. We have three v devs Each one of them having eight drives in it. So no they do not have to be the same Well, let's see Oh, you lie about to hear recheck your videos about zfs. I hope you have a great date. Yes Uh, yeah, the enable cove backups um The thing we use vm for is What am I gonna say there? The the vm is used for the fact that we can also roll our own storage with it But I know there's actually a way you can do that with the uh enable cove backups have that option as well um enable cove is kind of it Like enable bought that company and that was a really good backup product like that one was one that just worked um There were some bugs in it a couple times, but overall they were good at fixing them and getting things going All right. Well, I might wind this down here. We've talked about all kinds of fun things here This is this has been a fun great conversation with everyone But I think it's about time to uh Chill out for me my voice is man a couple days of uh traveling in chicago and talking to everybody My voice is a little bit worse. It's not terrible But it was that's another nice thing about riding the car. So Yeah, the backup server is raid z1 uh I'm fine with it. It gets the job done It's nice when because I have more than one backup copy. I've actually got multiple copies of it Uh, so I don't worry about it being a z1, but my critical stuff is a minimum raid z2 if it's really critical it'll be raid z3 but It's uh, what's on there is not that critical if it went down We'd be minorly inconvenienced I and I mean very might like it would break my lab. That's really what's on there now It's everything's lab um all the production stuff moved over to the colo. So Um, have I heard of a hypervisor called sunlight? I have not but there's so many of them out there Um, I yeah, I I can't really keep track of them all It's not not when I've um, yeah sounds like a solar wind project. I'll I'll go with that But nonetheless, I'm gonna get back to it I I'm going to chill out a little bit here and I got Stuff around me that needs to be done. I need to work on those next series of projects, which does include um, the at zen admin video that I'm going to work on zen cli and if I Um, here, I'll share this again in case anyone didn't see it earlier But yeah, check out this documentation on the cli stuff and as I pointed out it has tab auto complete I'll make sure to include that in my video Uh, thanks time for the xcpd demo overview documentation. I'm currently working on learning it I still find myself uh, translating from vmware lingo. We haven't worked with vm having work Yeah, a matter of fact, this is what we're doing internally is training our team who is very vmware Um smart like they know the we have vmware certified people that work for us So we're doing the same thing. We're transitioning their lingo uh to doing it and once you know virtualization, it's really It's not that hard to learn. It's like anything else once you kind of know that oh, it's not vmotion It's zen motion. You're like, oh cool. It's very similar at that point. So Yeah, all right. Well, thanks everyone for uh joining and I will chat with everyone later hit me up in the forum sit me up on the Vlog thursday at lornsystems.com. Like I said, I've been replying to people. I think I have been let me look again I don't think I had any yeah, no new ones in there, but I've been emailing people back and forth So, uh, you know, it's trusty and our forums is the best place to find me. So that's where to hit me up That's the easiest place, but hey vlog. There's the at lornsystems.com. You just want to say hi. Thanks