 I said, this one, there we go, I can see it moving now. Oh, you gotta start over. Is it working? Oh, that's the crowd. Do we have audio now? I think so. I can probably- Yes, Sam. Yes, yeah, all right, cool. Now we can start over. I've been talking for a minute about nothing. So last Thursday was the offsite company meeting and it was a lot of fun. I ended up playing a, what's that instrument? Oh, accordion. Accordion. It ended with an accordion. So it was definitely a good time. That, yeah. Yeah, we should make a business technicalities video about that one. Maybe too good. Maybe too good of a time. We all had a lot of fun. At least I'll find one, hold on, there's a photo in here. So this is, people ask like, how many people are at CNWR? I think we can, does this button work? It's a blurry you. Hold on, is there one that's not blurry? No, a little less blurry. Not everyone else is blurry. Yeah. So there we go. This was the big company party. We were just impressed at how many people are there. So there's a lot of people at CNWR. I mean, there are spouses too. There are spouses there too. But it wasn't everybody. We were missing at least four employees. Yeah, that didn't make it there. So that was last week. That was, we're just going to be some business and technicality talk about some of the takeaways about having a two-day management meeting. Yes, it is good and very bad. Yeah, it's good and bad. What? There's a Brett off-screen over here, chilling out with us. Everyone's hanging out in Tom's studio today. So accordion hue. Oh, that's right. I pull up the actual accordion picture, which is I'll spare you the audio. Yeah, no. The accordion needs some work. The accordion needs some work. One of the keys is stuck. So it's playing some sort of like a D-flat or something. It's not the only keyboard we can use. I have a pretzel in my mouth. That's the best part of that, is that I'm eating a pretzel at something to play an accordion. It made the video. I think it really kind of brought it all in for what was going on there. All right, in a more serious note, let's talk about technical things, because that's what this channel's for. Check out our Business Technicalities channel. We have more business videos coming for that. And we're almost to monetization. So just go do a bunch of viewing. So just go watch it and put them all on in the background. And the first level of monetization just gives you the joins and the super chats and stuff. And things like that. But let's start with the Trunass update. Jason's going to think about this one, the ZFS bug. Did you read up on it at all? I read on and off about it. And I came to the conclusion that I was unlikely to trigger it in my use cases and stopped reading about it. So the really too long didn't dive into the code of it is really simple. If you're copying files to your system, there's a check if dirty to make sure that that file's been committed to the system. It turns out, and there's a debate about the timeframe. Somebody said 2006 and someone said 2013. I don't know which one of these is true. Is it dirty commit under certain circumstances? Doesn't check to see if it's been fully flushed. And if before it's fully flushed, this is an absolute crazy amount of timing that has to occur. If before it's flushed, you try to make a copy of that file, it will grab the unfinished copy of the file. And that is where the problem is with the ZFS on there. So the likelihood of it happening is zero. I don't think anyone was ever. Zero. Not zero. Clearly happened to somebody. So it's just barely above a zero. It does require you to be locally using the file system though. It does not seem to work over a share. It doesn't work with ICE-Cuzzy. Yeah, there were very specific use cases that I saw people getting broken in. I think containers were potentially one of them. Yeah, because you make copy containers and things like that when you're building them out. So it's extremely unlikely this happened, but it's still worth patching. I have patched my systems right away because there's not many changes. There's no worries. They fixed a bug that I reported with their syslog having an extra character in the stupid. If you tried to put in an IP address to send syslogs somewhere, if you had an extra character, it would add. It was in the code. It was easy to fix manually, but now it's fixed properly. So there's a couple of minor bug fixes they had in TrueNAS. Easy to easy one to fix. The reason they took it so seriously, I thought this was just good about open source because you imagine this bug being in a proprietary vendor product, say I can produce it one in a million times, then it goes, yeah, closed. Yeah. And so enough people are reporting it and then maybe still closed. And maybe still closed, just closed a lot. But the team, if you go in the XC, I'm sorry XP, TrueNAS forums, people wrote scripts for this, people all started running our machine. It became a game of who could make it happen and who couldn't. And someone wrote a really slick script and then it became bragging points for how often you could run the script because it was also how fast your machine was. People were like, this man either reaches it once, which was kind of fun too. I liked the community involvement with open source with that, I thought it was just clever, but hey, it's fixed. And they take, IX systems does a lot of the code commits for the ZFS. They write tons of that, so they were deeply committed to getting it all fixed. And no one wants to have any integrity problems in ZFS. It's like the reason. Of course, and then there was an EXT-4 problem that we had in. Yeah, same week, right? Same week. EXT-4 data corruption issue too that had been persistent for a little bit. They had to roll something back. I think they rolled a kernel version back because of it. Yeah, so it was kind of, we had two file system bugs. None from ButterFS, by the way. ButterFS is going, ha ha. We always would pick on them before. ButterFS, the MongoDB of file systems. Yes. The MongoDB of file systems. There were a lot of people. The only people I think they can wrangle ButterFS properly is Synology. Yeah. Because Synology seal, but I've talked to the engineers there and they're like, it's not by accident that we're using MDAM RAID in the background and using ButterFS in the foreground because you can't let ButterFS control the hard drives directly. That's just bad news. Yeah, Alan Jude talked about it. He dove into the topic on two and a half admins two episodes ago. He had some really good points he made, but it's for people, the original trigger was someone running, I want to say Susie, and they were compiling a kernel. And when you're doing a kernel compile, I guess there's a bunch of file copying going on. There is, yeah. That's when all that happened. I mean, were they running it? Oh, okay, they were, this was on a local ZFS system. It wasn't on a NAS, okay. Right. In that sense. I thought there were some weird edge iSCSI use cases where you could potentially trigger it. I didn't see anything for iSCSI because it has to, what made it exasperate the problem was the block cloning. They've temporarily disabled block cloning because that would bring the race condition closer to happening. It would increase the statistically likelihood that this would occur. So it's kind of a weird, it's a very niche bug, but the amount of engineering time was absolutely huge. The number of people involved in this was pretty wild. Yeah, I'm trying to, I thought I saw some sort of weird way that if you were doing like DDo plus this, plus iSCSI, you could potentially cause it to happen, but I can't find any of that stuff now. Yeah, it's pretty niche. Now, on a fun note, we'll talk about the Unify update real quick. So we finally got this feature. Where did it go? Let's share it out. Are you gonna show the other feature? Yeah, I'll show the other feature too. So we have a more detailed admin activity. That is like their big thing they're selling is that they have this. The problem with... That doesn't, that doesn't look useful. Yeah, it doesn't. Well, it actually is. Hold on, let me switch to a client where I can see something. I think I can see in the VR, we can. I just wanna show our client names. Pull this. There we go. Okay, that's maybe a little more useful. Yeah, this is a little more useful. I like this feature. Click here to review. Thomas changed settings for your security. Click here to view. Oh, okay, nevermind. It's just dumb because everything's description instead of having a column for the username. Right. Yeah, you have to click on each one. So you can actually, like Kyle changed the printer front port to PoE. He actually changed it to name the port to, because he moved the printer in the office. I like it that they have all this. The stupid part is this still doesn't go over to Syslog. So this lives only here, even though I'm exporting everything to Syslog. I can't consume it in my sim, which is actually where I want it. Yep, I can't consume it the place where I want it. I can't create flags or rules around like, hey, when someone does a port change, that seems like something that won't happen often and matters. Now this is the fun one though. It does this now. Oops, there we go. Look, you can see the packets moving. I can see packets moving. So thanks to Unify for adding this. And games. But by the way, where's my, I have to grab and slide for everything. I don't have any. What's the over under on this screen showing up on like NCIS, like how long? I know my wife likes that. I like how it has all these loss devices. Doesn't know to do it because they're not in the same switch. It doesn't know how to deal with some things. Cause there's two different sets of switches that they don't have because we split our network, but super chat. Yeah, super chat. See, we got, oh, the SSH question here. There's another SSH thing we can talk about too. If you want to talk about spicy SSH man in the middle of text. Yes, the Tar Tarian. Is it called? It's got a weird name. Hey, Tom, is there, there's an SSH key issue in the news, Coby version and Truday scale. You have to add the keys manually for replication to work. And the newest, hmm, you have to add a manually or do you just have to accept it? Like, is it just a matter of it not accepting your remote host key and you have to manually add it or SSH by hand. So it adds it to the known keys. It's a good question. I'll have to set it up as a test on there because I haven't since I updated to the latest. Well, no, I did it since I uploaded 2310 and I had no problem getting it to accept. The only thing that's kind of stupid and I don't know why they did this. They don't have a way to accept the SSL certificate automatically. So you have to, you can, my, I'm okay with that. Oh, that they don't allow you to accept it? Yeah. Well, then it doesn't automatically set up. You have to tell it to, you have to set it to port 80. Yeah. And it works. Okay, okay, okay. If you drop the HTTP, you drop the S out of HTTP you can automatically set them up. So. It was more of a like, there are certain options where like just clicking yes on a key where you're probably a little looking at the thumb print. Like no one compares those thumb prints. Like requiring you to manually do a little bit of extra work when it's like important stuff. I'm sometimes okay with that. Yeah. I agree with that. That's the, I don't know. I need to do an updated video on how they do it. The, the auto magical way it works is nice. I can show people that aren't familiar with this. I'm going to actually going to check to see if it, I don't have any new systems to add a key to, but I can show my existing keys. So we go here. That's going to be under credentials. And no, not directory. Sure. I'm an idiot. Hold on. Local directory backup credentials, SSH key pairs. And it has part of my private key, but I don't care. Those are easy to roll. You can just regenerate the key pairs. What's that? The secret part of the private key, the rest of it's the same for everyone. You don't know that? Yeah. It's going to discover. Yeah, it's still working. Then the data protection. Yeah, these are all syncing still. So, and I just updated this yesterday to the, it came out yesterday and updated it. So it's all working on mine. All the, all of these use SSH keys to manage them. So, and they run daily. Actually, some of them run hourly. Oh no, they're all set daily now. So that still works. I'll have to look into that and see if there's actually a problem with that. Maybe, maybe I got to generate a new one and I have a new TrueNAS system I can load to test that. Let's see. Or is the other one that someone had for SSH? It was up, I think, a little bit. Yeah, go up a little bit. But it's the, what is that SSH bug called? Oh, I dropped that in our Slack yesterday. Yeah, I think we were both in the MSP room. You can pull it up real quick. Oh, Terrapin, there we go. This is the latest SSH bug. But it's pretty specific how you have to trigger it and it only affects, they have the man in the middle of you first. It only affects like cha-cha poly. I think you have to be using, it's down here somewhere. It's a very specific cypher you have to be using. Not all cyphers are even vulnerable to this. So it kind of depends on what cypher you're using. I'm using cha-cha poly, so I'm vulnerable. So that's the thing. Yeah, CVC cyphers are effective, but GCM cyphers are not. Yeah, the cha-cha cypher was the only thing of relevance there, because no one should be using CVC cyphers anyways these days. Right. So, but a lot of people on the latest ones are using the cha-cha. So that's definitely- Yeah, that came up during the ice alcohol yesterday. That's how I learned about it. And they asked how serious the problem was. And the general statement I made is, if you're to the point where somebody's able to do a man in the middle of its act, you're probably in trouble anyway. This is just one more angle they have for you on that. So definitely just patch, update your SSH. It's unlikely. Also, could exposing this stuff publicly? Yeah. I do everything over VPNs and then SSH, double encryption. Corey Thompson nails it. Another unifying management feature with no enterprise purpose. Yeah, we were talking about it before we started. And my guess was that this wasn't actually, it was some web designer just playing and said, look what I can do and then some executive goes, that's really awesome. And then it ended up in the code base. Yeah. Well, then the same thing. I mean, look at the, I gotta admit, I really like the RGB lights for personal reasons on the new switches. I mean, I can't, I couldn't sell that to a client. Yeah. And it's neat that they can label the VLANs with the RGB switches. Don't get me wrong. Very cool. For home users, it's, how many times have we in this month gone out to a client and needed to label the VLANs on a? No. It's just, once you set them up, it's a pretty rare use case. And I'm usually defining the ports like, hey, I'm gonna set port 16 to this and port 17 to this. And I plug the thing that goes in the port. I have VLANs so that I don't have to physically move cables. Yeah. So like the ability to physically see, like it's more the opposite is true. I plug this into port nine, and now I'm gonna put port nine on the VLAN it needs to be on. I'm not gonna light the VLANs up and pick the one that's the right color. Yeah. I'm gonna go with a no on this question here that someone just asked. I'll log into my system and tell you, but I feel confident no on that question. SNMP supported natively? Doubt. As a stepchild of all network stuff. Yeah. SNMP is what it is, but to my knowledge, there's no SNMP support in a dream machine. I mean, even the SNMP support on like the access points and stuff is not great. No, they have it. Yeah. I was surprised. I don't know how good it works. It's not very good. But so it's in here. I can't speak to you how good it works. So it is in here. I mean, it's just, I'm sure just open SNMP. Yeah. So how good is it? I don't know. Does it give you the information you want? It's the real question. It depends on the end how much trickery they're doing under the hood, right? So if they're doing, for instance, if they have a switch module, right? And they're just controlling the switch module and all of this stuff is done in hardware, then you're not going to get counters and stuff out of it most likely because that's going to show up to the OS. It's just a single interface. Yeah. Right? It's things like that, honestly, that make that difficult. I want to see if they have their video. Where'd it go? I guess they have it. They have it on their YouTube channel. It's going to pull up the video. I think enough for you have seen the video on the flashy lights thing that he does. Don't get me wrong. Techno Tim did a video on it and it makes his lab look awesome. Like, I think they know their audience who's buying these things. There is definitely a lot of enterprise use, but this the enterprise use for the flashy light thing? Not so much. Maraki has RGB today. Sure. No, yes. No, not this. The switches don't. I mean, they have an RGB status light. Yeah. And the color matters. Like what the color turns matters. Like when they're discovering the cloud stuff, they RGB cycle and then they turn white once they get a connection. And then they turn orange if they error out. I'll admit too. Yeah. Yeah, it's the front status light. Yeah. I'm liking this feature. I mean, I have full RGB control of my Unify, one of the best units that has that on there. I can change the color. Like that's novel to me. If your target is the gamer market and all their RGB sexiness, then. Yeah. I like that they have this feature. They nerfed it on a few of them though. What I really want is if I color this Ports View Lands green and this Ports View Lands red and once a trunk port does both, it better be orange. It's going to blend the colors. Yes. Yeah, if you haven't seen Techno Tim's lab, I liked it because he does it different. Everyone else goes with these really dark themes. Dude, he buzz everything in white. That is definitely, oh, he's got a video on it, but where is, somewhere's got a picture of his whole lab. Yeah, he did everything in white lighting. You can find his video on his Home Lab tour, but the fact that he did everything in these really light colored lights, his lab looks really cool. Matt is DMing me on Slack and I'm responding to him and he's like, don't respond to me, I see you live on YouTube. You must be watching. Yeah. Pretty cool on that. The next thing, because we're not going to be here too long, so we've got an event we're going to, certainly. Yeah, what Corey said is right. Yeah. What Corey, what do you say there? It's good enough. Good enough. Yeah, there's features missing that make Ovic angry. Yeah. It'd be nice to get more info. Wishful thinking on the trunk port there. Yeah. Just give me a separate color for trunk ports. That'd be kind of cool. Yeah. That's actually useful. Yeah, so I don't plug something into a trunk port. Or give me a brightness value that can be given by the number of ports or VLANs not pruned. So they start like gray and then they get brighter white the more things they have open. I don't know. Do you recommend running TrueNAS on bare metal? I can cut the sentence right there and say yes. Because every, the number of support problems that people have in my forums and under forums by virtualizing TrueNAS. So we were talking about the TrueNAS before this thing and hyper-converged, right? Sexy marketing hyper-converged word and that would be the one case where I think that I would not run it is if you really want to run containers and VMs then run XCPNG on the bare host and then run TrueNAS as a guest but I would expose the raw disks to TrueNAS and new device mapping instead of doing like pass through OS level stuff. Yeah. And if you have a full HPA cards you can pass through awesome. That is the least worst way to run it that way. That is how the Oracle, the ODA Oracle Data Appliance has worked by the way. Oh is it? That is 100% that they spin up a VM, they have a boot drive, they spin up a VM, all of the HBAs are attached to that VM via hardware and then they, it's not NFS there, well it is NFS in the end but they run ACFS which is their cluster file system on the top of it and then the volume is NFS mounted back to the host back to the down zero. Yeah, that is interesting. I will mention, cause I just finished this new page on my site and no this is not another invite for YouTube people to de-dost my site again which is what happened last time. I set up a whole creators we love page. This is a lot of the YouTube people I follow. I wasn't, I was on the mission to do thumbnails for your channel and I realized their channel's name thumbnails too much and I didn't want to deal with it. So there's a list of YouTube channels. The more static is the podcast I usually listen to, network related podcast, IT related podcast. So these are my usual listens. Like I'd mentioned two and a half admins earlier, dark net diaries I think everyone listens to. Yeah, Jack Reach Center kills it. Unsupervised learning, risky business. I love Patrick. He's just the perfect right amount of Australian snark talking about security and the art of networking hit and miss but cause I don't listen to every episode but they've had some really good episodes with different network engineers and they're talking about things. So everything from Cisco, I get to learn about a lot of the real enterprise stuff that maybe I don't see because I'm not in the Juniper world but it definitely a good podcast if you're looking for some recommendations but head over to laursystems.com you can find those, find those on my site. Let's see, Matt's wondering where we're going. Atom are going to the IT and the D. So that's the thing. Question, Unify Express or Nekate SG-1100 for home with basic VLAN segmentation. Ooh, good question. I, so move this real quick. Some people may have noticed the thumbnail I chose is a Unify Express. I'm testing it. I've had it in my lab set up and running data through it. I like this little device. It fits in here. It's thin, it's small. It's basically a mini UDM but it's like 100, I think it's like 150 bucks. It's got the controller built in. It supports up to five devices. It's got Wi-Fi built in too. So this is your Wi-Fi device. And yeah. Does it play nice with the rest of the APs? Yeah. Then that's useful. Yeah. So I think for home users, this is a killer device. I think this is great. And I do like the Nekate, my downside of it. The way Unify does firewall rules is atrocious and that's where I become a little bit mixed. It's not too hard. I just think it's atrocious because I'm used to something that gives me more logical screen for rules. They do the thing where you can't just type in a port. You have to create port groups and named ports. And then for reasons I can't understand, when you go to mouse over them, it doesn't tell you what the ports are. Everything's too quick to see what the rules do. And I'm like, this is stupid. But that's me setting it up. Once it's set up and if you don't have too many rules, you may not care. Yeah. I mean, if it's basic, yeah. If it's basic just routing, then sure. Yeah. Either one? And for home users, most of the time you can build, you can just create another guest network and you can actually create multiple guest networks. The nice thing is as you create guest networks, they're automatically segmented from your network. So you can say, here's my IoT guest. Here's my actual guest network. And here's another guest network for my kids stuff. And then your stuff is on its own network. And that is a few clicks to do. Hands down, I couldn't sell you something easier to do to make that. Oh yeah, this is another thing that Willie's pointing out that, yes, it's important to note he can only manage four other unified devices plus itself. So when they say five, they mean one, two, three, four. Why? They just have that as a limitation for how many devices you can, I think it's because it's so low cost and low powered. That's my guess. They want you to buy the big boy machine. They don't want to, it's one of those things like, you don't want to make a product that- I can borrow that so I can make it do more than five. Yeah, I know. But yes, as soon as I'm done testing it, I'm going to SSH into it and I'm sure I'm positive that this limitation can be changed. So, I mean, five devices for something that's this cheap I think is fair, because once you jump up to the UDM, it's like a couple hundred devices. So, and UDM is not outlandishly expensive compared to if you're buying 10 more devices that are attached to it. Yeah, what's the use case for it? I mean, that doesn't, it only does network devices, right? So like, typically you would have that plus a switch, plus two APs maybe, right? Three APs? I think it's the ideal home user stuff and now they don't have to figure out where their cloud key goes. This is the cloud key. So, my review of it in case anyone's wondering is going to be very positive because I actually think, like if I want to recommend something to home users. Yeah. That, do you know what else it does? Auto updates out of the box. Would you recommend it to small business users? Maybe, I don't know. Yeah. I'm mixed on it, but honestly, for, here's my problem and sorry, as much as I'm a PF Sense fan, PF Sense does an auto update and this does. So, I'm big on things that auto update because if you're not going to manage it, you're not going to pay me to manage it and you're not going to manage it yourself or you're under the mythos that you'll actually log in and update it. You're probably not going to. So, this thing by default, unless you turn it off and I like defaults doing it because here are any of the fault. No one changes the stand defaults. That's a whole CISA thing. Secure write default, secure write design is only a thing they're pushing right now. They do that. This auto updates out of the box. That. They just did that. So next year when California requires that they don't have to go back and push new updates down to all of them. Yep. Yep. And it's also an access point. It's nice that it works well with the other access points because some of the other things like I'm going to pick on Maraki here, you can buy a 68 CW, right? Which has cell and it has wifi in it but the wifi is completely in a different walled garden than the rest of the wifi stuff. It won't mesh with it. It won't, it's just, it's not, it's gross. Yeah, it's a different, if it's not a compatible product line versus this works with all the same things that you can get. And I was actually thinking, I'm going to put together like a budget idea kit. This plus like when eight port POE, which I think is about a hundred bucks. Yeah. Does that have any ports on the back? Just one. Okay. Yeah. So it's not going to do, the downside is this isn't going to do failovers. Too small for that. No, you wouldn't want that. But yeah. Home users aren't going to be able to get prep addresses and stuff anyways, typically. Exactly. So we'll be able to do that. It's not, it's this, it's a budget use case. So. So 90% of the cases, that's the only device. Yeah. Inferno part, if I was sitting in a small apartment and I need something better than the consumer garbage that's handed to me from the ISP, I'm holding it in my hand for, I think a reasonable price. So ubiquity, didn't they just come out with a cable modem? They did too. Yes. So, you know, here, steal our idea or my idea, that with a built-in cable modem. Yeah. Why aren't they making that too? Can I use the Mac mini for PSense? No. Well, actually, wait. Yes. If it's running the x86 one. Correct. Well, you can't, there's not an ARM compilation. There could be. It could be done, no one's doing it. Yes. The answer is yes, if it's x86, probably still yes, if it's M1, but it's a lot more work. Yeah. It's a lot more work. How do you manage and set up the UDB keys and Linux? I went through learn link CV video and it seemed command driven. I see the accounts saved in the key. There's no way to manage UB key with a GUI. Yeah, there is. No, not for one Linux login. But the UB key manager runs on, there's no Linux version there, right? But if you want to use it for PAM to actually use your logins, there's not a, that's the downside. I don't know PAM plugin for. Not understanding UI for it. There's a PAM plugin, but you have to go into PAM config and actually add the key. Okay, okay, okay. That's what they're talking about. They haven't, that I know of, no one's built like an auto install UI tool that sets that up. So Jay did the video on it, because I did a similar video of how to do it with SSH keys. It's cool, but actually one of the dumb problems is I actually learned you can't do it for Windows subsystems, Windows subsystems for Linux doesn't support it because for reasons that Microsoft won't answer, they turn the feature flag off in SSH to add external keys and you can pass the USB through to you Windows subsystems. So that was gonna be the question. Is this say you can't pass the USB device or I mean, in all honesty, if you're to the point of trying to pass through USB devices, you're probably in the land of you can figure out how to recompile SSH. Right, but why do you have to recompile it? I don't know, I'm not gonna worry about that. How many Unify APs for a 4,000 square foot house? You know, I need to do an updated video on this. It's all depends on what's in the damn walls. Is it plaster? Is it plaster? And is it the old plaster of Paris or is it the new drywall? Is it the metal stud construction? It doesn't have the like mesh, the birdcage mesh. The birdcage mesh? Yeah, if you have the birdcage meshed up, I'm sorry. One per room. One per room. Each room's a fairy cage. Fairy cage. Two, four might go through it. Five, eight, probably not so much. Yeah, so that's definitely hard to say there. I had one AP, sorry to interrupt, but I had one AP in my house on Grozio. Yeah. That was 6,200 square feet. That's cause your house is cheaply built. No, it was not. It was just built up. It was just built up, yeah. So it's been in the middle. Yeah. Yeah, you can get away with them. The Unify APs, the LRs, I am shocked. I'm in a basement. My AP is in the rafters and it reaches outside through the brick, through the porch cement. I can go in my backyard and it works. Absolutely amazed at how well that works. So that's things. What do we think of Cisco Duo? Works well. Yes, it does what it says in the box. Does what it says in the box. It doesn't work for WMI or remote, but that's pretty much all of the tools in that class. Yeah. One thing to note and to my knowledge, this is still a thing. It fails open by default when you install it on Windows. Yes. You can make it not fail open. It's just, you have to switch your registry change. You, and caveat there is if it fails closed and you need to be very careful with that because for instance, say you're running Hyper-V with your firewall on as a guest and your Duo on that and it fails closed, then you can't actually log in to start your firewall if it turns off. Yep. So understand the caveats of failing close. Failing close could be bad. Similar to Bitwarden under password managers actually. Where does the other half of that question? Yeah. I think maybe you were the good, I don't know. I like Bitwarden. I don't know the rest of your question. I wanna be able to manage accounts in the key like I manage accounts. He wants the manager thing. And I think that's your support on Linux. It is. The Ubiqui manager. Yeah, Ubiqui manager's in there. I think that's fine. Yeah, absolutely. Firewall as a Hyper-V guest sounds like a nightmare. Just it's a nightmare right after the word Hyper-V comes out. Yeah. I posted yesterday on LinkedIn about the VMware changes Broadcom made and are my dislike of Hyper-V and all the MSPs that I'm connected with. They love Hyper-V. They started bandwagoning. Why do you just use Hyper-V? And it's like, cause it's a piece of trash. Well, both of us have done, our experience of recovering with Hyper-V has been terrible. It's just so much work. We had a client with a big, one of the biggest Hyper-V clusters I've ever seen. And they pay for like premium support with Microsoft. And they finally gave up with Microsoft and called us. And to which I told them, I don't have a solution for the problem he has. He is only able to access, he has to create a new user and that new admin user has 15 minutes that they're allowed to view a session in Hyper-V to get the screen to work and then the screen goes black and stops. Microsoft support was all over. Their answer was to rebuild his entire cluster. He's like, yeah, no problem. I'm moving to another hypervisor. I am not rebuilding how big, I forget they had like six or seven hosts, tons of virtual machines. He goes, I'm rebuilding this and reloading everything. Yeah. I mean, some of the bugs are just so weird. There are very, very large cloud-based VMware providers out there, huge, giant ones, right? I know of nothing similar for Hyper-V, right? And to my knowledge, Azure runs on some sort of Linux back-ended thing, right? So even Microsoft isn't using it at scale and they're really pushing Azure HCI, right? They're really, the writing is on the wall there, I think that it's more of a toy. Yeah, it's more of a toy. That's gonna totally get, everyone's gonna let a fewer comments on that. Yes. The MSP is finding that word dead. Here's my favorite thing. So this was posted, someone asked this Hyper-V dead. This is the Microsoft forums. This is a Microsoft employee response and I love this response because it's by a Microsoft employee. We decided, and because we simply don't have the time and resources to keep breaking the free version because people ask why they killed the free version. They have a market cap of $2.7 trillion this morning. Their cash on hand on their September report was, I think it was 2.5 billion cash on hand, but we don't have the resources to keep giving this away for free. Yeah. It's a support issue. Yeah, they don't wanna, yeah. And I only expect them to raise the prices more in it. I have no love for people who wanna, if you wanna go on Hyper-V, go for it. I'm not with you on that. Can I have PF Sense and Unify Network? In the same network, I want PF Sense Firewall but I like the Unify Network. Yes, I have a ton of videos on even how to set up VLANs with PF Sense as your firewall and Unify as your networking. It's a fan favorite here on the YouTube channel because I talk about it so much. So yes, you can. Unify Firewall and PF Sense Firewall. I don't know why you'd want to, but because enough people asked, I think there's 70,000 views on the video where I explain how to make them both work on the same network with a double NAT. I don't know why you wanna do it, but enough people wanted to do it that I made a video on it. Just a double NAT and then crying inside. Yeah, I don't know why people want double NAT. I'm wrong, just so I get corrected here. Newer versions of Azure, it is Hyper-VL, so. Oh, it is? But it's not the version of Hyper-V that's in Windows. It's like a ring zero Hyper-V. Yeah, well, the other thing too, what the Hyperscale companies run, like the argument of whether or not they use KVM or Zen, which they use both that run the backend of AWS, you can't call them the same Zen you have. It is so customized, it is so rewritten that it's not the same. It's not the same tool that you're actually using. So Microsoft killed the free type one hypervisor, but not the type two hypervisor, but on the backend of Azure, they're almost certainly using that same type one hypervisor. There's not packaging or public use. I would say it shares a code base with it. It's not the same, they're not just running a bunch of Windows 2022 servers. It's not all that works. Let's see here. Oh, this is someone after Jason started. Give me a CLI, can you switch any day much faster than clicking through a million GUI options? Especially if you have to do the same thing a bunch of times. Better yet, give me something that I can run like Ansible against and I'm totally happy. Yeah, why can't we just automate this with Ansible? Yeah, see a Rista. That's actually one of my biggest complaints about PF Sense is that I have to do- Everything in the UI. In the UI, and I'm perfectly content running. I know my IPFW commands very well, but if you run them, they suddenly disappear next time it updates all your firewall rules are on. Yeah, you have. So each reload of PF Sense, it was funny, I used to do an advanced class at this Linux event every year on how to hack away at the config.xml file. That was the solution was pushing everything config.xml. And then, yeah. So it's, you're right. But for people that start with PF Sense, that was a common complaint. There was this old crusty BSD guy there with a big beard, fully long gray beard, one day Jason. It wasn't me. And, but he always had that same gripe. He goes, I can do it from the command line and I know BSD commands and I just do it that they should make it automatically save what I change. I forget it. And usually, I would never update the IP firewall rules because the chains it makes are actually stupidly complex. It's like running firewall command or firewallD on a red hat box. Like, you're not, once you do that, you're no longer manually managing IP tables. See, it's insanity. But I will occasionally log into a box and add a static route that I need and forget to do that. And they also go away as soon as it reloads. Yeah. Oh, let's see. I got a couple more questions because we're gonna be taking off here in about 10 more minutes. What is the best way to set a backup for Churnas? I just built my primary server. You're looking to build a backup server. Replication. Yeah, build a second one. Build a second one and replicate it. Put it somewhere else. And if you're looking for a cheap way to do it, Brian Moses has a, if you look at Brian Moses Churnas, he does these budget builds. Matter of fact, he's, he bought a bunch of these Topten boards. They actually work really well. They're raw motherboards from Topten and he has them on his eBay account. Cause he bought them all like from Ali Baba and then he was eBaying them for what he paid for them. Don't use the solid states they come with. No, no, no. Well, these are just motherboards with six data on them. So they were low budget built in two and a half gig micro ATX ports that were half the price of like ASRock. So there's different ways you can do budget builds. And for doing offsite set up tail scale, I got a video on tail scale. You load it on two different Churnas boxes. Now wherever that Churnas box wanders, wherever the IP address changes doesn't matter, the tail scale will automatically figure out where the things are and replicate the data between them. So make a friend that also has a Churnas box and put your backups at each other's house. Yes. Yeah, we're talking about having a buddy system for that. Wendell from level one techs and I have all the extra storage. Wendell also 45 drives, send him few of them. He's got even, I think he's got over a petabyte. And so we're like, why don't we back each other's things up as YouTubers? Yeah, as YouTubers, why don't we start backing up to each other and save ourselves the cloud fees? Yeah. No, I mean, my struggle with that is your uploads not good enough. Yes. Yeah. And the, I'm the, because I'm not using an ATEM, some of the other guys who are using like the ATEMs, they're producing, if they do a shooting day, here it comes about, oh, I don't know, 400 gigs of data for one video shoot. Yeah, that can be a lot if you have all the recordings. It's a bandwidth problem. If we were closer, we'd just do it all site to site. Oh, good. Someone else, you have a LinkedIn file or Jason, love your videos on LinkedIn. Jason's been doing a whole video series on LinkedIn. He's easy to find. I had two mine yet today. If you look for Jason Slagle, he's a easy guy to find. You gotta do it while you're doing this. Same time. Kill two birds with one stone. I'm gonna start doing them portrait mode, even though I hate portrait mode videos, but somebody pointed out I should really post them as shorts on YouTube. I'm like, oh, why am I not doing that? Yeah, they are popular that way. Shorts. Yeah. Just doing restream, just drop it up to restream and let it post. I don't know if restream supports doing shorts. I don't know. This is one of the last things I'll mention. I have a whole video coming out on this as a topic, but these are the new prices for VATES. This is the XCPNG bundle prices. Yeah, we had a spirited internal debate about this. Yes. And by the way, I would pull up the VMware pricing, but it's all been nuked. Yeah. VMware doesn't have a price, but this is, I'll drop this as a link. I can get the VMware pricing. I just logged in the partner portal, but yeah. Well, there's some confusion around that, according to people in VMware, when that it's not, it won't let you commit something. I don't know. I don't have a partner login to look at it. It wouldn't let them commit it or something like that, because there's some hang up on it until the powers that be at Broadcom bless the new pricing. Yeah, that could actually be the, I was actually... They won't let the deal go through. I was actually expecting for me to log in and then... But this is a discussion we were having just prior to going live is we obviously are push. I have talked a lot about and we've got clients using XCP and G. I think it's absolutely the best alternative to VMware. As far as when I look at the virtualization landscape, it's what we're going to be pushing a lot more of now with VMware changes. Yeah. I mean, it's awesome. There are... I will openly say I am not the biggest fan of the UI UX, but Tom and I have had some pretty big discussions about that. I'm just used to the VMware UI UX. The Proxima UI UX is very similar to the VMware UI UX and the XCP and G X01 is completely different. Yes. Not everybody likes that. I like it, but I'm me. But I'll actually pull something up because people are probably wondering what this looks like. The... Whoops, where'd it go? Not that one. This one. This server, here we go. VMware has pricing on their site, on that partner side. So we should make a betting pool on the over-under of them walking this stuff back because eventually the statement that all these companies are locked in and everyone is using VMware so we can do whatever we want. Microsoft tried that and it didn't work. It worked for a while, but after a while you start to erode the knowledge way because new people aren't learning this. Yep. Yeah. It's a short-term, not a long-term play. This is the new XO Lite and the theme that they built for XO Lite is the new theme that's coming in a new version of Zen. So this is better looking and Jason probably thinks this is probably more acceptable looking. Yeah, I wouldn't like that. This is, they're gonna release this probably Q1 of 2024. Yeah, and I'll pull a regular XO up so we can compare and contrast. Yeah, I mean, I want the left-hand board to not be dumb. Yeah. Ooh, look, I got updates. Yeah. Let's install. There should be an easy way for me to look at a pool and the VMs under it without having to go through 40 clicks to get there. Like that? Yeah, well, I mean, that's four more clicks than it takes me. How about the pool? Like, do you have a shared cluster? Yeah, you can, yeah, I'm gonna clear this one. You can do it by pool. Okay. Yeah. And then, yeah, okay. So it's just, it is a difference in management concept. Yeah. Like I'd rather have, like, let me set that left-hand bar to like the stuff you have along the top there. And that's how the other one is, honestly. Yeah. I like that everything's copy and paste when you're in stuff. So if we go into Eric's, like all these, you just click them and it copies the clipboard. That stuff's handy. There's a lot of little elements. Yeah. I mean, I generally like the copy and clipboard type stuff. More places are having it. I think that's nice. Hey, look, all the patches loaded already, you know. Way patched on. The patching's so easy in it. One thing I'll say, and this is my video that it's, it'll be published tomorrow. I uploaded it, I don't got time to, maybe tonight. One of the things I really like about the patching system, so I started all the way back in 2018 with version seven of Zen, or is XCPNG, because before that I was actually using Citrix. And if you dig around on my channel, you'll find old Citrix videos I was doing. Then Citrix kind of screwed up and then they launched XCPNG. Kind of? Kind of screwed up. There was a great, let me count the ways. I know. Well, there was actually a talk of by, I think it was by a former Citrix person. It was at one of the big open source conferences in Europe. It was how to destroy a community. And it was all about Citrix. I mean, they, I think they kind of ran like the SUSE playbook too, a little bit. There was a whole lot of like, yeah. It was just a mess. Well, what happened was they went from 7.1 to 7.2. And when you updated, and as always, there's a Yula to agree to, you actually agreed to downgrading all of your features behind a license. And everyone's like, hey, all these things just broke on my server. Yeah, that's the, you have a license, but that's not the license that supports those features now. Somebody should send them to semver.org. That is not a minor version. That is not a minor version update. By definition. A major version. Yeah, you deprecated features, but I'm behind the license. But I've been using XTP and G all the way since seven, their first release, I think was 7.2. It's been a couple of years now, but I have consistently loaded update after update without breakage, which is stupidly amazing to think that I have not broke a single machine with updates. I've had machines that when you rebooted and they didn't reboot because they had a hardware failure that we didn't know about. I've had those type of issues, but as far as the updates being the cause of the problem, that has not been the issue. And that is amazing to me. Yeah, we've definitely not had that with VMware. I've definitely not had that with PF Sense. That would be a very rarity for that to be the case. There's very few companies that I feel comfortable with updates. And by the way, if we don't have that with TrueNAS, I mean, come on guys. I was just showing they use the wrong field type on a password. There's always some little, I always trust TrueNAS to store things properly. I don't trust their VM system to work extremely well. I don't trust their applications not to get stuck deploying in a circle randomly. I don't trust them to give me good Kubernetes logs. That was a weird one. I had one of them broke. I don't know. My answer was just delete it. I deleted the whole Kubernetes thing and rebuilt it. Yeah, that's pretty much how most people troubleshoot Kubernetes. That's how you troubleshoot Kubernetes. That's... Well, I recommend not using Kubernetes unless you need to because that is the accepted troubleshooting method. It's basically your Hyper-V problem. Just rebuild it all. It'll work. Yep. Sunburst was a... Some... Sunburst was SSL, wasn't it? Let me look. Yeah, what was Sunburst's vulnerability? That one's not in my head right now. I've heard, I remember, but I don't remember who it was about. I thought that was... SolarWinds. Yeah, I thought that was SolarWinds. Yeah, that was SolarWinds. Yeah, okay. That was SolarWinds, not VMware. And by the way, go read the SCC stuff on that one. That's... The SCC's got them in a lot of trouble. Yeah, the licensing. They have no licensing. There's no license. There are no price lists to download right now, only academic. Oh, really? Yeah. So VMware officially has no pricing. And I don't think that's because they're getting ready to offer everyone a big discount. All right, well, this winds it down. So we're gonna head out to IT and the D. Yeah. Thanks everyone for joining us. It was fun. Check out our business technicalities channel. And me and Jason got to do these tech support ones. Yeah. Sunburst was possible via VMware vulnerability. I don't think so. It wasn't. No, there were several that were Horizon. The one for SolarWinds was not. It was something else. It was a series of stupidity because their keys were in their GitHub. Yeah. And that was where they blamed the intern. Oh, the intern must have did that. Yeah, cool. You have an intern running critical code base. Nice. Oh, man. You know, that's actually, I'll share this one last thing. One of my favorite stories was a new, fresh out of school database developer. So they had a developer role and they were given, they got the job, sits down at the desk. This is just a wonderful Reddit post too because he had a lot of encouragement from the Reddit community that it wasn't his fault. But because they said, no one should have given you full DB admin privileges on day one, but they did. Everyone does. Yeah. I mean, they shouldn't, but they do. They handed him the training documentation. So what he did in the training documentation for how to move things around was start copying and pasting it into his terminal back and forth. Turns out the training documentation was built on the production system. So when he started copying, pasting things, it blew the database up. And he didn't know it was actually doing all the commands because he didn't even realize he had full admin on everything right away. He thought it was like a test environment that he was in. He just didn't know. He says, I was my first day. I was so nervous and so excited. I got a new job. And he goes, I also got fired that day and they're threatening to sue me. What do I do? I'm terrified. You know, he's just young and they're like, dude, they are morons. They will lose the lawsuit for handing a guy on day one with no previous experience on his first job doing it. He got a lot of encouragement from the community, but he felt terrible because he not only broke the database, they found out their backups weren't working at the same time. So they couldn't restore. So it turned into like, you need to go home because someone's gonna kill you. Yeah. Then he threatened, there was something like they had a lot of legal threats, but it was funny because this just comes down to like just don't give these people day one. That's a really good business technicalities talk because I think that you'll find, especially in our industry, the MSP industry, that's actually really common. Like we're undertaking a really big push right now for role-based access control, lease privilege and every single MSP tool we use, but it's a journey to get there and a lot of MSPs and businesses, you are an admin. You just get an admin account. I mean, most of them, you don't even have a separate admin account. Your account is just a domain admin and that is definitely, yeah. And I noticed, you know, in contrast, we do things a little different. We should probably, maybe there's a little, after tomorrow, I'm doing a lunch and learn with the CNWR team. We had a security event that happened last Friday and all of our tools actually worked perfectly fine. So that's why I didn't do a video about it, but maybe I'll do a video about what happens when all the tools work perfectly. They didn't. They didn't work fine. No? That file never should have been able, like AMZ should have caught that file. It never should have been able to be executed. That is a huge miss on standalone because it doesn't AMZ. Okay. Then we should probably debrief on that. I was, I still have the file. So we should play with that some more. I was the first person to put the hash in. I got, I talked about this on ASL, I got shaded through every word, the download button in Bing. Like what a dumb moronic idea. Yeah. They downloaded it directly from Bing to find this file. But one of the other things that had happened was the, we had sent a one prior to the merger with CNWR. So not all of, because they hadn't been trained on it, not everybody had full privileges to actually act and send the one, because we actually do take the time to train the people on them and not give them full admin privileges to things. So that's kind of a journey. We've now upgraded people because we, we did it on the fly because the, it turns out some of the people that were in there didn't have as much access to do the investigation. They were able to isolate and get, they contain the incident. They just weren't able to do the investigation. This is a thread sighing stuff. Yeah. SolarWinds has not disclosed initial access for the breach. Oh, okay. They said it was either a third party, password spraying or yeah. One of those. Okay. I wasn't sure exactly. They had a lot of dumb, I was reading some of the SCC stuff. I just researched it while we were here. Yeah, it's a, we were driving down the road for sure of least privileged access for our people. Yeah, and we're still, there's still a constant level of internal auditing on there about it. Yup. I had the least privileged of all. No, we still have access to Ninja. Oh no. And I do not. He does. He's read only. He's always gonna read only. We give him read only because that's how he would do the billing. Yeah. I don't have access to a bunch of our tools. I don't even have an account. Like you can help us buttons. Don't even have an account. Don't want it. One, it's one more thing I have to do. Right. And two, like I shouldn't be doing it anyways because I'm not gonna follow their process. So if Matt's in here, he's gonna say something along the lines of damn right. I like this. Just in time philosophy of only using credentials you need to have extra steps to different credentials. Yeah, and we actually have that. We should probably talk about some of the tooling sometimes. That's very technical. Auto elevate. Auto elevate's awesome for that. Yeah, there's, that's how we handle it now at CNWR. Even the techs. Even the techs. Like that's a huge thing, right? Like our tier one and tier two, and I think even tier three techs, they do not have local admin rights on their PCs. Our techs do not have local admin rights. Fight me in the comments. All right, we're gonna leave that there. That's a good topic though. We should, this is a good, this channel topic of diving into exactly how we handle all these different controls. I think that's a, that's something we'll talk about. All right. Thanks everyone for joining and we're gonna go hang out with a bunch of IT nerds. It's, I love the title event. It's networking one beer at a time. So that's what we're gonna go do next. We hope you can drive home. But I'll be fine. I, yeah. The question is if I'm gonna ride with Tom there or if I'm gonna drive up separately. Yeah. It's directly, I don't stay late. So we'll figure that out. All right, man. I don't want to stay late either. So that works. All right. Later everyone. See ya.