 Welcome to vlog Thursday, number 375 with Jason Slagle. We're going to talk about some tech debt. I figured I'd throw that in the list because Jason wanted to join me on the homelab show for this topic. Yeah. I know he has a lot of opinions on it. Uh, super busy complaining about offshore vendors instead, which is also very relevant. Offshore vendors have their risks associated with them. Yeah, it's more, uh, there's risk patterns that people don't consider, which is if you're dealing with an offshore agent that makes $3,000 a year or even $6,000 a year, how much money does it cost to bribe them to give you what you want? Yeah, well, this, this topic, um, has been played out many times. I mean, I worked in the automotive space and it's because there was less internet news back then, but circa 2002, we'll just say one of the very large automotives, um, got caught in that predicament. They had the entire credit database offshore because that was the cheaper way to get programming done in the early 2000s. That was the first rush of let's put it off store and the company quickly figured out the contract was not that lucrative, but stealing the data and identity theft was very lucrative. So yeah, these are certainly an issue reports of echo. Is it my echo? I don't hear an echo at all. Yeah. I think someone else has an echo. Yeah. Oh, you know what? The reason someone has an echo is they click the live stream and somewhere else to have a tab open with their live stream and they started at different times. I have certainly done that and thought someone had a, you know, cause it's like, there's so many tabs open across. I have like three monitors. And one has groups of tabs open. Find the sound. Yeah. I have three Chrome windows that probably among them have 200 tabs open. It's pretty bad. Yeah. There's, I like that. There's no tooling coming out to help group our tabs to some rational. I am the problem here. We're not going to solve the problem of closing tabs, but maybe we can group them into projects. Well, every once in a while, I just right click and go close all to the right and go yellow. Yep. I hope those are important. I've really started putting a lot of stuff in log seek. That way I can stop doing that. And so I have it and being more concise and purposed about my research. But, but of our technical minds and the everything is shiny. We got to click on it and I'm going to read that later. Lists we make it's hard. I'll bring this up because I've seen a couple of comments already right. We'll just leave it here at the beginning. Yes. True. Nascor is getting sunset. Essentially. When does free BSC 13 reach end of life? I, let me look. That's going to be essentially, they said there's the two long didn't read the articles and stuff is they're not going to go to free BSC 14 with your next core, which means the sunsetting is going to be more related to the sunset of free BSC 13. So whenever they stopped doing security patches for that will unless they changed your mind, but they at least have Chris Moore made a statement like this is as far as we plan to go with core. There is running into too many issues and not enough and Jason's actually a long time BSC guy. So he understands the issues probably better than most people when it comes to the BSC community not being as big as it was or active as it was not enough people involved and yeah, it's expected. So 13 the 13 stable train support ends in one year in 10 months. So basically January 31st, 2026, but so true. Nascor is going to 13 1 13 1 mainline support ended seven months ago. So I don't know if they're back porting the 10 to and 10 three security issues at least. Probably. Yeah. But yeah, being being that course based on BSD it's it's the foundation understanding on it's not that they, you know, someone's like, well, they should just I always like those they should just like donate resources and completely update and update all the kernel for BSC. I'm like, yeah, they don't have those kind of resources on people. People realize how much work that actually would be. Yeah. And I mean in free BSD that again as you pointed out long time free BSD user still although I'm not very active. I'm still on puppet at free BSD.org. If you email that you I get it one of the members the amount of even mailing list activity on the chat hackers lists over the past years is way down. Yeah, isn't isn't what it used to be. I'll be doing some more lab on this because we actually have some internal stuff we are going to be converting. But yeah, the migration path is easy to get from there. They've they've purposely made it really easy. They're still using boot slicing into my knowledge. I'll be testing this with the 24 version which the release candidate just dropped about an hour ago. And the far as I know the it leaves a boot slice. So if everything goes wrong, you have a rollback point. So you can figure out how to do it again and hopefully get it to go right. Yeah. Yeah. I mean, I as long as it imports my biggest concern which I echoed you yesterday was like I have some of these that have nonstandard networking configs like is it going to pull over my 9000 MTU because if it's not, it's going to have bad time. Right. Right. It's that's all the validation testing we have to do is upgrade it. You know, do we have the proper because the bridges don't work the same of how they set up in BSD versus how they set up in Linux. So how will it translate those? Those are real questions. I have mine on independent network cards. So none of my upgrades were problematic at all, but I just had a network card with some VLANs on it and actually I had multiple network cards within different subnets pretty minor. So I probably wasn't the best test bed for that, but I will definitely I can load up another TrueNAS core system. Set it up the same way Jason has the production one set up and we'll just try it. And that's the that's the benefit of having a lab. Yeah. I mean, and I have lots of them and we can that this TrueNAS box that I messed with now, although I won't take it back into production here pro tip. I inherited one that used a Dell perk RAID controller is H710P and the num nuts that set it up basically set it up as 12 single drive RAID zeros and presented those to TrueNAS. Not a great time. Do not recommend learn from learn from whoever I inherited this from mistakes. Yeah. Put put the controller in IT mode or walk away and sometimes you have to reflash those Dell controllers. There's a ton of forum posts on which firmware matches which one. There's a big discussion in my forum where someone has the whole write up on which firmware supports which model. Yeah. And then because they're Sash drives, right? We had a double drive failure in it, which I can recover from because one of the drive is only predictive failure. But anytime you're working with Sash drives, data recovery becomes like considerably harder because you can't just attach them to things. You got to have you got to find specialized controllers that can actually talk to the things. Yeah. So definitely not always a fun time. Speaking of which, we will this is what I'm doing right now. I just dropped a few minutes ago or at least showed up in my controller, not my controller, but my TrueNAS here. This is one of the lab systems. This is the HL15 and I'm going to move it from 23 to the release candidate. So there we go. It'll do that. It'll probably be done by the time where before we're done with this vlog. Apply pending updates continue. I'm glad they fixed this. There's a flaw they had that I pointed out that was really silly. It's one of those security things that you're going to laugh about because it seems that you could see how this could be missed. So they had a password. Anytime you want to download the config file, there was a password to say, Hey, give me the root password download config file. I pointed out in the forums where I think maybe I opened a bug with it. I said, did you know if I go to upgrade and switch release trains, it auto downloads the config file. I was like, you may as well just remove the password or require a password there. But if someone's trying to get your config file, it's actually not hard to get. Yeah, I think that I seem to recall it only requires you to put the password in if you're using the encryption stuff because even the normal upgrades was letting me download the entire upgrade train. You don't even have to switch trains. You can just do an upgrade. Yeah, that upgrade thing prompts you to download the config and there's a checkbox in there that says like, is this encrypted? If so, I think you have to provide a password. Nope. I checked that box. It says it's downloading the encrypted secret seat. Yeah, they've eliminated it. This still exists in core, but does not exist at all on scale, which I'm fine with because if someone has root level access to your TrueNAS. Yeah, it's who cares if they download the database is also my other answer to that. Like if once they're in his route, they can delete whatever they want. They can see what they want. This has been a common complaint people had. And I don't have a solution for this that people like, well, they can go in if they have root and see my cloud credentials. I'm like, yeah, I don't know. Don't let them have root. Like, I think you're misunderstanding what root means. Yeah, that's how I feel like some of that. I gave somebody all of the access. I gave somebody keys to all of my doors and how I'm angry they can see my dishes. You know, of course they can. I think it was all the way back in 2018. I was on. I was with Kyle and from Huntress and it was when he was still doing the Tech Tuesday things and I was talking about backups and saying, hey, it's really good when you tie something and I used TrueNAS an example, you know, don't use the same credentials as you do your ad. You don't really want ad log and you want to keep that separate. Someone told me I didn't really know what I was talking about and that didn't matter that much. And I was like, even Kyle's like, no, these have to be separate control planes and Kyle's like why and then three more people started in the comments. Yeah, you guys are wrong about this. And I'm like, we're wrong about keeping separate control planes from where your data plane is versus your control plane. Like, yeah, I mean, I honestly, I'm okay with you joining it to 80. There are valid reasons to do that, but I would not have the admin users be active directory users. Right. I've always pled that to be separate because one of the things really cool in TrueNAS is snapshots present as shadow copies to Windows systems. And unless someone has root level access, they're immutable, which is actually really great. So if a ransomware hits you, it doesn't mess up the VSS when they're pulled off of a TrueNAS share. You can actually still be those because Windows only gets read only access because they're controlled by the snapshot policy on TrueNAS, which is great. It is saved a number of people. We have a, we haven't talked to them in a long time. It's a large city in upstate New York that we sold TrueNAS to set this up for. And actually, they think this because it saved them from a ransomware event. That's exact scenario. Like they did get hit, but they just hit rollback in a TrueNAS and shrugged their shoulders and go, well, that's annoying. And this was a lot easier to recover from. And then they probably got hit again. Oh, yeah, yeah. I don't think they got the budget to actually put in proper protections, but now they have this cool rollback. Yeah. So that's going to be their method of dealing with it is just rollback on a crown. Yeah. Well, this is going to rollback every time we're hit, you know, just hey, that takes us into the topic. I think you wanted to talk about technical debt. Yeah. You know, Avanti was in the news again, and they just can't stay out of the news, but this is one of the things I've pointed out to people because they had people worried about, you know, should we have VPNs exposed, etc., etc. And I'm like, well, you don't see these problems with WireGuard and OpenVPN. The technical debt carried over from the folks at Avanti is wild. Like they're running still kernel too and some of the base OS and where there's smoke, there's fire. We know this in cybersecurity because if there's one SQL injection and they're doing the minimal effort to back it, there's 10 more we're going to find in these things. But it's just hard for these companies and what's worse is these companies that buy it. I think this is probably a lot of business politics as people say, why do companies even use Avanti VPN? And I think it's just really good business politics. Get it there. No, I know exactly why because Juniper brought it in. So it became the default Juniper VPN client. That's why people are using it. And Juniper sold it 10 years ago to a private equity firm who has pretty much done nothing with its sense. That's the last firmware updates are from when Juniper sold it. That's it. And I actually didn't even know that because the places that I know using it, it was still called Pulse Connect or it's not, it was not Avanti at all. I mean, it's been a year since I've used it, but yeah, and it was a Fortune 500 company that I last used it with. And it was because they had Juniper firewalls on the edge. Yeah, it's I post this. You know who Gasey the dog is because he was the one that actually did the post. Let me zoom in a bit here. But these are, he took it apart and look at this curl, curl version seven open SSL one pearl. That's a that's an old pearl. Yeah. I mean, look at this stuff unzipped from 15 years ago and I know I had to use the usual meme posts and things like that. Kevin Beaumont posting about it was hysterical. But yeah, what I actually the funnier thing digging into the history that I learned was the name before Juniper bought it. It was a radius server combined with something else and I used to call it still belted radius servers. Oh yeah, no. Okay, it's that. Yeah, I actually had no idea. I know what that is. That was that was very popular back in the ISP days. Yep, which Paul funk and funk networks. They were I went and dug in some of the history through there. It was kind of interesting like it's this has been around a long time, but it's real demise was. Yeah, is Juniper networks bought it for 122 million, but then they sold it in 2014. So 10 years ago, they sold it, but a PE firm bought it and from what I've talked to for a few people, they put a little polish on it and made the interface look newer, but they did nothing on the back end, which is of course why it can't stay out of the news now. Yeah, I mean, to be fair, it's not like the other the other ones you mentioned that don't have issues. Open VPN notably has had several CVs, right? Like, but they tend to be fixed relatively quickly because they can't hide those things are hiding in plain sight essentially. So they yeah, and you know, at this point, the engineering and validation work required to modernize an entire stack is relatively high, right? Like it's the especially the kernel, right? You bring the kernel current if they're still using like something dumb like tap on the backside, right? Like there's all sorts of other potential problems that might require client changes. It's yeah, it's a whole thing. Yeah, it's one of the reasons it's well, it's not something companies are willing to do. It's something private equity firms feel like they're never going to do because they'll just sell that asset out like they bought it 10 years ago and they sold it to enough government entities, including CISA that they've like we cashed out on it. I don't think they're going to fix it. No, I mean, somebody said, let's see. Oh, I am logged in. I'm actually logged into Streamer so I can press the button haha the I agree. I am avoided LinkedIn ran so far about how I think private equity is terrible for industry because there are definitely it's there are pluses and minuses, right? It's allowed some of the cool things that have happened and it's allowed our industry to grow. But somebody was talking about it recently where basically it's like fish that eat fish and they get bigger and bigger as they go. It's like that game where you just become bigger and bigger and then eventually you get so big that there's no one else to eat you. So you basically just pick the meat off of the bones of everything you've eaten and then just toss the skeleton away. Yeah, and that is the business model of PE. Yeah, and you've been firsthand experienced many of this in the MSP space. Watched companies kind of fall apart. Yeah, it's it's tricky because we know our friends at Huntress. They've taken some private equity money and venture capital venture capital. There we go. It's not PE. There is there are nuances to the differences in there. Yeah, but this concern comes up because it's the thing and this is something me and you know, Kyle has done a great job of maintaining control and not allowing them to dictate to him exactly everything. And these are this and it's nuanced of, you know, we know what we know, but it's hard. Will they eventually straddle them with technical debt as has happened to other companies where they don't want to spend the money to fix it because it's too expensive and we want we rather get new sales and focus on that. It's it's not easy. Yeah, I mean, some of that has to do with startup culture to write like a little known thing. There are some little known caveats to startup culture that people don't realize. The one is that early early, especially the early super genius technical people that tend to make these startups, they tended they care about their work, right? They care about their stuff like the technical stuff there. They tend to be a little nerdier. They tend to take lower pay in higher equity is part of that. What happens is over time, these companies grow and the corporation sucks a life out of these people like it 100% does, but now they've got so much time and money and equity involved in this stuff that they'd be dumb to not see it through the end that those stock grants and that vesting curve, it tends to end four years in, right? So like four years after you start, you'll tend to see exit. You'll tend to see a bunch of people exit because their stock's all vested, right? And it's time for them to go. But when the company sells those those developers, the super smart ones that built the products of the ground up that were largely equity based, all their equity now cashes out. So now they're just underpaid in a job they hate. Yeah, and they leave and they leave. So you lose. So those those exits, they cause almost all of the technical resources to just immediately bail, right? And then you're stuck trying to train an entirely entirely new team that doesn't understand where all the bodies are buried, how to support this thing, right? So some of it is not unique to the PE or the VC. It's just life as companies get bigger people. I mean, we've seen people we know and like leave Huntress even, right? Because, you know, I think they're three, four, five hundred. They're a non-zero number of employees now and it's not the same culture as it was when it was, you know, Kyle and Andrew drinking with us at Connect. Yeah. Those it was interesting. I mean, I think the first time I flew down and met with them was in 2017 and it was like 13 people working there. It was from 13 to I know they have publicly said they have over 300 people now. Yeah, that's just that's a maintain that culture. This is something we've talked about at CNWR is how do we maintain, you know, with the people we have and the culture we have, we're not trying to do hyper growth that will break it because that's not ideal. I like the fun discussions we have today, like we were talking about someone who's got some critical infrastructure to protect different ways to protect it, the different networking protocols that might be ideal or not ideal for how they want to set it up. And those are the, you know, the fun custom projects we get to tackle at CNWR, which are a lot of fun. They are fun. They are also, I occasionally learn my lessons. The SCO one is a lesson that we should do a debrief on that one. Yeah, the dozen settled 100% debrief video to that one. Jason did a dumb and did a project that I thought other people could help out with. But as it turns out, as it turns out that SCO server from 1998 doesn't have USB. It's not on the network. It's not. It's it's a whole thing. And so I put in other people that are Linux aware on this box and they are lost. They do not know how to administer a SCO box. So I keep having to get pulled back into it and I'm ready for it to be done. So as soon as Eric does that last piece when I write when I'm back for spring break were we're going to push that through to finish and walk away and never look at it again. Yeah, that there's a perfect example of massive technical debt. Yeah, they have a SCO box that's been running since 1998. It's amazing that it's working. It's not even a raid or a yeah, it's three individual drives one of which is mostly dead at this point. The second which is dying now. Luckily, I have a backup of it, right? Like because it's on the new box, but at this point I think it's like two months old. Yeah. Yeah, that it is wild that these drives have been spinning for 20, oh, 26 years. Yeah. I mean, it every time I turn it off, I'm like, is this going to be the time that the drives don't spin back up because it's coming like those drives. They sound like they're tired. Yeah. Of course, that sound the drive made in 1998 is a tired sound to me. Yeah. A long ago. Maybe they always sounded that way. Yeah, I mean, they didn't like they definitely there's definitely a the bearings are tired noise that scuzzy hard drives from that era make like it's it's normally you have the spitting noise. There's a wine like I don't know what frequency of that it's at, but it's somewhere in the like four to five hundred Hertz frequency that they start to make when the bearings are going bad in them. Yeah. It looks like the true NASA actually updated to the latest version. Yeah, I saw something your video went away. I can see your I can see down on the bottom. Yep. I'm at Dragonfish RC one. Sweet. I presume that's like Debian on the backside then. Yeah, this is all Debian on the back end. So it's easy to play with. It's a familiar environment for me and you. We both are. I don't know you more of a red hat person or Debian person. I'm more of a red hat person. Okay. Just from Enterprise. It wasn't for a long time. You only ran Redhead and Enterprise a bunch who has changed that Debian doesn't have a huge enterprise following. It tends to be more fun too. But yeah, they I'm comfortable in there. I'm not as comfortable with some of the like snap and flat pack and stuff like that. Give me my dev files and don't and I mean I wish somebody could just take system D away. Yeah. The other thing I'll mention here is can disable maintenance mode now that it's upgraded connected. So it worked. And the stable maintenance mode. So yep, cool. I'll fire up my VMs on this. I've been doing that. I did some before and after benchmarks. I have a pharaonic system running on it. So you look at booted. So everything the upgrade survived and the VMs still there. Yeah. I mean, I would expect that to go well. Yeah. All right. Like that's that's a pretty Debian has good major version upgrades red hat. Not so much. A free DSD does with some caveats that have come over the years. I think the like 11 to 13 change required more space on the boot partition than you typically got than the installer tended to give before that it could all be done. They're still using boot slicing. So even with even if you move from core to scale, I like that they kept this so you can simply reactivate a previous version. So you know, here's what I'm on now. But if I want to reactivate, I could just go and reactivate this one. They got all the options there, which really nice that Trunas is like kept up with that. Yeah. Yeah. I mean, it's it's there. I'm glad to see more things doing that. The first thing I came across that did that was at five load balancers and I actually think it's an awesome feature to enable quick. Yeah, rollbacks. The they're doing it now with the version 24 of PF sense. Yeah. Their rollback is actually really clever that comes out in the new version because they do the same thing. They do a ZFS snapshot boot slice thing, but when they do the update, they actually boot slice, then activate that slice, update that slice. Then on reboot, it'll boot to that slice. If it doesn't reboot, it's got I think 60 seconds to boot and then the watchdog timer says didn't boot it resets the previous snapshot active and reboots again automatically for you. So for remote upgrades, we don't have to sit on our, you know, sit on our hands and pray. What was it that the one that was either here or one of the ones that Buckeye when I upgraded it, one of the switches, it sat 40 minutes. Yeah. Go catalyst switch 40 minutes. I had told everyone it was dead. I was going to have to drive to the Colo in the morning to deal with it. And then like 41 minutes later, it came back up. Yeah. That's always and time goes at about half speed when you're upgrading things remotely. Yeah. Generally, that's the rule we have found. Yeah. So I'm happy to see P.F. Sense is going to make that a little less risky where we can just push these out. And matter of fact, I feel, and I know this is slowly coming, they don't have an official release date for it, but when they get some of their central management, this will allow us to push an automated update where we can just look for ones that fail. Like, yeah, push it, it'll automatically reboot if it fails. So it brings back to a working state. So the client's not down. Awesome. That's the ideal situation. Look for failures. Look for which ones were successful. You know, mark those as you don't have to deal with that and look at the rest. Yeah. It's it. So some of this is a relatively unsolved problem, right? I asked you when we first did this stuff, like, how do you handle it? And I know that TrueNAS, they sell an appliance that has two control planes. It upgrading storage sucks because if I'm using it as an iSCSI backend for XCPNG or VMware or something like that, like, you basically have to shut down all of the VMs running on it or migrate them off, right? Which is definitely somewhat onerous. You know, now that they're running Debian, I wonder if they can go to the in-place kernel upgrade methodology. Well, because they're still installing it as an appliance, essentially with the boot slicing like there. So I don't think that's going to be they they're opening up. They have a new dev feature they added in 24 to allow certain packages to get upgraded. But if they did do package upgrades, that would certainly be great. They're still treating it as an appliance where there's a whole slice created, which still has a reboot, which of course has time attached to it. Yeah. And depending on the server, that time can be a non-zero amount of time. It can be like a solid five minutes on a newer box with Numa and all the other stuff to actually even start booting. Yeah. It gets a little, it gets a little hairy. Oh, so she, like I said, when they were rebooting things remotely in your colo. Yeah. I mean, it's just like you basically have to have a plan for it, right? So it's like I need a plan to migrate all the VMs off of that that can't go down and then before I can upgrade it and then to migrate them all back when they're done. We manage it. It's just a pain in the butt. Yeah. And I think it'll be easier with XTP and G because one of the nice things, one of the features that I think is cool. You tell me, I know you're not super keen on this interface, but if you go over to the storage, one of the things that'll let you do is select all the disks on a storage and just migrate them. Yeah. You can do that in VMware too. And we do that, but like, takes time. Yeah. I mean, when I was just doing that with this box that broke, I got everything off but the one VM on it. But I mean, it took a solid six hours to get, you know, the six or 800 gigabits, gigabytes of data off of it. And that's these are all 10 gig connected. Yeah. So we need 100 gig connected all flash arrays. Yeah. Well, I mean, that's basically it, right? Like because, you know, this one, the source here was spinning us. The destination was all flash. I mean, and it was doing 100 megabytes a second, but it's still a non-zero amount of time to transfer that amount of data. Yeah. That is right. Like you just rearrange a bunch of stuff and move terabytes of data back and forth multiple times, right? Like that time was measured in days. Yes. It was why this project took so long. Shuffling these around because there's two pools, 20 terabytes left. I deleted some of the stuff. There's still 20 terabytes left on this one. And oh, this one, I moved everything off. So but yeah, there's still 19 terabytes left. I haven't shuffled back over. It's not that much. It was, I got two blue copies of the data, but either way it's 19 terabytes doesn't move in a minute. No, like even, let's see. Wolfram, you know what the best tool for doing this? Wolfram Alpha. Remember that? Oh, yeah, yeah. I love that. You can type 19 TIB at 100 megabytes a second. It will take two days, 10 hours and a one minute. If you can saturate a full 100 megabytes a second, not megabits. Yeah. And that's the other problem people run into. I see people always pushing for like, oh, just upgrade to a faster network interconnect. I'm like most of the time I'm not even saturating it. I'm running at the speed that the spinning rust will move data. Yeah. Correct. Yeah. I mean, 100 megabytes a second is just about saturating gig, right? Like with overhead, it's probably gig speed at that, right? Like, yeah. I think I posted a screenshot on Twitter. I think I was able to sustain about 6.9 gigabits moving the data. So I moved relatively fast. It's actually shocking how fast this is the next little project I'm working on was showing how fast you can actually move data around provided you built your ZFS array and split it up properly. Because I'm really shocked at how fast that this system actually under storage at a 6.9 gigabits a second. It's six hours, 43 minutes to move that data still six hours, six hours. I mean, that's the whole problem with the 20 terabyte and 14 terabyte drives is that, you know, the re silver and you in ZFS can take literally days to happen. Yeah. You see, I got these broken into three raid Z twos. So you get, you know, a pretty good speed boost because you're spreading the rights across a lot of spinning rust and these actually will sustain really fast. I was surprised how fast this was the right. I mean, but each of those is three rights for every right you do three because it's raising to yeah. So it I think the true ass boxes do a good amount of right combining and they make very heavy use of that art cash to try to not once our cash is full, it gets a little bit slower if you're doing random. Yeah. This is something that I don't recommend. We had someone reach out to us and I told them there wasn't a solution for their problem. Well, I take that back. There's a solution coming next year for their problem. Did you ever use in production Ddupe Vita back up sure now so they much smaller disk and main location would use cost. I really don't recommend this and what someone run into is Ddupe works good at very small scales. The moment it gets bigger count constantly calculating whether or not that block exists within the existing block so it can be duplicated even if you have a pair of Ddupe MVM ease on there has a very high cost. They just did a big rewrite of the code. I think Clara systems partnered with I X systems. They did a major push to the ZFS code base to fix this and everyone's like exciting is fixed. I'm like, no, it's fixed now in the beta essentially. So you're not going to see it in production. I think TrueNAS says they're not going to get it until next year before they put it in code into their TrueNAS systems. But yeah, there's a there's a massive downwards quail as you add small files or just really files in general and data the data volume and deduplication. It's I almost wish they didn't have it in there so people wouldn't ask about it because it just doesn't perform well. Yeah, I mean, there are good use cases for Ddupe, right? Like it, but there are very bad use cases for Ddupe too, right? Like it if you're if you're using it, say, NFS and you're storing like a bunch of to a XC PNG box and it's like a ton of windows or even a bunch who or whatever hosts Ddupe is probably going to work pretty well, especially if you build them from clones, right? So the base disk is the same on all of them. Yeah, if you're using it to store backup data that's compressed or anything that's compressed, it is not going to do anything except way CPU cycles, right? Where D where Ddupe is interesting is it destroys your right speeds, but your read speeds are good. So for archival purposes, if you have data that's duplicated by the way, because there's you don't can he as Jesus, you only if you have a bunch of a series of encrypted things that you're storing on there, there's not going to be similar blocks between them. So it's got such a narrow use case. It should be like under the advanced and I understand that I'm going to. I should not do these things. Actually, it's on by default. Isn't it for a lot of these data sets? Okay. No, no, no, I'll see it. LZ is on by default. He's on by default, which is another thing. Trunas did do this. If you go to any of the data sets, one of the things that they added, which I really like, this actually started in version 23 is if you go here and you edit and you changed the sync to standard or disabled they give you a warning. About that. So I think warnings are good. It's I don't worry about it. The I talked about this and I'm going to do a video on about sync being disabled, what data you can lose, which is the couple seconds of commit if there was a catastrophic failure. Most of the time it doesn't matter if you're using something with a transactional database and it's for business production. It matters a lot. Yeah, it's a file server. You're probably okay if it's a database danger where Robinson. Yeah. Yeah. Ooh, when would you use ZFS copies greater than one? You ever used that? No. It's interesting because I don't the concept is if you you can actually have ZFS duplicate the file twice in the file system. I guess for integrity and BitRot checking. Yeah, it's basically parody. Right. If you look at architecturally how Seth works or you can go back into something like I think a left hand. I forget was the company that HP bought that used to do this via sands. If you could have distributed ZFS that would be awesome. Right. We're you know, you have more than one copies with locality spread out across multiple systems, but on a single system. I don't know. You gain maybe a little bit. Yeah. I've not really had much use case for it and this has been a topic that keeps coming up and I'm going to do some updated videos on this topic. It's become this too. Yeah, your thing. My thing is I don't think the performance boost is there as much as people thinks it is now used to be a much bigger deal because of the overhead of processing that many extra packets because this is why they most system tell you how many packets per second they can process. So if you push things into less packets, yes. Now you've got a boost, but that was great advice 10 years ago in less so with 10 gig networks or more than 10 gig networks. I think probably like I SCSI benefits from it more than 100% it does. Yeah, but I don't think you're looking at more than throw number out here. Like I've heard only 15% gains. 20% is what I've definitely heard and seen. I want to do some benchmark testing. I have I have a big pile of churnass over on my desk because this is one of the tests I'm going to do is build this on. Actually, I'm going to use the HL 15 server for this. I'm going to test MTU 9000 and MTU 1500 to do a series of pharaonics benchmarks and see what the differences are. There are a lot of other caveats there too, right? Are you running TCP off load on your NIC card, right? Because the CPU has to calculate the check sums and stuff of all of the packets. Then having more packets hurts. I mean, the other the biggest thing is is the amount of data you can transfer in one round trip time. Yeah, right? So like it if the locality is close, you know, we're talking sub millisecond with serialization delay at one gig or 10 gig. It it does matter, but like that's one of those things that if everything else is perfect, then going to 9000 can eke you out like another 10 or 20% if if you're running spinning rust don't even bother because it is also something that if you mess it up will make your troubleshooting that. Yeah, not fun and you get one switch in the path somehow something connects and one switch in the path as a 1500 bite MTU and now you have a really bad day. Yep. And I've run into oddly one of the problems. This is how I learned how bad hyper V is a recovery. It was an ice because he network where someone didn't realize exactly what you said. There's a switch in between that did not have the MTU separate and end up corrupting. I don't know exactly how to manage this, but you know, corrupting all their ice because he extends and they had sure to have snapshots on the Z vowels. So I was like, oh, we can just roll back to Z vowel turns out that doesn't work at all on hyper V hyper V does not care about Z vowel rollback. So it turned into quite a recovery project and I'm like, why is hyper V so stupid? It's all I felt about that. I mean, the other caveat there right is like only ever do MTU 9000 if you do an entirely separate storage network that one you can put your switches at 9000 and that's pretty safe because they'll pass 1500 but you should never like, oh, I'm just going to put my storage in this one server that talks to it over the same network at 9000 because it could be talking to other devices on the same network that are not also at 9000 and there are so many places where that will just shoot you in the foot. Like I we often and for my complicated I scuzzy deployments, I use separate switches and everything for them, right? And so there it's easy and a no-brainer because literally everything plugged into that fabric is 9000 to you. But when it's not that way, it's definitely a lot more like, should I really do this? Is it worth it on all flash? I would on spinning rust, probably not. And then people who stuff it in a VLAN. Yeah, then and then you have other factors that you run into. I think I did a video on this years ago on some of the challenges with Unify because I think at the time and I don't know if this is true anymore. You wouldn't set it to 9,015. You had to add another 15 bytes of overhead onto it. So you set them at like 9,015. So they would talk to the other switches at 9,000. There's all little things you have before you turn on the connection to your storage server of rice because he make sure you've checked all of those things probably 9,020 honestly because that's the size of the IP header. Yep. That sounds right. Yeah. Windows has that issue as well. I believe in Windows you don't set it to 9,000. You set it to 9,000 plus some over 1020. Yeah, that's usually and then if you have 802.1 Q in there like so VLANs, there's also some caveats there. If you're doing Q and Q, there's some caveats. If you're doing GRE, there's some caveats. Like, yeah, with over. That's the other place I tend to see it used right if you're doing overlay networks. Maybe you want to go to like 2,000 so that you have room for the overlay network to still pass 1500 by packets. Yeah, I this is one of those fundamental topics that I like when I can do a video and answer questions and I tried to be brief like my wire guard video and I explained why you actually set the wire guard below 1500 so it's encapsulation is more efficient in a transport of 1500 might be fun. I don't know. Maybe we should do this because mean, you know, the fundamentals and I think understanding the fundamentals is huge because this is why we see a problem when someone set something up where like he didn't set it to 14. Why should I have to set that to 1400? Yeah, 1452. 1452. What is this clamping? Why do we need that? Yeah, especially going over the Internet because PMTUD you cannot you cannot rely. Oh man, you're super nerding me out now. You can't rely on PMTUD working over the wide Internet at all because so many routers just block or severely rate limit ICMP messages to and from their control planes. So you just may not get the, you know, the MTU exceeded message back if your packet has DF set like it's all sorts of interesting things. Yeah, I need to revisit. I don't touch that stuff enough. I need to revisit. I have a book for you. Okay. The Comer book. It's the most boring book you've ever had, but it is like the definitive guide on how TCPIP works. I, yeah, I need to, it's like I have it all in my head, but it's one of those, it's falling out of practice. So I'm not as good at it. And I need to, I want to swing myself back into that. This is an interesting question. I'll throw this as an interesting question. Maybe. Wes has asked me and or us to. Yes, we and I suspect I can work out some sort of deal in trade with him. So we'll see. Yeah. I have access to it. I haven't logged in yet, but I have a code. Yeah. Wes runs, Wes and a few other people are friends of ours. They run and past cider. It's a new startup. They're really smart people. We definitely support them. They're awesome in the MSP channel. So if you don't know, if you don't follow them, follow those people. Kyle Christensen and what's Wes's last name? Spencer. Spencer. Wes Spencer. Yeah. Not Wes Spencer. Remember? Yeah. I have a shirt that says I'm not Wes Spencer for silly reasons. Uh, troubleshooting advice on churnass being, being unable to join 80 domain times out 34 seconds using DC is to QNAP. NASA's properly joined to domain all the same subnet no firewall ACLs using the same DNS. Yeah. That'd be my question is that same DC doesn't have like 8888 is another DNS server because most people don't realize they expect that that's a priority order that the DNS is used in for the resolvers. It's round robin round robin all of your you set your churnass to be okay. So it is in and that DNS is hopefully pointed at the Windows server. Yeah. I don't know. I would need to see TCP don'ts to see your logs. Yeah. And try to start diving through it. One of the things I've done to fix it. It's I recommend this. This is the same thing with permissions 80 purge it purge those things out reset them back up. You don't there's even if you look there's some command line ways to just dump it all like clear all the settings that get tied in with SMB. I don't know what people did before me when I start troubleshooting something and I also I don't want to spend a lot of time solving it. It's almost easier to just say let me just go through and clear all these settings save reboot. So all the services are restarted and try to join again and people like oh I didn't think to do that and joined right away. I'm like yeah just yeah it's like there's something stuck that well yeah the other thing I've seen and I think you were hinting at this as part of it look to make sure there's not an active computer account with the same name in active directory like going to ADSI like don't even go into Ada go to ADSI and see if you can find it because I've definitely seen the case where this thing used to be part of the active directory domain you rebuilt it but the old computer account still exists and it will definitely fail to join if that's the case in weird and mysterious ways that don't log very well. Yeah that's kind of the weird and mysterious ways that don't log in well sums up active directory for me it's always if you treat it as LDAP and Kerberos your life will get a lot easier conceptually. Yeah it's why it's my way I AD I actually trained on it in early days and T4 was when I first started and you know when you did the promote to DC and all the fun stuff that you had back then and the more it progressed the more I disliked it. So that's why that's why I'm more of a Linux person today is because of the way ad is I can blame them. Linux is getting a registry to now a system D. Oh yeah. I still like I partly want to read it. So you know who Michael Lucas is with his books right. He's got he writes fiction books as well and I love anytime someone says system D. He's got a love story tech fiction book called Savage by system D. It's it's it's an erotica book. Laugh that it exists. Apparently he does every type of play on tech humor in this book and I'm like every time I see him I'm like you know I still haven't read I love seeing Michael Lucas and I bring it up and he just laughs because it's a he says it's one of his most reviewed books because once nerds find out about it they can't help believe how they were also savage by system D and just he calls it something like free BSD erotica or something like that which also makes me laugh. I'm looking to see what else you got in your list here. Oh yeah. Not a ton I kept it like yeah I got stuff to do we're working on that new AI stuff. Oh yeah man it makes a good bartender. So I've no I let a llama run out to open all night and it got a little slower this morning so I'm going to see I restarted it. I basically just saved the model out. No it's still last night it was responding in like a second and now it's taken like a solid 10 seconds before it's giving me an answer. I'm curious why. Oh well then something we're doing internally. I've been playing a lot more with AI and Jason's built a dedicated machine over you know you are for it. I've had just a few in my lab. It it's been fun because there's so many little things it can do and I I like the private ones because I do not want any of my public data in there I'm but I am using things like the Google one for data that's more real time because something fascinating to me that Google did I said and I purposely tried this because I heard it would work so I went to Google and I said give me five YouTube titles for the latest Unify release. It understood what was in the release that was my prompt my prompt was dead simple it said talk about OSPF talk about this it had titles with the feature releases because I didn't have to tell it to go find the site. It knew what I it inferred everything I was doing for it went and read the Unify's release notes. It put the release notes in the titles into the video I was making. I'm like okay that's useful. Yeah it's it's cool. I think that for some reason Olama is not using the GPU so I will have to troubleshoot that later. Yeah there's I do not have Olama on a GPU so I'm used to kind of I don't get 10 second responses but I don't know but it must be something I'm not running a thread or I'm just running in like a rise in seven. So well this is a yeah what did I tell you this was less than a thread ripper. Yeah it's a thread ripper so it should be absolutely insane. It is absolutely I mean this box is absolutely insane but I mean it's got an A6000 in it. So it's also got an insane GPU. Yeah that that's going to be making sure that it's taking advantage of pretty automated for setup on there. I most people are using consumer GPUs which I I thought about because I got a nice fast GPU in the system I'm running now for my video editing. I thought about setting up a Docker container with it in my system because you know when I'm not using it for video editing my video card just kind of does nothing yeah yeah do automated things. No sure for sure like it's a it's it's pretty cool. It makes a great bartender. Yeah part of that's the use case I've trained it actually saved the bartender model out. I got the system prompt that I fed it with I think is good enough that I wanted to keep it. So now now in a llama I can do a llama run bartender. Yeah it'll make all the big streaks for us. Yes it will. There's the efficiency we were actually writing some letters with it things like that these are not things technical people are or really general no one really wants to write like notice letters to clients on topics and things like that that you go hey we're going to make a price change or something like that but being able to use AI for that is so helpful. Even if you're not copying and pasting it you're getting that boilerplate it does this with even code when I've had to get things written in bash I know I could muddle my way through it but it gives me a nice structured framework for me to start before I want to put some functions into it because by the way I suck at coding I'm not Jason I did not spend time doing that. Yeah I mean I increasingly suck at coding I feel like all my knowledge is slowly evaporating out of my brain. Yeah that does seem to happen. Setting up snapshots and replication for my backup Trunas do snapshots get stored on the main server and move to a backup server or store straight to the backup server. So the way it works with ZFS you're never backing up the actual data live when you snapshot if I snapshot at 1pm that 1pm snapshot is what's getting sent to the destination for replication so the snapshots and depending on your retention policy which by default if you set this up in Trunas it will replicate all the data all the snapshots with the same retention policy as a source that's all the default checkboxes so all your your data with all the snapshots are all sent to that backup server but the backup server doesn't contain it's not when the it only contains that last snapshot so snapshot at 1 and you write something at 101pm that data is not on there even if it takes an hour to get the data to there it's only sending that last piece it's not real time synchronization it's point in time synchronization. Yeah it's and it's copy on right right so like that's the other thing to keep in mind is that like it's not like it has a copy of the data is a snapshot and then it's the existing data is still there so it is using the snapshot but this the changes after the snapshot are stored over here right is a copy of that data that's changed so it's yes kind of it but it you wouldn't do a snapshot just straight to the other end right it's it's having those snapshot policies in point in time matter right Eric was fixing I know as soon as he showed me a screenshot I don't have to look at anything actually also show me screenshot of the checkbox is I know what client it is because they're a frequent flyer with us they him I told them many times not to do this but there's a box that is not checked by default called rebuild snapshots from scratch if they're missing on the destination there's a reason you may not want to do that because if something goes completely wrong on the source you don't want to replicate the destination but I know exactly which client this is when Eric was consulting with them like oh they did it again he's like yes they they go and mess with the snapshots and make changes because they'll the destination your backup will be read only and it doesn't matter if it's read right as long as you don't write to it the moment you write to it ZFS goes wait a minute I tried to replicate but it seems to be new data over here at this destination so I'm going to just stop the service I'm like you have to tell to rebuild those broken snapshots because someone has messed with your backup servers but they're the good news is starting in churnass 23 they in I think this is probably can only be done in scale the new version of syncing is on there syncing now can do extended attributes I haven't done a video on this but I want to you can check the extended attributes box we have I have one customer to one in Germany as consulting with they're actually using this and it works quite well you can synchronize to true dances and when a write happens to one even the extended attributes occur immediately as fast as syncing can get the changes over on the other side that does require if I have if I'm not using ad for example and I have you know user one thousand one thousand one thousand two etc all my users have to be exactly anybody yeah but I mean I presume that uses KQ on the back end and it's just listening to all changes on the file system and just secretly replicating them yeah yeah same thing works really I use same thing still quite a bit it's such a great little open source project for synchronizing files maybe there are definitely the case that we're going to use open source AI internally I don't know if there's a use case where we would deploy open source and a clients just from a it is kind of the Wild West and it's kind of not super user friendly right it's not something that a general customer if we have a devops client that might be using it but they're usually using their own they don't call us for that so I mean I could potentially set up a llama and have people point at it instead of open API where they would use the open API API but like yeah there's not an equivalent a good equivalent to like chat GPT even the even the Gradio interface you get with private GPT is not it's not super awesome yeah me and Jason are going to be doing a talk coming up in April in Toledo on this as a topic talking about how to navigate companies you know what does or doesn't work what's overhyped what's actually real for using AI in your business you know it's I actually someone pointed out that December of last year was peak AI hype cycle because in first quarter the mentions have started to slide down for Fortune 1000 companies mentioning it in their publicly traded really call that the forward looking thoughts so they're starting to see a little bit less mentioned but this because we're getting down to practical use cases exciting we're looking at the real what can it do what can it make easier for us and you know that's even why we're using it is because now it's got some use cases that make it easier for us it yeah I think it was the case it was the case where everyone had to add AI but no one knew what to do with it yeah so I think you posted was it you that posted that mean about that like that all these companies running around saying they have AI and then all these like executives go what are we doing with it is how we have it and yeah it's it's basically a solution in search of a problem in many cases I think we're finally starting understand the problems it's good at like making drinks yeah I mean if you have a if you have a lot of different ingredients around and you wanted to make a drink recipe fun fun thing because we have the unrestricted one throw Clorox bleach and is one of the options and see if it suggests it will it will it make a drink yeah no it's good for us yeah actually I'm going to do that they're they're I've talked to if you follow me on LinkedIn it's sometimes where I do my longer form posts I don't know where else to put them so LinkedIn's where they end up the I've talked about this and a couple of occasions and there are absolutely cases I use it for an easy example when I'm done with this live stream so far we're 56 minutes and we're going a little bit longer here but I'm going to take the tool called Taja AI because I have not found a way to get maybe I'll try the latest olamas see if it's better at it but Taja AI is a great job of creating chapters for all of my live streams I think fabric can do it yeah I know I want to it's one of the things I want to test today is is fabric and do it because I mean Taja is great but if I can save myself 1795 a month but this is also one of those myths people get into it's going to take our jobs the reality is I was never going to hire someone to do chapters on my thing it's just it's not a job worth hiring for it's a job though that for $17 if someone will work for $17 a month and do I think I do six to 10 live streams a month because I have my four blog Thursdays then I do sometimes Sundays and I do the home lab show so if someone was willing to work for $18 a month to put chapters in all those shows yeah cool you can replay it may actually have to do it less than 1795 then you can replace Taja yeah so I look at it always more of an assistive tool because it's a job that wouldn't have existed not likely it's not worth someone's time to do it but it does a great job of lining up all the chapters and makes it easy for people who go well I don't want to watch Tom and Jason rant for an hour but I love to see the answer they had on some topic that's in this list and they can jump to that point simple as that yep there's actually a fabric recipe for making YouTube chapters so when I didn't I didn't realize he had that I pointed at yesterday's business technicality video that's easy I want to see if it can do it I'll let you know what the output of that is yeah so but there's there's a weird way that AI looks at things it's a very much a predictive word system it's really just fancy spell check but it's handy for a lot of things like that but there's times when you just one of the things you can't trust it with this that's not where it's good at all yeah it's not if you need something that's very fact driven oh yeah it will still lie so at the connect wise event I was at last week that the shirt they had it right a boom had a bar or Morse code on it and if you decoded the Morse code you got a discount 20 percent discount code for itns but it had Morse code on the front and the back so I started taking a picture of it and throwing it at chat GPT and asked it to decode the Morse code and the first couple times I did it said the picture was too blurry or whatever the case is the literally the last time I did it it literally just made up an answer it said it said SOS help and a couple other it did not say anything related to SOS or help it it just assumed what it would say yeah it was it was pretty bad yeah this is here I'll share one because I thought this was so funny and this is the exactly the problem it argued wouldn't I can't say it argued with me I did a post on LinkedIn about banning tic-tac so the I had it do this I was like hey please make me an image of an authoritarian government taking control of things and it's what it did it has these planes in it right I said I want it without the planes give me pop art without the planes yeah there's the planes again yeah so I said those are so planes in there and it says I can't I can't remove them basically it says I can't see the images is the next response something like that yeah I'm here to help and I can't directly see images of control the output in such a precise manner so apparently no matter what there's going to be a plane if you also notice I've done a lot of video summaries in here South Park oh South Park style stuff it makes up puns really well so if everyone wonders how I got better at puns over the last year I'm not going to lie AI did it I was like give me puns on this topic and it's actually one of the fun promises pretend you're the register and give me a snarky version of this news article and boom it writes register style headlines with a brilliant snark what happened here I pressed buttons it did things in theory window that one share where'd it go oh I have to restart chrome because I didn't have to give her permissions it did it it I'll I'll send you what it did here inside of a hey if I can save 1795 a month I'm all in it needs it's gonna require a little bit of for some reason it put the chapter titles in all caps whatever it's it okay though like I'll put it in the AI playground channel so if a 50 terabyte main server with a 20 terabyte used backup server won't use 20 trade only use the difference from the last snapshot to main server I don't completely understand the question or I'm missing something yes that is correct okay I mean I believe it's correct basically he's saying the backups are only going to be used these space how is it how is it okay I actually only played with this peripherally is it creating an entire copy or is it transferring over a file that represents the the zfs so each one of the snapshots is a block level differential so for example renaming files doesn't have any cost to it other than the blocks those got renamed so if you have for if you look at my snapshots there there's very little used in between on my snapshots because they're only and I'll show you where this is it's actually it tells you how much you're using so if I go here to gray log and I have these gray log changes a lot so if I find the snapshots in here you can see the daily changes I do a daily snapshot in gray log and those differential snapshots are 19 gigs on this one or I'm sorry yeah it's only using 1.7 gigs out of the 19 gigs of data that's in this 1.9 gigs so I change only to help this time I use 5 gigs that day 1.7 but that initial snapshot you did if it's 50 terabytes with 20 terabytes used how big is that initial snapshot is it 50 or is it 20 I think that's a question because they're all incremental to each other they need all of them to it's a chain it's a chain yeah so you would the amount used is going to be let's say I'm averaging one one gig a day to make it easy yeah I've got seven days there's seven extra gigs that those will be attached to plus the 19 yeah but okay the 19 that's used but if you had a hundred terabytes of dry space you're only using 19 it's not going to be a hundred plus one a day it's going to be 19 plus one of that yeah yeah because if you look at the data set it'll actually tell you look at how much data is used here yeah okay that that's that was I think we're driving to and that's that was my understanding of how it worked but I just wanted to make sure I wasn't like is it actually importing yeah oh wait I wasn't showing the screen derp oh haha thought I was going to be yes so it'll show you how much overall this data set has in it based on changes because that isn't how much gray log is using that's this data set along with its 13 snapshots and it's 13 snapshots have yeah yep that works the way I thought it yep now the it's funny because someone did a there's a great video I probably should share it it was done at one of the recent I might have been a BST or is a ZFS talk it's about an hour and a half video someone walks through because someone says hey how do they deal with when you delete a snapshot in the middle of merging everything I'm like it's brilliant it's also the talk it's a hour long deep dive on just how they do that merge in ZFS it is really complicated math basically they just reconcile the differences correct yeah but the way they do it someone breaks down an exquisite detail like every step of the way walking through the code of what has to be done to make ZF it the person dives into all the ZFS magic like all like how much code base is there to make these functions work yeah I trust they work but it's also cool seeing you know how the sausage is made or how the snapshots are unmade oh there were people the work people putting in that pizza email address you have that now makes tickets so somebody door-dashed subway nice yeah we created a series of food alias emails and then we kind of forgot about them when the merger happened then then they started creating tickets and connect yeah I mean the number of emails I get about pizza Wallace whatever that is is pretty high mm-hmm yeah they're also may or may not be sliders at Lawrence systems dot com yeah that's the one that the pizza Wallace once comes to okay yeah we like white castles and also known as sliders here in Michigan so that's where you put them in yeah the ZFS commands like you know I will encourage everyone here if you are going to be doing this stuff for a long time start to try to learn some of the command line tools because when things break the GUI doesn't always help you but the command line tools almost always will yes yes I was just showing that on one of my recent videos and I have another one I'm doing on specifically because there's a lot of confusion around how log works the the way the intent log system works is a little different than what people think they always think it just is a right log but there's a reason it doesn't accidentally have the word intent written into it it's something that is written there but as a actually is writing it to the drives and holding it there it's still streaming it to the drives the synchronization is what's happening it's I got to put the whole graphic together for this I think to be explaining it better but I have a video where I did actually exactly I wrote down because it's one of those I want to be very concise and don't want to steer people wrong so I someone wrote and I linked this in my documents someone wrote their PhD thesis on ZFS and so that was my source material because I thought it was such a good I understood how it worked but someone put it in better words than I could have put it in of how like the intent log works to where it writes there but they actually only look at the intent log if there's a failure because it's really a rewind log to stream the data back to where it belongs that's why you can lose a zil matter of fact you can take a zil in and out dynamically without restarting server you can do it while it's running while it's under load it just flushes it and says here you can have it back and yeah there's a lot of nuance to that you can't lose though like the metadata you can't lose the if you do I'm going to show what happens when you do so you you have Humpty Dumpty you do there's no way to put them back together again I reference I can't remember if it's Jude he works for Clara systems him and Wendell Wendell it got excited at level one text when Linus destroyed his server because of ineptitude and lost about a petabyte of data there was a unique opportunity Linus had another machine they took that one out of service and Wendell actually said you know I never have an opportunity to have unlimited time with the ZFS server I you know people want it done but they want it done yesterday so there's now yeah yeah there's not usually a way to really methodically walk through could you recover ZFS so he did recover it and because he had months to work on he took it on as like a personal project him and a few other ZFS aficionados kind of bounced ideas each time they would get stuck and they started writing code to kind of put Humpty Dumpty back together again and they were able to get like 90 something percent of the data back which is amazing and but yeah it's definitely one of those one of those fun things well I forgot I chatted you when I put in the comment box well that won't work publicly anyway so yeah I don't know if I'm doing there yep I like it was even a great academic paper from that YouTube video that's great yeah I'm an architect construction by day took me six months to get solid enough understanding of Churnass to be able to recover from hardware or use your fun times yeah you know one of them my favorite things about being a consultant really comes down to people do things I would never expect him to do me and Jason if we were experienced systems architects if we set something up we set it up from start to finish like a very sane method so we never get to see why would anyone set it up this way yeah and figure out but it also becomes this like a light goes on our head and it's a stimulating thing to try and unwind it and solve it it gave me latex did it yes that's amazing yeah cool yeah I mean we that we run into that all the time that like it's like how how what I kind of want to interview you to get the chain of thought that you use to get from what your needs were to where you landed today because I just can't get there yeah it is there's a particular job and we know that the person inherited it so we're by no way throwing shade at them they inherited such a mess they called us and I think Jason knows what I'm talking about the one with lots of windows VMs spread across multiple servers with a weird three-part system there's like they inherited that and they couldn't they had the same question for us what thought press probably brought the person to building it like this and we're like we have no idea we're just gonna unwind a lot of this and untangle all of it but she go no that would have 10 gig connections but they were only using one gig that was one of the head scratchers yeah the I mean we we kind of been noodling around that on the on the backside right like that we get a non-zero number requests that come in that are essentially other MSPs that want help or backstop with this stuff and we're trying to figure out how to actually make that a thing that we can provide in a way that is fair to all parties involved because what I can't be is a fire department right like I cannot be the I only make money from you when everything is down and it's super high pressure and you need me to drop literally everything I'm doing in run and save this like that's not a viable business model I am not the fire department right you want to fund me at fire department tax levels you I can be the fire department but you know my relationship with you isn't going to be like you don't pay me any money until the sky is falling and then you expect me to drop everything but there's probably a happy medium where you know we do some sort of retainer thing I don't know yeah and I like that idea because if you if you're coming at us like hey I got a bid can I talk to you guys next Tuesday about it cool awesome you know keep us on retainer we love to help you with your bid we love to help you with the thought process of how do you get this set up in a sane way how would you sell this how would you configure this because doing it all at the beginning before the you know systems in place or even if it's a question like hey should I take on this project here's the situation I don't know these tools can I lean on you as a resource or is this a don't touch it and suggest it's a rip and replace no problem we have years of experience we have a good team we want to help we want to do it under duress all the time yeah it's I mean it while I thrive on that like high stress like it is really hard on the team and even on me it's really hard to do for like where it matters like I'm I'm doing my own ZFS data recovery now from that raid system like but again it's the situation you just described with Wendell like I don't need the data like I want to prove I can get it back so you know I'm in there occasionally I have on another screen over here I have the iDRAC for the thing open watching the DD rescue I have running run right like but it is very frustrating to get that in in like hey I set this true NAS box up I didn't follow any of the guidelines that I should have used I had multiple drives fail no one is paying attention to it and now I need all the data back it's like can we get it back I don't know maybe depends on how bad Humpty Dumpty is broken do I am I going to drop all of my other recurring revenue clients and all of the other projects that I have on hand to drop everything to pull in the very expensive and very technical resources to do that for you when we don't have a previous business relationship I'm sorry I'm probably not yeah and that's where it kind of becomes a challenge I was when I was on the 45 drives podcast they shared a really fun story one of those long pauses take a deep breath and explain to the client what they did wrong which they spent about $50,000 on the wrong hardware because they thought they could build out a really fast set cluster with consumer drives they don't understand Q depth yeah oh yeah Q depth is a thing yeah so they couldn't figure out why this the large $50,000 investment didn't perform and they were tuning like well the tuning is replacing all these drives the other thing is even if you don't do consumer drives one thing that people often overlook is that the status back typically only has a Q depth of two right in one of the things that you get with sass even if it's the literally the same planters in the same controller on the backside sass has a Q depth of 128 I believe it is and that for right combining that Q depth is huge yep absolutely huge yeah that is a it's simple oversight to just not understanding because it's a scalability thing if you tested it in a limited space you're like wow look how fast this is but when they hit production they built this one system then they bought several more of the same system to build all the stuff but then like oh wait it doesn't perform the performance doesn't scale and it's because of that Q depth problem and this is a random IAPS problem too right it's like it yeah it's it matters a lot more for random IO than it does for sequential and it can definitely we had one at Finnecke response last company I worked at we bought this really nice Hitachi saying and I'm like oh man with you know 105 spindles like the fact that they're say to drives doesn't matter that much oh it hundred percent mattered that thing was a dog for random IO because of the Q depth issue yeah hey I just want to say thank you Chris for the donation much appreciated and it's not generated by gg4 so it's in it's got that weird that weird L symbol at the front of it oh yeah yeah we just got 20 pounds added to it I don't want any more pounds any more pounds we're trying to lose that while we talk about having chat GPT makes our drinks snapshots and replications tasks that you know it's the best way to run a backup server syncing exact copies I love replication so I use them for backing up all all my data both at my business side and my YouTube side they're great real time synchronization I use syncing for example for all my business documents because as I change them if I save a business document I like to know that it's replicated immediately and it works well the reason I don't use syncing for my videos is because every time I'm creating a big file it would then have this hammering effect of all these big files I'm shuffling around syncing also isn't doing block level looking at the underlying basically how ZFS does the block level changes that are occurring on the disk so if I actually move to a folder a bunch of let's say nine terabytes or nine gigabytes maybe nine gigs of files I just created for a video it'll also go hey look new files let me re-sync them hey let me re-sync these everywhere so it'll be less ideal so it comes onto your use case also revisioning you can turn it on and sync things so you can set up all your revisions you also have snapshots which work as revisions so think about that from a hey what happens if someone destroys this data how long will those snapshots last two weeks maybe how long did you set the retention for because if I deleted your file today you only have until those snapshots expire to be able to retrieve it so you have to like put all that into context to decide which backups right for you why not both have all have have all you sync thing and then you snapshot it that's actually one of the things I do with my business documents but you ever use VMS mm-mm they had an interesting thing that file names had versions in them like so like when you edited a file it was no longer like file semi-colon one it became file semi-colon two and you could still reference file semi-colon one it was just an inherent thing built into the file system Oh that's actually kind of cool yeah much neat well we went an hour and seventeen so I have to go to lunch yep and I got some stuff to do so thanks everyone should join and we'll see you next time see you later later