 And welcome to The Home Lab Show, episode 61. I had to look over my shoulder real quick to go, wow, 61 episodes. Today we're talking about Mycroft Mimic 3. I added the word Mycroft in there because if you look up Mimic 3, apparently it's a movie. And- Yep, I saw that on YouTube when I was looking up, you know, whether my video showed up or not. I'm like, oh, that's interesting, there's a movie. Okay. Yeah, there's a movie by the same name. And we're not talking about the movie. This has nothing to do with that movie. We'll just show it out there in the beginning. This is gonna be about the text-to-speech system that's within Mycroft because this is a really cool topic. Jay's been working on it. Then we have some DevRandom. We have a few things that got changed and we didn't wanna stuff the title with everything that changed because that just becomes difficult. So we're gonna stop for the main topic and then we're gonna cover Q&A about a few new updates and things that were released. But first, let's thank a sponsor. And that is Linode. They've been with us for a while, sponsoring this. They are literally, if you've downloaded this from our website or visited the Home Lab show website, it's all hosted on Linode. Jay maintains the software on there, but Linode is where all the servers reside. They have been a great sponsor to the show. They are a great place where you can get started with any of the projects. So if you just don't have the room for a server or it's something that you think would be better public-facing, Linode's a great place to put it. Use their servers, not yours. If that's the use case you're looking for, head over to Linode and use the offer code, the Home Lab show. And thanks a lot for being a sponsor and yeah, get started on your next project with them. They've been great. This is something we're gonna have to host locally here is the Mimic 3 engine. I'm assuming we have to start with it being locally hosted, right? Well, so I think some discussions in order about a little bit about Minecraft. We already had a full episode about it, so I'm not gonna go over all of that. Our new listeners maybe a background of what Minecraft is. Yeah, exactly. So because you have to know that first, right? Well, you actually don't have to, but it just makes more sense that way. So what's the first thing that you think of when you think of buying one of those voice assistants from Best Buy or one of those stores? That they're spying on me. I think they're spying on me. Right, and that seems to be the general consensus and it's not wrong. I'm big on IoT and I don't own an Alexa or any of those other things. Sorry if I accidentally activated your smart speaker. Cancel, cancel, cancel. Cancel, cancel, cancel. Yeah. Cancel, cancel, cancel. Bycroft is an open source voice assistant and it's something that has been a part of my channel for, I don't know, I wanna say three years or more. It's been in the background and I've actually showed it in a few videos but I didn't really do anything with it. But anyway, it's an open source voice assistant that doesn't spy on you and that's awesome. It hooks into Home Assistant and many other things, the integrations and it's kind of crazy how many there are. And they're coming out with a Mark II which I have a prototype on my desk for the few people that are live stream that will buy few usually grows or the podcast audience is obviously not gonna see this unless I could just kind of teleport the thought into their brain, that would be creepy. We don't wanna do that. You know, and don't worry, you can certainly look up what these look like and you can even watch a wonderful 20, was it the 2021 April Fools video or Minecraft in its own review and reviewed itself. So Jay has a great video on that. Because people are asking me, do a Minecraft video, do a Minecraft video. I wasn't ready yet because I really wasn't in a place where I felt like I deep dived enough to do a video that people would like on a technical level. So I'm like, but I have to deliver. People want Minecraft, April 1st is coming up and I'm like, well, you guys get what you asked for. It's not quite what you asked for, but it's what you get. So I had a lot of fun doing that. And the reason why that worked out so well is because it's a good sign when you can SSH into your voice assistant. That's a good sign. That's a very good sign. Like I don't know if people generally jailbreak their voice assistants, but you shouldn't have to do that. I'm just making an assumption that you can't by default and I have a feeling I'm probably not wrong. So with Minecraft, you have the source code, you have the ability to SSH into it. You can do whatever you'd like. I mean, it's yours to, I mean, you could break it. Don't break it, right? Don't rm-rf slash, that's the worst thing you can do. But if you could do that, it's not, you know, it's under your control. Don't do that. So the topic of today is Mimic 3. Mimic 3 is the text-to-speech engine that powers Minecraft. And you could consider the Minecraft devices as sort of, oh, like reference units basically for Mimic technology. Basically, you could buy an all-in-one unit, the Minecraft unit like I had in the original video, or you could just get a Raspberry Pi, install it on that, do that for free. You could put it on your laptop or desktop if you'd like, which would be actually something I want to try on my end. I've been meaning to get around to that. Ooh, okay, we're not gonna install a laptop. What's that? Would it work on an older laptop? That might be interesting because it's got speakers built in. Well, I think it will because there's a certain point about Mimic 3 that I like the best. But Mimic 3 is the latest version of the text-to-speech engine. So before the issue was that even though it's a voice assistant that's on our side, it needs the cloud to do processing, which kind of might make some people feel uncomfortable. Now, I'll be clear. I don't feel like using the cloud at all by itself is a red flag. It can be, absolutely it can be, and it often is, but when you have an open source technology, it's not uncommon to utilize the cloud to process some of the workload, but the selling point of Mimic, or I shouldn't say selling point, but the cloud is not required for Mimic 3. You could have a device offline, no connection to the public internet at all, and it could process speech for you. So that basically means by association because the Mark 2, which is coming, I think it's September if I'm not mistaken, that's going to have Mimic 3 built in. So you'll have a voice assistant that can do that. Now, before you couldn't do that because older Raspberry Pi boards were even more constrained than they are now. So processing speech is just not going to happen. On one of those older Pi boards, and the Mark 1 of course uses one of the older Raspberry Pi's, and Mycroft is not, they're not hiding the fact that there's a Raspberry Pi inside the case, okay? That's pretty clear, and they mentioned that too, which is also why you can build your own. But one of the coolest things though, I mean, the fact that they have, like, I don't know how they did this exactly because I'm not on their development team, but they made the newer Raspberry Pi boards, or they made the code efficient enough so Raspberry Pi's even can process a synthesized speech. So if you want to try it on an older laptop, I would pretty much guess it's gonna work fine because your laptop that you consider old is probably more efficient than a Raspberry Pi, at least not more efficient, but faster. Yeah, it has more processing power. Yeah, so I think, yeah, most likely it would. I mean, I installed it on a computer myself, but the computer that I installed it on, it was an Intel NUC. Honestly, like that Intel NUC is probably more powerful than a Raspberry Pi, but the fact that it can run on a Raspberry Pi, that's pretty cool. So that's a pretty good metric when you develop something. If you can get it to run on that, chances are it'll run on pretty much everything. I think too there's a lot of challenges with language because we all speak differently and we use words maybe in different orders and we're just talking alone about English, not getting into the diversity of other languages. So I think that makes it a little bit more challenging versus Raspberry Pi's doing object recognition to figure out what a banana is or an apple is. It's actually something that's been around for a while. There's some Python libraries and projects for that, but speech is a challenge in building the data sets you need to understand speech. I think when we talked about it in the Minecraft episode, I think that was why they hadn't spent so much time was building these models to constantly be listening to the variations of the way someone says even a single word or the phrasing thereof to understand the context to be able to turn it into a command. So I think it's taking a lot of, probably a lot more hard coding work to make that work is not just having something powerful enough to do all the work, but all the coding that goes into understanding language. Right, right. So yeah, I mean, there's just so much to it when you have a Raspberry Pi and you try to get it to do various things and making the code run efficiently. But yeah, in the object recognition, I totally forgot about that actually. I think we were speaking about that not that long ago. So yeah, there's definitely a lot to be said about that. Now Mimic 3 can be downloaded as like a dev package. The source code's on GitHub. There's a dev package for AMD 64 for Raspberry Pi. There's a Python package. There's a Docker container. That's just off the top of my head. So if you want to run it on something, there's different formats that you can download it in to get that to work. So I definitely recommend checking it out. And the cool thing about it, I mean, it's just free in every sense of the word. It's not like free, fill out this contact form and then blow off the salesperson. Don't answer his phone calls. Every starts calling you to get that download link. You actually just go and download it, which is really cool. And that's also a good sign, because if something's behind a contact form, that can kind of, not that contact forms themselves are bad, but if you're forced to fill one out, that's, there's a catch to that. So I've been talking to MyCraft for, you know, on and off for a long time now. I don't even remember how long. And it actually was not the April Fools video that got their attention surprisingly. I kind of thought it would, but I don't know if they actually saw it before, you know, I started talking to them, I don't think they did. And then we decided to time the release. Well, I should say I decided to time the video to release at the same time as Mimic 3. So today's actually released day for Mimic 3 and released for Mimic 3 itself was set for 11 a.m. today as a recording time, the same time the video showed up as well. So it's out and you can download it, you can hack on it. And the reason why I kind of decided it'd be great to focus on this in a video to get the word out is because it's an open source project. So if you want a project to contribute to, here you go. It's an important project because if we don't step up and develop an alternative to the, you know, devices and voice assistants that are stealing our personal information, then we have no hope. We have to step up and develop something MyCraft has done that. Obviously, you know, he would probably benefit from free firmware and all these other things in our smart plugs and light bulbs and things, but this is a great step in the right direction. So I created the video today just to kind of talk about what Mimic 3 is, get the word out, but also it's the first video in my MyCraft coverage on the channel because I do hope to do additional videos for MyCraft. So I think even though I showed MyCraft in an April Fool's video, and then again in a open SUSE review, I had him and I both. It was kind of funny. We were kind of like talking back and forth, although I was behind the scenes with SSH and telling them what to say in response to what I'm saying, but it actually looked like a conversation that was really fun, but those are just fun videos, not, you know, MyCraft internally focused videos. So today's the day. So Mimic 3, I figured it makes sense. It's the text-to-speech engine and it's out right now. That is really cool. Yeah, and if some things, well, let's back up. Can you just load this or does it have to integrate with MyCraft? Can you just grab this text-to-speech engine and start playing with it and start, does it have like a framework and API? How does it integrate? I'm so glad you asked that. So when you install the package, the first thing that you get is the Mimic 3 command right in your terminal. So you could type Mimic 3 and then a quoted string, for example, if you want it to synthesize speech. It has the option of playing the speech back on the speakers of your device. If you have a device that has speakers, you could definitely do that or you could have it create a WAV file that you could then use and put wherever. So if you think of like, you could put it on a server and have it like generate synthesize speech right in the, right in a cron job even. I think about you have a cron job that just kicks off a script and it'll maybe produce, I don't know why you wanna do that in a cron job, but you can. So you could get a WAV file or it plays back over the speakers. There is an API. It's- I think I have a use case for this. So you start scripting this and then you start having it just outputting so the computer can just tell me, please open the pod by Georgetown. That's true, but I mean, if you think about it, you could have it, let's just say you needed some kind of assistive technologies. You can actually roll this right in if you wanted something to announce what time it is because maybe your vision isn't such that, because let's be honest on any operating system, if you have challenges with vision, you're not gonna get the panel thick enough to make the text of the clock big enough to actually see it. You could actually have something like this just read out the time or if there's some way to hook it into a calendar, if your calendar has an API, you could probably have it read out your calendar appointments as they become due. There's just many other things you can do with that. So it does have an API. So what I was able to do is just, I have the reference device as well, be actually the Mark II prototype. So I could just actually hit the API right from a terminal, make it say something, which is pretty cool. It also gives you a web console. I forget which port it is. So you can go into the web console, which I'm pretty sure is being served by Nginx. It just gives you a text box in your web browser. You can just type text into, click the button and then you can get that text right out to you and there's different voices, different countries even, different accents, right? So you have all of that already. So, and that's just the core of it. And then from there, I'd actually be kind of interested to see what people do with this because I'm sure there's, you know, you of course, just now, but other people like, I know what I'm gonna use that for later. Well, and I think one, our first missed opportunity is we're 15 minutes into the podcast and we didn't have it do the intro to the podcast. So that's, we, if we do another one on Mimic, we will have it do part of the intro. Also, someone said, it's probably great to have it talk to the phone scammers. And I'm like, I see that in the chat. That's an amazing idea. Yeah, like I can just have it script. Like you guys can just all, all the phone scammers, you can just have a conversation with my, with my automation tool. That would be great. Yeah, it'd absolutely be a good use case for it. And yeah, it's getting me thinking now too. Cause ultimately my first thing was it was, I just want to see what it would be like to have a voice assistant on PopOS to say, hey, you know, name, launch my browser or whatever I want to do. Not that it's that much effort to really click on the Firefox icon, let's be honest, but how cool would that be? And what kind of a talking point would that be if someone was over and I just told my computer and opened an app and it did, that just be a lot of fun? We're all inspired by Star Trek. So I mean, we watched, they talked to it and that's at least one thing that's consistent throughout the Star Trek series is they really got the, the language systems figured out one for the universal translator, but two, so you can just talk to computers and they do things in a more natural way. That's the real vision they had in Star Trek was like, hey, this could be possible one day. They just didn't realize that it, most would be done by large companies that would try to figure out how to monetize the fact that you exist. So I'm going to attempt to do something right now. I don't have a static IP. I should have set that up. As long as the IP didn't change, we're going to see if this thing can actually announce the show, let's see what happens. All right, we're doing this live here. Welcome to the Home Lab Show episode 61. Oh, that's beautiful. It came through very well. I like that. It sounds so much better than the Mark 1 because if you like play back the April Fools video and there's even some of the Mark 1 in the new video, you'll hear the older voice, but it just sounds so much better. No, I definitely like that. Oh, let's try this one more time. All right, we're waiting for it. Hello, Tom, have you tried any new hot sauce lately? Oh, I had to too, cool. That is so much fun. And I was able to do that by just typing echo and then a string in quotes and then it's in a video, but you just pipe it to curl dash x post the IP address. It's a little bit more than that. Then there's an API string and you give it the target, which is going to be like your speakers or the target could be the other device where you want it to play back. So I gave it target server, which that guy right there is a server. So it receives it and then speaks it. I now want to put some of these around the house and then when I hear, because my studio is in my basement, my wife will occasionally yell something unintelligible, but then I can untell as we see something back or I can simply type into the speaker, target living room. What did you say, honey? Exactly, no, that would be, I could mess with my kids, it'd be kind of funny too. Oh yeah, the kids when they go in there, have it tied to your refrigerator, build it with home assistant. I'm gonna give it a challenge. I'm gonna give him a sci-fi reference to speak. Let's see if he's able to do it. I'm sorry, I can't do that, Dave. Yep, so the power of open source, what can I say? Yeah, no, this is really cool. This is definitely neat. Now you have a whole video on this release as well, is that correct? Yep, it's about 15 minutes long. Somebody in the comments mentioned that my audio was out of sync in that video. I hope, I'm sorry if that's the case, when I played it back, it was fine. So I'll double check that in a moment, but yeah, the video's out, it's about 15 minutes long. I show some examples via the command line and also give a behind the scenes look at how I did it with the April Fools video. Obviously it's not the footage from back then, I didn't go back in time to where I was doing it, but I re-enacted how I did it back then. So you'll see at some point in the video how I actually made the mark one conversate with me in a video, actually also in the open source review as well, which was I think now the version or two ago, wow, time flies. Yeah, for sure. So that's what we have for the Mimic. Anything else on the Mimic or we wanna talk about ZFS? Well, yeah, I especially wanna talk about that because I am offended by how easy this was. And by offended, I mean like, so many other people and companies out there do it wrong that when someone does it right, I just wanna just shout from the rooftop and just set an example. And there's a lot of that lately, like for example, TrueNAS being able to, has its OS on a completely different volume and its settings backup and data is all separate from that. So you can switch between different versions of TrueNAS, which is really cool, that's doing it right. And then when it comes to PF sense, you and I were talking about ZFS and how it's good to have. So I'm like, okay, I will go ahead and give it a shot, but in my case, that means a reinstall because I'm not on ZFS. So what I did was I, like any good person does, they take a backup, I took a backup, I grabbed that backup and I put it on a flash drive that was just, I think I had it, I think it wanted like fat 16, fat 32, I think it supports both. Yeah, you drop it on there and then on the other flash drive, I loaded PF sense by downloading the ISO using DD, I copied it over there, booted my firewall from the USB stick and then I had the other one with the config file on there. I just told it to wipe everything and reload. And when it booted up, everything was like it was before I did anything, like all the networks came back, the subnets, the VLANs, the rules. Let's give a little context to this real quick though. You jumped a little bit ahead. So what, when PF sense 2.6 was released and the PF sense plus 2202, I think it was, they both started with ZFS as default. Now you could load ZFS prior to that, but it was not the default install and it's here any of the default to Steve Gibson says is very holds true. Most people next and yes, the way to install. What happened though was after they defaulted to that, they've added more enhanced features to support ZFS. So they've actually give you some functionality in the UI. Most recently with PF sense 2205, you get a lot more functionality with the ZFS options. But of course people asking Tom, how do I switch to ZFS? And there's not a switch to ZFS. But as Jay said, if you reload your PF sense box, grab that config.xml file, make sure you have a backup of it. And as Jay did, he popped it on another flash drive during the install process. So one flash drives your installer, one flash drive has the config.xml file. You just wipe, reload, but then while in the reload process, you can do it after as well, but while during the reload process, cause it's faster, you can grab that config.xml file. And when it reboots, it's just, it's right where you left it. All your settings are there. Like Jay said, the VLANs, the settings, you don't really have to monkey with anything. You don't have to spend time goofing around to get it set back up. Everything just restores as if nothing happened. So now the only thing different is you have ZFS functionally, you had a little bit of downtime cause you wiped and reloaded your firewall, but you didn't have to spend time assigning interfaces or anything cause you're reloading on the same firewall. They've done a nice job, as Jay said, of making that really simplistic to do. It's why I see people asking, there's actually a discussion a couple of times in my forums that happened where, oh, I take full images of my hardware device. I'm like, you don't need it. Just make sure you have the thumb drive laying around with a copy of the operating system on it and your config.xml file. Those are the only two things you need to reload PFSense and put it exactly how it was. And considering it's an XML file, you can open it up and make sure that it's actually readable, right? Cause we always tell you test the backups and simply reading an XML file does not constitute testing a backup, but it is something that you could do to see if you can open it in a text editor and there's not a bunch of gibberish in there. So yeah, I mean, it works just fine. And like Tom said, it was just, I mean, I waited until everyone was sleeping and then I just did it. And there was just nothing. I thought knowing me and my bad luck, I was going to break something. Something wouldn't come up right. But I actually thought that maybe I would have to reboot again just to get the settings to apply. But no, it was there. Like everything was there. I did choose to like not have it save the DCP leases when I took the backup because I don't care. It's just going to, everything's going to reconnect anyway. So yeah, pretty easy. Yeah, and that's a neat option that I think they only added after 2.6. In 2020 was in that feature came out. So you can, I think back up the leases before that was an option. This just wasn't a checkbox there, which it can cause a little problem if you ever have to reload a firewall. If it doesn't save the DCP lease table, leases are obviously kind of static. I mean, there's your static leases that are absolutely in the config, but it's the all the other ones that were handed out. If you don't save that table, it'll rehand things out again, except things that already have the lease won't ask for a new one until that lease expires. So you sometimes can end up with conflicts. We actually run into that a lot when people swap firewalls just in general. They just upgrade a firewall and they're like, a lot of things aren't working. I'm like, reboot all the things, power off the switch, power it back on that usually will prompt many devices. If they see a status and link change, a lot of them will go, oh yeah, I should ask for a lease again because I even know my leases and up, I see no link change. IoT devices are hodge podge random. They may have to ask for it. They may have to be power cycle to ask, but yeah, the process on the PF send side is pretty painless. Sometimes on the other side, just reboot things. That fixes all the things. Have you ever noticed that on residential routers, just the kind that just, the wifi routers you buy at like Walmart where a device will reboot and get a different IP address, but usually things ask to use the same IP address again when they go to reconnect, but I've noticed that some of the residential ones that strangely enough, devices will constantly get different IPs. Whereas with the F sense, it just does a great job that unless you change something, it's just gonna, I'm not saying you can rely on DCP don'ts. I'm not saying that, but what I am saying is that every time I've experienced this that everything gets the same IP address again, more accurately than most I would say. Yeah, it's the way PF sense holds onto the lease table. I believe even over reboots, it's only on a reload that it loses the lease table, but I think that's not the same for some of the community ones, not community, but not community, but like the consumer ones is what I'm looking for. Some of the consumer ones, if I'm not mistaken, don't, they don't care. They're like, this is a temporary table held in RAM. And oh, we rebooted. Time to hand them all out again. And I think that's common from a design standpoint. No one thought to even take the time to write some temporary file. Of course, the limited amount of rights you could do on a consumer one, they don't have a lot of storage on them. They're, well, not carefully crafted. There was even, I tweeted today about another attack on a big collection of different consumer routers. Some of these sound like there were some unknown vulnerabilities in them. And they're, yeah, they're not well crafted. They're not always well vetted. People are constantly finding problems in them. And just because problems haven't been found, it's only because, almost all the time, only because no one has really taken the time to poke at them. So I'm not going to say that they're the best. It's becoming clear and clear that they're not great to use. And I think a lot of our audience probably knows that. It's the general populace using them that is causing some of these issues. Yeah, you say that. And then literally today, there's a modem upgrade that I have to endure from Comcast that I kind of just put off because I don't really need it, but they think I do. And it's, I say $50 long story, but they're going to bring a new device over. Like, yeah, it has Wi-Fi, has this feature, that feature. I'm thinking, eh, I don't care. Like just put the stupid thing in bridge mode so I could use PF sense and just turn all that stuff off and also kill IPv6 because I'm not an ISP, but hopefully everything goes according to plan. But it's just every time I have a new modem that has all these features built in, I'm like, yeah, what I have is really good compared to that. And it always works. It's just great. I mean, doing it yourself. Now, another topic I want to talk about is there was a new release to TrueNAS scale. And it's another update to it, but I took the time to really do some in-depth testing. So I like scale. It's not, people think I'm disliking the product but it comes down to use case as I always tell people. I love the fact that they've built some of the Docker images in and they have a lot of features around that. The challenges that have come with it, specifically in my enterprise realm is the storage consulting I do for high-performance storage and it's not high-performance yet. So I did a video on this already where I broke down all the numbers, but essentially when it comes down to 4K and 8K small block size writes that are really intensive, which, you know, that could be a database application. That could be a bunch of small writes because of some workload that's running in a VM and you're using it as a storage target, it falters a lot. Now, what I've not been able to prove and it's just a matter of, it's a lot of work to set this up, a few people commented though to me in one of them, I think DM me on Twitter this, this is a Linux problem where Linux has a harder time with small writes high IO, allegedly according to two different people that messaged me. I don't know that this is true until I really test it. They didn't give me credentials that made me go, oh yeah, this makes sense. But I thought about reaching out to maybe like Wendell from level one and asking him to see if he's seen some of the same things, if it's a BSD versus Linux controversy, because they're neck and neck on larger writes of larger blocks. But when we were doing the testing and scale being based on Linux versus TrueNAS Core being based on BSD, that's the only stark difference between them is those small writes. But when you start using the larger writes because this is exactly, we took the same hardware, reloaded it with scale. So we ran it on core, ran our battery of tests, reloaded it with scale, ran our same exact tests again. So we're dealing with the minimal amount of variance other than the operating system being different. All other things were the same. And that's the only substantial difference we've seen between them. So I think it's interesting. I don't know, I know there's reasons even like Netflix to this day is built on BSD for some of their delivery because they could do better kernel tuning management with it. And BSD has been around for a long time doing tuning for things like that. Specifically, contributions to the BSD kernel come from the people like at IX Systems to get all of this tuning. It's not like they started yesterday doing it. They've been doing this for years. And as product bases and code bases mature with enterprise-level use cases, which BSD has a ton of them, there was a lot of engineering that went into solving small write problems because it is an issue. You think of what a lot of the world runs on today, what are cloud applications, especially ones at scale? Tons of database writes, tons of small writes. You can say all the layers in between and you're like, but aren't they running self-clusters? And they spread the load out over this? Well, sure, but at some point somewhere there's a tiny four kilobyte text type that someone did and updated some status on some social media site that eventually had to be translated into a small write and someone tuned the performance to take that little bit of 240 characters on Twitter, turn it into something that got written to a disk somewhere. And I'm willing to bet somewhere is probably a BSD machine where it landed on. I don't know exactly, it runs the back end of all these hyperscale companies, but I know we've consulted enough with the TrueNAS Core to know that BSD lives at the heart of a lot of these systems. So I think that's kind of the broader answer as to why you see a lot of really high-performance systems being on free BSD and Linux, not that it can't do these things, it has not been tuned to it. I mean, TrueNAS scale is still a pretty new product and the team is working diligently at solving these problems. So they're not going unaddressed. People are like, they need to address it. I'm like, they will, it just takes time to do. And they have integrations are doing right now with Kubernetes that has its own complexities that they're integrating. On top of that, they're also building clustered file systems. So there's a lot going into this project at once. That's why we sell people be patient with it. It's not that I don't like it, but I don't know that I would run, if you have a production workload that has that demand on small rights, it may not be the right system for you. Do you know what scheduler TrueNAS scale uses? No, I do not. Because I don't know if that's the reason, but I've noticed that, I've noticed IO problems. I mean, I use Linux on everything. It's not as bad now, but I remember it often being the case that it comes down to which scheduler they go to use. Like, I remember when I first started even, there was a big push to try to get the BFQ schedule, scheduler to be the default in distributions and whatnot. But what I would notice is that if I was backing up my hard drive or copying a bunch of files to a external hard disk or something, like I couldn't even browse the web. Like forget using my computer on most Linux distributions. I just can't. Like, eh, it's gonna take an hour to complete. I better stop using my computer completely until that's done. And that got really frustrating. Like even sometimes writing to a flash drive would be a problem. And it's not as much of a problem now, but it still does become a problem. And I think that could even be why Papa West decided to come out with their own scheduler for their distribution because of that same problem. The IO scheduler issue in Linux is just one of those things that it's like, why is this still a problem? If I can't copy files to a hard drive without the entire system slowing down to a crawl, it needs to be fixed. There's no excuse at this point. Like it's been years. Like what's going on guys? Like this is not the way it should be. But again, it is better nowadays, but it's still not great. But I haven't really noticed it as much if at all on Papa West because, literally I was just copying the entirety of my unedited videos from TrueNAS to a local external hard disk attached to my Thelio. And it's fine. I didn't even notice a blip. So I don't know if that's the issue, but that's the first thing I would look at. Yeah, there's the other issue. I think someone mentioned it, chat smb.com. There is also a challenge when you have thousands and thousands of files and you're trying to write them using Samba. Samba needs some retooling to be able to scale. There's working on it. It's on the roadmap to better deploy some new technology into it in a way it handles that, but there's some files. So sometimes people realize they have a 10 gig connection and a full flash array. And when they're moving large individual video files, as I do, they get this incredible maximum performance. And then they try to copy their collection of 10,000 memes over. And they're like, but the 10,000 memes is less than the one video file Tom moved. And it's taking longer. I'm like, yep, welcome to Samba and writing 10,000 small JPEGs or PNG files somewhere. It'll have a much bigger problem. And it's someone said it's slow because of user space, but it's a little bit more complicated than that. I thought about doing a video where I break down the complexities of it because Wendell's actually dealt with this a lot. I think it'd be a good thing where he's could talk it to, it's the way it utilizes, it's something about the way that it has to do these lookups. It's not just a user space, kernel space problem, but there's some other functional problems with its design. I can't remember what it is, because even Microsoft has moved to a different way of handling it. And Samba is going to follow the way Microsoft is. It's something with the way it has to pipeline things, not through the kernel, if I'm not mistaken. I could be a little bit wrong on that. I remember reading about it and said, that's interesting. I understand it better, but then the knowledge fell out of my head and I can't articulate it right now. So I need that white paper in front of me for how to get that to work. But it is also why a lot of people don't do Samba for those type of services. Like you're not going to use Samba if that's your use case you're gonna use. It's still gonna be faster, probably to even use something like NFS. So there's other ways to handle that. I think another topic worth bringing up really quick though is, I mean, I need to rant, but I'm gonna try not to sell it as a rant. But why on earth are these problems with shared file systems still a problem at all? Like NFS, even an NFS4 has locking issues. Maybe not as bad. They've made improvements, NFS still sucks. Samba continually makes improvements. It still has problems. And now just the other day, as I'm writing the book that I'm wrapping up right now, I am faced with the decision to remove SSHFS because the maintainer decided I don't wanna do that anymore. So I don't even know if they're gonna find a new maintainer. If I can even trust SSHFS at this point because who could tell the future? This is literally developing right now. So I'm probably gonna have to remove that out of the book. So there goes that. Now we have, you know, Kubernetes, even containerization has happened since, and actually everything has happened since NFS because NFS has been out forever. But virtualization, you know, Kubernetes, we get into these new technologies, cloud, all these advancements, and then you have like overlay networks. That's an amazing invention, wire guard, that's great. Why the heck can't we put some actual attention on shares that don't suck? Like these people that put all of these great attention and their skill set on these problems and come out with amazing solutions, we need a new technology to share files. We just do because everything we have sucks. Yeah, and I think what part of the reason is it's solving for the minority versus the majority. The people who have those problems are the one offs. And I think that's where Samba works for 99.9%. Who it doesn't work for is, you know, the consulting work we do has been kind of specific to people in the graphics industry where they're literally working on films you've heard of and they have to do 3D sets and those sets require 20,000 photos to run through some type of parsing tool that does an automation to them and they're all a bunch of tiny little, you know, image files that then eventually get stitched back together into a move fast. Kind of weird how it all works. I don't know why it works that way. Someone else engineered that part of it. I'm just here to solve the or explain or look at better ways of connecting with the Samba problems with it and how we can make that faster. And I see someone even mentioned like I SCSI and that is a solution we do sometimes providing I SCSI because I SCSI is kind of like a workaround for file sharing. It's not a sharing service necessarily. It's presenting a block device over IP. So you've just kind of kicked the can down the road. Like I don't know how to get files in small batches to you fast, but we can wrap it in a protocol and send it over IP. Doesn't allow sharing, but it kind of kicks the can down the road of being able to have a large block device that can write things to that's part of a NAS, but then still have that fast interactivity. And it literally is sometimes a solution we have from a storage design standpoint to what we'll get the job done because we need some huge amount of storage that's unreasonable to stick into a Windows computer. But we can spin up a true NAS running I SCSI and make a block I SCSI mount and have a 100 gig backplane connection that allows it to do all these small writes to be able to get the task at hand done. So yeah. I mean, I do agree with a lot of what you said. It's just the way I see it though, what makes this strange to me is that companies complain that I've experienced, they complain because you think about all the companies out there that are saving a ginormous number of small files or doing deep learning or whatever shenanigans are getting into where they're processing a lot of data and there's a lot of companies out there. I often feel like when people are sitting down and they're talking about how do we design this? Well, the conversation will shift to these are the options. We need to go with one of these options. And then it's basically down to pros and cons of each then they choose the one that sucks less and they just live with it because what are they gonna do, right? Their IT team is just so spread thin. They're not gonna go invent a new sharing technology as part of their, unless maybe they're Wendell Wendell could probably do that. Everyone who's not Wendell though would not be able to do that. And I really feel like it would be an amazing benefit to come out with a modern file sharing solution cross-platform that actually works. And I know that's a big ask but then I've also seen the community come out with some amazing things that has blown my mind. So if the community is able to do that I would say that other people are as well. One thing that I've noticed is that some companies do solve these problems. Amazon being one, they have a solution that has fixed some of the NFS issues by, I don't wanna call it a fork of NFS. I'm not gonna get onto the AWS bandwagon here but they develop something but they're keeping it to themselves and they're not like letting other people have it. So you gotta use AWS if you want that. So there is a little bit of that going on where you have the cloud providers they will tweak things in such a way or maybe fork something, but it's theirs. I wanna see some replacement that's cross-platform that'll actually hook into modern file systems, hook into modern kernel features. They could probably blow away NFS and Samba performance with the right tweak. So that's my challenge to the community. I hope somebody will feel adventurous and take that on because I really feel like Enterprise IT needs you. They need this. So Samba, not the biggest fan, although I use it NFS, not the biggest fan, I use it. Now SSHFS is off my list because who knows if it's gonna get a maintainer. So, and that's a little scary when something like SSHFS is in potential danger of being removed from distros. You know, and I see people mentioning here in the chat and I agree with this because I said this at the, I mean, I add the full context to it. I said this is the end of the video where I did the comparison to the latest TrueNAS scale and core and the latest release. As of right now, do I think TrueNAS scale is production ready? Depends. Could you use it or trust it for your data storage needs, like, you know, a Samba share or a iSCSI extent? I didn't find any reliability problems in that aspect of it, but the implementation of Docker and the stability of updating and deploying apps, I don't think, and this is based on me even reading forum posts, there are several people with forum posts where they talk about the instance breaking and not allowing any of their Docker containers to come up. It just screwing up sometimes. And then my, I didn't see replies. I didn't check back recently, but it was people who reached out to me. And now the part that confused me was they were using this in a production environment. So they're like, hey, Tom, can you offer emergency help to sort out why all my Docker stuff is broke in TrueNAS scale? And I'm like, not really. I mean, I'm not a Docker expert for one and two. It's beta as far as I'm concerned. And I see people then posting in the forums, but after two days, I didn't even see a response in the forums that are questions about not being able to bring up any of their instances and also not having a good error log to go off of. It's not that Docker is the problem, but it's a new integration that they're doing with the team over at TrueNAS scale. It's cool they're doing it. It offers a lot of flexibility, but for production workloads, now the good news is even with this person, they didn't have a data integrity problems. The FS is still well engineered and well integrated into there. So the data's all there. They just can't get the Docker containers to come up. They're all in some stuck state. So once again, from production standpoint, if you're asking, can I store my data on it? Yeah, sure. Can I run my Docker images on it? Sure. Are they absolutely bulletproof reliable? Well, I don't know. If updates randomly break them, then no. And these people have built quite a few containers on there. So they're using it from a storage efficiency standpoint really well. And I get that, especially in a home lab, why have 10 servers when I can have one server do 10 things? There's a good reason for that, as long as it's done reliably and doesn't create a giant monster for you. Which is why, I don't know if you're familiar with it, Jay, but this is, I mean, do a follow-up video on it because I think Wendell did such a great job. He called it the forbidden router. Basically, he built an XEPNG system that does everything. It runs his firewall. It runs a bunch of all of his services, all running inside XEPNG and one single beefy system, because it's his home system. He didn't want to have several servers at home. So he did what he titled the forbidden router, and it's a three-part series Wendell did on there. I thought about reaching out to Wendell, see if he wants me to do, as I don't think he did his in-depth on how he configured PF Sense. And I thought about doing, because people ask about it and why not? I'll build a forbidden PF Sense router. It is for fun and walk through the details of that, kind of a fork off of Wendell's project if people are interested. It's pretty cool. It's fun. It is, it's a really cool. I liked his title for it. And his title is the reasoning of, can you do this? Yes, it'll also test some of your skills, test some of your learning, and it could possibly test your sanity if it all falls apart with an update, because you've integrated so many things so tightly together, and you have to update, because there's always security things, there's always issues. So not updating is great in terms of stability, but bad in terms of security, but then your updates to keep you secure can affect your stability so that it becomes some back and forth. But Wendell did something rather clever, which was he did a drive pass through for PF Sense. And the reason he did that, instead of putting it within the hypervisor, when you do a drive pass through, if the hypervisor goes belly up and all the services go down, you go into the BIOS, you boot off of the PF Sense. And PF Sense now can boot, because he's also passing, he's had a couple of extra network cards and he did a pass through with the network card. So PF Sense is already using a pass through. So if you reboot the machine off of the SSD, in theory, PF Sense will go, hey, there's the network card I was already using and then boot and run, even though your hypervisor's broken, at least you're online, because that's what matters, because you're gonna have to get online to read download and troubleshoot all the problems that occurred. So one thing I wanna mention though, disclaimer, I haven't really looked very closely at true nascale yet, because hearing all the people saying, this needs to get fixed, that kind of makes me wanna wait until the next major release, just a thing. But I recently ran into a lot of work where when I was updating the book, Mastering Ubuntu Server Fourth Edition, I'm gonna get that plug out there, it's coming. The chapter that's changed the most, it's a toss up between AWS and setting up a Kubernetes cluster because it just didn't work. Like everything I did and how I did it in 2004 did not work in 2004. And I know that true nascale is not based on Ubuntu, I'm not saying that, but they do share a common ancestor with Debian as I understand it. So there's been a shift towards container D rather than using Docker as the containerization engine within Kubernetes. And when I went to update the book so that it's using container D rather than Docker, which is the correct thing to do, it, nothing worked. Like it kept all the containers just fell over, they were broken, nothing would work at all. And I think it took me a good week to figure it out. And all of the how-tos and blogs, I'm not gonna say they were wrong, but each of them were missing something to get it to work. And that's one of the reasons why I intend on doing a video on this. I don't know what containerization engine they're using on a true nascale, if they're actually using Docker or container D, but if they're switching over to container D, there could be some issues because it's been said that container D should be just a drop in replacement. Anytime I hear that, oh boy. Oh boy, that scares me. The only drop in replacement that's ever worked well for me personally has been MariaDB replacing my SQL with that. That's fine, at least in my case, but when you deal with containers, you're also dealing with a container engine. A container runtime is the term that I'm supposed to be using here. And that's gonna be a problem for some people. I don't know if maybe their developers are running into some of the same problems. I don't know, it could be that, but there's definitely some challenges right now as things migrate over to the new normal. Yeah, I think it's a lot of this the way they're integrating it. It's a big undertaking. And it's probably, I understand how hard it is to code a lot of this stuff. And simultaneously, some of the same team, at least the same company, they were able to pull off TrueNAS 13, which went really smooth. I didn't really have any issues with 13. There's gonna be a 13 U1 come out because I mean, there were still some bugs. It wasn't bug free, but a lot of them were minor bugs and there was work arounds for it. There's like a problem with not being able to replace a drive. There's a couple of UI elements that didn't work right. Like not, so if you could still do, there was just UI elements that made it goofy. I found those before as well. There was like a, you couldn't unlock a nested dataset from one of the menus. You always had to go to a second menu. I reported one of the bugs as like, this is, it still works. It's just got a UI element bug where it puts the data in the wrong spot. Well, that's pretty bad if you put data in the wrong spot. Yeah, yeah, yeah. It is well, yeah. Just to summarize the point, I've been, you know, as someone that's been using Linux for like two decades, actually more than two decades now, it's, I get thrown through a loop myself when something like this happens and we migrate to something else and then it doesn't work as expected. It happens to all of us. So even developers as skilled as them, I mean, they're gonna run into the same problems I run into, you know, every now and then something throws you through a loop and you gotta figure it out and make something work that technically should, but isn't, that's the most egregious and frustrating, but they're getting it done at least. So just give them some more time. Well, and it's worth noting too, and I made sure to comment to this in my last video, they removed the pip installer because people were loading all kinds of extra things. And this is where from a developer standpoint, you know, it is pretty clear, true NAS, whether it's be core scale, these are not full blown distributions. These are appliances. And what I mean by that is they're meant to be used the way the developers intended to. They didn't give you an operating system and say load true NAS scale as an add-on for you out OS. They're spinning the entire thing into an appliance. And what happens is sometimes the tech support problems get increased because people started installing their own Python packages and started modifying the underlying OS. So now when they say this doesn't start, they're like, well, that's a weird error. How did you get that package on there? So you can also sometimes that can be a contributing problem to building an appliance and providing support for it is people monkeying with it. Cause they actually, they have someone did a request, hey, in the new version, the pip installer's gone and they close the ticket going, you're not supposed to use that. You're supposed to use R UI. Don't go in here and start adding packages cause it messes things up. One of the first things I did with scale as a test and ended up reloading it immediately after cause it broke. I said, oh, cool. It's based on Debian. This is before it was even in, it was in its earliest releases, its night lease. I taped app get update and it worked once and then it didn't boot again. I am so in agreement about the appliance thing. That's exactly what these are. That's exactly how you need to treat them. If I was like back in my beginner days and I had, I still had my Ansible script that I have now, I would absolutely point that Ansible script over at TrueNAS scale and have it configure everything like it is on my local Ubuntu system because why not, right? You can make your, you can have the same dot files and the same packages on everything, including TrueNAS, but now that I'm more seasoned, I have learned the hard way that these types of things you consider them appliances, if you don't want to consider the appliance, you could absolutely just create your very own Debian server with stock Debian and install Samba, install NFS and all the different bits and pieces that you need to build a NAS, you can absolutely do that. But if you're going to customize things, you should do that and not use TrueNAS in my opinion. But to be fair, I would have done the same thing when I was, I'm not saying whoever that was that, you know, broke it is a beginner. I don't want to make that distinction, but if it were me, I probably would have done that earlier in my career. So I'm sure they get a lot of that. I'm sure they do. And this is the reason, and I've covered this before, maybe I need to revisit it again. I work with the team over at 45 Drives and they've done a great job with like cockpit. And this puts a nice UI that allows you to do a lot of the storage management and everything else based on a standard OS. So you take the standard OS, you can load your ZFS on there and that OS can be a Buntu, that OS can be whatever, you know, the base underlying is a cockpit supports, but I think I did my demos all on a Buntu, but now you have an Ubuntu based system that people are very familiar with that you can do whatever you want with, but you would like, and I understand the need for it, especially when I demoed it, we were managing 30 hard drives with it. Well, 30 hard drives from a command line, that's a little bit more tedious when you're building out the ZFS off. Not that you can't do it, not that you can't script it, but it's a little bit trickier to manage. Doing the cockpit UI with the 45 Drives and they have it all open source, they made some open source contributions to add a ZFS manager, a Samba management, so you can use a nice web UI to manage all of that functionality pretty easily. It works really well. So that's where you probably wanna go. Matter of fact, they go a step further if you go over to the 45 Drives channel, they take Proxmox and then load cockpit, I believe with that. I think they have a whole video on that now. So they show how you can build out even further integrations because some of these things that you wanna build out like that that aren't just appliances, they're full operating systems, you can. True NAS is more the exception for it. So they're saying, I wish cockpit had better ButterFS support, but understandably why it does not. Yeah, I don't expect that. ZFS is pretty much, when it comes to, ButterFS has used so little compared to ZFS, I'm not saying it shouldn't be used more, it's a good system, Jay's got a video, Audnet as well, as it dies deeper in a topic, we've talked about it before on, I don't know which episode, a couple episodes ago we did talk about ButterFS. So, but yeah, ZFS is still the king. And by coincidence, I wore my shirt today, it says ZFS is a cult with integrity. So. Yeah, I guess anytime you set up something in cockpit, if it doesn't support ButterFS, you have to have the comment or a pop-up window that says, I can't believe it's not ButterFS. Yeah. I'm sorry, I'm sorry. Oh boy. Yeah. All right, I think we covered all the topics, we got anything else on the dev random list, Jay? I have like a zillion things, but also an intro to a nine, probably not for now. Fair enough. All right, well, this has been episode 61, Mycroft Mimic 3, for those of you that find that Mycroft Mimic 3 is a movie, that's what we're talking about here, I did leave a link so you can get down there. Let us know, send us some messages, tag us on Twitter with what interesting things you have it saying, and I fully and totally expect at least some percentage of you have it swearing. So. Don't. I don't think that's a bad use case either, by the way. Well, yeah, you know. The childish part of me says, that's probably the first thing it's going to say is a string of profanities. And how many of us that have been into IT as long as you and I, how many of us have installed BonsiBud, even though it's spyware just to make it swear? Come on, I mean. There's a lot of people that have done that. Yeah. All right, well, thanks for joining and see you next time.