 Now it's working. Technical difficulties. Now wait until the last minute to press the Go button. And I don't know. Sometimes it don't go. Also, I had a virtual machine that decided it didn't want to go away. It just said it's not shutting down. So it makes for an interesting first few seconds. You're like, turn this off, turn this off. Well, that's not going away. It has been a wonderful day so far. Welcome to Vlog Thursday, number 371, security, homelab, tech talk, and live Q&A. I didn't pick a specific subject today. I figured I'll just riff with it. You get busy doing stuff. And stuff, for example, was kind of being involved on the outside to an extent. I have a lot of friends that work in security. Lots of my friends at Huntress. And obviously this whole screen connect thing has been quite the mess. They work tirelessly to come up with the best solutions possible of getting things squared away with it and trying to protect people. And I did my part as well in terms of I went and Jason Segel was very good at putting together the list of like, hey, let's break these down into action lists of people to call. So you reach out to people, even if you may consider them your competitors. Matter of fact, I ended up having a really interesting conversation with a local MSP probably about the same, maybe a little bit bigger than us. They've been around a little bit longer. Talked to them. They accepted the fact that, oh yeah, you're right. Our screen connect might not be completely up to date and they fixed it, which was great. That's the outcome I want. I don't wanna see anyone, even if their perception is like a competitor, I don't wanna see them get hit with this bug. And the reality is the bug is so bad, which was also why we did the kind of awkward live stream where people are like, oh, you're not talking about it. I'm like, it's so bad. And the problem is once people see how bad it is, and I highlighted this in my screen connect video I did, the dedicated one I just released, that yeah, it's absolutely actionable from understanding how the flaw can be triggered to immediately having your system taken over. It's amazingly fast and it's happening right now. Still currently, some changes were made to help mitigate this, people were notified. I took the time to just call people. I called school districts, I called, well, I tried calling a lot of people I couldn't never get a hold of. I just kind of left messages with people because they have no way to talk to their IT department. So I left messages, you really should shut down your screen connect server, just tell them that, tell them to Google screen connect flaw, tell them to Google screen connect huntress. Like there's plenty of information out there. And that is one thing that's very disheartening about it is it feels so futile to try to call these people, especially when you look them up and they hadn't patched in years. So you're like, oh, now I'm gonna call them as if they're really gonna patch these systems. It's a little bit jading. I like that the people I had good conversation with, even some of the school districts awesome, they responded and I was like, great. They said they're working on it. They thanks for the heads up. There was some really positive, but there was a whole lot of, this feels like such a waste because these people, I talked to one person that could have been E or, yeah, we know, we're gonna get around to it eventually. I was like, you should probably make that a priority. Yeah, I know. Thanks. Click, just the person really, I mean, maybe they're having a bad day, but they're definitely gonna have a worse day if that gets a pot. So that's why I didn't have a whole topic. I mean, I kind of got that on my head and went back to working on some of the other projects. I'm working on a really long video right now. I see really long. I haven't figured out how long it's gonna be since it's not finished, but it's the whole getting started with XTP and G. I figured it's a good video to do. I think it's about at the 30 minute mark right now. So I'm trying to figure out how much longer it needs to be. I've realized the videos that do well on YouTube. So it must be what the audience wants or the algorithm wants or maybe both the audience and the algorithm want are these longer in-depth start to finish from let's load this off of a USB stick to building virtual machines type video. So it's gonna be very thorough on there. That's, it's taking a while to put it all together. Long form videos are much harder to do because you gotta figure out where all the cut spots are and how much recording. And especially a video that's a tutorial on a virtual machine system, it's a lot because there's so many things and so many steps. I don't wanna miss any of them. So I'm trying to be very detail oriented. And at some point you go, all right, I'm gonna be using local storage, but do I do a dive? Do I add 30 more minutes? Cause I have like a 20 or 30 minute video just explaining how storage works in Zen. Do I take that video and insert it in? Do I do a video on that and insert it? Like where do I do an entire recording of that and insert into this and add 30 more minutes to it just explaining storage? What about my backup and DR video for disaster recovery? Do I add that one? Which is also like 20 something minutes long. Do I add that in to make it one super video? I'm trying to sort that out in the editing phase. AT&T, oh boy. Do you think it would be a problem if most ISPs do symmetrical interconnections? I don't know. I think the demand isn't there for symmetrical as much. It's my assumption. Most people want downloads, not uploads. People like me who do YouTube things want uploads, but that's not many people. If you're trying to share data with someone or you're trying to offload data or host something, yes, you want uploads, but it's a niche want, not a want from everyone. I don't know where the demand level is on it. Too many conflicting streams. I didn't, who else is live streaming right now? I guess I don't, my other friends like craft computing live streams on Wednesday. I don't know who else live streams on a Thursday, but I'm curious. Any tips on starting an MSP from scratch? First offerings, ways to get a company build a report to larger projects? No, that's the business technicalities question that I'm not gonna dive into here. We've talked about that topic starting in MSP. I know it's on there. If you look up, I think his name is Kyle Christensen. He's doing a series on LinkedIn about it. You can find a few people that have to that topic. I don't know, I don't have any, I don't feel like I give the best advice for that. Like not in, it's not, I'm less interested in the business side. That's why, well, for two reasons. One, I let Jason and Brett mostly do the business activity videos. So I do have a good video of an interview with Riley Chase that I added. If you wanna learn about the history of hostify and SaaS business. But yeah, I'm not as, like I said, my passion is a little bit less with the business side and the MSP side. I often ask people first, why do they wanna start a business? Cause I have a video titled on the business side. My job sucks as in a business plan. And a lot of people unfortunately, they just want, or maybe they were sold a book on why you should be an entrepreneur. And they think that's the right path rather than finding a job because if you don't have a sales pipeline, and this is where almost all the MSPs fail, one of the biggest segments of the MSP business is like third-party marketing companies that charge you $10,000 plus a month to get marketing leads. Sometimes your marketing spend as an MSP is dramatically expensive. It's hard to get people to switch to IT. And generally MSPs aren't good at marketing. So they end up spending a fortune on the marketing and still barely getting any leads from it. If you don't have a good connection to people and you're not willing to put the time in not to learn networking from the RJ45 and OSI model concept of networking, but networking with other humans, you're not gonna ever do well as an MSP. I see that the FCC was talking about making the ISPs do symmetric connections. I wish they would. Oh, I mean, cool. I hope they make them behave better. That would be, my first concern isn't that we have symmetrical, but we have better behavior by them. Sure, all takes symmetrical as well. Oh, there's a moon landing going on. Well, that's cool. Unify Cloud Gateway Ultra, such a shame it can't work with Hostify. Oh, yeah. Oh, Raydall is streaming with Techno Tim. I didn't even see that. So we're all in a large group, a Discord group including Raydall and Techno Tim. I would not have landed on the same time slot as them. So that is, I generally would have looked, but I wouldn't kind of heads down. Hey, and we have Veronica explains in there. She's among us. There's a lot of us technical YouTubers that you may notice we may reference each other. Also, you will find out through some of us talking, we all have a large Discord where it's just us, we're all friends and we like to talk about the YouTube landscape and everything around it. I kind of wanna join my other friends and harass them in their live stream. So can we do, I'm curious about this. So we will do a screen share because if they could do a screen share in my insane time they probably wouldn't, but this would be funny. I'm gonna just put a smile, I won't do it. Cause I'm not gonna have time to interact with them and this, but we can stream each other and get that whole inception thing going that would be funny. But yeah, definitely this is, you know, they're both wonderful YouTubers. I like both of them. Having chatting with them all the time is actually great. I love techno Tim. I had Raydall on, he's been doing the whole Linux challenge thing, which has been fun and his Linux challenge went really, really poorly before, but now I feel bad. I wouldn't have timed it to be. I just kind of arbitrarily chose now to do it. I didn't realize there's was gonna be an afternoon stream in the stream. We had combined the streams. Even though I know there's some movie that said don't cross the streams, but hey, such is life. Good people subscribe to both of their channels. They're definitely solid. Even a color stream matches, you're right. We all have, well, it's not just a color stream if you look at my background and not Tim's, but Raydall, these are the same tiles. See when they're in focus? We even, they're different colors. He went with the, I went with the gray ones. He's got like the white and blue ones, but yeah. Unify is about to do dynamic roaming. How long do you think for them to do it correctly? I won't hold my breath. I can't think of that many requests that people have. Most of the requests are gonna be like, hey, give me more enhanced VPN support and things like that. Cool, they're throwing in OSPF. I can't say that's a popular demand for the people that are using Unify, but hey, I welcome them updating it. What about the new Ulta Labs self-hosted controller? Well, funny you ask about that. Guess who I was hanging out with today? The people from Ulta Labs actually had a good conversation with their technical people. I will be doing a review of the self-hosted controller. They gave me kind of, you know, we talked about roadmaps and where they're going with it. I told them, I feel very positive about what they're doing. I gave them feedback on some ideas and stuff like that, but yeah, my overall feelings to Ulta, I should have, I must do that in the title, but I don't have the demo set up yet, but my feelings in Ulta is I like where they're going. I like where their head's at. And they think, I think they have a bright future. They have a very hard future because it's not easy to be a newcomer in the hardware space and then say, hey, let's go ahead and I'll get there. Because, you know, they asked me very real questions or like, well, what's going to change your mind about switching this? And I'm like, here's the problem. We're doing large scale ubiquity projects. We're doing large scale projects in general. And I, we put our company name behind what we do. How do we switch what we know to something else? What's the benefit? What's the killer feature that you have that the other people don't have? This is a real challenge because they've come up with feature parity with a lot of things with the self-hosting controller. This is going to bring them to a feature parity with some of the hardware for Unify, but it still doesn't give you a complete ecosystem. It doesn't give you, if it doesn't have any advantages and the price is only marginally better or the same, it's hard for a company to go, well, I got to put my bet on something I haven't used before. I'm not having problems with the things I'm using now, which is a lot of the Unify stuff. So it's definitely challenging to switch, but I'm going to take my time and review them. I'm doing this completely, no charge to them on purpose because I want to give them a fair shot in the market. And I think they're, they are doing the right thing and asking people things. So yeah, there's definitely, I'm looking forward to it the best way I could say it. What are my thoughts on a big cellular outage today? I think someone messed up on a route or something. How could I describe it? Who did I see that I liked their meme? Mostly it's just about memes to me. Where did that meme go? It was my favorite one I've seen so far. Can I find it? Because I thought it was the best one. Come on, there we go. That's what we think happened at AT&T. Someone pushed the wrong button. Could you talk about shared storage with XTP and G in orchestra, especially ISECSI versus NFS? How do you prove its concept with a small business cluster? Absolutely. That's a great topic. I like that. Let's go over here to Zen Orchestra. We're going to pull it up in just a second. So let's go to, yeah, a lot of tinfoil. Oh, yes, people, look. I want the crazy people to be right once. They're never right. They're always telling me it's a conspiracy. It's never someone screwed up DNS, BGP, or blew up a router somewhere. It's always like people thinking it's really complex and they're realizing it's just human error. Human error is so disappointing. Yeah. Is the Cattle Limiters relevant in 2024? Yes. Absolutely. It's always DNS. I have to look, because if I did, I will let you know to see if I got my confirmation from ubiquity. I have a lot of unread emails today, so I will be going, if they confirm me, I will be booking it. So for sure. Back to the question where someone had asked though about Zen Orchestra, let's share this tab instead. The ISECSI versus NFS. Let's just cover that really quick. I have both set up here and actually, let me take the filter out. So I do have a ISECSI setup, but my opinion is don't use ISECSI. I, you can set these up right here. It's set up to these two servers. It works. I can turn on multipathing. I've tested it before, multipathing works, but I just don't use ISECSI. I have this setup as a demo to make sure it works. Anytime I'm doing updates, but in terms of ISECSI versus NFS, the problem with ISECSI is it's not thin provisioned in Zen, and I believe in Proxmox as well. ISECSI is thick provision. So I generally avoid ISECSI and prefer NFS. NFS is a lot easier. If we go over here to our storage again, we have plenty of storage that is your NFS. Like here's our production NFS store. If you notice, I have a lot more things on this one here. There's a lot more drives. There's what, 14? I think yeah, I got 14 VMs on this one. If we move over to our storage for, where's the other storage? The lab storage. There's 24 more VMs on this one that's NFS. So you might get the idea that there's that lab and turning lab, there's another, and 23 here. So 23 there, another 23. That's a lot of, you can see I have 47 VMs on my NFS storage and one on my ISECSI. Yes, I definitely favor and prefer in production environments, NFS over ISECSI for doing that for a person to ask that question. What does thin versus thick provision mean? So this is interesting and a good topic. If you are doing thin provision, if a VM, so we'll go find some virtual machine here. If Bitwarden says it's using the disk here, it says it's using 25 gigs right now. How big is it actually? Well, in a thick provision scenario, such as ISECSI, 25 gigs means 25 gigs. In a thin provision scenario, it's only using the amount that it expanded to. So once you load up Bitwarden and load some software on it, I think it takes like 12 gigs. So the thin provision is only using 12 gigs. The thick provision would be 25 gigs. Now, you can over provision within provisioning versus you cannot over provision with thick. So you have some risk, but you have some gains. When you're doing things on a thin provision, you can say, well, I mean, I have 10 VMs and each one of them has a 100 gig drive, but I don't have 10 times 100 in terms of storage, but because none of them are all using it, I actually can store all those VMs in one thin provision storage. I just have to be smart enough, and the system will give you warnings if you don't do this, that knowing if all of those were to expand to their full capacity, they would exceed the capacity of that particular storage repository. Then I would have to make sure that I migrate them somewhere else to allow them to expand to whatever their size is. So thin provisioning is definitely a big efficiency gain, and one of the reasons you really would prefer to have it, but that's one of the reasons I like NFS so much better. There are so many file sharing protocols. Can you talk about the differences? No, there's not so many. The reality is each file sharing scenario comes down to the way you want to share things. Like, do you want to share them between Linux machines? If NFS is probably better, are you dealing with Windows? Then you definitely need to share them with SMB. That's your best compatibility layer when it comes to file sharing. So that's not as much that there's so many of them. It's kind of use case pretty much universally, and what you're gonna see in the business world is SMB sharing. This is where all the Windows machines talk to. Matter of fact, the most effective for me with my Linux machine tied to my TrueNAS server is they both speak SMB. So all of my video editing is actually done over at SMB. Does SMB support trimming, reducing the VG size, using thin frishing? I know it does it during migration, what about? It just sits there in the same store for years. They do have the option to reclaim space. So if you go to the different storages on there, yes, you can reclaim any free space by clicking that. And it should reclaim that on there for you. I don't think it does it automatically. I think you have to kick it off with the reclaim free space. I recently set up XCPNG, the storage ice, because the multi-path through two 10 gig debt cables and TrueNAS, it seems like this would be faster in NFS for a SR for virtual disk. NFS, I find fine. I mean, it's not necessarily that multi-pathing makes it that much faster. In a lot of your limitation on a 10 gig connection comes down to what are you connecting it to? Do you have a fast enough series of drives to connect it to, to saturate 10 gig? And matter of fact, 25 gig is cheap. And you said, yeah, you're only using two 10s. 25 is cheap, you know, pop in NFS on a 25 and it might be better. There's another advantage to using NFS and that is because with ice because you're presenting block device, when you do NFS, the NFS has all of the individual files on it. This allows you to on the storage side snapshot and restore and understand which file is which in case you ever have to recover them. Recovering ice guzzies way harder because if you corrupt the block storage, it's way harder to sort out corrupted block storage versus the VDIs, you can see all the individual VDIs. That's another big advantage you have with NFS. It's just easier to manage. There's a lot of file sharing protocols though Tom are that are still platform agnostic. I mean, there's all kinds of random things out there but what you see in the industry doesn't mean there's not something obscure that doesn't exist. I mean, technically FTP is a file sharing protocol that exists. I don't think anyone should be using FTP here in 2024 but positive someone is. So the new TrueNAS and XTP set up by data should just redo the storage. Yeah, I really, and my storage videos on XTP and G have always said use NFS, you know and this is one of the things like, do I include this in my getting started video? I might add a section for storage to show how to do it to show the, but I've got a video dedicated. It's called XTP and G storage explained. I break down all the storage connectivity availability in XTP and G in one video. Time to use ice guzzies or friends don't do that. Don't use ice guzzies. I mean, ice guzzies is a block storage device ice guzzies not file sharing ice guzzies block storage. Thanks for your question and run proxmox and XTP and G plus an orchestra of a home lab. I like how proxmox you can check the discard option for the VM, which enabled trim inside the guest with NFS as an SR, is it possible to use LACP for storage connector failover multi pass? Yes. Yeah, as far as I know that works. I don't know, I don't think that you can actually do multi pathing though for storage. If we go to new storage and we choose an NFS store, I don't believe you. Yeah, there's not a multi path cause I believe there is multi path NFS. I've never set it up. Ice guzzies on the other hand does, if you go, you do have the option for multi pathing in there. You just have to go here and enable multi pathing. So you have that ability here for ice guzzies, I don't think they natively have multi path in there, but you can do bonding with the network interfaces to get redundancy as well. Many still use FTP. Yes, Veronica being that she comes from the world of COBOL programming is certainly aware of FTP usage, probably an extensive amount of FTP usage. I prefer NFS just works and you can map users via the Sys permissions. See, maybe out of scope of here, blocked picture ice guzzies, when I say blocked storage, even though yes, they both can be connected to Zen, Zen handles them very differently than each other. They both can be used for the same purpose, but they are not the same at all. So with ice guzzies, it's picture having a scuzzy drive is kind of what this means. I for internet or whatever it held it, I guess is what it stands for ice guzzies. You're basically taking a hard drive and stretching it across a network cable to another device. That's block storage. You're interacting as if it's a hard drive. So that's how block storage works. And that's what ice guzzies is block storage should. So would there be a use case for ice guzzies? I agree with the being easier, easier. No, ice guzzies more difficult in a recovery situation. NFS is easier to recover than ice guzzies. I don't have a use case for ice guzzies. The only time people have a use case, I think for ice guzzies is when they can't use NFS because they have some special storage server that doesn't support anything but ice guzzies. There are some Dell servers that, I can't remember the make and model, but there's like certain ones they only offer ice guzzies as a storage option. And maybe you have one of those servers. I have a friend that has that. That's one of his reasons that he doesn't like was Zen is he wishes they would solve the thin provisioning problem. And he just has this old piece of hardware, but it's got a lot of storage on it. And it's all done in ice guzzies. So it is what it is. I don't think it's been talked about for too long. I'm actually appreciative of the organization level by which the VATES team works at. So they're building out of the new version of XCPNG, or I'm sorry, Zen Orchestra is very methodical. And I prefer that because I don't prefer the way some companies give you a messy version that feels totally discombobulated. Like, hey, we updated the UI. We only brought over some of the features. So sometimes you spend time in the old UI, sometimes you're in the new UI. Good luck, help, you know, and it's difficult. They have a very concise way they're doing it. And I think that is a better approach. It's gonna take more time, but you get a better product that's more polished in the end. So at least some companies encourage FTP for scanning email. Yeah, Veronica, I have seen some things. Oh, I'm sure the old stuff you find that you're like, this runs this company? What? NFS that is. Can I move VMs from Iscazi to Enash here? Basically move it from different SR. I'm going to do it in a few to test. Oh yeah, this is actually one of the really nice things about the ways an orchestra works. There's no, you can go from thick to thin provision system. So if we have something, and let's just go to one of my, I have a lab machine running. Hey, look, it's my Debian 12 lab. Well, I need something on the other server. Debian 11 lab demo. Oh, here's my Olama. But I should probably boot this up. I've been running some internal stuff with Olama. Anyways, I didn't really have to boot it up. But it's, where's it at? Okay, this is on the production NFS. I'm going to go ahead and start. Well, it doesn't matter. I can show you that it works live as well. So currently this is running and it's booted. So let's go ahead and migrate. So currently it's on that NFS, but if I want to migrate it to Iscazi, I can just hit okay. Oh, it's not, there's not enough free space. If there was enough free space, it would actually let me migrate it over there. So apparently what's on the Iscazi that doesn't have enough free space? No, there's not much on here. Oh, wait, that is on a different server. Production, Iscazi production. There we go. I chose the wrong one. That was my fault. I don't know which one I chose. It wasn't the right one. Either way, it's now migrating. You can migrate from NFS to Iscazi from Iscazi to NFS. It doesn't matter. It'll just move them perfectly fine. Iscazi is fun with old servers, but generally I don't see it. Yeah, I don't see other than people who just kind of are stuck in that ecosystem. I don't see a lot of people on there. I have no problem with the GUI as an orchestra, but a fresh paint job would be nice. The light looks cool. Hopefully the same design principles to orchestra. Yes, the light is a sneak peek on what the design principles are for their upcoming one for sure. Enjoy the late vlog. I haven't been able to catch any lives during today. I hope you are doing well. I am, and thank you very much, Bill. I have used Iscazi to provide storage TV for apps where the share has been turned to align with the DV disk block size, otherwise you just do it in the file system. Yes, in my storage architecture video, I talk about that particular use case. That is a very recommended way to do it. Sometimes Iscazi can have, especially when you do the tuning properly and the block sizes for database type of applications, Iscazi can be really more performance oriented for doing that. So there's use cases for Iscazi. I shouldn't say there's none, but there are more, the people who know, they know. Like they're like, oh yeah, we're using Iscazi because we have this really high end transactional database. We boot up this server. We have the server mount, the Iscazi mount, and it does all the transactions in there. So yes, I've seen more telnet in production than Iscazi. Oh, I believe that. Anyways, my llama has been moved. So now if we go to the llama, we go to disk. Hey, look, it's on the Iscazi and I can start it up. It's no big deal. But now it's thick provisioned. So it's taking up a lot more space. But it boots the same, though it's actually physically the same server. It's just a different way it's presenting. And there, now I have my old llama system running over here. Now I should be able to migrate it back even while it's live. So we're gonna head and move it to production, NFS. Not enough. That's a weird error. Not enough free memory. I don't know what I broke to cause that. I'm running a lot of beta stuff. So I am not the example. I do know it migrates when it's off. So we'll migrate it back. Cause I don't want it there. I want my old llama back on production NFS. So send it back. There we go. Does an Iscazi not support sync and always work in asynchronous? I don't think so. I think Iscazi can be synchronous. Do you know if the new XCB and V-Vision will support Iscazi? No. I don't know where that is on the roadmap or if it's even a priority at all. I think that would be a very, very low priority because most systems, most with the exceptions that I noted, like some of these older Dell systems, most modern systems have the option, most storage servers in TrueNAS being one of the really popular ones out there, they have support for Iscazi and NFS at the same time. Have you seen any cases of XCB and GE connecting to TrueNAS to your fiber channel? No. We have zero of those I've ever run into in production. Someone says, yeah, Iscazi will honor whatever the client sends to be at sync or async. That makes sense to me. That sounds true. Is this dumb or no? Server running XCB, GE create TrueNAS VM and pass the HPA card through then create smaller VMs run Windows, other videos, software. I want to make a demo box for my security client. I would never run that in production. I think it's fun to run for non-production workloads. I mean, a lot of people do it, pass the HPA through. It's a demo, but it's not anything we would ever sell to a client for production reasons. I last used Iscazi on a NetApp SAM with redundant controller. We tend to only use the NetApp dish shell standalone with HPAs these days. I think Wendell would approve. Yeah, the NetApp ones are popular. I've not really had much interaction with NetApp. I'm sure I remember the other one. It's owned by HP. We've had too much interaction with those. What's the big one that HP acquired? Three-par. That thing's gross. I mean, the three-par server, the performance is terrible. The support is only more terrible despite them paying for the support. But I don't think I have any clients that I know of or maybe we haven't talked about that part of their infrastructure. I'm not familiar with anyone running NetApp. Like I know a lot of people run it. I just haven't really interacted with it. Oh, let's see. Yep, gonna be a testing box in our own office for access and a vigilante. Yeah, I mean, whatever works for them. That makes sense. Sounds like most operations are asynchronous otherwise requested. However, in Shurnass, it's sync not supported when it comes from a guest. No, you can turn sync on ZFS is different than how the synchronous writes work from the device connecting to iSCSI. The synchronize, how it happens inside of Shurnass is going to be different. The settings you do in Shurnass, whether it be for an NFS dataset ones, whether it's an NFS dataset or a Zval dataset, where you set those synchronizations are different. And they're gonna function a little bit differently. Shares Tours I'm using for my proof of concept is a small flash net app box, has a great NFS implementation kind of a thing, but does fine with iSCSI as well. Can two Zen orchestras manage the same XCB and G-Pool or do you need to create a Zen orchestra HA? You don't need HA, so you can actually have. So two Zen orchestras manage the same pool, not a problem. The Zen API allows for a one to many relationship. The database is all stored inside of Zen. So the XCB and G host have the database. Zen orchestras just reading from that database. So they don't have any data in it. Matter of fact, the easy way to show this, if I go over here to our lab servers, and here is our, well, let me actually switch over to our lab one. I need to switch. So here in our lab one, go to servers. Here is the Ace Magic AM20 box that I have right here. And here's all the things running on it. But if we disable it, matter of fact, we can do this to all of them. I just turned all these off. Zen doesn't know anything. When you disconnect them from the servers, Zen has no idea. So now we connect them all, suddenly it knows them. So this particular one, which is called Rise and Labyrinth, and we're gonna go ahead and open it up. We're gonna switch to another one, share this tab. This is also a connection to Rise and Labyrinth. You can have more than one Zen orchestra. These are two separate instances of Zen orchestra, each one both connected to the same server. It's perfectly fine to do that. Share some of the music for me, share that one. Really random question. Have you ever been curious to try a CCI exam? I've stopped lying to myself and said I'm gonna take an exam. I've never done an exam. And I kept saying I should do those one day. And I realized that's never gonna happen. So because it's never gonna happen, I'm not going to kid myself and say that I'll ever do any type of certification. I have no certifications. I'm almost 50. I just don't think I'm gonna bother with them. I've somehow made it this far working. My first tech job was in 1996. And I don't know, there weren't certifications really, but I mean, there was NetWare cert and a few others that are not relevant anymore. Then came the early Microsoft certs, but somehow I never needed any of them. So yeah, it's not on my to-do list. Well, I'm not gonna kid myself and put it on my to-do list. Zen orchestra use Redis for a store. You can build a Zen orchestra in HA technically, but make sure it's an active passive theory. Oh, you're trying to build a HA copy of Zen orchestra? I mean, the thing is if Zen orchestra goes down, it doesn't matter if the VMs don't change their state because of what Zen orchestra is doing. So I don't really have any use case where I would want to, and this is a weird debate someone had with me. I think in my forums, they were really determined to do exactly that to try to build this high availability Zen server by rewriting the code. I'm like, good luck. I don't know what the use case is for it because Zen orchestra can be turned off, but it doesn't stop the host from running. What am I drinking? Bullet rye. So for anyone wondering here, we've answered the question now. To be honest, all the certs I've done have exactly 0% use case on the job. Yeah. You don't need Zen orchestra in HA. I was just answering someone's question. Okay. Yeah, that's, I see what you're saying. Yeah, it's just weird. A lot of people, it seems like a question that comes up a lot, but I think there's, I just want to make sure there's no misconception that Zen orchestra does not have to be running for the VMs. Matter of fact, if you look at our lab server here, and here's the VMs on it, but if we turn these off, you can't see the VMs, you can't interact with the VMs, but they're still there, and they don't stop running because I wasn't connected to them. What is on it? There's a lot of things on the server. We keep a lot of different stuff in our lab whenever we want to spin something up. When was the last time I started this? Oh, a month ago. I should fire up my Windows 10 lab. I know it's, I forget about it sometimes, and then there's updates that need to run on it. So I'll let this one, I'll let it boot because it'll want to run updates. Just kind of all those things I was thinking about here. Actually, this is a new feature they added where you can add notes in Markdown to the VMs. You can also get more colorful. Oops, hold on. I'm having trouble typing today. There we go. They just added this feature, so I'm not as familiar, but you can, there. Now we can make Windows red because why not? I thought this was just kind of a cool feature for advanced tag creation, for editing tags. We can fix the Tom one too. So let's select the Tom tag. We'll make Tom green. There we go. Oh, whatever. I got more than one Tom. New features are fun because why not add these things to it? Fitting existing XO backup config can be restored to another XO causing the issue with the original. Hold on, I missed something. Have you ever presented a DEF CON? No, I have not. Not presented at DEF CON. Is it true there's no way to do synchronous rights over SMB with TrueNAS CFS? I am not understanding the question. Synchronous rights with SMB. I mean, SMB works how SMB works. So maybe I'm not completely understanding the question. Oh, do you miss to talk about Zima board? You know, I forgot I put the Zima board in the thumbnail. I've still been testing. I'm loading a new version of TrueNAS on a Zima board, but there wasn't any particular Zima board talk I had today. It's just what I was playing with, loading up TrueNAS on it, getting ready for my getting started with TrueNAS video. Yeah, it's in beta two right now, so pretty soon we should be out of beta. If an existing XO backup, the config, can it be restored to another XO without causing issue with the original? Well, I believe when you do the backup restore, it overwrites it. So when you backup and restore the config, it overwrites the existing config. So hope that clears that up too. Yeah, you want the Zen server in high, well, I mean, if you want it, I don't always recommend people put everything in HA. If you don't know what you're doing, you can cause your own problems with that. People think they want HA until they realize some of the challenges that may or may not come with running things in HA. Especially when they goof up connections and don't have the quorum set properly and fun stuff like that. Ah, the Zima board though, so I have, and let me pull this up because this just came out. Actually, we can pull it up on the, where is my TrueNAS Mini R, pull up on that one. I have not updated all of these yet, but the new TrueNAS came out today. So this is why I had the Zima board because I've been playing with it on the Zima board, but yeah, there's a lot that changed in this new version of TrueNAS. So this is also a video, I'll get the video out by this weekend because I started updating my TrueNAS servers. I don't do the video when I first update them. I wait till I do some workloads on them and then do it, but yes, there's a lot they changed in here. I believe this is one that has all the, they have some SOM improvements. I know there's a feature though, ZFS read layouts, should be this version that has the updated memory. This is the thing I wanna make sure is that we have the updated memory in there. So hopefully last modified 2421, beta, beta, beta, beta. I'll have to read through this and figure out what it is, but there's, they're working on fixing the memory so it auto scales better for the ArcCache. That's the part that is, I'm looking forward to. But I'll read through all the notes. I haven't really read through the notes yet to figure out what the good parts are. Hey, Tom, I'm in Zima, and I was feeling good about getting comfy with ruining most of my home ecosystem with Docker compose files now, building a new share net server and need to learn Kubernetes. You're in sad mode, feeling good company. Well, good luck learning all that. That is a lot, diving into Kubernetes. I don't do much with Kubernetes though. Yes, you do need three minimum to set up XCPNG. Probably next week, I'll get the review done. I've been running it now for, I don't remember how many days. Quite a few though, it's still my main firewall and I like it, it works great. I haven't had any issues with it at all, which is good. I mean, my test because I got distracted with other things went from review to really long review. Looking at the uptime on it right now. I've been running it for three weeks. So yeah, it's three weeks of runtime on it so far. So my review got more thorough. I was having an issue with new versions of TrueNAS scale breaking my Active Directory negation, any advice. I don't think we have any production clients with TrueNAS scale and Active Directory. One of the reasons we still mostly go with core is it's very, very dependable. So I don't have enough experience with it. I've been told it works fine. I know in the lab, I think the last time we tested it worked fine, but I don't know if there's any bugs with it. I had posted in the forums on that. You enjoy my walk around the data center? That's awesome. I enjoyed it too. It was so much fun hanging out there. If you haven't seen the data center video, it is definitely a fun deep dive into the data center. Walking through the power systems, the rows and rows of systems. It was just a good time. And I had such a great tour guide. Incredibly knowledgeable. Their site manager was just brilliant and so full of good information walking around there. The apps area of TrueNAS has everything I want to run. So just making sure I understand what it's doing and how to troubleshoot it. New server arrives on Saturday. Awesome. Hello Lawrence, have you tried Multi-Wan VPN with PF Sense? Multi-Wan VPN. What are you trying to do? Thoughts on how the client failover? I mean, I can't remember if I've mentioned it in a video but it's not hard to set up. If you have Multi-Wan, there's a parameter you can add to your open VPN so it'll choose both. I mean, we actually use it at the office. We don't have that many failovers to test it very often. It works. I remember I was playing with it and we know it works but I don't really have any other comments on that other than that. Well, we use a lot of Synology in production. I have several Synologies here running in production. I just don't do my video editing on a Synology. I do my video editing all on a TrueNAS and we certainly have some pretty large deployments of Synology. They're very reliable. I'm trying to think, we were just talking about one of our clients today. I can't remember, I think they wanted to buy some more Synology servers but yeah, we have Synology completely in that aspect as well. To be fair, this is just my home lab. I've had my Active Directory for many years. I use it to Windows Server file shares. It's been hard to move away from. Yeah, I don't use Active Directory at my house at all or I mean, we have it kind of at our business. Very little though. Would you consider virtualizing TrueNAS to pass you under Proxmox to take advantage of the hardware? No, I don't because of the headache that it can create. I prefer to keep them separate but I completely understand people who go but I don't have the budget to buy multiple fast servers. Yes, you can virtualize it. I just don't recommend it unless you pass the HBA through and it's a fun home lab project because it gets you learning all the challenges that may come with it. So I think it's a good thing. I've never sent that run in production at a client but I completely get, especially if you're a home user that may or may not have the budget to buy separate systems or you're someone who's going, well, I can afford them but then the power would eat away at my budget as well. So consolidating it into one system may make more sense. So there's reasons it'll work. There's ways you can make it work. It just requires some effort. A multi-way and failover works fine with PF Sense. I use it. The only issue I have is that hate shifting IPs like YouTube you need to refresh the page to failover. Yep. How do you go about choosing it? I have a video that dives into the comparison of those two of how you would choose them. I've done a whole, I got a whole breakdown of the chart for features and everything else. Tear to my apps breaking on TrueNAS after the upgrades. Yeah, that's, they're getting better at it but yeah. Tom will never virtualize TrueNAS for production. Yep, that is very true. We, when we sell a solution to a client we also support the solution we sell to a client. This is the reason why. Any good cases that put in two, to put two motherboards in, I don't understand the question at all. It's something I'm playing with college, having nice group policy, automatically setting up printers and stuff on my mouse computer. Yeah, I mean, you can get good at Active Directory and reality is, there's some staff we have that are really good at Active Directory which is awesome and wonderful. I'm happy that they're able to do it. What are you using production just in SSO? No, I don't even like SSO. I mean, for our business side CNWR, yes, we are using SSO because that makes sense. For most of the other apps we have, a lot of them, we just use the different app authentication that they have, but for a lot of the business applications, yeah, I'll admit we're using SSO with, this is on the CNWR side of the house. It's all tied to, well, not everything. Some don't support it, but in many places I would say we are using SSO. I saw a negative videos recently about TrueNAS saying it can't support different sized disks like Synology SHR does. I've never used it, but saying the same drives all around makes sense. I mean, you don't want a hodgepodge of drives if you care about performance. And ZFS, this is not a TrueNAS thing. There are certain limitations to ZFS. There's, someone wrote a good article a long while ago and I think you can still find it. It's called, I'm gonna say reasonably good. It's a use case of why not to use ZFS. It's like the hidden cost of ZFS, watch out. And they're not wrong, ZFS has some disadvantages for homelab people who will like to just expand things as they want, whenever they want. ZFS has a lot of rules. I've done a whole video on expanding ZFS and people who watch it go, oh wow, there's a lot of rules around it. That's not for me. That's fine. I'm not saying that it's the end-all-be-all solution for every situation. I will say though, the reliability and integrity you get from ZFS is second to none. And those same level of features that you get good integrity out of, a good trust level of integrity out of things like Snology, but it does not come with the same level of performance. It does not come with the same level of, I would say hardcore integrity for creating lots of snapshots and moving around with the absolute confidence I do with ZFS. Hello, any advice on data migrations? I'm sure you're asking. I'm currently using replication to move data from bigger drives. Can app start normal? Can app start normal after replacing the old drives with new ones? Yes, I don't know. Replication is good to use. I'm not completely understanding your second question. Raise the expansion coming soon, but you should not use different sized drives. Yeah, coming soon though is quite a ways off. It's been coming soon for a long time. And I believe there's still certain parameters around raise the expansion. It's not just insert, drive and magic happen. I believe there's still some rules that had to be followed for it to work. Yeah, I don't understand the question here. Proxmox converts storage virtual disks as ZFS dataset. I mean, Proxmox has Ceph built in for their hyperconverge, so I'm not exactly understanding your question. I've imported my pool to a new server. It does take some time for the apps page to see the apps that are totally running, but they do run right off. Okay, that's interesting. Finally, moving from VMware to XCPNG. Well, welcome to the world of XCPNG. Migration went smoothly using their import from VMware, only had to rebuild the DC from scratch. Yeah, the import tool they've done a lot of work on and have really engaged with the community to get feedback to make it better. So I've been excited to see the progress they've made on that. Ceph takes a lot of CPU as you found with the client who hired Homelabber. Yes, yes it does. That is, yeah, because you can, and these people don't take, they don't take the time to read the manual because you can go through the documentation on Proxmox and they will tell you the requirements for setting that up. And it doesn't say connect with one gig links and hope for the best. It says you must build on robust hardware to build our hyperconverge system and expect it to be performant. That is definitely a challenge. What is hyperconverge? I'll let you Google that one. You should Google some of these. I am not Tom GPT. There is a limit to how many questions I can answer before I just feel like not answering. I mean, I'll answer a lot of them, but the hyperconverge storage one, it's an oversold thing as far as I'm concerned because people don't realize when you do hyperconverge there's challenges that come with it, but I will answer the question. The hyperconverge storage idea is if I have a couple servers hyperconverged, magic storage is how I always hear people say that. It's more of a marketing term, but essentially you're taking the storage and replicating it across multiple hosts. So if I have the VM running on host A and suddenly host A goes away, I've replicated that storage across host B and possibly host C all at the same time. So the storage is in sync between all the devices, meaning I don't have to worry about data loss. Can you migrate from Proxmox to XTP and G easily? I don't know. I mean, you can use things like clonezilla or any cloning utilities that'd probably be the easiest. I don't know if there's any import methods for them. What bourbon tonight, bullet rye, Tom GP. I will eventually do a training model on all of my videos because that would be interesting. Proxmox does not give you a way to import disks. It's all CLI, not the VUE. That's interesting. Sorry, yawning a little bit. It's late. It's late for me. Yeah, you can import disks. They do have import options inside of XTP and G. I've never really thought about being able to import them. Matter of fact, you have the import for all the different storage options. You can import from URL, which is cool. I actually like the fact that they do this so you can set up servers and then grab these. Matter of fact, it's just novel because you can do it not just from here. You can do it from the command line, of course. You can build scripts that pull in VMs and import them. It's a neat feature, but they support XVA here. But if you're doing disks, I believe, yeah, you can do VHD, VMDK and RAW disks can all be done here. And if it's ISO, you can still do the ISO from URL as well. Those are all features in XTP and G. Docker, pole, but for Zen, kind of. Kind of curious time. Do you guys still use MDT a lot? I know Microsoft is no officials for it. What is MDT? Oh, the deployment toolkit. We use a tool called Imibot. So no, we don't use that. We use Imibot. To me, Proxmox is great for CLI and more advanced users. XT is more for the average user. Both are good options, so I wanted something that I could migrate from easier. XTP and G is probably used that I've seen a lot more in the enterprise than Proxmox. Proxmox is definitely super popular with the home users because there's so much customization you can do with it. But the businesses aren't looking for one-off customization and hacking things together. They generally just want their hypervisor to run their virtual machines. What would you recommend migration path for machine running with RAID to ZFS but rather not reinstall and reconfigure apps? I mean, there's not really an option. If you're running some standard RAID array, there's not, you can't really just migrate. You just got to set up the ZFS and copy the data over to it. Where does Imibots fit in when you have an RMM, essentially why both? Imibot is not exactly an RMM. It's kind of close. It does have some RMM-like features, but Imibot is, let's see if you should pull it up on her site. Really, their genius of Imibot is all the installing that they can do. The fact that they, we have these different Imibot scripts that allow us to deploy workstations. We have set up certain clients that let us load over 900 applications and counting for Imibot. It's kind of just a cool auto deployment and it's just a really neat, Darren is awesome. I've got to meet Darren a few times. He knows what he's doing. He built Imibot and it's just amazing. He's got a good team behind him. They have a ton of integrations. By the way, completely not sponsored. We're just being honest about a tool we use. XDB is easier to migrate from and to. Proxmox gives you no-friendly option to export important VMs. That's interesting. Since you're using Blumerra and it's an XTR, you still use Sentinel-1. Well, yes, because Sentinel-1 does, there's always, when you start looking at these tools, some overlap between them, but they're not going to solve it. Unfortunately, there's no one tool to rule them all. So while you do have some XTR features in Sentinel-1, you also have some of those features in Blumerra, but Sentinel-1 has different detection tools for endpoint detection versus Blumerra is more endpoint SIM monitoring. But between the two of them, they give you good correlation data for security. Go-to slog device in your production, TrueNAS servers, whichever one that comes with the TrueNAS IX systems boards. Most of our product, a lot of our production stuff is actually IX systems hardware. So it's the one that they put in. I never looked at what model exactly it is that they put in there. I don't remember the name of it, but it's usually one of the, I can't, I don't remember. It's in one of my reviews I did of it, but if you look up the hardware for the TrueNAS server, you can find it. I don't, we got to remember, when we're building TrueNAS for high-end production servers, it's usually custom hard, or not custom. It's not one-off cobbled together. I bought it off the internet hardware. It's IX system stuff. I use Proxmox for a lightweight uptime kuma instance in my locations with a script I created using a deployment quick. Yeah, it works well. I like up, I love uptime kuma. It's a great project. Just want to say thank you for the time I contribute to the community. Awesome, it is, I love doing it. This is, I couldn't pinch myself and get a better job. Like wake me up in my dreaming. I like what I do, I'm not going to lie. Sounds interesting, reason for using it over Intune because Imibot is usable. That is the best way to describe it. Intune can't hold a candle to the capabilities of Imibot and the usability of Imibot versus Intune. I have an awful lot of questions I've ever asked. What do you use for office documentation? I'm trying to pick up something I can self host used for all things IT. We're using Hudu right now for documentation. That's our commercial business tool. So good bourbon, yes it is. Have you used HA proxy enterprise? No, I have not. Uptime kuma is good for basic up, down and it's free. Yes, all of those things. It's just simple, gets a job done, lets me know if things are working or things are not working and sends me notices. It's one of those things of simplicity. Plus one for Hudu, we use it as well, works well and is very flexible, can customize it to your needs. Yes, this is true. We've got a bunch of custom integrations and things like that that we've done for Hudu but yeah, it's definitely, it's part of the integrated stack that we have of things. In case people are wondering what I'm talking about, I will pull up their site. It's hudu.com, I thought that's what their domain was but build a strong foundation for IT management. I have talked about building an open source one. One funding is hard to get people to pay for it, not easy, hard to put a business model around it and I kind of said, I don't have time to do that. Yeah, it's definitely, as much as I'd love an open source one to exist to compete with them, I haven't seen it and I don't know who would build it. And building something as extensive as Hudu, it's not rocket appliance, but man, it's a lot. Are my friends still doing their live stream? I'm curious. I am wandering or winding down a bit. Obsidian open source? No, obsidian is not open source. Yeah, the, I use a tool called LogSeq, that's my preferred knowledge management tool. Connect your notes, increase your understanding. This is the, I love this tool. I've mentioned it a few times. I don't know that I'm qualified to do a video on it but for those interested, this is my preference. It is open source, that's one of the reasons I use it. It is logseq.com. I'm calling it LogSeq, as far as I know that's what it's called. This is where I have a pretty good mass of notes of things of whatever is going on is being done inside of there. Here I can share this. Hold on, what can I share? Yeah, that's all shareable. So let me go ahead and stop, present, share. And you can look at how I use LogSeq. Here's the fun part I like. Here's all the little knowledge graph dots for how I connect data. Here's a script for the tail, here's a script for the net bird tail scale video and things that are going in there and stuff like that, which that probably had some passwords in it now that I think about it. Let me see if those passwords are even, I think they were the temp passwords I had for that. Let me look, that I share some passwords. It wouldn't matter because they're passwords but I don't think that they would let you log in. Let's see. It's not up right now. So none of those, are these passwords or not? Oh no, these aren't passwords, nevermind. The user is admin at net bird demo work and Docker compose. Okay, no passwords were in there, so I don't gotta worry about it. I can just pull that back up. But I really like this view here. I think the graph view is just novel to me because it just shows all the interconnections between everything on there. Is there a front end for Proxmox that users sign up and it provision everything from? I don't know, I've not heard of one but I don't know if one that exists. Who do you just edit IPM support as well? I think it's a way to go in this category but it's a good start. Some auto discovery or auto population from like an RMM integration would be nice. Yeah, I know where he's like I said, there was no password. I thought there was passwords in it. The system's off right now. It also had to do is boot it up and change those passwords but I'm not that worried about it. I'm gonna be reloading it anywhere. What do Jay and Jason use? Jay from Learning Lakes TV uses a tool called Standard Notes and I don't know what Slagle uses. Jason Slagle I think kind of sort of uses a lot of different things. I can't say specifically because I just don't know. I don't think he's, me and him have gone down the path. I've chose log seek. He's tried out a couple of different tools. You know, the typical journey of let's try lots of different things. I stopped trying things and said, I'm using log seek. And so now I have log seek. And here's my like agendas I try to make for stuff that I'm really bad about doing. I mostly have a lot of notes from time to time, things I'm working on. Oh, my live courses I'm working on. That's my notes for that. But maybe I'll do a video on how all this works. I've been using it for a while so I have a lot of things in here. Tweets I saved and all kinds of fun stuff. Videos I might wanna watch later. Yeah, he was talking about obsidian. I don't know if he's actually using it. He talked about it but I pointed out that I don't like putting my data in proprietary systems and I don't know. It kind of turned into a weird discussion we had. He kind of agreed with me but I don't know if he stopped using it or kept using it. I don't know. I don't like my data in proprietary systems. Everything you see in log seek is all done in markdown. Log seek itself is open source. This is why I like it. I don't know. We pay what we pay for it. I'm not sure maybe. I don't know how much we pay for things. I'm just gonna be straight up honest. I don't know. I mean, we pay a lot for things. I know the financials. I could probably find this information but yeah, you gotta remember though we're managing a lot of things and we have about roughly 30 employees as well. So $400 a month doesn't sound like a lot for things for us. It's just a scale that you start operating at. So it's really, this is the hard part about when you start an IT or MSP, all these tooling costs because they have so many seat minimums. I was just talking to this because I'm an investor in like FIN and we had a discussion about where the minimums are set and things like that because FIN security has got some new stuff I'll be talking about soon. But among those things is gonna be, just share the screen, doot, doot, doot, doot. So this is a cybersecurity company that I am an investor in and they build security awareness training. I think they did do an excellent job but these are struggles you have of how do you set the minimums? Cause no matter how low you set them someone's always angry at how low those minimums are set and it's sometimes like we're trying to service the larger community but we have a minimum need. Huntress has some of the same struggles as well with their company. So yeah, costs are what they are for some of these things but you overcome them because of the efficiencies because once you start adding up what labor is saved you're like, oh, that's not much at all because it saves us this many hours in labor and automation. You still have security concerns with overlay networks like NetBird. I mean, if you have any VPN, NetBird being among them, I mean, they're overlay version, they're overlay VPNs but whatever the VPN is you're using, yeah, at the moment you start allowing connectivity into your network you have now opened up threat surface to it. So yeah, there's always good reason to be concerned. Yes, it looks expensive but once you have over 4,000 devices you're like, yeah, $2 an agent to manage 4,000 devices with 30 people suddenly sounds more reasonable. Does LogSync have cloud integration? No, it doesn't itself have cloud integration. They're working on their own synchronization server but they leave it up to you to synchronize it. I use SyncThing with LogSync, that's my solution. I have SyncThing on my phone, I have SyncThing on my computers and so that's my solution to doing all of that. No, I just realized I missed a phone call. I've been trying to put my phone on mute a lot more. I think I'm happier with my phone just off while I'm completely muted. I still don't think I hit minimums of fin but I don't think it's worth it because I spend zero time there. Yeah, the fin is great. The minimums for many MSP tools raise the cost for many small one or two people MSPs out there but understand the need to make money just like everyone else. Well, it's so complicated because when you're looking at it from the perspective of a company that has thousands of customers, they have a lot of data and that data, unfortunately, always ends up pointing in the same direction. There's no way around this. The smallest one in two person MSPs are the hardest to support, the most phone calls and it's because they're at an operational maturity level of usually early on in their business. So they end up consuming a lot of support time. Then you sign a deal with an MSP with 5,000 end points. You're like, yeah, we signed the deal, they pay the bill every month and we never hear from them. So you start targeting those audiences because they're easier to support and you don't have to hire as many people to do it. So it's kind of a balance to keep. It's just an unfortunate, it's not saying that specifically every small MSP shop is needy but it does feel like a lot of them are needy, unfortunately. So yeah, it is a challenge when you're trying to set those minimums and it does raise the bar for entry for starting an IT company, starting an MSP business and I don't want to see the bar raise so high that it becomes insurmountable because we need more people in the space. Hi, Tom, is it possible to automate copying data from one pool to another in TrueNAS like a primary array to an older pool, spinning rust? Yeah, absolutely, just use ZFS replication. Yes, I guess obsidian does do markdown. Okay, I haven't used obsidian. I just ruled it out because it was proprietary. I cool that it actually saves things in markdown. So that's a plus one for obsidian for using standard file formats. I will definitely say that. Very good insight on the minimums. Yeah, it's a challenge. I'm someone who sits on like advisory board sometimes and talks to them about it. So I get to hear both sides of it and trust me, the passion is there from all the companies. They don't just stick their nose in the air and say we're too good for small companies. Matter of fact, the struggle is how do we be more inclusive with small companies? How do we help these startups? This discussion is echoed through all these large companies because they were all small companies and many of them are very passionate about how do we help the industry? How do we help the community? So definitely, it's one of those balancing acts that are always hard to keep. Even myself, I had to get rid of the retail operations a few years ago. And I can't believe now it's been five years ago that I got rid of my retail operations because it was 2019 when we wound our retail down, which is coincidentally before the world changed in 2020, but it was one of those things. Like I don't know exactly how to deal with this. Like I don't, I want to send the people somewhere but I'm just not good at handling it. I don't have the staffing to handle one-off computer repairs. And I ran a retail store for a long time but I said, I'm just not making money at it. And I'm not saying that you can't make money at it. I'm saying I'm not good at making money at retail stores. I was really, that was my thing I told people when they said, why are you sure you're done? I said, I'm not good at making money at it. Simple as that. Well, can't you figure it out? I said, dude, if you have a solution for me, double thumbs up, but I don't have a solution. So I'm going to close it. And we stopped doing retail. So 2019 was when I closed retail. And we had a retail store going back to my very first one was in 2005. So from 2004, from 2004 to 2006 or 2007, 2007 was a retail computer store. And then we got rid of all that. And I have a whole page on this, by the way. I don't know if there's more I should put on this page but I will mention this. I put a timeline together. People ask, what was your first computer? So I put that on there. When did you start working in tech? 1985. My first assistant men job, 1998. IT director, 2000. Started Lawrence Systems, the company I have now, 2003. Suburban Electronics. This was the first retail company I had. And can I make this bigger? Here we go. That was my first retail. This, these changed. So 2005, I had a partner, 2007. I got rid of the partner. And it was one of those things. Like this is when I launched PC Pickup and this was another retail venture. Started the YouTube channel technically but didn't really do much with it until 2017 and 2019. Stopped retail operations. And then 2023, merged with CNWR and here's what I'm doing today. But I've been keeping this up to date on my website for anyone that's curious what Tom's up to. You're not wrong that all the problem is most of the tuning was built on the backs of smaller RMSPs. This is not isolated to the IT sector by any means. Retail's hard. Yeah, that is a, retail is not an easy business to figure out dealing with the general public. I'm not, I did it for longer than most. I mean, that's a pretty long run I had. If you look from my second venture in retail which was from 2007 to 2019, that's not a bad run. What is that? 12 years I had going with retail in a few more years before that which was also an electronics repair. So the 2005 to 2007 was an electronics repair run we did. So yeah, I've been there for a minute. You can make more money recycling hardware than trying to run a repair shop. Yeah, there's actually another couple guys that run a recycling company pretty close to us. And that's what they did. They stopped doing retail computers and started doing recycling. They are right. That was their strategy exactly. And it seems to be working for them. I like them, they're really nice people. Historical nonprofit volunteer wants to pull back with Azure. Have you seen smaller orgs and businesses that are pulling back from cloud migrations? We're seeing all kinds of companies pull back from the cloud. That's kind of the niche we're in is people who go, the bill came, the cloud's expensive, what's next? And you're like, wait a minute here. I love the 37 signals moving away from the cloud This is, there's a podcast on it. Let me just throw the podcast down here if you want to listen to it. But yeah, it's called Leaving the Cloud by 37 Signals. And they talked about how many millions of dollars they saved. So really, they have a transcript if you prefer to read, but it's a good podcast episode. And they just talk about what the change is to working and moving away from the cloud and how much money you can save. It's just a wild, like millions were saved. They thought they'd saved this much, they saved that much. And it's one of those things that the cloud and the companies, despite compute getting cheaper, your bill only goes up every year and they're not giving you more for what they're charging. Matter of fact, they're just charging you for the same amount of data you have, but they charge you more. And because that doesn't scale very well, it works out to be with all these price increases over time that you end up with a cloud bill that suddenly doesn't math out very well for especially a lot of large operations. There's things that are gonna be cheaper in the cloud. My email is in the cloud because it's easier and cheaper. But when you get into any large datasets, that's not the case anymore. AI Advancements, Google's big query of Vertex and SaaS is still gonna take over the industry. I don't know about takeover. This is the part that people have a hard time with. It's the way statistics work. And you can think of it this way. And this was a fun statistic to play with for a while. Every year, there's two ways I'm gonna say the same thing and they're both accurate, but they both will lead you to a different conclusion. Every year, iPhones market shrinks. Every year, they lose ground to Android. That's a true statement. Every year, iPhone sells more iPhones than they sold the year before. Every year, they sell more iPhones than the year they sell before. You're like, wait a minute. If they're growing, how does that change? Well, the market's expanding. The market for phones is expanding greatly. Therefore, you have a couple of competing things and it's all in the way you look at it. So when you start thinking about the statistics, like, oh, here's this other service, it's going to overtake debt service. When the reality of the market is, because of the way the market expands, the demand for these services expands, the people that were doing this thing over here or still hosting or still having their own data center are still doing that and still growing it, but you notice that there's also this other thing, but it didn't replace the other thing. This isn't an evolutionary change in what we're doing. This is an expansion into doing more things. So it's not that I see this as an overtake. This is more of a, oh, and besides the things I'm doing here and besides the things I'm self-hosting, also there is available this other thing for these other data sets. So yeah, it's kind of interesting when you look at the market that way and kind of think of it from a big pictures perspective. Too many managers think the cloud is a magic place that they don't need texts to keep it working. Yeah, but there's a lot more people getting slapped with the bill and the bill came and they go, the bill is expensive. So let's talk about that. And now we need to save money. When you have private AI models and people wanting to manage data more effectively, you really don't have a choice. No, you absolutely can. The fact that I'm able to run these Olama and other systems, as a matter of fact, if we look at my Olama, let's talk about my Olama. I'm not sure if this is something I'll do a video on, but being able to do things and so I do have some memory assigned to this. Granted, this is the Olama. If you're not familiar with Olama, let's pull that up. I am not an expert on it, but you can set these up and this is the self-hosted AI stuff. It's just beautiful, it works so well. It's amazing that without even going as far as having a video card to rely on, or I should say a GPU that can do better processing, I'm actually able to get incredibly fast queries out of a system with 20 cores and 64 gigs of RAM and I'm able to query this quite fast with the Olama. I've actually got the Dolphin Mistral and CodeLama loaded on here. So I can now query locally and have these large language models help me process code and do Regex and write Python and Bash scripts for me on demand. And it's hosted at my office and this is great. It doesn't take a supercomputer, it takes a reasonably fast computer. So I think this is where the huge opportunity lies with a lot of this is the fact that we, as people who have access to HomeLab hardware, are able to run these models. Matter of fact, if you happen to have a gaming system, you can leverage your gaming system when you're not playing games to use that same type of hardware, a good GPU to build these models or even process them at a really good speed. It's pretty impressive how well they work too. VPN plus RDP, still waiting for it to go remote desktop, probably. But VPN plus Guacamole is definitely another option. Guacamole is pretty cool. Do not publicly expose Guacamole. I don't think that's a good idea, but putting VPN then Guacamole, much better idea. No open source solution for Vertex or BigQuery yet. Yet is the important model. It's surprising how inference is cheap on these 7B models. Yeah, I think this is the cool part of AI. This is the part where AI actually gets me excited is not when some big company holds it. It's really about when it gets in the hands of the people, if you will, the homelabbers, the tinkers, the technical people playing with it. That's when this gets interesting. You know, I kind of, I ranted, I call it a rant, I don't know how to title this, so let's read it. But this is kind of related and I'm pushing people more towards this. This is something I wrote today this morning because I was just, I finished the migration of my server. So let's talk about building community and owning the platform. I started my forums back in 2019. Here in 2024, I have over 6,000 registered users. For those who don't know, forums.lorencestors.com. I keep my forums public, allowing the solutions and all the amazing community members to post there to easily found with search systems that has led to a good amount of traffic. I currently get over 7 million hits a week from over 50,000 unique visitors. I am posting this because people should know that locking their data in audiences and proprietary platforms is not the best way or the only way to do it. I choose to build my forums with the open source tool, Discourse. And this is my real crux to why I wanted to post this. Relying on proprietary platforms for community building is akin to building your house on rented land where the landlord can change the terms of your lease on a whim or worse evict you entirely. That is why tools like Discourse represent a beacon of hope. They're not just forums, they're fortresses of freedoms in the online world, a foundation of community that you and your members own. You're immune to the capricious whims of platforms giant, platform giants who view your community as mere data points in your algorithms by owning your platform, you ensure that your community and its invaluable interactions remain yours, portable and persistent, a digital agorah that can't be walled off or whisked away. It's all about reclaiming the web's original promise, a space of unfettered connection and creation where communities thrive on their own terms. And this is just, you know, me when I wake up in rant on something, but this is one of those things. I just really don't like when these platforms are too locked up by too many companies. It doesn't really fit with the way the original internet worked. And this is the internet I wanna see for the future where we build these communities. And I believe this is where AI is going to help us because they're gonna go through these wonderful, Wendell brought this up and he was so on point talking about this on The Home Lab Show. So go back to The Home Lab Show, the last episode we did where we had level one texts as a guest. And, you know, Wendell had a really good point he made where the challenge you run into right now is AI is really, really good at creating garbage. It's one of its best features. It creates that SEO content, garbage, regurgitated, unoriginal articles and combine that with Google. Google runs around and gathers up these articles and goes, hey, this is what you're looking for. Let me bring this to the top of the search because there's a thousand things that say this garbage content and anyone who has any technical knowledge will look at those articles and go, that's that garbage SEO spam posts, just stuff. So while AI is cranking out garbage and Google has got an AI system reading garbage, you suddenly have to go, I guess I can't Google that anymore. Where are the watering holes? Where are the places of knowledge? Oh, look, there's these people like level one texts running their forums. There's Tom Lawrence running his forums. Tom Lawrence works in the industry, Wendell works in the industry, shares a lot with the community, excellent write-ups. I don't have to search it. I can just look through their forums which also have a search, their index I can find the things I'm looking for and that is a better future to have these little microcosms, these places where you don't have to deal with the enchantification cycle while people duke it out. One company writing garbage, one company sucking up the garbage and you trying to navigate around all the garbage to find that. So yeah, that is why I, yes, that's one of the reasons I've run my own forums. That's why I sell posts. The audience is mine. I do not have an intermediary between my audience. In early days of Facebook, Facebook actually used to tell you who your users were. They would give you the user's email addresses. That is far, far gone. And one of the things about it is you don't own your audience at all. If you build your groups, you build your forums on Facebook, Facebook can change the rules. And if you decide to move from Facebook you can't take your audience with you. And this is something people learned even with Twitter trying to move their audiences were really hard. There's no way to export all the people that follow you or to let them know because the platform, well, kind of pushes down on the algorithm any link you may have to where you're going with the forums. So this is one of the things that is just solid in terms of why I do it, why I continue to do it and it has a cost. It's free for me to put my forums on Facebook. There's a cost. Not just in hosting fees and email fees but there's actually a time cost of maintaining the forums but to me it's worth it in the end. So that's my rant on that. The goal is to never go through in shitification because no one wants to buy my forums or at least I'd never sell them. Why would I sell them? Wendell is based. I like Wendell. So true. Many others searching for TrueNAS and XCB and GINFO. Yes, which channel I point them to and much appreciated. Corey Dockrow has a couple of books that inspired me recently. It says The Internet Con. If you haven't read it, really good. Highly recommend that book, but yes. Oh, this is so true. Old car forums are miles better than any Facebook group. Reddit comes close behind. Yes, car forums are great. I used to participate in car forums a lot more when I was in the cars. Now it's motorcycle forums but the answer's the same. Those old PHP bulletin board looking forums, I don't know if that's what they're running but they always have that old look to them. I like them. I'm gonna find the best information in there. I don't understand people who think things should go in Discord. That is me, old man yells at cloud. Kids these days putting things on Discord? What are you thinking? I think the attraction elements is searches, lower, pursue friction and solve a problem or searching for a forum requires more effort but the effort's worth it. AI, automated plagiarism, yeah. Running a BBS back in the 80s, yeah. I was watching an older video, I think you were recovering from a dirt bike injury. Do you still ride? Yes I do. I was out yesterday. It was nice here in Detroit. We had unusually warm weather for February so yes, I went out riding. Blinking border text. That's a good idea, I like that. The old BBS border text stuff. All right, I have wandered around on this meandering journey of vlog Thursday number 371 for an hour and 42 minutes. I'm going to wind it down here. I'm a little tired but thank you all for joining. Thank you all for coming here. This was wonderful as always. I think it's time to go chill out and watch some YouTube or something. Oh, let's see. Oh, great, now I can picture is you on a year old with a cycle. I do think the year olds are cool but I don't own one of those. I can always, as I like being off topic for a minute so our last parting message will be, what does Tom currently ride? Where is, there we go. Just before I washed the mud off, some of the mud off of it. This is my little Honda. This is my current toy that I ride on. Yes, it's only a hundred and one of these 125ccs but I still like it. So yeah, this is what I ride today. I've had a lot of motorcycles. I've downsized just to do this one now. So you went to Flint last year when I meet you and never got to call back. I can be, it can be challenging nailing down where I'm at. Email vlog Thursday at lornsystems.com would be a good way to do that. So yeah, I still have the older CT90 as well, the predecessor to this motorcycle. So yeah, absolutely. All right, well, thank you everyone and I'll see you next time. Take care.