 Welcome to the Home Lab Show, episode 59. And this is a special episode because for those of you watching it live, there's three boxes, not two, because we have a special guest. Techno Tim, how are you doing, Tim? Hey, good, real good. Thanks for having me. Yeah, no, this is fun, your 100 days of Home Lab and the challenge with it. I like that there's not just let's play with stuff, but a commitment to doing it, because I think that's what really gets people going forward on it, you know what I mean? And picking these different projects. You seem like the ideal guest. And if you haven't heard of the 100-day Home Lab Challenge, that video will be linked in the description. Tim put that video out, we all participated in it. You know, it's always exciting when we submit a video to someone else putting in projects. I really didn't know what Tim was going for, but the end result was amazing. It was. Thank you, thank you. Yeah, just to expand on that a little bit. Yeah, I've been noodling on this idea for, I don't know, like six months. And you know, I kind of bounce some ideas off. You guys, I explained a little bit, but I've been noodling on this idea for like six months. I'm a software developer for my real job. I actually took a little bit of time away from work to do this. But I'm a software developer and software developers have had this challenge for a while. It's the 100 days of code. And I see it pop up in Twitter all the time. Yeah. And I thought like, hey, like, you know, the landscape of Home Lab and infrastructure and automation is in a cloud, non-cloud, hybrid cloud networking storage is rapidly changing. Like we need something for people in Home Lab. And that's where I started thinking like, hey, what, what if we did a Home Lab challenge? And then, yeah, you guys participated in my 100 case sub video, which I greatly appreciated. It was awesome. Yeah, I was, I was a little vague. I was like, hey, you know, I want to do a 100 case celebration, but not about me. I want to get people motivated about Home Lab. And oh, by the way, I think I might do this 100 days of Home Lab challenge. It was definitely interesting like editing your guys' videos. And thank you so much for trusting me with your videos. It was very surreal as it is now talking to you guys. I've listened to you guys for years, years. And you guys are the de facto standards for a lot of, you know, focus areas for me. So it's very surreal hearing your voices, but talking to me and not, you know, just on a video explaining how to do something in a Home Lab or technology or networking or Linux. You know, interesting point. As much as I've chatted with you via Twitter and other places that we talk on Discord, you're right. This is the first time we've, I think, done a live or interactiveness like this. And we're doing it live in front of a whole audience. There's a bunch of people watching, so. Yeah, yeah, yeah. I mean, we've chatted a little bit. Yeah, but this is the first time like doing anything live. Yeah, I mean, I have my own live show, but it's always been by myself. So I greatly appreciate it. It's awesome. Awesome for sure. And I love to collaborate. You know, it's just the greatest thing to just, you know, get someone that, you know, has something in common, just talk about that thing and just geek out about it. It's sort of the, one of the greatest things about content creation when you get to share that with other people, not just people watching, but someone in real time. It's just really cool. Yeah, I agree. I agree a lot of fun. And there's, this is also, it's kind of fun because each one of us actually, interesting between the three of us, I do a lot of the hardware stuff when it comes to some of the physical servers and set up Jay, really strong background in DevOps and Tim being a software developer. If you work in a large company, we are the three personalities that exist to make that infrastructure work in the back end. Someone has to rack the drives and physically build the wiring and the servers. Someone has to do some build automation to get those DevOps things set up. And then someone's got to write the software this all runs on. Full stag right here. I like it. This is the full stag. That's what we should have called it. You should probably mention Linode. There you go. If you don't have us three to set up your full stack, Linode does, you can get your homelab projects spun up very quickly or maybe just have a homelab project that's not suited for your environment because maybe you want it public-facing. It's a great place to test everything. A lot of the projects we talk about here, Linode has been a long time sponsor of the show. They have been, where we host everything. So we're not just saying nice things about them because they sponsor it, but literally because we use them. So if you listen to this podcast, it was literally downloaded from Linode servers. Jay is actually the one who maintains that in particular. So if it's not up and running, it's actually not Linode's fault, we can blame Jay. But so far he's got a- That happened. I did break it once. Jay did break it once. We had a really fun incident. One day we'll do some debriefs on that, but nonetheless, although we thank them for sponsoring the show, they've been great to us. And we look forward to future sponsorships from them too. So that's, we have an offer code to get you started. So head over to linode.com slash the homelab show. Thanks. We keep the ad reach simple. It's something simple when we can, except for homelab, we like to just overcomplicate everything because connecting 10 things to one thing to this thing to that thing is just awesome. I actually know it's calculated and it's planned and there's a reason why everything's there. So we should probably talk about what the 100 days of homelab challenge actually is where people can go to join up on this, how they do this and how they get started and try to get as many people in on this as we can. By the way, I'm impressed with the number of people using the hashtag on Twitter. So I'm watching that grow exponentially. Yeah, I'm blown away too. My Twitter has never been so active. You know, not that I have a lot of followers, but I try to interact with anybody. I try to meet people where they're at, whether they're on YouTube, Twitch, TikTok, Twitter, anywhere, you know, I always try to respond, but man, it's kind of tough keeping up with Twitter the last couple of days. But yeah, it's, so the easiest way to get started is I built this website, 100daysofhomelab.com. It's actually self-hosted. It's built on top of open source. I containerized it. I built it in my own homelab at home and I self-hosted out of my homelab in my Kubernetes cluster. So like, and it's running within my environment. And that's where you can go. It's just a brief overview of the three things that really I'm just hoping people do. One, just commit to the challenge. Just say, yep, I can set aside, you know, one hour a day for the next 100 days to work on something in my homelab. Tom, as you mentioned in the video, starting with the goal is fantastic. Like I totally agree. That's a lot of where people get stuck is they don't know what to do. There's, because there's so many and it's funny because in my early days, as I think I'd mentioned having a TRS80, there was very little you could do with it. You got a book and you started typing code because there was no websites in the 80s to go and copy it off. So you just started typing code. So you're very, very purpose-driven, but there was a narrow amount of things you can do. Now it's analysis paralysis if you looked at all the projects. But if you start with like, I want to build something, a web server, a 100 days of homelab website, or I want to host my movies internally or something like that. You start with some of those goals and then everything will align itself around it, essentially like, I need storage, I need compute, I need these services to run, I need to run Plex or I want to run MB or any of the other options that are out there. Yeah, yeah, I totally agree. And I always mention too, like pick something you're passionate about because at the end of the day, like that'll help encourage you to do something and it'll be much more rewarding at the end too. Pick something you're passionate about because you'll learn along the way. And then that's a carrot at the end too, is hey, I did this thing that I really like. And then so too, it's publicly commit and you don't have to publicly commit. I know that people do or don't or like or dislike social networks, totally fine. But we ask people if you are, hey, share it. I created a link on there that automatically creates a tweet for you and tweets it out. So you don't even have to type anything. I think the commitment part is really important. Jay was, me and Jay started being friends a few years ago but where we really started being a little bit closer in terms of where we're aligned with creating content on YouTube was I think sometimes the term you may hear in business is like an accountability partner. And it's just having more people in the community to encourage you to do things. And so me and Jay started doing that which actually led to us meeting you. And now we have a group of us YouTube creators. You notice that we all kind of share some of the other things, Wendell and Veronica explains and you and a list of other YouTubers we've all kind of gotten together and said let's all talk about these things and encourage each other but also this is where the HomeLab challenge comes in because once you have more people willing to help you and kind of encourage you along the path people really are, if you have a question just not me and Jay but other than me and Jay like other people are going, hey, let me help you get to your goal. What part didn't you understand? Hey, here's the command I used to make that work or solve that problem or here's the hardware I think works best with this. And hey, look, it's on sale, here's a link. That encouragement, it goes a long ways to getting people from feeling alone about getting started to being part of a community committed to it. That's so important because I've often felt like I'm in my own little world that's just segregated. Like I could be around like 50 people but still kind of feel like I'm kind of on my own because my, I mean, they're talking about, I mean, a lot of people like sports, sports are awesome, it's just not my thing personally but of course they're gonna talk about those things and things they're passionate about but hearing conversations in a crowd about technology just doesn't happen very often. So Tom and I go to PenguinCon and it's like, I feel like I could talk about, I don't know, randomly SSH. I guarantee you probably a third at least of the crowd knows exactly what I'm talking about. And then you get like-minded people in content and people realize, you know, that's actually a lot more popular than I thought and how many people in a crowd are into technology and they don't even mention it because they assume that everyone else doesn't know what they're talking about but for all they know there could be several other fans there. So I feel like it's also kind of bringing this awesome geek stuff that we do into the mainstream. Like we have this hobby and it's fun and we're not as nerdy as we are in the movies. We're actually kind of cool and we do the technology thing and you could do that with us. Yeah, I totally agree. That's a good call out too. Like you do hear people talk about sports quite a bit and a team they're rooting for or a team they don't like or an neighboring team or a rival team. I feel like we have those kinds of conversations either in Discord or wherever in our own group amongst brands and technologies too. People root for brand A or brand B and I don't like brand A or brand B but it's that same kind of rivalry you feel with sports. I feel that same way when I talk to people about how I set something up or I use 10 gig or I don't use 10 gig. Those kind of rivalries and those things that come off are one, great learning experiences but two still connect people across different experiences. So as much as I'm not thrilled with the distro wars which we can call it but there is something fun about it and the reason why is because when you're passionate about something or even myself, I just did some videos recently on hypervisors and I talked about KVM and I talked about Zen and I talked about Proxmox, I talked about CP and G in order for me to even get the Proxmox video and one person was still upset that I did it all he says I was not qualified to do Proxmox which made me laugh but I said to be in video it took more research because I don't usually use Proxmox I think it's a good solution so I don't hate people that use it by any measure. I'm on team XCPNG but it forced me to learn deeper features I don't even use it also forced me to learn deeper the other side and I think that's sometimes where the rivalry comes in because you want to be right about your product you want to be an advocate for the product or the distro or the editor that you use Vi versus Emacs but it forces you into this competitiveness because they're like I like it because it has this feature and when someone says that you're like oh but mine I think has that feature hold on let me read so I think that's actually some of the fun that comes out of it it's not all bad it's as long as you keep it lighthearted if you will but there's certainly I use ArchPy by the way kind of people we know it's another joke in our industry but either way in some bigger picture it does make it kind of fun and another thing I wanted to point out too is that this isn't always the case obviously but there is a subset of people that work in IT that maybe they work in a small IT team they could even be the only IT person which I think should never be the case because you should never rely on just one person to maintain your infrastructure right that person never gets a vacation but usually what happens is that it feels like you against the world because you have to maintain the servers for the company and do the best thing and other people in the company really don't know what you're talking about so if you want to talk to a coworker about the best way to tune the TCP IP stack they're going to look at you like I work in accounting but then when it comes to HomeLab there's no shortage of people to talk about and you could literally build at least somewhat equivalent system like work or maybe just set up the same software on your side and then try to experiment with it then you might have a colleague that's also a HomeLab fan that you just chat with online and they're like, yeah, did you ever try X because that would totally eliminate like a couple layers of your setup and then you start to realize that the stuff that you do for HomeLab if you do this for a living can go to work and then help you be more proficient there or if you don't have a job in IT it could totally help you get your foot in the door. Yeah, yeah, I totally agree and just to kind of expand on that a little bit about sharing information with people something I've noticed too in HomeLab especially just over the last couple years is people are willing to share how they solve stuff too like people kind of want this platform to talk about what they did and why they did it how they solved a particular problem and oh, by the way, here's all of the code or documentation on how to do that too and it's fantastic. So through that over the last couple years I feel like I'm working on this bigger team this bigger team that I've had the 10 years prior because I've been doing it for a long time after college I had this old crusty computer that was like already 10 years old that was in my basement and that was my first HomeLab computer and it was me figuring out stuff by myself for 10 years and then eventually it was like, well people started asking me how I did this or how's my blog at home or what network equipment do you use? And now it's to the point where there are communities built around this even shows and channels built around this where people can share these ideas and it's just great. I mean, I feel like we're all in this together and on top of that like standing on the shoulders of giants like you too we have these great channels that are putting out great content of how to do things and people really, really study those guides and take it word for word but at the same time if they deviate from a standard they'll be sure and tell you why and how they did it and why they did it. And I just feel like, man the last couple of years have been fantastic for HomeLab, for me personally just because getting into it with content but at the same time discovering this like huge community around it I feel like I'm on this team of like a thousand and it's awesome. For all of its flaws YouTube has been in a big picture overall great place to share a lot of this knowledge. There's a lot of different communities not just HomeLab, it's greater than that but amongst the people sharing different technology ideas putting them together including a lot of cybersecurity people like my friend John Hammond he does a lot of deep dives on reverse engineering and things like that. So even if you somewhat wander a little bit out of HomeLab and come back over to things like the cybersecurity side of it because it's a fast growing market that needs more people in it it's been a great facilitator having all these different videos available to you like the library of knowledge that you can watch for almost free you do have to watch some stupid repetitive ads so that it's a but that's a pretty minor inconvenience in comparison to what you know a college career costs to learn a lot of these same things. And your friend John sounds pretty nice but if he ever says anything about opening up a park please stop them. I know, being an IT guy named John Hammond I mean, come on. Yeah. I'm surprised his handle isn't like Jurassic John or something like that. You know, you may as well have fun with it, right? Yeah. Just own it. You've been named that. It's just I think it's interesting to hear how people got started with HomeLab and maybe we could kind of do that. I feel like how I started I'm just gonna make a guess like probably one what one third of HomeLab is bands that have started this way and this is gonna be extremely common. I started because I was studying for a certification actually certifications. I've had several of those. And when I was looking into how to get a certification in IT usually people will say, well, you could just download this emulator and it just perfectly matches the real switch hardware and all this and you could just install it on your computer and learn that way. And then I remember thinking why would I want to install an emulator? Like if I'm running Nintendo games I'm totally fine with that but I don't wanna emulate hardware. Give me the hardware. And I would just go on eBay and I would just buy some really old devices and then just start playing with them because it's just so much better to get started or to study for a certification when you have your hands on the thing. And then from there, it's like, oh, I could run a file server. I could have virtual machines running and then just kind of spiraled out of control. And here we are today. You know, the thing that really got me started and I've posted it before maybe I'll do a dedicated video because people always ask me like, what is some of the old servers? And my oldest server is before I was even using monowall. I think it would have been probably a Mandrake. Mandrake Firewall. It was a Linux firewall from like 2000. So 20 over 22 years ago but I used to run a series of unreal tournament servers. So I had to have a firewall that had the capabilities to handle all the different servers that I had set up. And, you know, I rigged it together quite a bit but it's where you really start learning and it made me learn more about hosting, setting up Linux servers because oddly Unreal Tournament 20 something years ago ran on Linux servers. The cert, even though it was a Windows for playing the game, it was, gaming in Linux was not existent 20 years ago. It's taken a long time for that world to change but it was a lot of learning I had to do and it really got me into Linux from a hosting perspective. I mean, I was already a mail server administrator in like 2000 but- I'm so sorry. I know, I know. I'll tell you, you will never learn as much as when you are a mail server administrator and on a Usenet post trying to learn proc mail recipes to stop the way to spam. That was my early days. But, you know, that was what I did for a day job. And then I took it all that knowledge home to actually do it. I was among the first to have a cable modem in my area where you could actually host things. So it was wild times. And on Unreal Tournament, yeah, absolutely. That was one of my first journeys into networking is just getting two computers on the LAN when I first started to just, I want this game to work between these two computers. And it took hours, you know, because when you're first starting out, you don't know how to do this stuff. And it's so alien to you. You're kind of just like reading and hoping and just it's almost like you're just trying random things hoping something is actually going to make it do the thing you're hoping it's going to end up doing. Then you break it, of course, and then you fix it and eventually it works. I think for a lot of people, the gaming is a big stark. Even the generation day, I mean, what do you want to do? You want to host your Minecraft server because that's more popular if you're among that age group or my son's at. Unfortunately, he hasn't seemed to taken an interest in hosting it. He actually had one of my staff set it up. But he was smart enough to know to ask someone. But, you know, it's still starting with the goal starting with something in your head. Like this is what I would like to do, you know, then the hardware, I seen someone post in the chat they had like an old R710, which is a pretty dated system but still very capable of getting something done. But they said, you know what to do with it? Well, come back to, you know, is there a server or a service you'd want to host? And if you even talk about the Raspberry Pi, we need to have Jeff Gehrling on some time because he is the Raspberry Pi King. And Jay has quite a few of them too, though. There's, you know, you start with something really simple like that and you can build all kinds of extensive things. Yeah, yeah. It's funny you mentioned Jay starting out with games. That's kind of, I think my first networking for Ray, you know, my brother and I wanted to play NBA Live 98 or something on our PCs. And I had no idea. Like I wasn't even that much into computers. So like I understood them, but I had one. You know, I was, I wasn't tech yet. And I remember just spending hours upon hours of like trying to figure out how to set a static IP address on a Windows 98 machine. I had a hub that wasn't even a switch at the time. It was a hub, you know, and configuring two static IPs and getting them to communicate. I mean, it took hours. It took hours, but I remember, I vividly remember, you know, network cables drawn through his bedroom to my bedroom and the hub in the middle because we didn't have enough cables and getting it working and just seeing the hub lights flash. And I honestly, I cried a little bit. Not joking. Cause I was just like so happy that I had figured this out, you know, this, this, this magic, you know, of networking and then we could play, you know, NBA Live 98 together, you know, even though we were sitting in two different rooms, but now that was my very first, like networking experience besides, you know, plugging in, you know, a phone cord in a modem and then, you know, anything like that. But yeah, it was, it was definitely that. And then that, I think that's what, that's what, that's what lit the, you know, spark that started the flame that then I started realizing, oh, I could, if I had two computers, I could, you know, do this, you know, but it wasn't until a couple of years later where, where I actually had one that was dedicated, it wasn't my main machine. Once I had a second computer that wasn't my main machine, that's, that's when it really took off because then, you know, I, you know, I stood up Windows Active Directory to figure that out, stood up Linux to figure that out. Then I started thinking like, hey, I want my own blog and, you know, use Dine DNS for my blog for years because like this is awesome. I can get dynamic DNS and people see content I write and, you know, it just, it just spiraled from there. But for me, yeah, definitely it was, it was a very, very related to gaming and getting, you know, two machines networking. It sounded like such a geek. I think my brother remembers that I cried, but I'm not gonna lie, I think I did. There were tears of joy. Totally fine. You know, the thing is you mentioned, you know, talking about gaming, it's one of two moments before the certification thing that kind of like really got me into technology. Video games was the first. I grew up without a computer until I was like 18 or 19. Before I had my first computer, I'm not even kidding. We couldn't afford it. It just wasn't ever gonna happen. So computers to me were like this miraculous thing that only the rich people could afford. No, I was just substantially poor. It wasn't that at all. But in my eyes, it was like only the wealthy people can have a computer. So I remember as a young teenager, actually my might have been 11 or 12, I'm walking down the street. I must have been very outgoing. I see this truck and this guy getting out of the truck and inside the back of the truck was like a mountain of computer parts, like cases. And just like, it looks like he's salvaging computers. I'm like, so what's the computer parts? And he was like, oh, yeah, I get parts and I build computers. I'm like, you can build computers. And I was thinking before that that computers are something that can only come out of the factory. I didn't have any exposure to buying a part and putting it into a computer. I've never seen that before. So the whole concept that this individual was able to do that was huge. And then later on, I became very angry as a few years later because I was a huge Final Fantasy fan and Squaresoft of Japan decided that Americans are not to have Final Fantasy V. And there was a whole bunch of drama about that. I remember going to a friend's house who had a computer, which was amazing. And he was playing Final Fantasy V in English. I'm like, how in the heck are you playing a game in English that was never released here? And he says, well, I have this ROM and there's this translation patch that people got together to work on. I'm like, okay, so people just didn't weren't satisfied or not satisfied. People weren't just accepting of the fact they're never gonna get this game. They decided to translate it themselves, work together, solve the problem and just release it for free. Yeah, I'm playing it right now. I'm like, oh my God. And then that was the time I realized that collaboration in tech is super powerful and we could solve problems that companies can't solve. And then from there, I just became more and more and more obsessed and the rest is history. So it's interesting you mentioned games because that was just such a moment for me that I was able to play a game that the company decided that I should not be able to play. Yeah, the other aha moment for me was, you know, again, I didn't get a computer until later in my life. We bought a Commodore 64, Commodore 64 when I was older from a garage sale and that was our computer, but it was already 10 years old but it was still awesome to me. But like my other aha moment, very, very relatable to what you said was, you know, when I went to college, my PC didn't have a network card in it. It just didn't have it at the time and I had no idea. I was the same way. I was like, what do I do now? And they're like, you know, you can just buy one for however much you forget what it was and then ask someone to put it in. I was like, how the heck do you put this thing in? Like there's no way I'm not qualified to do this, you know? And I asked my RA at the time that, you know, Rodney, I was like, Rodney, can you help me out real quick? You know about computers and I saw he had, you know, the cart full of computers in his dorm room and he opened it up, put the card in, you know? And I saw how easy it was and that's when I was like, never again, like I can do this by myself. But that was, yeah, I agree. It was, I was blown away when I learned you can, you know, customize and expand computers too. It was, it was another aha moment for me where I was like, okay, I can do this by myself now. Then there was lots of tinkering along the way in college because I was in the dorm rooms and I could do whatever I want. But yeah, I totally agree. Let's go forward a little bit because you go from I'm not qualified to having some really solid Kubernetes and hyper-converged infrastructure videos that I've been watching. You covered Harvester, which is really cool. A few people have asked me about it and I've just simply replied, I did watch your video on it and I think it's really cool. And there's some complexities to some of these but what brought you all the way? Because you do, is it your day job that you do a lot of Kubernetes or is that a more of a hobby? Oh, yes. Yes, I'd say, so what brought me to Kubernetes? It was really, so I, you know, I've worked at different companies. I don't want to say bounced around because it sounds like the wrong term but I've moved to different companies and along the way, those companies have been using Kubernetes. When I first started, I had no idea but I was on a team where we were building software and the way to get that software into their production environment was to put it in a container. So, you know, I was on a newer team, we were building a newer project. So I said, we're gonna take the newer platform at this environment and we're gonna put our code inside of containers and we're gonna ship it into this thing called Kubernetes or however you pronounce it. And, you know, we're gonna play this game because that's the, it seems like the easiest way to get our code into production. So I learned about it a lot at that company but then, you know, after I left and I had my home lab, I, you know, I write software outside of work too, you know, tons of little, you know, websites, bots, just fun things and I wanted that same pipeline. Like I wanted to be able to write code, put it in a container and put it in production, my production in minutes, automatically and not touch anything. And so that's where I was like, I'm gonna try Kubernetes at home. I know it sounds crazy. I know it sounds complicated but I'm gonna do it at home. And it was all like driven from this, me wanting to get, you know, repeatable code into production as fast as possible. And, you know, not have to mess with the alternative ways is, you know, write my code and then maybe SSH and put that inside of some, you know, service and restart that service and then have that spin up. And so I, you know, I adopted a lot of things I learned at work and applied them to my home lab but then at the same time, I then learned, you know, exponentially more about Kubernetes working at home than how I can apply to work. It's like this, you know, this cycle, this feedback loop of I wanna learn something, I do it at home. I've found out something awesome. I can apply it at work. Oh, I learned this at work. I'm gonna apply it at home, you know, and it's this awesome feedback loop. So, you know, I do do some Kubernetes stuff at work. Even today I work at a small startup, really awesome startup and it's pretty awesome. We all wear many hats. Part of that is Kubernetes. A large part of that is writing code but it could be many other things too. So, you know, it's a little bit of DevOps. It's software engineering. It's, you name it, you know, any title you can think of we do to, you know, have, you know, services out there and running, even some networking and stuff, even some tech support at some point in time. It's pretty great. It's pretty great going back to a smaller company and wearing all these hats that I kind of wear in my home lab too. So it's pretty fun, but yeah. It's not been such a great feeling to work for such a company because it's sometimes like, some companies just see technology as the thing they have to do that they don't want to do, but we just kind of have to do the technology thing darn it. And then you have companies that are all about it. And I think working for a company that lives and breathes technology is exactly the type of company that's that a individual that lives and breathes technology needs to work for in order to be happy because if you can't work for a company that hates technology, it just doesn't work. Yeah, yeah. And we are a technology company too. So that kind of helps that, you know, our product, our digital products. So it really helps. But yeah, it's allowed me to grow and explore so many things. I wouldn't have time to at home and vice versa at home. Then I learn all of these things and I can apply them at work. I think, you know, monthly or something and at least monthly, I pick up a few things at home. Then I, you know, I get to try at work. And vice versa, I learned so much at work too because, you know, it's really focused on, you know, having a solid, you know, dependable, highly available service for, you know, millions of customers, you know, I get to apply some of those principles at home. I know I don't have to, you know, I'm not serving out millions of customers at home, you know, handfuls, dozens maybe, but still I get to, you know, absorb some of that knowledge and, you know, kind of pick and choose and apply that to stuff at home. It's great. I think an important aspect too, and this is something me and Jay have talked about. There's a lot of others on YouTube and I'm loving seeing this. We're not people who read this from a book, stood it up our lab and that's it. We've showed you the basics on how to get it going. You gain a lot of knowledge deploying these things out in the field. I've always liked that old adage, no battle plan survives, contact with the enemy. And you can read the book, you can get it all stood up, but actually servicing things at scale, you run into lots of quirks. One of the things you had mentioned in one of your videos was you talk about things like disk pressure because you have to think about performance and everyone, you know, they want the magic of hyperconversion infrastructure and high availability replicated storage with something like Longhorn. But you're like, by the way, to actually make that work and get the IOPS you want, you need some incredible hardware. And there's sometimes design considerations because there's always budget considerations which are driving design considerations. Because sometimes people say, why don't you have this configured hyperconverge? So it's magically replicated. I'm like, because that cost the IOPS, that cost the interconnects between have to be, well, they become the weakest link in the number of IOPS you want to get, but that's not something you think about from a lab standpoint, that's something you know when 10,000 people, 20,000, 100,000 and keep scaling up are using your product and it's online facing, you go, oh, I don't have the database activity at home, I do in this production environment to I can see where everything stretched then. I mean, it sounded great to replicate it that can't actually be replicated at the speed the data comes in because I didn't plan enough or I didn't have the budget to build the interconnect. So speak to that just a little bit of what some of those learning lessons. Yeah, so I think there are lots of lessons along the way when building any kind of system, a lot of people I've noticed gravitate towards the stuff that I do that's highly available. I mean, you guys probably see this too whenever you mention something that's highly available, fault tolerant, people are really interested in that because having something that's highly available means more uptime, but it also introduces a lot of complexity, a lot of complexity that I've either stumbled into, figured out on my own or didn't read along the way and figured out myself, but it is tough. For instance, people always want, you mentioned disk pressure, people want this magic solution that's like, hey, let's spin up a Kubernetes cluster, let's throw 30 containers inside of there. Each of those containers are two, three, 400 megabytes or even maybe gigs. And then all of a sudden Kubernetes says, I have disk pressure warnings. People are like, why do I have disk pressure warnings? That's just a warning basically saying, hey, Kubernetes disks are running out of space. We're not gonna do anything else until you clean some stuff up, otherwise Kubernetes won't run. Probably a little more technical than that, but you kind of have to plan some of these things out. Some of them you'll discover along the way, but when you started getting into things that are highly available, you got to think about all kinds of things. I didn't even think about load balancers, preventing like split brain scenarios, keeping services up. If you have stateful services, those can't really be scaled well. When I say stateful services, one of the, going back to HomeLab, one of the things people like to host a lot is like Plex isn't designed to be scaled. It's designed to run as one. You run two, you run into a split brain scenario where one thinks it's the one that has all of the data, and then another thinks it has all of the data. And then so sometimes when you send that traffic to individual services or pods, each one thinks it's the one that should be running and doesn't know about the other. And so a lot of times people think like, hey, if I'm running Kubernetes, I have Plex, I'm just going to run five instances of Plex because I want it to be highly available. It doesn't work that way. These systems has to be architected and designed in a way that they can be highly available, which means if you can make them stateless and put your state somewhere else, like in a database or somewhere else to where these pods are, I don't want to say dumb, but they're stateless to where they don't know about each other. And the only way they do is from some other service or database or something else out there. And so those are the kind of things I run into a lot, things that I've had to learn along the way. I have very simple, even code I write that I don't think up front like, hey, I need to scale this service. I go to scale it and I'm like, oh crap, I didn't. I have local account of something in memory and now that count only exists inside of that one service. So I can't scale the service until everybody think about it. So, you know, and the same goes with storage too. A lot of people, you know, think, I mean, Longhorn I think is pretty awesome. Seth's pretty awesome. Rook is another way to get, it's basically, it's kind of complicated even for me to think about because I haven't done a ton of stuff in storage. I've done some, but you know, Longhorn is this way to get highly available block storage inside of your cluster. So basically you could have, it's kind of hyper-converged to where the storage is floating around inside of your Kubernetes cluster, just using up available extra storage on all your nodes. It's pretty cool because you can replicate that across all of your nodes and your servers. And then if one goes down, they can just point all of your services to the other place and then re-replicate on another one. And Rook does that same thing, which is based on Seth. I haven't done a ton in Seth, but you know, it's that same idea where you can replicate storage to a Seth cluster using Rook. But, you know, storage is another thing where even simple things like NFS, like people are like, I want highly available NFS. I'm like, I have no idea how to do that. Do you know? I'm like, I have no idea. I have no idea how to do that. I was going to mention, because 45 Drives is in here, maybe they're still watching. They have some really good topics where they dive deep into Seth, but we're a 45 Drives partner. So, you know, when I'm not doing YouTube things, my business hat is on. We are building solutions for clients. And someone asked when we built our petabyte storage server for one of our clients, like, oh, Tom, I can't believe you didn't use Seth. And I'm like, why would I? Even the 45 Drives consulting team, they don't start with the most complicated solution. You build up to the solution and no more. You want to be able to support this. It's actually supported on TrueNAS with their internal team that we provide this for. TrueNAS is easy for them to manage. And the adding Seth as a layer on top of it would serve no purpose other than a layer of complexity. So it's weird how many people ask me those same questions because there's always those second guessers when you do make a video. Why would you do it that way, Tom? Modern file systems all use Seth as if Seth is an actual file system as opposed to distributed away. And it's like, look at ZFS in the backend and we only need to present it as an SMB. They're just dumping data sets for a system to process data sets to do calculations. Like, I don't know where Seth fits into this workload that they have. Because it's, I think they have eight compute nodes that just look at SMB data, cross an SMB share, process it and give output. That's it. I know where to go for storage questions. It's Tom all the time. Yeah, I do it all the time myself on my end. I've had some very interesting problems that only I can find because of this ability to find issues that other people are like, how did you run into that exactly? Like, Jay, in his NFS stale file handle problem you had, I think that was one of them we troubleshoot a while ago. Well, I never solved it. I thought I did. And the issue was that for whatever reason, like I was switching everything to flat pack. I really like flat pack. I, but, you know, I started, my as well scripts would remove the dev package for Firefox and install the flat pack, for example, same with Underbird and the other things. And then what I would notice is that Firefox would lose the ability to upload files. Like you could click upload like a YouTube video, right? And then the file picker dialogue comes up, you select the video that you wanna upload, you click, okay, nothing happens. It's like it never got the request. And there's errors in the logs a bunch. So it's not like I didn't know what it was saying. And then I traced it down to, you know, something flat pack related. Like, and I still to this day, I had to stop using flat pack for Firefox and Thunderbird because something that I'm doing basically makes Firefox useless. So it randomly work, randomly stop working when it comes to uploading files. There'd be like XDG document portal errors all over the place. And I'm like Googling for this. And it's like, am I really the only person running into this? I could wipe my machine and just, you know, continue to, and set it up. And what I, you know, I think I could think to blame is that it's expecting something to always be mounted. And because I'm using NFS that auto mounts, it just really kind of, it gets confused with open file handles. I mean, these are only problems that somebody gets to at the end of the homelab journey, right? Cause it, I mean, usually most people are fine to just take the Dev package of Firefox or whatever and just call it a day. But, oh no, I have to go a different direction. And because I go that other direction, I find bugs and I find them everywhere. So I mentioned to Tom, like you use flat pack. Can you just like open up a file? Does it work? Oh, it does. How does that work? Yeah, it's funny you mentioned NFS and problems because I run into them too and they're ones that I've just discovered along the way. Like I love NFS, like it works great, but there are some use cases where it doesn't really work well. One that I always run into is like with SQLite databases. I haven't really figured it out, but it's sometimes in containers if you mount your storage over NFS, like SQLite databases, like write some kind of lock file and it can't read that over NFS. And so things don't work. And so yeah, those are definitely the things you'll explore along the way. And sometimes like you mentioned at the end of your whole map journey, I have people who spin up NFS and then they get their containers going and then they mount all of their, they build up all their infrastructure to get this going. And they plan on NFS being their storage of choice for their containers and then realize, hey, this container uses SQLite and I can't do it over NFS. So yeah, those are definitely the fun ones to figure out. And sometimes you figure them out really late in the game, like you mentioned. I made sure to mention, because windows have to deal with that a lot in my day job. But when I did my TrueNAS file permission videos, one of the things I covered is you probably got the permissions right, but Windows hasn't learned that you got them right. So you have to log in and out of Windows. People were telling me, you shouldn't have to do that. I said, oh, I can agree with you all day that if I adjust the permission in TrueNAS, Windows should reread the permission of that file. But I will tell you that it doesn't all the time. And I don't know what circumstances it does or does not. But I know if you log in and out of Windows, it then reads the permissions again when it reconnects because TrueNAS is usually not part of, unless it is, but it's usually not part of an active directory domain. There is still a lot to that. And having storage persistence is, like you even said when you're spinning up these Kubernetes, if they spin up before the storage is attached now, they don't know what to do either. So it's very critical to getting all of this automation set up and making sure that these things work. Yeah, yeah, yeah. Yeah, I totally agree. Kubernetes, though, is kind of nice because it'll, well, the Kubernetes way is basically crash and restart until things work. So it's part of Kubernetes, like if storage isn't there, it's going to say, okay, exit, crash, start. Again, is it there? And it'll do that over and over. I mean, eventually it'll start like backing off, but yeah, I've been there many times with my containers. I probably have a few in my own environment right now that are in a crash loop, but that's par for the course for Kubernetes. Could even be a race condition, right? Because I'm sure you run into that many times too. Yeah, yeah, yeah. Like, yeah, a database connection it needs is not there or it is there or I wrote some code that's supposed to be asynchronous and I didn't do it right. And it's waiting on a database connection. Yeah, I've been there. I think it was there last night with something I was working on, yeah. My personal favorite is when it checks for the storage before the storage layer actually came up. So you have to kind of go in and give one a time out. So it kind of just waits longer as the one has a chance to come in. And so there's better ways, obviously, of having things start up in a certain order, but that, yeah, so many times. I've been there. Then DNS, like we could, I mean, we could complain about DNS. DNS is its own episode. It is, it is, it's, I love it and hate it at the same time. Yeah, absolutely. Yeah, we were, I need to do a video on it because that's literally what my team was chasing some crazy web DNS from a client we just took over. There's so much to it. And I'm like, ah, like sometimes you lie on me for some of the really esoteric complicated things. All I know is if you look up DNS security trails, you can find some DNS history. And sometimes what you have to do when you take over a client and everyone let everything expire, you need to rebuild DNS from scrapping on the internet to figure out where everything used to be. There's the whole topic there. Yeah, I mean, there's a lot of, we could even have a rant episode, but we won't, we're not. Yeah, we won't. We're not just here to rant. We're absolutely not going to do that, though. It's too easy to commiserate because we're all mad at DNS. So one, so I think in, you know, getting back to the 100 days of homelab thing. So I definitely want to make sure that, you know, we get people on that because it's really a great way to get started with this. And it's such an amazing idea. Like, like it really, it really is an amazing idea. And some people out there might be, you know, still hesitant to get started. And I say, just, just start. Like, like Tom says, you do have to have a goal. Yes. But you could also just install some random Linux distribution on a computer in your closet that you're just not going to use anyway. And of course, yeah, I could probably break the install and have to redo it a few times. I mean, just do something. That way you're kind of like getting into the habit of doing something or just kind of like installing Linux if that's where you're going to go. And just kind of like get your hands on it. And I feel like if you spend too much time, you know, deliberating over what to do, then you could just probably end up never doing it at the same time. But like Tom says, you still do need a goal eventually, but got to get started. Yeah, yeah. Oh, go ahead, Tom. I want to just comment real quick though. I mean, Jay, we just covered this in episode, the episode before this one, so episode 58. We talked about using some of the hardware, the random things like even broken laptops. My staff had a laptop with a smashed screen. That was one of the servers for some project he had at home for a while. Cause he didn't want any, like we have plenty of stuff at the office to play with cause we have a pretty big lab environment, the office that all my staff gets to play with. But, you know, having some of these broken laptops like yeah, he took it home and he's like, cool, I can run a few things on here and it's quiet. It doesn't, you know, by the way, it hasn't built in battery in it. So if the power goes out, you know, it's, it already has a UPS. The UPS and a KVM because it's got a screen and a keyboard mouse on there. You literally have a self-composed server environment. Yeah, yeah. I totally agree. And yeah, so the 100 days of home lab stuff. Yeah, so I appreciate it. Yeah, I've been, like I said, noodling on it for a long time. I know that some people, you know, don't know how to get started, don't know where to get started. I honestly think even if you say, hey, day one is planning and that's totally fine too. Like it doesn't have to be hands on a big part of having a home lab or building systems is design and planning. So day one could just be thinking about it, planning it, you know, setting aside some time to document or diagram or explore or read. I mean, that's totally fine too. It doesn't always have to be hands on. You know, and I think then if you just do that daily, you know, it could become a habit. You know, I work at, I've worked at companies where we have these daily standups. And so I'm treating my one-tweet a day like my daily standup. Like, hey, here's what I'm working on or here's the challenges I'm faced with today. You know, and here's how I'm going to solve it. You know, in standup, we usually do that and scrum, you know, they do that a lot. Like, hey, what did you work on yesterday? Would you work on today? Do you have any roadblocks or any challenges that you need help with? And I'm treating this hashtag as one big scrum for everyone in home lab, you know, treating it like one big scrum because I think it's important to just get in the habit of like focusing on what you want to accomplish for the day, saying, here's how you're going to tackle it. And then hopefully, you know, the next day it's something you tackled or finished or completed and you're on to the next thing. If not, that's totally okay. On day two, you could be working on the same thing from day one. You could be working on that in day 90. You know, and that's another thing too. Like some people are asking like, hey, do I start with you? And no, you don't have to start with me. If I was on day 100 and you started on day one, I would love it because that means that this is living on. This is going, this is bigger than my 100 days. It's someone's 100 days, you know, in a year. It's someone 100 days, you know, six months ago and you can see that along the way. So yeah, no pressure to start on day one and you didn't miss the train. The train I hope will be leaving every day and it didn't leave on the day that my train left. Your train could leave next week. Your train could leave next, you know, next year. But I hope that people will join. People listening to this episode have to check off a day. That's right, you know. That's right. It was almost my day four. It was almost my day four, but I was like, okay, I'll, I did retweet it, but it was almost my day four. Then I backed up and deleted the tweet. And then I was like, okay, well, I'll just say, hey, I'm going to be on the show. My day four, I mean, I could even say here, but yeah, it's just, yeah, more. So day four for me of 100 days of home lab, what was it, flux updates? So I run flux. There were some flux updates. I've been kind of pushing it down the road, but updating my flux, which is kind of coordinates my Kubernetes, it's so meta, but it's so awesome once you do it. And then I created a bot that I wrote some code for that runs the retweets on 100 days of home lab Twitter. And I got that into production. I just need to do some logging and make sure that it stays alive. So it's my day four is doing that. I want to mention too, in case anyone's not on Twitter, Twitter for all its faults, it can be problematic. Don't follow problematic and don't click on whatever is trending because it's going to be stupid. Unless it's the 100 days of home lab that's trending, you can follow that, but feel free to participate in there. I see people trying to avoid certain platforms and things like that, but I found Twitter to not be too bad. Any social media can turn into a social distraction, but if you keep your vision focused on there and narrow and just ignore that whole, you know what? That's what we need. There's probably a web browser plugin that just removes the trending. Cause every time I look, I'm like, I want to click on that cause it's, it's following. Like I want it, but I know it's going to be stupid. Like I know it's going to be something that's not going to be useful to me. It's certainly not going to be tech related. A lot of people might be interested in what Kim Kardashian is doing. I'm personally not interested in what she's up to these days. So trending is just not going to be something for me. I see someone in the chat scene. So Dunkel says, Twitter is the least worst. We'll go with that. Yeah. That's a good point too. Cause it's not only Twitter. So this hashtag, obviously a lot of social places support hashtags, use it anywhere. Text your friend. You could text your friend and say, this is my day four. And if that friend is holding you accountable, that's awesome. And also, and not to just plug a Discord, but we do have a channel dedicated to this. Exactly this. Some people don't want to do social networks, but they're totally fine with chat clients. And we created a channel where everyone has their own thread on their hundred days and people are actually participating. And it's really awesome. Our mods thought of that. I was like, this is, this is great. This solves the problem. And if you still want to text your friend with a hashtag, you absolutely can. Yeah. You can go as basic as text is a little more complicated to discord or go all the way to Twitter. There's a lot of different options on here. So. Absolutely. So, yeah, that was awesome. I don't have anything. I mean, I feel like I could talk about homelab forever, but I do have to say that if we talk for more than 10 more minutes, there's going to be sweat like pouring down my face because this studio heats up quick. But man, I feel like I would still wish we could have like a two hour episode chatting about this. I'd love to come back. So I think that's a segue, but yeah, I'd love to come back. Thank you so much for having me. I've looked up to you guys for years. Like I said, years, even before I started my own YouTube stuff, I'd go to you, I'd go to Linux for, I'd go to J for Linux and I'd go to Tom for a storage and PF sense and TrueNAS, or it was, you know, it was FreeNAS at the time. But, you know, I, and I refer to you guys's channels all the time. And it's not just lip service. I know, you know, when people ask about certain topics I'm not as educated on, I'm like, go to Tom. He wrote the book on it, you know, go to J. Absolutely has a, you know, a very focused, you know, tutorial on how to do exactly what you're looking for. You know, it's interesting when I did that. Oh, I'm sorry, go ahead. Well, I think it's really important. I mean, YouTube is a very, I never see, cause I've seen some people that have this in their head, other creators from when they first get started thinking it's a competitive space like business, but it's much the opposite. Cause even business isn't that competitive a lot of times it's more about collaborative. And once you understand that in, like I said at the very beginning where the full stack between three of us, when you work in these environments, no one person knows everything. I don't have time to be a Kubernetes expert and a TrueNAS expert. I mean, you watch my ZFS, which by the way, I'm wearing my cult of ZFS shirt. But if you watch my ZFS as a cow video, like there, once you really take the time to understand ZFS, like, wow, it's a really complicated topic. I'm like, yeah, at some point my head's so full of that. I don't got time for Kubernetes. I just watch techno Tim's video and follow the instructions. That's kind of weird to me when, you know, the whole, I don't want to call it popularity because when I think of popularity, I think of like, you know, a recording artist that gets millions of plays or something. That's what popular means to me. But it's interesting, like I'll go on a Linux group and I'll just make a simple comment. Like, yeah, I agree. That's a great distro, right? And then there's like a bunch of shocked faces. I'm like, what's wrong with people? Like was my comment just agreeing that I also liked that distribution such a weird thing to say that people are shocked and it just keeps racking up these surprise faces. And I'm like, oh, right, right. They recognize me from the thing. And I swear to God, no matter how much time has passed that never normalizes. I never expect that. I never, I'll like be scrolling through YouTube, you know, after work on the couch, you know, on the TV and I'll see my thumbnail come up. I'm like, that's weird. Okay, see my own face on TV. Okay, that's just weird. Moving on, no, I totally appreciate it. I think one thing you'll find is I'm super excited about technology. I think Tom could also agree with this. Like I'm just not the popular feeling person. Like I'm human, everyone's human and you know, it is what it is. But I really appreciate that. I just love to help people in any way that I could be of service. Yeah, and just to thank you. Yeah, and just to echo what Tom said earlier too, like, you know, about it's not competition. Absolutely is not, you know, in the beginning, I'll be honest, I thought it was. Getting into this, I thought it was and it's absolutely not, you know, I thought that in the beginning, not that I thought I was competing with people, I thought I was just competing for eyeballs. But what I learned through this journey is that, you know, what do they say? A rising tide lifts all boats. And so if we're in this category, my goal is to grow the category and not my channel because growing the category will rise the tide for all the boats in it. So yeah, it's, you know, those things you pick up along the way and I totally agree. And again, it wasn't like, yeah, I never thought it was a competition, but, you know, from the outside looking in, you know, that's what you think, but it's not like that at all. And it's fantastic. And I think all of us, you know, creating content in this category is just gonna help all of us too. I think it can be that way though with some people, there's some competition. I feel like doing business right, don't compete. Right, if somebody was to hypothetically, you know, because Tom and I are both a business, right? So if someone comes to me and they're like, I need to know why I should pick you over Tom. I'm like, flip a coin. Or like if Tom gets the gig, he's gonna do a great job. If I get the gig, I'm gonna do a great job. You go with the other one of us, you're gonna get a great result. Flip a coin, do whatever you gotta do, but do both or which Linux channel should I watch? Should I watch yours? Watch them all. Literally just watch every one of them. Like that's gonna be my answer because that owning everything does no good to anyone. It just doesn't. I have another friend I'm helping out and he works at the high level of cybersecurity. But we were talking about it, like his first concept was, well, I looked for this content, I didn't find it. So I thought maybe I shouldn't produce it for YouTube. Come back a few years later, he's like, oh, I needed to produce it for YouTube. And look all the views it gets and things like that. So it's fun talking to people like that because sometimes people get nervous about whether or not they should post the content on there, but honestly, the amount of content there is on these deep technical topics, whether it's setting up storage servers or build automation or Ansible, once you build it and once one person starts watching it, many people start watching it, now people just expect that content on YouTube. You go back, YouTube didn't even expect this content on YouTube. This wasn't like, I mean, I really digs in history. They were a dating site, originally not even the place but now there's a place for learning. So overall it's just kind of cool to where it's growing and the more of us that participate in it, the more people that come to YouTube as a platform looking for that content. But there is one really huge problem I need to address that really, we need to do something about this because it's really bothering me. So Tim has a hundred thousand views and his like editing skills and video quality is far better than mine was at that same time. Oh yeah, yeah. How the heck does he have just a hundred thousand views? We need to get that pushed up higher. It's not good enough. A hundred thousand subscribers, not enough. Yeah, like if you're not subscribed to him, do it. We need to push that up there because if he's this good at a hundred thousand views, like when he gets to where I'm at now, I mean, oh my God, just absolutely subscribe to his channel. Let's get those numbers up there because they're way too low for what he's doing. Thank you. You're too generous. And you know, I say the same thing about YouTubers that maybe people haven't heard about too, you know, or Radow or Christian digital life. I mean, they're coming out swinging on video zero, you know, and I'm like, wow, these channels have, you know, great content, very focused and have great editing skills, you know, on, you know, and their first year. I'm like, yeah, I think the same thing too. But yeah, I do appreciate it. There are so many channels out there that, you know, that I think deserve more subscribers than they have. So I always encourage everyone to subscribe to every channel that you're even mildly interested in whether it's mine or anyone's. If you looked at a video lunch, just hit subscribe. That's a good signal to YouTube to say, hey, this channel is interesting and I'm going to recommend it to more people. So if you got anything, even one tip out of any video, just hit subscribe, hit thumbs up, hit subscribe, leave a comment while you're at it too because all that helps the algorithm, those are just signals to YouTube to say, hey, algorithm, this person liked this thing, you know, and we'll recommend it to people who also like this thing. So I always encourage people to subscribe. I mean, it's like clicking thumbs up on a post. Just do it if you like it. Just do it and you're not going to regret it. I mean, your content's great. I watched the video, I believe it was yesterday when I'm mistaken and I was really impressed by it. Everything was spot on, camera, I liked lighting, the editing, the narrative was great. Yeah, it's just good stuff. So your YouTube career is going to be pretty damn good. I'm just going to say that right now. I appreciate it and I appreciate it. I mean, just a little meta about editing that video, it was pretty awesome and again, surreal, like that everyone was like, yeah, sure, I'm on board, hand me your video and just say, go with it. And then I had to figure out how to tie the story together and it took a while. That was the first time where I felt like I was a newscaster or something, reporting on something, it was kind of, it was just a different take and it was really fun. So I hope it turned out okay and you guys are happy with it. Oh yeah, it was great. All right, so go subscribe to Techno Tim's channel, easy enough to find. I'll have links in the description of the video by the time this is the live stream is over, I always got to go back and update the links. It'll be in the podcast. Thank you very much Techno Tim for joining us and looking forward to you joining us again in the future. More episodes. Thanks so much for having me, I really appreciate it. Go get started on your home lab challenge, your 100 days of home lab. So that's right. All right. Back to work first, but yeah, right after that. For sure, thanks. See you.