 It shows up over here. I don't know if I, I don't think I got to click anything. If you set the time, I think it automatically starts, but it never starts on the time I've noticed. Yes, it did start on time. Perfect. It says we're live. First time we're using StreamYard, so excuse our rough intro here. So this is, yes, we love new tools, even if we don't know how to use them, but we'll figure out how to do it live because that's just more fun. Part of the live stream experience, if we mess up, you get to hear it, if you're watching, you get to see it in real time. All right, this is the Home Lab show, episode seven, containerization. This is Tom Lawrence and we have special guest, Alex. Go ahead and say your full name, Alex, because I don't want to mess it up. Hey, it's Alex Kretschmann. Perfect, and then we have... Jay LaCroix. Yes, and the containerization topic, we figured, well, it's an in-depth topic. It needs someone with a lot of knowledge on it, and Alex happens to be pretty much an expert on this. He's been working with containers for a little while and I'm excited about the show. I probably won't say a lot because I'm just going to be absorbing the knowledge with the audience here. And I think the same is true of me to some extent. So I think we should have Alex introduce himself. So just tell the audience about yourself, what you're into, and any places you like to send people links or whatever you want everyone to know. Well, hello. Yes, I am Alex, as you know. I am the co-host of the self-hosted podcast, self-hosted.show, and I'm joining these two fine gentlemen on this here show, which is pretty cool. Thanks for having me. Our pleasure. So, yeah, I mean, I've been involved with containers now for a while, 2014, I think. I was involved in some of the earlier stuff with getting Docker into Unrayed, founded Linux server.io, around that sort of time as well, or co-founded, I should say. What else? Yeah, I mean, I think those are the pertinent things for you all to know. And we have a good show lined up today. We've got a lot of stuff to talk about, what is a container and some of the more basic stuff, but also moving into Kubernetes and whether that's right for people at home and self-hosters and homelabbers and that kind of stuff. So, yeah, it should be a good show. It's going to be an awesome show. So, yeah, it's kind of hard to keep the secret about Alex being on the show because I'm like really excited about it. I've been following self-hosted. I think I might have listened to every episode. I'm pretty sure it's possible I missed one. That's the beauty of this space. There's always someone, I mean, no one knows everything. And I think knowledge overlaps in some ways. I know a lot about containers. I have a video coming. I'm not going to talk about that right now, but Alex knows it better. And when someone knows it better, I love to learn from someone who knows it better. And it's not a competition or anything like that. But what it is is that in this space, we trade information. I might know X, Alex might know Y and we get together and we learn something. I think it's healthchecks.io, Alex, you were mentioning on self-hosted a couple of times. Isn't that the one where you have a string on a Cron job and it lets you know if it has it running a long time? Pretty sure that's it. Yeah, that was one of the recent episodes. I was doing some stuff with ZFS snapshots. And I wanted that to happen every hour as part of a Sanoid Syncoid type thing using Jim Salter's tool. And the thing about Cron jobs is they just sort of fail silently in the background and you have to go and check or create some kind of elaborate monitoring system to ensure that these things are actually working. And healthchecks.io is a super simple way of hosting a small service that basically listens for you to hit a certain URL and you can do that with Curl. So when you run the Cron job, you have your script and then in the final line of the script, you just put Curl and then the URL for healthchecks with a UUID in it, it receives that ping and then you can set a bunch of rules to alert based on whether you get a ping within a certain timeframe and stuff like that. So if I, for example, don't get a ping within three days for my ZFS backup, I will get a notification to my phone to say, hey, dummy, go fix your backups. Yeah, and I learned that from Alex by listening to the self-hosted and now that's a tool in my tool chest. So that's, yeah, and that's a brilliant solution and I love it. So we wanna talk about containerization today and I am going to let Alex talk because I definitely wanna hear what he has to say about it. I have some thoughts as well. I think Tom is just going to silently nod his head in the back while we talk. Well, I know what a container is. Yeah, I know what a container is, yeah. That's the part of the show notes I understand. Well, in its simplest form, a container is just a process isolated in RAM by the kernel. So a lot of you have VM knowledge, right? In that domain, you have one kernel that is isolated to that virtual machine. You spin up another virtual machine, you get a brand new kernel and the kernel is the bit, if you're not familiar, the kernel is the bit that sits between the hardware and the software and does all the translation to, when you send a request to send a network packet across the network in software, the kernel is the bit that translates that into something that comes out of your ethernet jack. And essentially what happens with containers is rather than creating one kernel per virtual machine, you have one kernel that is shared across all the containers on that system. And the way in which the kernel controls the Linux kernel, of course, the one true kernel, the way in which that works is using something called namespaces and C groups, which is a very low level kernel technology. And effectively each process, let's say Nginx as a web server, for example, each Nginx server has its own view of the world and it has no idea what else is going on on the rest of the system, even though they're sharing the same kernel. So that kind of sandboxing, that kind of containerization of processes is pretty awesome. Now, that kind of leads you into wondering, well, okay, I have my process in this super secure sandbox and it can't see anything else that's going on. How do I allow that container to map to a certain port or something so it can actually listen for web traffic? Well, you can allow basically like a firewall, you can create certain rules and there are lots of different rules. There are rules for bind mounting data with volumes, there are rules for ports and networking, there are rules for Sysctl parameters, all that kind of stuff. And so you can allow these different parameters through the containerization firewall for want of a better word and allow these different processes that are actually isolated to talk to one another. And that's how when you have more than one container running at once, you could have five different Nginx web servers all running on the same system, all listening on port 80 inside their containers and still you're able to run those services without them clashing with each other. I think it's interesting too, it works in my head I should say very similar to the way NAT works with a firewall. You have a lot of devices behind the firewall and you have a NAT translation. So I want to open up port 80. It doesn't matter what the server runs behind there. So I think once you start wrapping your head around it that way and then having worked with TrueNAS quite a bit with their IOCage which is a containerization system in the other kernel, the less than TrueCurnal, the FreeBSD and the same concept. I think that's where sometimes people get a little bit confused is how you like map storage to it or map the networking to it. But sharing the kernel space ultimately and kind of the bigger reason for running containers is there's a lot of efficiency there because you're not reproducing everything as you do when you do VM. So for every virtual machine they all have their own kernel, their own networking stack, their own resources individually. So I think there's a lot of advantage to being able to do this from an efficiency standpoint. This is probably much why kernel, why containerization became popular. We can pack more into a singular machine. I love that analogy, you know, containers are NAT for the kernel. I kind of like that. Yeah, it's at least what works in my head because my background is much heavier from the network engineering side. So once I started grasping the concept of how containers work, I'm like, oh, I see why the ports are different here because I noticed that's the common question of, hold on, the containers running on port 8080 but I map it to port 80 because I want it to be a web server or 443 and 40443 for there. But once I think you start putting it around, you're like, ah, it makes a little bit of sense. We have a question in the chat from DV movies 1999 about, is it possible to restrict open ports to a certain IP address? Yes, you can listen on certain interfaces very easily, just specify that at runtime and yes, very easily done. Yeah, you can, because the full networking stack is in there, all the same rules apply with the firewall rules and everything else. So you still have the permissions and everything else that you can do. I see Jay's reappeared. Excellent. Yeah, the whole PC just froze, everything. Uh-oh. Force it off, all that fun stuff. So yay. We also have another comment in the chat from Grayson saying that he really likes Proxmox and you can run containers on Proxmox because it's a Linux hypervisor based around Debian. And Proxmox after the box supports what are called LXC containers, which are Linux containers. And they're a very similar technology, but slightly different. And the primary difference is that you can run system D or other init systems inside these LXC containers, whereas something like a Docker container or a kernel container for one of a better word is that's not really possible. When PID 1 dies in a Docker container, the container goes with it. And I think this is probably a good time to mention that Docker is kind of like the Kleenex of containers, right? It's just a brand name. Yeah. It's not just a thing, it's the brand recognition. Everyone just assumes Docker was that Docker is the first company, I think, to really commercialize it and create a whole repository which we'll get on that topic a little bit further down and whether or not that's the best place to download random five million images they have on there and whether or not they're all secure, which spoiler alerts are not. Yeah, I think that's so true because even I fall into that same problem where I'll hear someone say or I'll say myself, I'm running this in a container. I didn't even ask or I didn't even say what kind of container it is. Everyone assumes, and even I assume when I hear it that they're talking about a Docker container, which may not be the case. Most likely is, but the fact is we have other container types. It's probably a good thing to talk about. We have, we mentioned LXC, what are a couple of our container types? Jay, you like LXD, don't you? I do, and that's one of my favorite technologies actually. And I'm kind of surprised that it's not more popular than it is. I think it has decent popularity. Canonical is like really pushing it. And my information might be a little bit out of date, but my understanding is it's LXC with some added stuff on top of that that Canonical has added on there. Canonical is not the only one that develops on it. I think they're the mean steward if I'm not mistaken, I don't know if that's still the case. But there's cluster options there. For example, you have failover between servers. I'm not saying you can't do that with LXC, but it's made easier there. They have ZFS or ZFS, how am I supposed to say that? Depending on what country you're from, but they have some integration there with ZFS so you can leverage that. There's just a lot of extra options on top of it. Now with Proxmox, like Alex mentioned, it's LXC. And when I was talking to people at Canonical, they want me to say Lexi and LexD, but I just can't make myself do that for whatever reason, tomato, tomato. But with LXD, you get some additional layers on top of that. And with Proxmox using LXC, I'm like, why aren't they using LXD? Probably because they're based on Debian and they wanna try to keep everything within the Debian realm. But sometimes I've wondered why Proxmox has made that choice. Not that it's a wrong choice. I'm just kind of interested in why they made it. But Proxmox aside, LXD is a great solution, in my opinion. Before I moved to Proxmox, I was actually using LXD for everything, virtualization related. And yes, it's not a virtual machine technology, but it served essentially that purpose. I had a server, a PowerEdge T30. And for some reason on that server, which is an entry level server, it's pretty cheap. I got it for 300 US dollars. Memory upgrades are not cheap. I mean, a memory upgrade for that server can cost more than the server. So I had like eight gigs of RAM. And I had to make that, you know, that RAM stretch pretty far. So containers were a logical choice for me until I got a dedicated Proxmox solution. But at the time, I really enjoyed LXD. And it's something I might even cover in tutorials if enough people want me to do it. It's a great point as well that we didn't touch on yet is the efficiency of containers. Because you are only running one kernel and not a dozen, one per app or a few per app, you're able to make your resources go a lot further. And that is such a huge benefit for older systems. More low power systems, like Raspberry Pi and stuff like that, for example, that just don't have the horsepower to do multiple VMs all at once. You're not emulating a network stack, a disk controller stack. All this stuff that these hypervisors have to emulate when they create a virtual machine. We've gotten pretty good at making those things quite efficient these days, but there's nothing more efficient than not emulating them at all. I agree. A lot of people nowadays, the Mac crowd, with the M1 processor, my understanding is that it does support virtualization, but the technologies around it are still kind of new. So some companies are going to be further ahead with that than others. But Docker runs on it. I'm not sure if it's out of preview yet, but when I was messing with it, that worked pretty well. You have less RAM with the newer MacBooks. I'm not going to make this the MacBook podcast or anything like that, but the point is some platforms have varying levels of compatibility with virtualization. You could have an old desktop tower that does not support virtualization at all, but you could run containers on it. Maybe you have a machine, and I had someone write in on my forums at community.learnlinux.tv asking me, I have this old Acer laptop has two gigs of RAM and a Core 2 Duo. I don't think I could do much of this, but yeah, you can run containers on it because I think two gigs of RAM will get you a heck of a lot further with containers than it would with a virtual machine. I mean, you could probably run one virtual machine at that point because you need enough for the OS. So like Alex was saying, I think the flexibility there is definitely something to keep in mind. Also, the spin-up time as well. I know a virtual machine booting only takes 10 seconds, but a container takes literally as long as it takes for the process to initialize seconds. Yeah, it's like an instantaneous turning it on. I mean, it could take 10 minutes to boot up a virtual machine on a Core 2 Duo. I mean, for all I know, with a spinning-rust hard drive, a container would beat the pants off of that any day of the week, so there's definitely a lot of value there. Then the other thing to consider, sorry, we talked about Docker. Red Hat, full disclosure, who I work for, has a replacement tool that operates in the same space but is architected entirely differently called Podman. And Podman is designed to run on top of container D and I think Run C in a demon-less fashion. So Docker suffers from being the first to market five, six, seven years ago, something like that. And when they released the original Docker, it was this huge monolith that contained everything that was needed to run a container except for the kernel part in this big, you know, monolithic blob. And over time, they've kind of spun out a little bit here and a little bit there into separate components. But what Podman and container D and Run C have done is they've basically created an open foundation on top of the open container initiative where you don't need a demon running at all. And the advantage of that is that you don't need a demon running as root on your system so that if somebody was able to compromise one of these containers, you've got just one less attack vector with some kind of root-privileged demon running on your system. Yeah, and that's an... I was really hoping we would cover Podman because it's definitely on my radar of things to check out at some point, which I hope to do soon. I haven't actually used it yet, but I've heard some good things, but then again, three people in my circle, now four with Alex worked for Red Hat. So of course I'm going to hear about Podman. I mean, there are some things, you know, from a self-hosting perspective that I rely on that Podman still isn't very good at, Docker Compose being one of them. But I do think, you know, you look at the trajectory of where and the kind of business focus behind containerization. Podman isn't really aimed at people like me doing self-hosting at home. It's more aimed at people running Kubernetes and doing, you know, builds for OpenShift and other Kubernetes distributions. And when you're in that space, the decisions that they make to prioritize certain features make a lot more sense. I can give you an example of Docker that might be of value to some listeners that was actually a use case in my professional career. I worked at a company, Long Story Made Short, they developed a GPS solution, so they developed the Linux distribution and the software all the way up. It was pretty impressive. And they're all Linux fans like me, so we got along pretty well. But one thing we didn't agree on is which distribution to run. Three or four people on Debian, some people preferred Ubuntu, and then we had two people that thought Gen2 was the greatest thing on Earth. Maybe it is, I don't know. But the management kind of didn't like this because it was tested on Debian and now someone on Ubuntu comes with, oh, I have a problem when I compile this. The general thing was use Debian, but not everyone wanted to. So Long Story Made Short, I created a container, a Docker container. It wasn't public, it was just internal. So the libraries built in that the developers used to develop their applications day by day. So at that point, it no longer mattered what distribution or OS the developer was using. They pulled down this curated Docker container that has all the libraries built in and then they compiled their app against it. That kind of defeats the run one thing mentality of a container because it had a bunch of things in there, but it was internal, but it became this portable development tool. So we basically were able to abstract the operating system, which is another reason why it matters because you could have Windows users, Mac users, Linux users, and you have this container that's supposed to be the same. There's edge cases, but for the most part, it's the same. You could pass it from developer to developer. They could work on it and they can, for the most part, have the same experience, which we could do as well. So talking about building containers, that's where I really got, you know, cut my teeth in the enterprise world was automating or improving the release process of a monolithic Java app for a bank in London. And my remit was to take that huge app and split it up into small, you know, microservices was the buzzword back then. And that kind of leads you on to, you know, building images and stuff like that. And you mentioned, Jay, that you built some internal images and stuff like that. And we kind of alluded to this earlier, but how do we trust the stuff on Docker Hub is any good? Yeah, that's a really big concern. I worked with a company that I'll, I think I might have mentioned this in an earlier episode, and I'll leave the name of the company out because I'm not trying to throw shade on anyone. But as the kind of company that kind of feel like is going to be on the news because they're going to do something boneheaded and get on the news like Equifax style. And I didn't want to be a part of that. So I had to resign. And the problem was that they just trusted every container, just get a container, just get a container. I mean, it's not that that's an invalid thing because obviously it's a great technology, but we have to kind of audit these things because I mean, everything is secure until it's not. And you have containers that are in a registry that, you know, for five years or however long, they've been fine. And then someone just sneaks something in there. I'm kind of surprised that we haven't had like a bunch of issues with this yet. So we have the tech press and they love to trot out these statistics that there are 60,000 images on Docker Hub that are insecure and stuff like that, which may well be true. However, may well be true. Right. So I guess what I'm saying is it's only a matter of time to where we have this huge company. I don't know what company is going to be. And we're going to hear in the news that they pulled down this container to solve a problem, which is a legitimate use case for it. They did not audit it. And then it has a back door. Next thing you know, some company or some, I mean, not company, but some hacker mystery and just exfiltrates all of the company's data and throws it on the dark web. It's only a matter of time before we have something on that level. Unfortunately, I hope I'm wrong, but we all know how these things go. They all play out the same way. It's like watching the same movie over and over again, expecting a different ending. But the truth, I'm sorry. Let's say I had covered kind of as well. One of the popular things recently already happened here in 2021 was the number of Docker containers that were compromised with crypto miners. They were installed back doors from defunct projects. And it's really hard. Docker wanted to create this really cool resource, almost like application hub like we have for our phones and things like that. But it's faced the same problem. Projects go defunct. The developer sells it to someone else or sells it to a thread actor who then goes, all right, I have all these and everyone loves this project. Let me go ahead and update it with a crypto miner back door that I can slide in and send all these magic coins using your resources. So my solution to this problem, I'm not sure if Alex is going to agree that this is a good solution. I mean, it's not that it's a bad solution. It's just, it might be more tedious than it should be. So I personally just get a Docker file and I just build it. I grab Ubuntu. So I mean, you do have to start from something. You have a container based on Ubuntu, Alpine, Debian, whatever. So you have that starting point and you could probably trust the Ubuntu container, at least I hope. But then I build everything on top of that. Docker containers have layers. So obviously every command you run in the Docker file is going to add a new layer. So you could kind of go between the layers. And then I end up with a container that I have built. I know it's in it. I maintain it. But the problem though, one is that if life gets in the way and I don't update that container for a long time, then well, there's vulnerabilities now that aren't being patched. So whoever's building it has a responsibility to keep it up to date, even if it's you that's building it. But I like to have that control. Now it could be argued there's quicker ways of building an app for sure. But I like the whole Docker file concept. I know there's Docker composed and others. And we should probably talk about the difference between those later. But that's one possibility is just build it yourself. Would you agree with that, Alex? Or do you think I'm, you know, that's just too tedious? It's a lot of effort, but at least that way you can audit what goes into the container. But it's, you know, it's an age old question. How do you trust where your packages come from? And Linux distros have had that sorted for quite a while. And you know, there are still attack vectors in that pipeline. We've seen, I think Debbie and a couple of others over the last few years have packages into their repos that were compromised. So you've got to be vigilant at all levels. The other thing about a Docker file is it could be running more than just apt install blah. It could be doing an MPM could be doing who knows what else building who knows what else. You've still got to audit the Docker file and make sure that the scripts it's running are fine. Ultimately, I am a pragmatist at heart. And while self building everything is the only way to make sure that it does what I think it does. Do I care enough to bother doing that? Probably not. Websites like Linux server.io exist. And they are, you know, a trusted source in my opinion. Obviously, I was involved with them for quite a long time. And the reason that I trust them is because they have a fully open source build pipeline at ci.linuxserver.io. So if you want to go and verify the images they're building are the images that you're actually pulling down and running, you can go ahead and do that. Also, all their source code is open as well. So you can go and if you want to just pull down their Docker file and build that locally instead. There are also this, there is this concept of a trusted or a certified image on Docker Hub. So I mentioned engine X earlier. That's a good example. They have a root namespace dedicated for certified images. So normally you would need to do Docker, pull, ironic badger image as, you know, like the username slash repo name. But with these certified images, you don't need to put the username in. So just be Docker, pull engine X, for example. And that's a really good way to know that you're pulling a trusted image without even looking at it. But again, you know, if you're not even looking at it, that probably means you don't care about the answers to these questions. So I think it depends on context too, because, you know, in my case, I'll give you an example. I'm using Nagios. I know boring, right? But inside my house for my own purposes, it tells me if my Wi-Fi access points have dropped, if one of them is not responding, or one of my internal devices isn't responding, I could get a container for Nagios, I'm assuming, and not build it myself, run it from there. And do I really care if it gets hacked? Okay, that's not a great day, but I'm one person. This isn't like a company with like 400 people. So when you're talking about a home lab, then it's kind of like, okay, as long as it isn't within reach of your personally identifiable information, like your credit card information or something like that, the most I think you're going to do is piss off your ISP, because your ISP is going to say, I think there's a crypto miner. I'm seeing some traffic there. That's not a big deal. But if you work at a big company, and you graduate something that you develop in your home lab, and you now run it at your company, now that has a vulnerability, that's going to be a much bigger thing. Also, I don't want to talk. I just noticed in the chat, sorry, Jay. Sparkly balls. If that's who I think it is. Hello to you. You're one of the original guys that wrote most of the Linux server containers. So long time no speak. Hi, little shout out to you. Yay. Celebrity. Okay. When we have a celebrity pop in there. I also wanted to make sure we don't gloss over linuxerver.io because it's an amazing resource. And I think we need to like take a moment to really understand how great of a resource this is and make sure everyone else knows too. Because when I first created, and the video comes out Friday, I'm not going to plug it yet, but it's a Raspberry Pi container video. I'll talk more about it later. But the point is linuxerver.io will have containers that will run on a Raspberry Pi. Now I have run into a situation where I pull a container and I try to run it on a Raspberry Pi. It might run. It might be fine. If they actually compiled it for ARM, they may not have and a lot of times they don't. So I did a video a long time ago about running containers on Raspberry Pi. And of course people will comment in, this container won't work. Why doesn't it work? Well, it was made for x86. That's why. But on linuxerver.io, I can't find a single container. I can't get running. They work and there's all kinds of apps there. A lot of the apps that you want to run in the home lab, like Smokepean, for example, there's a container for that. That was actually one of mine back in the early days. Funny story about that one. Oh, wow. I actually put my home IP address. This was when the next server had like 5,000 or 6,000 total pulls from Docker Hub. They're at 15 billion now, by the way, which is just... Wow. That's huge. So I put my IB's, Ironic Badger's residence as one of the end points in Smokeping. And I still get tens of thousands of pings a day from people running that Smokeping container. We need to maybe push something out to them, update your container. Oh, no, I've already done it. But it was the default config for so long, probably three or four years before I really noticed. That is hilarious. Yeah. So I mean, I guess since I've mentioned it a few times, just as a quick aside, I did a video that's going to show a 10-node Kubernetes cluster running on Raspberry Pi. It's going to hit this Friday on my YouTube channel, LearnLinuxTV. And I bring it up now because linuxserver.io is a part of that video. There's a whole section near the end that tells you, you know, now that I showed you how to build this Kubernetes cluster, what do you run on it? And then I go into linuxserver.io and first I have the nginx container, then the Smokeping container is the second example that I give in that video. So kudos to you as well as everyone else that had a hand. Well, back in the early days of Docker, it was even worse. The Docker Hub was even worse then than it is now because nobody really knew what was going on. So me and two or three other guys sort of took it upon ourselves, even though we didn't know what we were doing either, to try and write some kind of a standardized container and then build all of our images off this one base image. And there were a few things we did that other people didn't back then. Well, we've got loads of the linuxserver crew in the chat now. We've got Roxy, we've got Stark. That's awesome. Hey, guys. And, you know, standardized permissions between what was running inside the container versus what was running outside the container. This is the PUID-PGID stuff. That was come up with, I think, by Lonix, one of the original founders. He came up with that idea. And what else? There was regular updates. You know, that was a new concept back then. We started actually just building the images every week, even if they didn't need it. Since then, there's been a lot of tweaks to the CI pipeline and the guys over there now do a fantastic job at keeping things up to date, only when they need it, as well as, you know, security patches and stuff like that. Yeah, there was nothing really like it back then. And, you know, it kind of annoys me because it was such a good idea to monetize Linux server. But in the end, Open Source has really absolutely fundamentally changed my life and it pays my mortgage and everything else like that now. And so I see it really as a gift to the world. That kind of thing. Yeah. And one thing the Open Source mentality and community has taught me is that when you have a solution in the control of the people, the people will drive it. For example, you can have a company say, this product is no longer free. We're taking it back. We have the trademark. It's closed. And we're going to charge for this now. And then the Open Source community is like, oh, yeah, about that. We forked it. Oh, I forked it too. Oh, by the way, I forked it. Yeah, my brother forked it. We all forked it. We have like six different versions now. And you could choose the best one that you want. So now you get more choice. I think it was one of the audio apps. I can't remember which one it was. That went closed source. It was one of those where you could have your mp3 collection presented in your web browser. That's a, that was a great thing. And Sonic, I think. Sub-Sonic, Air-Sonic. Yes, back in the day. Yeah. That's been through a whole bunch of changes, isn't it? Yeah. We had like a whole bunch of Sonics come out. You know, I was, I almost wore my Sonic the Hedgehog T-Shirt. I wish I did. But anyway, we had all these different clones and forks. And then you have Open Office, which I believe is still open source, but you know, they rub their people the wrong way and now we have to leave our office. It takes over. So it's like the community around open source. The technology goes the path that it's meant to go. And it's like nothing can hold it back. And the technology can flourish and be what it is and evolve the way it should. So I think it's changed my life too. In different ways. It's a big part of my mortgage as well. But all things considered, it's like for me being, you know, heavily ADD, my hyper focus is Linux and Linux, you know, I'm able to make content and make money off of it. So it's a mutually beneficial thing. So moving things along a little bit. I have a good, I have a question, which I think we can bring Tom in a little bit more, because I'm aware of the talk. I'm fine. I'm absorbing a lot of this too. So you are a self-proclaimed VMS guy, right? Yes. A little more. I don't want to say old school, but you know what I mean when I say that, right? You're doing VM since VMs became a thing. So you can call me old. Okay. Old school. That's fine. It's not a derogatory term. It's just, you know, that's what you know. So when does it make sense, do you think, to use a container versus a VM? Kind of what Ajay alluded to is if you're running older hardware for a home lab, definitely it's something to lean towards because you don't have the resources. Not everyone has. I have a really nice stack of hardware. Most of our stuff actually runs in individual VMs, partly because I am confident and secure in how I can lock each one down to prevent any type of potential lateral movement between them. So from an utmost security standpoint, each one of the things we have running as an individual VM with its own network stack with its own set of rules feels very self-contained and restarting or working on any of those doesn't feel like it'll break or jostle, so to speak, all the other things that run in our stack. We self-host some of the tools we use for our business like our invoicing and things like that. I like knowing and confident that it's on a single spot. I know it probably could be done in a container and probably could be done efficiently and securely, but I lean a little bit towards that individualness of each system. And by the way, not all of them even run the same kernels are on different versions for different support reasons. I know this is sometimes having a little diversity in there that can be a little bit more of a challenge in the container. So if you have certain dependencies that have a different kernel requirement that may break your container need or even in the case of running a Windows, we have some Windows VMs in the stack. We have to virtualize them. We can't just run a Windows VM inside of a Linux container properly. Right. I think it's important to understand too that some applications, at least in my experience, they don't run in a container. This is rare, though. I think the majority of applications will. But a company I worked with in the past that I again won't mention their name, they were containers or bust. It had to be a container. Like under no circumstances could it be anything else. So I had an application and it's an open source app. I can't remember the name of it, but it was written in a very horrible way. Everything was like an absolute path, absolute path and trying to run that in a container it broken so many glorious ways. So I was forced to run like engine X sub filter calls to change the communications and URLs in real time to force this thing to work in a container. It took two months. I was able to do it. It was a very, very hard thing to do. It would have been easier to run that in a container. Another thing to keep in mind is some vendors won't support you if you're running it in a container because they themselves don't support containers. Now of course they should get with the program. Let's be honest. And I'm sure this doesn't really matter to most of our listeners because this isn't like enterprise IT podcast but it is something to keep in mind though. Atlassian used to be like this. I've worked with them a lot. They had a container for JIRA confluence and things like that but they wouldn't support you if you were using it and it took them a long time over a year to certify that so you could call and get support. So if it's something critical you have to understand is it meant to run in the container? It probably will. If you're a home lab you probably don't care about anything that I just said. And that's the nature of the podcast obviously but the reason why I bring it up is because a lot of times what we start in the home lab we kind of present to the employer as hey look what I did. I think we should use this. 100% yeah absolutely. So I think it's important to keep in mind that some apps won't run in the container maybe because of the developer not doing things quite the appropriate way or maybe the vendor doesn't support it because they have stigma against containers which is unfounded but some people do. So my opinion is if you could run it in the container that's great because it's going to use a smaller footprint. If it's a virtual machine it's going to have a larger footprint. But there are things like monitoring demons and things like that. The Telegraph is a great example that I used to collect system metrics on Linux boxes. I don't run that in a container because I have to mount everything on the system into that container and at that point why not just run it on the freaking host? I generally default to running stuff in a container unless there's a good reason like that. So I have about 30 containers running on my home lab. I know people with 90 plus. I don't know what all those self-hosted apps are but there you go people do run those things. I think one of the other reasons and the opposite happens is too to what Jay said Bitwarden is a good example and so is discourse that I run my forums on. Both of them provide you the container to run it. They do not. I mean they give you some vague like here's all the code if you want to try to figure out how to assemble yourself or just get our Docker container that we have built and just go ahead and pull it. Bitwarden does a nice job of this. They've properly set it up. So all the data is where it should be and it just replaces each module, the series of containers. I should say that Bitwarden runs on but that's actually their support model and you buy it from them. That's how they maintain an update on-prem solution is they have a tool that automatically grabs any modules that need to be replaced and goes ahead and replaces them. That's easier. I mean you technically could go through all the source of Docker and build it yourself but then you'd be building yourself every time there's an update and that would be tedious. So some of these other as you say companies that have gotten with the program Jay understand this is a better pipeline for delivery for managing it. So I don't even, I've updated Bitwarden in the last year quite a few times, all the different vary changes. Every one of them has gone perfectly smooth. The same goes for discourse. My forum software over the last couple of years has been plenty of updates. They actually have a whole web tool that updates and the web tool just does the back end pulling of all the Docker images. They give you a command line way to do it of course too but being able to pull it all through a web interface and relatively quickly get the latest versions right from them already tuned and ready to go and dropped in are great. It's a great methodology. I think it is. I think you hit the nail on the head because I mean I guess I don't fully understand why this is the case that developers will build an RPM not just one but like a Fedora RPM 1% to us, Amazon Linux, OpenSUSA, whatever else and a dev for Ubuntu, a dev for Debian and it's like do they love their mental health? Is that important? People talk about packaging formats all the time and for me containers are the universal packaging format. Exactly. I don't care what base image you're running it on just give me a standardized interface and that's what a container gives me. Exactly. What about databases? I run quite a few databases in containers. That's probably going to be blasphemy to some people but for me it works really well. Yeah that's actually something I'm going to be looking into here very shortly for various projects that I'm working on because this is something I'm actually researching because now that I have the video out the door it will be out the door. I can use my new Kubernetes cluster for whatever I want. When I'm making a video I'm going to tear it down build it up, tear it down, build it up. Now that the video is out I can run my stuff on there I can run a bunch of containers. I'm going to try smoke ping a number of others but the question is where do I load the database? I could load it on another Raspberry Pi. I could have a database server as a VM as a container. I have all these different options that are available to me now and it's something I'll be looking into. What are your thoughts Alex on that? At a starting point where you need a database server and you already have containers would containers be the way to go for you? Would you just leverage your Proxmox server? Well the bank in London had this ridiculous Oracle database setup that was just nuts frankly and that was something we investigated very heavily on containerizing and eventually just decided we valued our jobs too much to even touch it. So we didn't go there. But for self-hosters and homelabbers I have kind of pivoted on databases in the last year or two. I used to have one MySQL, one Postgres, just one version of each database technology running on my server and I had those two or three technologies. Recently though I've kind of moved my thought process to doing containers as kind of logical groupings or pairings. So let's say you have a container next cloud I think I run the MySQL backend is a great example I run the next cloud app, the front end and then I also run a MySQL database dedicated container just for next cloud and then the next app that needs MySQL I spin up another MySQL alongside it and I give each of these databases their own volumes and mount them with Docker and it makes the backing up of these things super simple I never have to go in and do MySQL commands on the command line and add users and add tables and drop this nonsense and I hate MySQL so much. Whereas when I was trying to do one monolithic container effectively running dozens of databases with inside one database container it got really confusing for me to administer which the whole point of doing them in containers was supposed to be to make it easier. And so stacks I think sparkly saying is the word in compose speak. Yes, Orange, I am giving in to peer pressure. Yes. So if you said you have, you know, you really don't like MySQL have you tried someone else's SQL? Okay, sorry. Oh my goodness. I thought Tom was looking at the dad jokes. We can all take turns at that. So here's a question for you, Alex. So when I'm looking at Kubernetes obviously you have a pod which is what a container runs inside of and you run containers on Kubernetes. So, you know, I'm thinking, yeah, I could have a database server in that same pod, you know, that's fine. But on your end, are you running Kubernetes? It sounds to me like you're just running straight up Docker. Is that true? Well, for my, this disillusioned to something I just said actually in terms of a phrase I think this might be a very British phrase but a busman's holiday. I don't know if that translates. I do Kubernetes all day every day for my day job and the last thing I want to do at night is come home and debug, I say come home. I'm working from home all the time. You know what I mean? And debug Kubernetes. I mean, there is an argument to be had that the stuff you do in your home lab translates into what you do at work and clients and stuff like that. But Kubernetes is just so complicated. It's such a big hammer to learn and administer and fix that, you know, when Plex is down at, you know, 10 p.m. All I want to do is watch a movie or a TV show at the end of the day. I don't want to have to go in and be worrying about YAML manifests and this, then that and the other stuff that could be happening. And so I just run a single host. It's actually my Proxmox host where I have those 20 or 30 containers on there. All running on that single host and, you know, in terms of what the Linux kernel can handle, you know, you're looking at 250 to 500 containers before that thing starts to get resource constrained. So my 30 containers isn't even scratching the surface of what it can do. That's a good point because, you know, my understanding, I don't know if Proxmox has fixed this. You have a LXC container and you have, let's say, two or three Proxmox nodes and you want to take one down for maintenance so you want to move all the resources from one to the other. You know, that container is going to stop. It's going to start up again on the next host, but a VM can live migrate. So I'm assuming then you would be able to live migrate all of your containers in one shot with one VM from one Proxmox host to another. It's my home lab. I mean, if my Plex is down for five minutes. True. So what? Yeah. Well, I guess in my case, there's three people that'll be at my door. Like the Plex server isn't working. Okay. Go outside. Read a book. You know, there are alternative solutions to that problem. I think one thing about that too, because that is what I say, but I also have this other mindset too for the YouTube channel because I always mess with these technologies that I don't really need to use. I don't need a Kubernetes cluster. I really don't. I need a server rack. I mean, I kind of do. Kind of don't. We can argue that. But it's a use case for the channel. So I often, I'm in that frame of mind. I'll tell my kids, yes, go outside, but I'm thinking, yeah, it'd be nice if I could live migrate that actually now that I think about it. That's sad. I think Kubernetes is a fantastic piece of technology. It's just incredibly complex. It is. All power to you. If you want to run it at home in your home lab. Great. I actually have a dedicated box that I use for OpenShift testing where I run Ansible and Terraform and I'm spinning up clusters most days and tearing them down and it's fully automated. But I don't want to have to rely on that level of complexity for just running a few self-hosted containers. And the overhead also of running Kubernetes is pretty significant, relatively speaking. You need at least three VMs for your control plane to remain in Quorum. Ideally, you want those on different hardware and then you start... You're building a very enterprise grade, highly available, redundant system with lots of moving pieces that could all go wrong at any given moment. And unless you're engineering this thing with multiple switches that are highly available and multiple power supplies it's just overkill, isn't it, for a home lab? It is except for one use case. One use case to keep in mind too, there's a subset of home labbers, I think, that want to get certification. Maybe their employer wants them to learn Kubernetes. Learning is a totally valid use case. Yeah, so maybe setting it up at home, I mean, you could argue, yeah, you're not running your Plex server on there, the things that your spouse cares that they want running all the time. But you might be drinking and Kool-Aid all the same, but then, of course, I think we do get a subset of people that... You might ask, why are you running VMware? Well, that's what my employer uses, and I'm trying to get that certification. Oh, yeah, rock on, that's totally cool. So I think the use case depends on the person, but I will admit sometimes I overcomplicate things, big surprise, but speaking of complication, I think one thing, I'm sorry? We all do from time to time. That's part of what they call job security, isn't it? I think so, yeah. Well, I mean, security onion feels like a little heavy-handed, but I've been running security onion at home of containers and everything else, because it's the easiest way to deliver a really complex, very high-end open-source SIM tool, which is ridiculous for at home, but it's also a learning experience. It's also the, you know, I make videos with it and things like that, but it's... I've encouraged a lot of people to run it because it's a good way to get your hands on the cybersecurity side of the house, to really dive into the packets coming into and leaving your building or home. Start there, because the problem scales upward, trying to put that in a business. You have a business, you learn all this stuff. I mean, they think you're talking another language any time you talk about what you're working on anyway, so the boss asks, what are you doing? And you can say, well, I'm just adjusting the turbo-encabulator right now. Okay, you're very... Quantum carburetor. Yes. Hey, Morty, you can't just take a car word and have a sci-fi word and make it a real thing. For Kim. Someone listening, no, I'm kidding. One thing I think, Alex, I noticed at the beginning I kind of cut out here, so apologies if you've already talked about this, but when you're getting back to containers and taking away the Kubernetes thing off the table for a bit here, I use Docker files as just what I use. I know you use Docker Compose. I've heard you say that many times, but then there's also other ways of bringing up containers, Docker containers specifically. What is your opinion on Docker file versus Docker Compose and things like that? I think they're entirely different things. The Docker file is basically the recipe of here are the ingredients I'm using to bake this particular container into the end result, the cake, whatever. Whereas Compose, for me, I know a lot of people can use it and do use it in development circles to build entire stacks of images all at once and then deploy them and what have you. I just use it effectively as something to manage the images I pull down. I see the Docker Compose pull and it pulls the latest version of those images and then a Docker Compose up and then it brings up those containers. So I use it as pretty much like what Kubernetes is trying to do as a desired state engine. I use Compose pretty much like that. I say, here's the state I want for my 30 containers, I want them to have these volumes and these ports and what have you. Then I just throw it all in one massive Docker Compose file and run those commands and that's the difference for me. Yeah, you get it back up and running pretty quick and I don't feel like me but sometimes something's broken, it's not working, I don't have time to fix it, delete the stupid thing and re-run the script. I can do it from my phone as well. This is another argument against Kubernetes and it's a home-labbing, services you air quotes rely on. When Plex is down or something is broken, I can SSH in from juice SSH on my phone and a couple of terminal commands later I've tailed the log file, I've then figured out all these containers fallen over and restart that container and then I'm good to go again. A lot of people in the chat also mentioned and we were going to come on to this but I don't know if that's the way to say it. I was just about to ask you about that. Which is a really, really cool project and there's this other guy on YouTube called Network Tim or Tiny Tim or something I can't remember his name and he does a bunch of really great content around K3S but it's essentially just a really lightweight version of Kubernetes I think made by Rancher and they run really well that destroyer runs really well on a Raspberry Pi and if you want to just learn a lot the Kubernetes concepts, Techno Tim thanks Chris. If you want to just learn a lot the Kubernetes concepts as a more developer type focus instead of an infrastructure guy which is where my tendencies tend to lean towards K3S is a fantastic way to get used to Kubernetes objects and YAML and all that kind of stuff. Yeah, that's awesome. K3S I haven't checked it out myself but it's on my radar but I have heard a lot of good things about it. Future video, you had it here first. You heard it here first. I don't think people will take me seriously if I don't cover that at some point. Mixing Network Chuck and Techno Tim that is right, Master Tim. Yeah, absolutely right. Which one are you talking about? Maybe there is a YouTuber with both names for all I know. Now, a couple topics that we have here because we have a few we want to definitely get to one of which is performance on Raspberry Pi with containers and what you can reasonably expect out of a Raspberry Pi. Well, the Raspberry Pi kind of sucks, doesn't it? I mean it's also amazing. It's a fantastic considering what you pay for it. I mean the size of it and the envelope and all that. We've got another Linux server dude now. Hey Johnny, how are you doing? How are you doing? I think the whole cruise here. Yeah, I mean the Pi is great for what it costs and everything else. However, I'm going to caveat that pretty heavily in the I.O. Roxydesk, I already said hello to you. I'm going to say hi again. Hi. The I.O. is just horrendous on a Raspberry Pi. The SD card is just not fit for purpose. And then you think to yourself I'm a USB boot so I've got to do this kind of custom it's not custom anymore is it? Enabling USB boot on a Raspberry Pi is not trivial. I've tried to talk relatives through it over the phone for a Pi I bought in England to do as a remote backup system and it was it was difficult, you know. Not easy. But I would wager that the Pi isn't really well suited as a home server. I know what I just said about being great and all but I don't actually run any in production other than on my 3D printers. I have a couple running OctoPrint on there and it works perfectly for that. But for a home server a home lab type thing if you have anything that's I.O. that's where you draw the line is what I would say there there are good news is there are a few other single board computer companies that do offer SATA support so there are ways to get non Raspberry Pi so it's not all single board computers that suffer from this but you're right the Raspberry Pi for I.O. intensive operations is generally not well suited. I run most of my applications other than Plex on Raspberry Pi but there were some things that I had to do to kind of massage that a little bit now there's all kinds of ways they can go about this I.O. problem and I don't think any I think the problem is regardless of which thing you go with it's not going to solve it 100% you're just trying to get that percentage as high as you can. What's worked for me personally is I would get one of those USB 3 flash drives that have a very small footprint they look like a mouse dongle so you don't even really notice it in the back of the Pi and I just buy the fastest one that I can now not booting off of that I'm still booting off the SD cards I'm not trying to like you know go crazy with all these different you know changes but I'll look at the directories that I access a lot like if it's serving a web page I will put the HTML files on the faster storage I'll sim link something if I have to or just change the config path my home directory I'll put on there as well so if you look at my USB flash drive that I'll have in my Raspberry Pi it'll probably have four or five things even bar log goes on there too just to give you an idea so that's a lot to manage though it still doesn't change the fact that Alex mentioned which is you're going to have issues so that being said it's just all about do you want to deal with that or you know are you going to have to move on to a VM server or something like that you know my biggest problem with the Raspberry Pi though beyond the input output is the price of the damn things I mean I know I also said a few minutes ago that the price is the best thing about it if you load this thing up and you get like a four gig 8 gig Pi with a nice case and a nice power supply and everything you're going to be spending 100 maybe even 120 dollars on this system you can go on eBay and build a used x86 board system with a case and an SSD and decent you know input output that you don't have to do any of this futzing around with and also it's not an ARM CPU so everything just works for it yep for the same money why wouldn't you do that I mean I understand the appeal of the pies because I have two or three of them around here anyway but I it's not rational it's not logical like I do agree with you but as an aside for some people the power usage is orders of magnitude less than a pie it's not it isn't you think it is but if you actually measure it it's not true so I have an I3 3100 something I think in my basement running open sense right now idle that thing draws 8 watts this Raspberry Pi running this hacker display behind me 6 watts right now really wow that's interesting so I guess it's the same so I guess when you compare it to like a power edge server maybe that's not a fair comparison because you know that one's orders of magnitude more powerful than the other but power usage can sometimes play into it to an extent maybe not absolutely it can yeah I think it depends on what are you trying to achieve what's your budget what do you have available how much is a power usage around you how important is I owe to you are you impatient if you are yeah maybe not a Raspberry Pi if you don't really want to wait for something to load so there's all these different comparisons but I don't I totally agree depending on what you're running you could make that argument that other solutions would actually run faster for less power so it depends on the person but I think there's also the cult of personality with Raspberry Pi that it benefits from because they have the community factor and that is the thing that they are truly unbeatable for there are loads of other ARM based SBCs out there but they just don't have you know the heritage I guess the Raspberry Pi does in making an Octo Pi we have one too it runs our 3D printers I mean it's an Octo Pi you get this image you set it up with very little I've done videos on the Pi screen projects which are the Pi signage we've actually deployed them there they work incredibly well and they're so inexpensive that if the board burns out or something happens to it I just take the image loaded again and it becomes a display server again to show menus that's a great point it's all part of that community support we actually we have an e-ink display we're setting up on the other side of the wall behind me that's going to have right now it has a bunch of cryptocurrency on there but once again it's a Raspberry Pi project with an e-ink display with a 3D printed that we use the Octo Pi to send a 3D print job to so we're reproducing Pi's now in Pi projects that reminds me I had a company I used to work for throughout this perfectly usable laser jet HP printer with a brand new toner cartridge in there it was just a USB printer and it didn't support the versions of Windows that they were running so they couldn't install it so I just took it off their hands I basically set up a Raspberry Pi with Samba and basically cups on there as well to share the printer and you know what I had a wireless printer just like that I just attached a Raspberry Pi to this otherwise non-wireless printer that has no purpose anymore because Windows users can't use it anymore and now it's a Linux user every computer in the house can print to this thing and considering a brand new toner cartridge and I might print 10 pages a month I think that printer is going to die before that toner cartridge dies I think Meanwhile I hang my printer off my blue iris box in the closet over there There you go so sometimes yeah it depends on the good idea that's what it comes down to Now do we cover every aspect of containers I think we down all the way to the end of our show notes or anything else we have in the container side so we're wandering off to the Raspberry Pi we could probably do a whole episode about Raspberry Pi I think we can yeah I think the only other thing that we probably don't want to spend too much time on this unless you want to Alex but I don't think you'll probably want to OpenShift should we use that what do you think are you an enterprise are you an enterprise then no probably not there are some open source versions of OpenShift available upstream OKD is available if you want to do that I actually wrote a blog post for OpenShift.com on how I automate all of my OpenShift deployments on VMware using the user provisions installer that's part of that product I'm a huge proponent of infrastructure as code and Terraform and Ansible are a huge part of that and so for me I think we could probably do a whole episode on config management and automation and that kind of stuff I think you just revealed what we're going to do because if we weren't planning on doing it we certainly are now if you want that information we'll put a link in the show notes it's cool stuff it's not just limited to VMware you can also use these config management tools to deploy to Amazon or to Google Cloud or wherever you want to deploy your stuff, Lino, Digizlotion whatever so yeah I think we'll definitely do an episode on that moving forward but OpenShift for people at home it's too much unless you're choosing it as a career path that's probably the only exception to play with it but it's a lot of learning it's a lot of RAM you need a lot of RAM for that thing my home lab has 196 gigs of RAM because of OpenShift yeah that's going to be out of reach for a lot of people because a lot of people I find they're kind of just starting out with what they have available and then obviously they build up their home lab from there but I don't think it's common that someone has 196 gigs of RAM in a box just in a closet they're trying to find a use for unfortunately that's everybody isn't it I would hope so I hope we all have a bunch of servers we'd love to repurpose but it doesn't always work out that way as much as there is a good commodity hardware out there off eBay to build your home lab but yeah I totally agree so I think we had a great episode about containers was there anything else yeah I think we I got now we're at the very bottom of the notes and I feel comfortable we covered all the topics wrap that all up we'll make sure we wrap these up into the show notes that will be published at thehomelab.show someone made comments about audio this is the live stream we will do some post production and links will be on the homelab.show for this and all the previous episodes if you want to download them wherever good podcasts are consumed we've got it registered everywhere now so thank you all for joining us any final words guys not on my end alright see you guys we'll trying to do this weekly we're not promising weekly but we are doing our best see you guys next episode though regularly regularly regularly regularly is a term that's the right word thanks