 Welcome to the homelab show. Is this episode 32 already? That's amazing. Wow. Yeah, it is episode 32, and 32 is one of those numbers that SIT people we remember. 32, 64, 128, and so on. Oh, yeah. I think we're one of those. Yep. Absolutely. While we're on that topic, it's important that you patch your system, so we decided that we definitely need to have a talk about patching. I've seen some myths. I've seen some unusual things. People have said like, I don't patch my system. I'm afraid it'll break. Linux isn't Windows, and we'll spend 30 seconds in the beginning talking about Windows patch management and what a nightmare it is. But before we do that, we do want to get a little housekeeping all the way and thank a sponsor of this show, and that is Linode. Linode is literally, if you're downloading this, how you got this show, especially because a couple weeks ago, Jay increased the bandwidth on his server because so many people are downloading the show, so that's a great thing to hear that so many of you like it. And many of you signed up for Linode and seen the stats, but it's a great service if you haven't, if you're curious. I mean, it's kind of a no risk. We have an offer code to get you started with Linode. It's a great place to start some of the projects that we've talked about, and it's not just something we recommend, it is something we use. I want to thank Linode for being a sponsor of the show, and let's talk about keeping it patched. That way, if you put something Linode, it's patched. Yeah, absolutely. I think that's one of the first places to start is that some things are more important to patch than others. I mean, obviously you should patch everything, but if you have something behind a firewall that's not exposed to the public internet, it's not as important to patch it as it is something on a cloud provider, or maybe if you've exposed something to the internet, obviously those things are the most important. But one of the things we'll be talking about today, too, is that for human, unfortunately or fortunately, depending on how you look at it, and we often forget things because we have things going on in our lives, and we can't always remember to do the thing, and then sometimes we're reminded we should have done the thing when everything goes downhill. So it's important to keep up. Starting to keep up with it. Now, I said I'd briefly talk about Windows patching. I think listening, because I was able to catch up on the latest Security Now podcast, and I tweeted the other day that Microsoft has done more to get people thinking about paperless offices than any of those little emails that say, hey, don't print this paper because they can't get patching right. So we're not gonna do all much on Windows patching. Basically turn it on, yes, do it, brace for impact every time. It's the thing you have to do that you don't wanna do that'll probably break something and probably has become the mantra for 2021 when it comes to Microsoft patching. There's nothing good about Microsoft patching. I mentioned Steve gets in Security Now, he's ranted about it, bleeping computers, got a collection of articles on how hard it is to keep printers up and running on it. And it's also one of the challenges that you face as IT. It's like Windows is just the beast you rest with. Nothing does amazing patch management, not Windows and not third party software is supposed to manage it for Windows. I see that as an IT professional who's been using and managing Windows service for over 20 years. So it is downhill. But that's the reason the enterprise world runs on things like Linux and BSD. We're gonna keep it a little bit more. Linux focused here, but it just patches better. It's a better system from the ground up than the way Microsoft currently handles their patching system. It's just, it's a lot less broken, but there's still some nuance to it. But I don't, what happens is, and the reason it's important to mention the Windows one is I think there's so much carryover where people have the same fear of things will break when I patch something in Linux because they have that same taste in their mouth from managing Windows servers. And like, oh man, every time I hit update on this, it's something else goes wrong, but that's definitely not the case. Windows is problematic, but Linux is pretty smooth in patching overall. I think so too. Yeah, I mean, there's some edge cases every now and then, but there is with everything. So I mean, there's probably nothing I could say that's ever 100%. But yeah, absolutely. I mean, Windows is kind of weird. It's just one of those things that hasn't really changed all that much. It seems like with Windows Vista, it became orders of magnitude worse. Coming off of Windows XP, Windows 7 was a little bit better, but it kept just being the same, which is kind of like, okay, we're in 2021, shouldn't we do things differently now? Then again, there's quirks in every operating system like that in Linux. As great as it is, has some pros and cons too that we'll be getting into. Yep, but the first thing is important that especially public facing things, and this is where I always have them on unattended updates. We make sure that is completely constantly patched when something is public facing, because well, with very few exceptions, the Apache one was actually interesting because this is a good thing. They found that when there was a recent Apache vulnerability found that people patched really fast. So fast though, that new version of Apache actually had the vulnerability, but it didn't take long for people to find it, mitigate it and patch these subsequent patches. But of course, it took a couple of iterations, but because there's not a patch release cycle, you don't have to wait until next month to do it. These patches pretty much get pushed out as soon as they're available for you to apply for most of the major operating systems. Yep, yep, absolutely. And one of the first things to bring up, I guess is the elephant in the room, which is uptime. We Linux people like to brag at the uptime. You could run the uptime command on any Linux server, tells you how long it's been running since the previous time it was started. And someone might brag, oh, I've been up for two years straight. Well, great, but you've probably had updates sometime in there that you probably needed to reboot. We'll get into why you might need to reboot or not in a minute, but uptime is just one of those things that I think people are obsessed with and don't really understand why. I mean, yeah, it is kind of cool, but at the same time, does it really matter? I mean, if you have to reboot, if it's not going to, I mean, it's not like we're running a business in our house, so I guess I am here, aren't I? But not everyone is doing that. So a little bit of downtime and rebooting isn't really that big of a deal, but going back to your point, yeah, I mean, if it's public-facing, you definitely have to do whatever you have to do to keep it patched. Still no guarantee that it's going to be hack-proof. Actually, there's no such thing as that, but I think the minimum that you could do is keep everything patched. Yeah, and for the most part with Linux, you don't deal with as many rebooting issues. The only time you reboot is the kernel pretty much. I don't think there's really any other thing that's going to cause you to reboot. So it's only when there's new kernel availability and that's not something that's happening on a daily basis versus the patch is a great example because it runs a lot of web servers. So as the Apache one rolled out, you'd run the update and then you just quickly restart the service. Like it would grab, hey, look, here's the new version of Apache now that we're restarted the service. And that is almost unnoticeable to the users. A quick restart of Apache maybe breaks a couple sessions. Generally provided you don't have an ancient system, it takes a long time to restart the service, you have a very minimal amount of disruption when you do that. And this applies to actually most services in Linux. Yeah, exactly. So I think one of the biggest differences between Windows and Linux is that Windows is going to keep bothering you over and over and over again until you give it a restart. That's it. I mean, it's just going to keep bothering you about it. And some versions will try to just pick a time by default to reboot, you could change it but they really want you to reboot. Now, when you update a Linux system, you can very well get a message that says you should reboot your machine but you are free to ignore that. You're free just to not do that. So you can just say, no, thank you, I don't want to reboot because maybe as the Linux administrator for your own systems, you understand when you actually have to reboot because sometimes with a Linux distribution, it's more of a recommendation but if you know exactly what was updated then you can make a more conscious decision as far as whether a reboot is actually necessary before you just go and do it. So if Apache like you were saying is one of the things that was updated, probably not. Could you see a recommendation? Maybe you probably won't see that either but if you do, you could say, no, thanks. I already know that was just Apache. I'm going to restart Apache and we're fine. So you can make that decision up and by yourself as far as whether or not you want to honor that request otherwise Windows will just send someone to your door knocking on your door. You need to restart your computer or whatever they end up doing. Yeah, it certainly pesters you a lot more. It's usually just a suggestion on there and kind of left up to you. Right. So one of the things that I think we should probably get out of the way is what types of situations may or may not require or maybe recommend a reboot. Now you mentioned kernel updates. Now there's a way to avoid reboots with all of this which we'll get into but I think that's a good place to start. With most distributions when you have an updated kernel it's not removing the old one. I mean, some distributions actually do when you install an updated kernel it replaces the kernel that you're running on which is kind of scary to think about but Debian and Ubuntu and many others they install the updated kernel alongside the current one. So that means in order to use the new kernel you have to restart because it's during the boot process that it actually picks the most recent kernel on the list and boosts into that. So just simply having that installed isn't always all you need. Now, if it's a application, a demon or some kind of service that runs like Apache, Nginx generally speaking, you could restart that service and you're fine. Some distributions will restart the service automatically. It kind of depends more on the package and how it's configured. So I remember, I think this is probably still the case when you install an update to MySQL in Ubuntu it restarts with MySQL process. That may not be what you want because your app goes down when that happens. Maybe you'd prefer to just restart it later when you want to but you'll notice that behaves differently depending on the context, the distribution, the package maintainer, things like that but it is often the case when you update a package the service will restart right then and there. And I believe that's all configurable for example in Debian in the unattended upgrades. Yeah, there's a flag for it that you could do which is another topic we'll be talking about that as well. But I feel like you as the administrator you have to watch for that. So if you know there's a critical vulnerability in Nginx and you update the package, you look at the output you noticed it didn't restart the service make a mental note, make sure you restart that service if it restarts automatically, I guess you're fine but just basically gotta keep your eye on the output. I know it's a wall of text but they have a general idea about what exactly happened when you updated. Absolutely. Yep, that's important. So getting into unattended upgrades. Now, in full transparency, I'm not fully aware of how other distributions do it because I'm basically a consequence of what I'm exposed to the most, muscle memory if you will but Debian and Ubuntu is what we're referring to when we say unattended upgrades but other distributions have an equivalent thing that you can use the idea being that it's going to essentially install all the security patches at a given time with Debian and Ubuntu you install the unattended upgrades package and you're pretty much all set although you might want to adjust the configuration. For example, you could put in an email address in the configuration for where the message is being sent to so you have some kind of notification that this happened you could choose which time you want it to reboot or even if you want it to reboot at all you could say, no thank you you don't want it to reboot automatically or you could say, yeah, 2 a.m. a reboot is fine go ahead and restart it you could choose how you want that to be set up and going back to the importance level if it's something that's not internet facing it's probably not a big deal to reboot it manually if it isn't exposed to the internet one could argue a vulnerability chain could still lead to someone getting into that server but that aside, you make that choice if it's not all that important it's not public facing, okay, not a big deal but if you have like a node instance yeah, you probably need to do something about that and make sure that it reboots and one of the ways I like to think about this is that I have some servers that I want to be online 24 seven and some that I don't now I'll talk about later how I solved the 24 seven one but if I don't care if it's down for example something I might use during the day but I'm not using it three in the morning it can reboot that's fine I don't care because I'm sleeping so whatever but if many people are using it such as the learnlinux.tv website I would probably prefer that not go down is basically if I can help it yeah, I do have mindset to automatically apply security updates which I believe is the default still for unattended upgrades so when you install it that's actually the default it's going to just apply the security patches you have to go in there and change it if you want to do more I'm fine with automatically apply security patches because they don't often require anything more than a service restart but I do have it set to restart because the fear I have and this actually applies to even my website I'm fine if it has to reboot in the middle of the day because a hacked website being up is much worse than a minor downtime because my website takes about 25 seconds I think from the time I type reboot until it's completely fully running maybe even less because I got it on one of those really high speed servers so it's like it doesn't take long worst case is there was a 20 second disruption on my website that someone had to experience but if they had to experience someone who defaced my website that may be less pleasant and certainly a more awkward phone call going, hey, you know what your sites got on it now? Yeah, that's never good when that happens so I mean unattended upgrades is like the minimum in my opinion and it's so easy to set up and on my end, I think I've talked about this before where my Proxmox servers shut down at like 12, 30 in midnight and I don't care, right? Because they're down, they're internal they can be down and it saves electricity it's more green that way there is a debate on whether or not it's harmful to servers to have them turn on every day I argue it's fine but for those servers they have unattended upgrades set up not automatic reboot they all shut down anyway at 12, 30 so I just make sure that it's scripted to happen at like 11 p.m. at night so that way when the cron job hits that shuts everything down they've already been updated so when they come up the next day which I have triggered at like 7, 30 in the morning about the time that I'm just getting started for the day then all my servers come back online so since you were updated the night before it's fine, I don't really care and then I handle the internet servers a little bit differently because they are being exposed I have a next cloud server that is now publicly available I have people using it all over the world but just a few people it can reboot it too in the morning so you just basically make up your own mind as far as what should and should not reboot and when and just use your best judgment Yeah, and you can set this up even on your own system but if you're like your desktop but if you're using something like I'm a big PopOS fan in PopOS it prompts you for the updates I just say yes all the time and I've actually because having extra media devices plugged in I've had to write a little script I call boot things and boot things is just all the things that my system does when I log in because I do shut it off pretty much every day I don't just don't leave my computer on when I'm not here at the office so I actually have it running flat pack update and the after get update it's just part of my boot up process so everything's nice and fresh every time I boot it up Yep, that's one way to do it too So one thing I want to touch on I don't want to spend too much time on this because I think this is going to be a little bit overboard although I'm sure there's a certain subset of our audience that are that's probably doing a huge subset of others that likes overboard so let's go there Yeah, but actually maybe I'm yeah, maybe everyone does so the whole idea of like a rolling update system so in cloud we have auto scaling slash auto healing where servers come online and go away to match the load but similarly we also have rolling updates which is probably something more likely to happen in the home lab a good example of that is like a virtualization cluster so let's just say you have I don't know, three hypervisors so you just move all the VMs off of one live migrate, they don't go down so now this server, the host server has no VMs running on it you can update it, reboot it when it comes back move some VMs over to it free up another server update that one, restart it, repeat until each of the three hosts were updated and basically nothing ever went down except for the hosts themselves now, I mean, in the home lab do you care? I don't know, kind of depends, right? Because the worst case, your significant other your kids are watching house on Netflix and or Plex or whatever and all of a sudden it goes down okay, they can resume their show in 15 minutes it's fine obviously at a company it's a lot harder but I think we as home lab people we still would rather not have to restart everything if we can help it because it's a little annoying, to be honest Yeah, for sure Yeah, and I also kind of wonder why are we even rebooting anything in 2021 if you were to ask me like 10 years ago would we still be doing this? I'd be like, no, I doubt it but then again, you know I have a computer mouse in front of me so I didn't think we'd still be using that either but here we are Here we are with some of those things they just haven't, that hasn't changed much Yeah, and the reason why I'm not going to go too deeply into rolling updates is because it really depends on what type of system you have set up whether or not that's easy to do or not I mean, especially when you get into database servers I mean, having like a database cluster becomes a pain and I've done it, you could absolutely do it but do you really wanna go through all that? I mean, personally I don't but maybe someone who enjoys database administration more than I do, maybe they'd be more keen to set that up And with the rolling updates out of the way now we get into the concept of live patching which is really cool Yes, this is really neat the way that this is all engineered, I like this It is, yeah and I think the problem with it is that it's a little confusing to people that are kind of starting out because most things, you know you get an improvement in the Linux kernel and you benefit right away So for example, the AMD sync feature in the monitors I forgot which kernel version came out with that but as soon as it did then everyone who had that monitor that had that feature in there benefited immediately there's nothing to configure, nothing to tweak it's just, okay, new kernel, new feature reboot you got the new feature, great that's kind of what we're used to, right? New features can improve speed and things like that and one of the new features in the past several years I actually forgot what year it was the concept of live patching in the kernel and if you asked me, you know years and years ago if that's something we'd be doing I'd be like, no, that's impossible but it's not because you can do it so the concept of live patching is literally injecting a patch right for a module or whatever right into the kernel as it's running so you don't have to restart for a kernel update you'll still get an updated kernel package that you install via your package managers such that when you reboot it'll be booting into the updated kernel but you'll be able to temporarily postpone those updates or the reboot because it's right there in the kernel injected live which kind of seems like kernel black magic to me but here we are it's a very real possible thing we can do now Yeah, being able to actually modify they've done a lot of engineering to it's kind of cool because it used to be way more monolithic and much harder to compile everything to the kernel and then as they started breaking it apart they solved for a lot of these pain points the way we load modules the way we everything if you remember was a add the kernel module recompile the kernel not just you couldn't just insert a module you spent a lot of time recompiling it and now come all the way here in 2020 we may still be using a mouse and not just doing waste commands for our computers in a fluid way but now we do have the ability to slip that code in and replace or modify the code so it can keep working and of course like on next reboot you're gonna get it without the slip in but that little slip in part like you said that's that kernel black magic stuff that's really cool engineering It really is I think the problem with it why I say it's complicated and not straightforward like most of the features we get is because there's no clear direction for most people how do I go about doing this? So you could download a patch you can put it in the proper directory there's some things you have to do I will get into and you could go ahead and live patch right then and there Now are you really gonna have time to look for these patches find out how to install them it's not the easiest thing to do so what people do instead is they subscribe to a paid service to do this for them so all the common suspects like Red Hat and Ubuntu and the similar enterprise distributions out there they have a paid service that you can use they subscribe to and it'll just go ahead and check for patches and inject those automatically for you so that way you don't have to worry about that and that makes things a lot easier but the problem is if you are using a little bit of SUSE enterprise and maybe like a Red Hat family distro and maybe an Ubuntu server you're potentially paying three different companies for the privilege of live patching which gets a little tedious after a while but full disclaimer they're a sponsor of LearnLinux TV but I really love their service TuxCare makes a product called KernelCare which supports multiple distributions for live patching so you actually can pay one company regardless of which distribution that you're using which I think is a lot better and they're a little bit cheaper too now Canonical has one service that's even cheaper than that which is actually free but only for three machines but if you only have a few Ubuntu servers then this is perfect because you can have like an enterprise level live patch system set up and working for literally no money at all so three Ubuntu servers it's easy you can actually set up live patch there's instructions out there I have a video on it as well that teaches you how to do it I think it's called the Ubuntu Advantage program last time I checked I think it was $5 per machine but the first three machines are free so if you only have a few then I guess in your case then that's great then you're all set you have live patching yeah it's this like feature that they have on there it's been a kind of a weird thing that that's like what these companies have settled on as the upsell but it is also some more complicated engineering to get it to work so I think that's probably why it's not it's still open source it's just that the facilitating tools that do that have the extra fee attached to it right and they're inconsistent too which is another problem because if there's a vulnerability out there that's pretty bad you're hoping that it's gonna be live patched but maybe the vendor doesn't agree with you on the level of importance and that's not one of them that's going to be live patched different companies will do different things one company might be better at it than others I was looking at some documentation before we hit the record button today and right on Red Hat site for this they say this isn't meant to eliminate all reboots we just basically patched the most important things but they don't patch everything and they're very clear about this so they don't want you to think like using their service you won't have to reboot your servers you probably won't but you might still have to whereas SUSA they advertise this mitigates reboot so they're more in on eliminating the reboots than Red Hat is so depending on your distribution of choice it may or may not actually be something you can really rely on you might still find yourself rebooting your server but at least we have the capability of doing this now whereas we of course didn't until recently yeah it's I don't know I should take a look cause I know you've there's sponsored your channel I've not actually tried the TuxCare but it is some kind of interesting to try that out and I'm sure you must have a video reviewing it to kind of show how they do that type of patch management right I do and it's a little old actually I was surprised when I saw just how long they've been a sponsor cause it didn't seem that long and I looked at the video I'm like wow that's my old branding so it needs to be updated there's some new features that are coming out which is why I haven't updated it yet but I will now another thing that they also are able to do I think this is worth a discussion is they're able to patch shared libraries as well because that's another reason why you might need a reboot so for example if you have something running and it's linked against a shared library you know maybe something like SSL that's been patched then you know maybe restarting isn't a service isn't going to work there's might be something else that needs to be restarted it just becomes kind of like this rabbit hole of trying to avoid a reboot at this point and it's not a kernel update at this point but you still might have to reboot I mean technically you don't have to if you could find out exactly how to restart everything that's using the shared libraries but that becomes a pain it's probably easier to reboot but then they have kernel care plus which actually live patches running services as well in particular certain shared libraries you can't say that it's going to patch everything but there's going to be some common shared libraries that they will live patch which eliminates even more reboots which is pretty cool and yeah I like the service quite a bit they're a little cheaper from what I've seen when I first looked at it than the others and they go after multiple distributions basically LTS style distribution so your enterprise Linux Ubuntu LTS they're not going to work with the intermediary releases of Ubuntu for example because they have a life they don't support Arch because they have a life but I mean no offense to Arch users I love Arch I always have at least one installation but it's just one of those things that you can consider I guess what it comes down to in the home lab is it worth the money maybe you could say the money by disrebooting and not pay anyone anything or if you use Ubuntu just use the three servers that you can use and if you only have one server that's facing the internet then you can just put the live patch service from Canonical on that server and maybe that'll just be enough to solve the problem Yeah and I see a question here and this is something you know towards the end here we want to talk about and I guess not as good as times any is when things go wrong when you're patching and I had a mail server that I maintained for years and one thing that did happen was well I had to maintain it for I think eight years and the updates finally did deprecate a few functions you have in the file this is something you kind of have to deal with it's dealt with relatively well for major packages but for minor packages it may not be as dealt with as well so when there's version updates that will deprecate maybe some function that's in the configuration files right you have to address that manually in some of them and there's you just have to kind of be aware it's part of being a Linux system administrator because if this system worked 100% all the time I mean what would me and Jay do? Right yeah there wouldn't be anything to talk about would there? Yeah. It's kind of awkward but you have to kind of look for this and it's very it does a good job of telling you I was able to ignore some errors in PostFix for about I don't know a good year and it would be for it finally wouldn't PostFix wouldn't start because of deprecated function I'm like oh I guess the time's up on the warning of there deprecating this function in future versions the future version is here you just have to keep an eye for it it doesn't seem to happen to often like on major things like Apache, engine acts and those they're relatively it's weird looking at Apache going I've been doing the same format for Apache Comp files for I don't know how many years but a long time and usually though if they have a whole rewrite of a version of the configuration file you'll get a whole update into the version of the software so you'll end up eventually you've been told this has not even supported anymore it is something you have to watch out for but I haven't really seen it come up too often most of the time and I'll use gray log as an example because when they did some bigger version updates part of the update was re-updating the changes and getting rid of things that were deprecated it did automatically for me in the configuration file it says no this isn't used anymore we removed it it just kind of is in the notes and you're like awesome you mitigated it for me and that's just really good coding on the developers part for the people maintaining the software. Third party packages or small one-off projects yeah that's a hodgepodge not all them update fine. Right no that's very true and it is the case I mean sometimes it deprecate things and they change things and you have to redo how you do things and if anyone listening thinks that's annoying to have to redo something because of an update I agree with you it absolutely is but think about it from our perspective when we wake up in the morning and determine that we have to redo an entire tutorial because it's no longer current with the way it is in reality so surprise get recording it's not a fun time for anyone I don't think but it's just a nature of the business we have this happen so it's just one of those things one of the many things that we deal with is these updates and you know what to do about them but getting back to the question what do you do about this? So it really depends on what kind of breakage you're experiencing like you were saying it doesn't happen with Linux very often it's very rare but I have seen it I had an Arch Linux install I know what you're thinking already as soon as I say Arch Linux but this has happened on other distributions as well but I remember this one because I remember how I fixed it where I had this old laptop that was already like three or four generations old by the time I installed Arch on it so usually with Linux older hardware is rock solid unless they deprecate something nobody is really going back and changing too much of the old code unless there's a vulnerability so your old latitude laptop is probably safe but in this case it wasn't so I installed the kernel update rebooted and it wouldn't boot, it was bricked like it just wouldn't start and what I had to do is just get a live CD go into a CHroot environment and I side loaded the previous kernel back into it and then it booted up just fine. And now normally with Ubuntu or Debian and other distributions it's a lot easier to recover because you just select the previous kernel to go ahead and get you started so that way you're not booting into the newly updated one that for whatever reason has a regression so for those distros it's really easy just hit escape at the beginning choose the previous kernel in your set but Arch Linux for example replaces the current kernel with the new one so there's nothing to go back to which is why I always make sure I have two completely unrelated kernel trees installed so I just don't have that capability but that's if it's a kernel update that breaks things if it's a service like Apache and X or whatever you'd probably have to tail the logs as you're restarting the service which of course is going to crash but why did it crash? If you're tailing the log you can see exactly why it crashed and then make a decision as far as what you have to do and honestly it's pretty much down to copy the error message and paste it into Google because nine out of 10 times that'll lead you to the answer. Yeah, the first result will be on Stack Exchange Matter of fact, with PostFix I remember the problem was it didn't even record a Googling it said like line 39 is no longer supported you just could comment it out because it was a deprecated function that was enabled by default and they didn't accept turning it on with the parameter that was used in the previous version it was some type of the way it handled certificates so it was like all that I do is comment that out and it's telling me line 39 I live until line 39, comment out oh look it starts up fine a lot of them each service you just have to go through the logs and we brought this up I think when we were talking about cool Linux tools LNAV is one of my favorites I team up split the screen I LNAV to the log file of the service I'm trying to restart because it'll highlight all the errors in red it'll scroll automatically I have that in there start service see what the error is all right I got the error pulled up down at the bottom of the screen then you go and type VIM and whatever the config file is for that service and away we go I figure out what is it that needed to be changed what needed to be updated what's been deprecated that needs to be fixed to get this service back up and running Yep absolutely now how can you prevent having to go through a lot of work to recover your systems and the answer is obviously using snapshots which depending on your type of server could be very easy to do or not so easy so for example if it's a physical server you aren't going to have the same tooling that you have with a virtual machine a virtual machine you could just go in the GUI hit the snapshot button before you do your update you could just reboot it or whatever you have to do if it doesn't come up something's broke you don't have time to fix it right now you've got to move on with your life you can just restore the snapshot and then you're back to the way it was that's always a great thing to do we also have tools like TimeShift which can be used on Linux it only works on distributions I don't know if they fixed this yet but you have to be running grub which most distributions are it'll still allow you to restore certain things but a full system restore may or may not be possible unless you're using grub there's also like LVM snapshots you could use if it's DFS you could use that but our FS has snapshots as well if all else fails you could take a clonezilla image if it's a physical server and none of those things apply to you so that way you have something to fall back to if the update doesn't go as planned yeah and for me I've really shifted to running all my critical infrastructure as VMs over the years and I mean containers are good as well and there's some stuff where the VM does have maybe Docker inside of it but that just simplifies things quite a bit especially you know you're just doing a new Docker pull for a new image and swapping it out real quick that temporarily goes really fast there's a minimal amount of downtime there and then I'm a big fan of for obviously a lot of you I've talked to a bunch of channel XCPNG and like Jason earlier it will do the rolling pool patch updates to keep the XCPNG server up to date but then all the VMs inside it's just process procedure because even some of our infrastructure is run on Windows you know we just snapshot it load the new version as long as everything's working fine I hold the snapshot for maybe 24 hours to make sure there's no issues purge it and the way we go life is good yep I really love timeshift I think that's something that if it's compatible with your system that everyone should check into you just have to have the extra disk space for all the snapshots and whatnot but it's so easy to use I mean it might take you a few minutes to learn a few things about it like everything else but it's helpful you can restore your entire system if you have to at least you'll have that snapshot if you need it I think that should be the mentality is if you're not snapshotting things somehow let's find a way to snapshot because if nothing else you'll probably really enjoy having that snapshot hopefully you'll never need it but if you do it's great to have it makes it so much faster especially if the problem's a little bit more convoluted especially when there's a dependency like some dependency updated and this has happened sometimes where the line of business application software may not be aware of the dependency that got updated this is one of the big reasons for snapshotting when you have those one-off tools so if a dependency maybe it's got a Java dependency and there's a new version of Java a new version of MongoDB and you have a tool loaded on that system that actually depends on MongoDB well that update may break it and then the problem you run into is the line of business application is not actually ready for the new version so they have to do something so that snapshot lets you hurry up regress back to the previous version like do I really need this version of Mongo just yet you contact the developer of the software and hey you apparently don't work with the latest update to Mongo and they go we will next week so then you just hold off on that patch an extra week but that snapshot is that saving point on there and this is where you get a little bit off because it's the line of business applications and those one-off toolings I'll mention things like Unify controllers and stuff like that your non-standard, your extra packages not your standard Linux packages like an Apache or MySQL those are more mature well integrated into the update process and generally don't give you those kind of problems whether you're running a mail server you have a whole lot of other issues if you're running a mail server but if you're running web servers and you're running WordPress and things like that those problems generally when there's a new version of PHP it's generally not breaking WordPress because it just expects the newest version of PHP and things like that those type of things actually integrate really well together Yeah they really do now I think one question that may come up if it hasn't already is okay but how do I know what vulnerabilities I should be patching against if I only have time to patch a few things right now and I want to just get the most egregious things out of the way I like the tool Linus LYNIS that's really great for doing an audit of your system it's a free tool that you can download they have an enterprise version if you want it but if nothing else you could just add their community repository to get the latest version installed you could run an audit within Linus and it gives you this huge report with oh gosh so much information in that report I actually have a video I'm doing right now that's showing how this works and you could take that report and you understand exactly what the most important concerns are in your case for your server that kind of gives you an action plan as far as what you should target first so if your biggest threat is an Apache vulnerability then you know you have to at least update Apache and make sure that the service gets restarted if there's a kernel vulnerability well you know what to do there so if you're using live patching you can make sure that the live patch tool is actually patching that vulnerability if Linus shows that it's still vulnerable perhaps it is or perhaps it's not but maybe it's better just to reboot it but at least you'll have some idea yes it's a good start to run that tool to figure out what might be hanging out there is it checks some of the configuration files like SSH as well too it's been a little while since I've run it but it kind of gives you a whole list of things you should be doing to lock down your Linux server a little more yep absolutely and that's a whole story of it by itself I mean we do entire episodes and entire videos just on you know individual subjects within that realm so there's a lot to talk about when it comes to security it's never ending conversation yes and it's better to be patched than insecure hopefully that rings true because man I've seen some really old systems and it could brag in about uptime that's how I know you aren't patching things especially if it drives me crazy because it's like no one cares right I mean I think it is kind of cool to an extent but at the same time it's like I'm avoiding best practices aren't you know I'm the coolest thing ever I mean it's cool it's cool if you're running a Commodore I've seen some of those in some long uptime I've seen a few I love when I see some of these articles out there you know someone's still using this old machine and those are cool there's really not I mean they're not connected to the net they're isolated hopefully from the rest of the world so they're kind of novel that they're still on and running for all this time but modern systems especially ones that are internet accessible even if they're not even if you have them behind your firewall and they're more for your private access especially browsers so that in there there's a reason browsers have gone to their own patching systems and having them up to date and are updated in real time because their browser is now the threat surface that you interact with with the greater world so the browser is the way in so to speak so you may be thinking I'm behind the firewall so I'm safe but then I'm like well do you use the internet at all with a browser you need to keep those up to date the latest version of Firefox added a feature to they found basically a hack in Firefox where people were in modifying threat actors not people not just a general public the threat actors were modifying the proxy to stop Firefox from updating to hold back patches that would get them out so Firefox now has a way it can go around the proxy to check for an update to see if you're running an out of date version of Firefox because the war against you know the threat actors and getting in your systems when they know everything else is patched public facing and you're behind a firewall the next thing is the places you interact directly with so yeah the browser from your personal standpoint just to round this whole talk off is absolutely you got to keep the browser up to date all the time that's whatever browser of choice you're using as Firefox or Chrome either one of them keep them patched so one of the reasons I'm always a skeptic before someone says well what about this other browser Tom I'm like how good is their patching that is that scares me so much like I've literally seen browsers that you know I mean people can fork anything they want it's open source so no disrespect intended but a browser is not something somebody should rage for you know someone gets upset with Firefox or Mozilla or whatever because they're you know doing something they don't like oh I'm just going to fork the browser and they do it and maybe they're part of like three people that are updating the browser are they keeping up with CVEs and then the constant counter argument is well it's an open source browser browsing engine so as soon as the libraries get updated it's all automatically fixed which is untrue because they're putting additional Chrome and UI stuff on top of it that could be causing brand new CVEs to happen so I usually squirm when someone you know it says I'm using insert name of browser here that I've never even heard of before and they're doing their online banking and everything on that browser I just have to face palm because I mean you could fork anything you want there's nothing stopping you but please stop forking browsers it's just going to cause problems at the end of the day it is one of those worlds we live in right now where many of them are based on Chromium so the open source Chromium engine and Google is the maintainer for Chromium the open source and then they have Google Chrome based on it Edge is now based on it so it also like Jay says where's that vulnerability found in there and if the vulnerabilities in the Chromium engine great Google updates it Google rebuilds Google Chrome with that updated Chromium engine that's patch but this is where you've seen the diversion so Microsoft Edge puts the Chromium engine but then their own flavor on top of it and they've had vulnerabilities in the essentially the sauce they've added on top of the Chromium their integration so this is that concern that comes into each one of these is even though they may start with a secure engine you think they're getting updates what that covers the engine part provided that they pull from upstream as the upstream updates come in but then you have to worry about the extras they've done on there the leak for a while in Brave browser because I know it's a really popular one out there where Brave mis-implemented the way they were handling the Tor protocol and they were leaking a lot more data than I thought it was just a mis-implementation it wasn't anything malicious but these are the concerns that can come up and there was just a privacy leak less the security issue but it's one of those things it's worth keeping a very well conscious thinking mind about this that's the best thing to do you really got to forefront your mind before you see some random browser going hey this one looks cool so one example of that is this browser I don't know how many people remember this one called Reconq it was for basically a browser made for the Plasma desktop because it was made using the Qt tool kit and you know there's a number of people that thought it was a great browser I mean to be fair it was good I'm not saying any of these browsers are bad from a usability standpoint but it was discontinued in 2014 these browsers generally don't last long it's hard to maintain them and sometimes the developers you know they're human they get tired something changes in their life they have to do something else they can't really spend any time on it or maybe they're not as passionate about it as they used to be or they can't find a maintainer or anyone to adopt it that happens a lot there should be like a website that's a graveyard for browsers because there's quite a few of them out there that came and gone and everyone's like yes this is the best browser ever it's great and then you know year later it doesn't exist yep that's so true all right I think we've patched this whole show up right Jay hopefully you got a better understanding of patching I do have a video called unintended upgrade using unintended upgrades to avoid unintended consequences so it's about this Jay's got some videos on his channel about you know Linux patching and things like that you can check out even if it's an older video he does have one that talks care full disclosure they are a sponsor of Jay's channel not mine but you know either way it's a neat service if you're looking for a service to patch it but those hopefully you got a more well-rounded system and hopefully you don't have any out-of-date systems that you have in place and if you have a Commodore 64 with some crazy uptime please send us a screenshot take a picture of that and tag us on twitter on that I'm a sucker for Commodore 64 I'll end up listing a video on you know going nostalgic on them so I always I'm a sucker for those articles because I played Commodore 64, TRS-80 especially big fan you know the old stuff so Tandy, yes all the nostalgia I don't know if we'll do a nostalgia episode or not this is the homelab show not the homelab but I do have a nostalgia friend we could bring on so an argument can be made for the fact that you know even in homelab we always have this and I know everyone's guilty of this we have this downloads directory somewhere probably on our NAS and it probably has every game that we've ever purchased since like 1995 let's be honest and ISO images of Doom 2 and what not so it might be fun just to kind of talk about running this old software because at the end of the day I mean you may as well have some fun playing Doom 2 on a Raspberry Pi or something it's a lot of fun now we want to go watch the 8-bit guy go live some nostalgia alright thanks everyone for joining us and we'll see you next time thank you