 This is not the video I planned on making today, but this is the one I'm making. The firewall died at my office, and I will absolutely take some heat in the comments for going Tom. Shouldn't you have had a HA system? You know, you set up a lot of them. The irony on that's not lost on me. As a matter of fact, I was looking at a few of the HA systems that are ready for delivery in the other room and thinking, I should have done that because I did have another one of these. And of course, it's just, you know, so I have a spare ready. But I wanted to talk about the process that is how to restore a PF-Sense system and how quickly, you know, we got everything up and running. There's always lessons to be learned even when there's failure. The first thing I'll mention though is I'm happy this happened on a Sunday because, you know, it's a more relaxed time to do it. I'm like, oh, this broke, and I have to get it up and running. It's not like it's in the middle of a day when there's a lot of people here and things are going out of my office. But it also happened on a nice Sunday. That's a little disappointing. But hey, let's talk about some lessons learned and get into this. Now, the first thing I want to start out with is backups, backups, backups. We all know we should be doing them. Was Tom doing them? Is that what this video is? Or would he be honest about that? I would absolutely be honest if I wasn't backing up because that would make me a hypocrite. And, well, I never want to be a hypocrite. So yes, of course, I'm backing everything up. Anytime we make any major changes to our system or minor changes or any changes, really, that's the rule. Any. And when I make a change, I download a backup file from my PF Sense. This makes my life way easier and lets me rest easier at night because, well, hardware failures happen. They don't happen often. And this is a NuttGate 5100. We've deployed a lot of these. We keep them out in the field for, well, until their retirement time. And they all work. This is the first one I've had die at all. Now I've had other models die, but not a 5100. I thought this was pretty odd. Now, the what happened is a little bit interesting. We'll get into that in just a second. But of course, anything that is critical to my business, I have another one of. So the restore process, because I back up all the time and we do the same thing for our clients. We keep these extra on the shelf. We don't stock like a lot of them because, well, it just doesn't feel necessary because it's such a rare occurrence that any of these have died. But because I had another 5100 on the shelf, I just restored the config file. And of course, that's what gets me back to making videos and doing whatever I want to do. The process is relatively pain free. And that's why I want to go over is what happened and how to do these backup restores to hopefully give you a little bit more confidence in how this process works. Because people know to back up, but not everyone goes to restore process and might be a little bit daunting for someone who hasn't done it, who's worried about losing a bunch of things. So let's cover how that works. Now, before diving into that, I will mention I use gray log to ingest all my logs. And because my 5100 no longer comes on, no longer posts, it does not give me anything to really diagnose other than it's not working. I was curious what the last logs that sent the gray log were. And I actually noticed I had this controller timeout. Now this doesn't surprise me because if we go here and load and we 5100 controller timeout, I had seen this error message a few days before on actually 310 2022. And I actually made a note to my staff because it happened in a day. I'm like, Oh, that's interesting. It had a controller timeout and then locked up. We restarted it just unplugged it plugged it back in it booted back up. But I thought that was strange that the last log it sent was that and it made me suspect that maybe the drive within it is going bad or some other problem with the board. Either way, because we had a spare, I was like, Okay, I'll go ahead and replace this probably on the weekend, but actually was going to be this evening I've replaced it because I had some free time. Instead, well, it decided to stop working right here at 127am. So really early in the morning, it just decided to quit responding. And things go down alerts go off. And I'm like, Well, there's nothing going on on Sunday, I'll sleep a little bit longer and go into the morning and go ahead and replace this. So that's where we're at now. Next thing we're going to talk about is the lab because that's where we do all these experiments. So we don't always want to test on production equipment. This is my lab system that I have that will be doing the backup and restore demo one. And it's really easy here. But I want to bring up a bug I did run into but the bug has been reported. So there's really nothing more you have to do just something that you may run into if this happens to you. That's this bug right here, we happen to use not just a j proxy and lots of other fun stuff, but also free radius as part of our tie in with open VPN for our logins. Now, we'll cover some of how the packages are backed up. But one problem I ran into is it didn't reinstall free radius, which I thought was really strange. There's a bug on this, I'll leave a link this if you want to read here. The solution to the problem is really simple. The package reinstall just skips that package, you just go and manually install that package and all the settings for that package will come over because those settings are saved in a config file. There's a name mismatch really minor bug. But hey, that happens. And they do have that it's going to be fixed in 2205 of PF Sense Plus and version 2.7 of PF Sense Community Edition. Speaking of that, we are on PF Sense Community Edition right here because it works the same in plus or community edition for these backups and restores. Before we go to the back of restore, one thing I want to mention here, whenever you set up a package inside of PF Sense, and we're going to go and look at packages, oh, I don't know, we can even look at something like firewall, PF blocker, and you'll notice keep settings is checked. This is the default option when you install packages that the information within this package does go to the backup file. As long as you didn't change this, you shouldn't have a problem. I don't recommend changing it. Maybe there's some use case where you don't want your package settings backed up. That's fine. But for the most part, I always leave things at these defaults because I would like that my backups include all my package settings. Now, how do you download a backup? That part's really easy. You go here to backup and restore, and then we hit download package information of note. You can, if you want, do not package, do not backup package information is an option you can check here, which overrides that and says, all right, don't include that in here. By default, skip RRD data. And this is the RRD data you don't really need. I'm not going to get into details on it. You can read more on it. Do leave that checked. Backup extra data. Now, I don't want to conflate anything here. Let's make sure we're very clear. This is not needed unless you want your DHCP leases database. This is different than your static assignments. Your static assignments are backed up, but there's a database that is all the handed out IP addresses. And where you can have a problem is if you do a restore that database, if you were to not back up that database, because it defaults as that's not checked, it would just start handing out addresses. Again, if something decides not to listen for a new address, an address could be handed out that's already assigned to another device. Could be a little conflict you're running to just something worth noting that you can back that up if you want to. It's just that a fault is not to back that up, not a big deal. And whether or not you want to back up the SSH keys, this is checked by default. So these are all the settings you have to do. And of course, it's up to you, if you want to encrypt this backup file, or I store it on an encrypted device that I keep locked, this is a pretty way easy way to do this. The problem you may run into if you encrypt it is, is you may want to edit the backup config file. There's a scenario we'll talk about towards the end about doing that. Now, once you've backed up this file, and I'll come here to the beginning to point out a couple of things. We noticed that like the ACME and system patches are out of date. That's why they're yellow. And I can update them to the latest version of I'm not going to bother. Now I can even delete these packages right now. It really doesn't matter if I wanted to get rid of this. We can just remove this package. Hey, confirm we're going to delete a package out of here. None of this matters. You can take a PF sense when we do the restore and have no packages installed the way PF sense works. And this is what's important. We're going to go diagnostics backup restore. And you know, there is a selective restore option if you want. And that's great for if you only want to restore certain aspects of the machine, but we're talking about a full config and full restore. So now we're going to go ahead and show what happens when you do a full restore inside of PF sense. Now we deleted one package, but it doesn't really matter because this config that we have right here, the one we just downloaded, we're going to hit select, and we're going to hit restore configuration. Are you sure you wish your stores configuration because what happens next, and we're going to go over here, because this is my demo lab, and we'll expand the council out a little bit, you'll notice that the system is about to shut down and restart. What it's going to do is it took that config file, and it's going to blow away everything on the system and properly set it all back up. I believe it, if a package is already up to date in the same version, it won't read download the package. I'm not 100% on that. But I do know is the system is going to go into a package lock mode that will show once it starts back up, because it doesn't care what config was on there. And it's now going to go back and completely restore the system to its state as of when we downloaded it. The only differences, for example, any outdated packages are going to be the latest version, because if it seemed like the system patches that I left in there were outdated, it's going to go ahead and say, nope, that one's outdated, we're just going to pull the latest and put this all together. So here it is going to boot, and we're going to fast forward a little bit so you don't have to wait too long. All right, the system is booted back up. And then we're presented with this right here, you'll notice that the packages aren't all running. So even though some are installed, they're not working at the moment. So the system is in the background. And what it's doing is, and you'll notice right here, first you get this message. And the next one is, if above message those played after a couple hours, use clear package lock button under diagnostic and restore backup and reinstall the package manually. What this does is it's going in the background reconfiguring all the packages, all your routing and basic stuff works, but all the add on packages as it goes through and assesses what needs to be updated and the configurations push to them, it puts a package lock on there. So don't make any changes to the system while we wait for this to happen. I've never had it take a couple hours, it's generally, I don't know, a 15 minute process. If you go here and we look at this virtual machine, we'll probably see it pulling some data. There's the boot up process. There's some CPU activity where it's getting things set up. There's some disk activity and some network activity where it's pulling down, grabbing things that it may need. It doesn't really take too long to do going to vary with your internet connection, how fast I can get those packages if it needs to reinstall all of them and also just how complicated of a config you have and the processor power that you have assigned to your particular system or the hardware that you're running it on. So we'll go ahead and fast forward through this to get it through till this part's finished. All right, and the system has been restored. Took about 10, 15 minutes for the system to be completely back up and running. Now, the next question I want to answer that comes up a lot is what if you're not restoring to the same hardware, you're swapping network interfaces, there's a couple of ways that this gets handled. PF Sensei, she has a question that'll ask if the interfaces don't match and it'll have you align them essentially. The other way to do this is go into the config file. And for example, right here in the config file is XN zero, which has an IP address of 192 1683 dot 217. Now, how do we find this? Well, we're going to look inside of here, and we're going to interface assignments. We can look at the way on, we see that it's XN zero, and it's three dot 217. What if you had a different card? What if you had a different type of interface? Well, you would go and download the config file to match that. It's a little tricky to match because you have to download a config out of the upcoming system. And if you're just adding a network card, you may not have that information, but you can go through and set it up, download it, then do the restore process. As long as you're matching the different network interface names, you can go through and swap it in the XML file. That's actually one of the things I like being that this is an XML file. Really, if there's any changes you want to make to your PF Sense, especially if you're making a group or mass amount of changes, you can go through here and like even things like that you can see right above the radius port interfaces and things like that are all set up inside of here. All those details are stored in here. So whether you're reassigning something, and I've actually done this before, for example, when you have to go through large DHCP changes, because you want to redo the network, you don't want to teach the go through and change those. Well, you can just go and do search and replace the XML file and re-upload your DHCP list that you have all your static leases assigned and then all the devices will get a new address that the new assigned network and set it all up. It's a lot of flexibility having everything in an XML file. Now, the last thing I'll make a note of is what if you have an older XML file? I don't know and consult the documentation for exactly how far back you can go. But generally speaking, if you were a version or two behind on PF Sense and you have loaded a new system and well, for whatever reason and circumstances, you only have a backup from that previous version, then that's just so work perfectly fine. Generally, I haven't had a problem doing this. We sometimes run into people that are last backup was quite a while ago and they're asking for help, but we're able to recover that file and bring it back over to the new version of PF Sense that are getting things set up. And as I mentioned, sometimes if it's different hardware, you do have to do the network interface changing and line them back up. But once again, that's just a matter of editing things in the XML file to make your life easy. It makes it very convenient. All right, links to that one bug I mentioned, which is really a minor one, but it's something of note that can have you do a little head scratch and so hopefully that will solve that for you. As always, have a more in-depth discussion over in the forums and leave your comments and thoughts down below. I love people that say hi. All right, thanks. And thank you for making it all the way to the end of this video. If you've enjoyed the content, please give us a thumbs up. If you would like to see more content from this channel, hit the subscribe button and the bell icon. If you'd like to hire a short project, head over to lauranceystems.com and click the hires button right at the top. To help this channel out in other ways, there's a join button here for YouTube and a Patreon page where your support is greatly appreciated. For deals, discounts, and offers, check out our affiliate links in the description of all of our videos, including a link to our shirt store where we have a wide variety of shirts that we sell and designs come out well randomly. So check back frequently. And finally, our forums. Forums.LauranceSystems.com is where you can have a more in-depth discussion about this video and other tech topics covered on this channel. Thanks again for watching and look forward to hearing from you.