 Tom here from Lauren systems and let's dive into troubleshooting PF sense. And when I say troubleshooting PF sense, I mean connecting to PF sense or connecting through PF sense. These are tools I want to talk about that are built in the PF sense that allow you to diagnose whether you have a firewall rule problem, the ability to talk to something internally on the network, what connections are going through the system, or maybe some config changes need to roll back to get it back into a working state that you may have had it before. I'm kind of going to cover the whole gamut of things that you can do inside a PF sense to help the your troubleshooting journey. Now that being said, everything is time index right down below. So you can just jump to the part that may be most relevant to you. And some of these will have more expanded videos linked in the description below as well, such as the tracking of packet loss. I've got a more in depth video and I've got a more in depth video on end top, but I'll still be touching them on them in this video as well. Before we dive into the details of this video, let's first are you an individual or company looking for support on a network engineering storage or virtualization project? Is your company or internal IT team looking for someone to proactively monitor your system security or offer strategic guidance to keep your IT systems operating smoothly? Not only would we love to help consulting your project, we also offer fully managed or co-managed IT service plans for businesses in need of IT administration or IT teams in need of additional support. With our expert install team, we can also assist you with all of your structured cabling and Wi-Fi planning projects. If any of this piques your interest, fill out our Hire Us form at laurancesystems.com so we can start crafting a solution that works for you. If you're not interested in hiring us but you're looking for other ways you want to support this channel, there's affiliate links down below to get your deals and discounts on products and services we talk about on this channel. And now back to our content. First place you want to start is what version of PF Sense am I running? This is the 2.6 release, but for some of these demos, they'll be jumping over to a PF Sense Plus version and it doesn't matter because these diagnostic logs are the same whether using PF Sense or PF Sense Plus, and they're going to be the same in the new version that's upcoming here in January 2023 of PF Sense. Well, we don't have an exact release date, but we suspect that soon it will be released, but that new version of PF Sense that is currently your release candidate still has the same diagnostic menu. So if you're watching this in the future after that release, that has not changed much. The interface looks much the same. Going over here, the first thing I want to talk about in diagnostics is going to be your backup and restore. The backup and restore is pretty simple. Download the configuration. That is a complete configuration including your packages. Do this before you start making a bunch of changes that way you can always revert back to that moment. Also, PF Sense keeps by default 30 revisions and that can be overridden right here for the number of backups that you'd like to keep. So if you'd like to keep more revisions, just go ahead and type a bigger number in there, hit save. Now, as far as these revisions, they are logged by which user did it, whether it was a change by the system. For example, PF Blocker automatically updates things. So it still adds an entry for that. That's something of note, because if you have something like PF Blocker making those changes, it may push out some of your older revisions. So it's still a good idea to keep regular backups that you have for the config history. But this config history is really helpful to be able to try and undo something you may have just done that suddenly something stopped working, you can try going back to that particular config and seeing if that's the solution. Also, if you're curious about what you changed during that config, whether you added a firewall rule firewall alias, you can go here and pick two different versions. So this is a modified configuration of the APC UPS CD. So we're going to do this and we're going to go ahead and hit diff. And we can see what changes were made and modified in the config. So it gives us an XML view of that really handy to do. But of course, sometimes that last thing you did was lock yourself out of the firewall. So how do we resolve that? You can go right over to the firewall console interface. And we're going to choose option 15. Now, this is where it can be a little bit confusing. We're going to press one to list backups. Usually if you lock yourself out, it's going to be one of the most recent one or two here, but the rest go off the page. If you press control shift and page up, you can jump backwards through all the different ones. So that's control shift page up in order to get to the other ones, because whenever you hit list, it just dumps them on the screen so fast that they scroll by and you may not know what they said. Once you know which one you want to revert to just press option two, choose the number, there's 30 of them in here choose one through 30, we could choose this and it will go through and restore the system. The system may or may not need to reboot depending on how big of a change that was. Also of note, if you change a firewall rule, you will have to do a rule reload really easy to do. Just go to status, filter reload, and you can just reload the rules. All you're doing is reloading the firewall rules and applying them because you reverted the config, but this will force it to apply any of that config that you did. Next is under status and system logs. I can't tell you how many times I see a forum post where someone tells me something wasn't working and they didn't post the logs. Always take the time to post the logs. Also go to settings. I prefer this. I don't know why this isn't the default, but forward reverse display show log entries in reverse order. I wish that was the default, but it is not. Check that box to just put some newest entries on top. This makes it easy when you're troubleshooting things like open VPN. I would like to have the latest on top so I can go through here and see if there's any particular errors that I need to address or look at when open VPN is not working. Also it's really easy to figure out things in open VPN by highlighting and right clicking and doing a search. This sometimes will solve all the basic problems because you're probably not the first person who have a misconfigured open VPN or some common error message that you may run into in any of these particular logs, whether it's an open VPN log, a package error or system or firewall error. When you're going to system, you also have general gateway routing errors, DNS resolvers, wireless kind of weird. Yes, you can do wireless on PF Sense. I have a video you can find on that topic, things with the GUI service. So this is like the login information that's in here. And I'll also make note that when you go to the settings and scroll all the way down to the bottom, you can push all of this to a logging server. And I do recommend that because well, it's better if you push it all to a log server that way you sometimes may see the last log before PF Sense decided to stop working, being pushed over there. Also, there's a limited amount of storage within PF Sense. So having your logging pushed off site really does help with diagnostics and troubleshooting. So you can look at things over time. I've done a whole video on gray log. It's a great logging server. If you'd like to put it inside of that. The next one I'm talking about is diagnostics and authentication. Whatever authentication servers you have configured in PF Sense, you can test how they work right here. You can put the user names and passwords in, test it against a radio server, you can test it against a local database. And it will give you the results, pass or fail, of whether or not that user has authenticate, or if that user authenticated and does work. And it gives you some other information, for example, with the local database that this is part of the admins group user all. These are really important tools because sometimes your authentication problem is bad user and password, but you're thinking you have a VPN problem that you have using this as a back end authentication. So you can test this right in here, which is really helpful. Now the next one here is doing a port test. This is something that will really help you if you're doing a port forward because you should make sure PF Sense can talk to it. And there's not something blocking it. Sometimes when people think they've done everything right in PF Sense in terms of port forwarding, they're not wrong. They just have a device that's denying access to it. So we're going to check this IP address here. We're going to check port 22. We'll leave it as source address any. And we can see port test was successful. Great. Let's show the remote text on there and hit test again. And when we do the test again with the remote text box checked, we see it responds with the SSH here. So we now know that this PF Sense can talk to this particular device. This is something that if you're troubleshooting port forwarding, see if this works because sometimes if the system doesn't respond to your PF Sense, that can be part of the problem with the port forwarding on there. Next, let's go ahead and test something like google.com and we're going to chest port 443. Now I'm choosing Google and then we can choose where we want this to come out of. Comcast or Wide Open West. I have two different connections on this particular system. So we can choose which one we route out it. So this is also another way to check the connection to see if it's a problem with one of the connections, connection A or connection B, whatever your different LAN or WAN connections are. So you can choose the source, which is also really helpful in doing some of that troubleshooting. And this is nice when you're doing it remotely for a client. Also of note when you're doing it remotely for a client, remoting into their PF Sense and going in and seeing what the PF Sense can see on the network via ports. It's also really helpful to see if something else is blocking. So a couple different options here that you have. Another really helpful tool is Diagnostics PF Top. This allows you to watch all the connections and where they're going and what they're connected to and the status of those connections. Now it supports a lot of modifiers. And for example, we've chose to just show the host 192.1683.225 and seeing anything that's talking to it, which of course is right now just this one computer, 172.161630. So let's go ahead and SSH into it for my computer. So we establish another connection. All right. And now we see 172.16169. My computer has made a port 22 connection SSH into that system. There's a lot of modifiers that you can add into this. Some of the other modifiers you can use are ones such as and and not. So if we say host 192.1683.225 and not port 443, we can now limit it to say, all right, only this SSH connection, or we can invert that and say not port 22. So we can see all the different 443 connections or any other connections we keep making to this particular system. So by filtering these back and forth, you can kind of drill down so you can watch the connections and where they're being established as they go across the different networks. We can also filter this by protocol. For example, we take that same host and we add PROTO ICMP and we'll drag a window over here and we're going to ping it. And as soon as we start pinging, then we can see the results in here going from my computer at 172.1669 to the 192.1683.225. Go ahead and stop the ping. Now you can also flip to this for things like protocol TCP. And there's other wild cards you can add, for example, we could change this to a full network by putting in net 192.1683.0 slash 24. And now we can see everything is going there. We can also filter further or drill down further by typing destination net dst net, and then the IP address range that we're looking for. You can also put SRC and this will let you choose the source network. Now nothing's coming back from that network over here at the moment. But we can actually fix that really quick. So now I have this device pinging one of the other devices on my network. And now we're seeing that traffic go across. The next really helpful tip is dealing with firewall rules and whether or not there's any active states using that rule. Sometimes you create a rule and that rule isn't matching and you'll see the states going somewhere else. But if we look here, we can see that this current firewall rule has states equals zero and state creations of zero. So we're going to go ahead and SSH to that particular server refresh the page. And now we see there is one state created for a connection going across just a really simple little thing that not everybody looks at. But when I'm looking at rules, this is where I go, well, that rule doesn't seem to match. We can prove that's not matching by establishing a connection that we think should be matching and then seeing if it shows up in that particular rule or if it goes to one of the other rules down here. For example, we see this rule has thousands of states because this is like the catch all rule. This particular network has no states on this rule and no states on any of the other rules. So there's only one state because I only created one SSH connection to this particular server that's catching this rule. Just an easy quick look for when you are matching rules. Now I've got an entire video dedicated to end top PNG. So I'm just going to touch on it really quickly. And we're going to look at what's going across, for example, just my wire guard tunnel right here and some of the connections. Matter of fact, let's go ahead and speed up the connection by doing an iPerf test across here. Then we're going to go ahead and look at the flows. Let's sort by actual thoroughput. There we go. Jump it all the way up to here. And we can see how much data is flowing across in real time on this particular connection. I can also click to any one of these servers such as this one here. Look at the traffic apps and data that's going across and start drilling down mostly it's ICMP. It's kind of weird the iPerf test gets labeled as target data speed. But that is what it calls that in terms of application identification. You can also go back over here and we'll switch, for example, over to the NSFW LAN. It tells you this host isn't in that LAN, which is correct. It's not. We'll look at the interface here and we can look at the apps overview and kind of see where the data is going. And it'll also let us drill down into any one of these. For example, if we wanted to drill down into how much Facebook traffic is on here, these ones are the ones talking to Facebook. So it's 192.168.1.70. Be able to go ahead and click on that and get more statistics on it. Like I said, I did an entire video dedicated to end top because there's so many different things you can do inside of here to track traffic and track history of it. Also, if you may have noticed that the host name is Pixel 4 XL, I happen to know that particular device is my wife's Pixel 4 XL. And she's undoubtedly on Facebook or doing something clearly that is pulling a lot of data and watching some videos or TikToks or whatever on there. Kind of get the idea here, but this lets you drill down and find those traffic flows. I have my video to end top link down below where I go more in depth on this and setting it up and configuring it. But it's a great plugin built in to PF Sense. Now let's talk about diagnostics packet capture. This is something that I don't use often, but when you have a troubleshooting task, this can be very, very helpful for doing this particular task. When you want to go grab something off one particular network, any particular family address, and you don't want to initially dump everything, but maybe you want to go check that one server again, this 192.168.1.8. And we'll just say a normal packet capture, we're going to hit start. We'll start that capture ISSH into that particular server. So we'll hit stop. And we can see the data from my system going in and the data back and forth from it. We can download that capture, open it up inside a wire shark and look at the connection. Look at the TCP handshake and we can see ISSHed in. Here's all the packets that are going back and forth. This is really, really helpful when you're troubleshooting, especially phones. When you're trying to figure out what connections you're trying to make outside of the PF Sense, being able to throw the IP address of one particular device in there and capture it or a particular port packet length is really handy. Of note, when you're doing this, make sure your PF Sense has room to store this data. So you can have it do different log levels, normal, medium, high, full, but make sure you have enough data if you're storing it on there. I've done a video also on using SSH and WireGuard directly with PF Sense. That's something that's supported as well. So it's nice to be able to grab a few packet captures like this, but you can use via SSH, connect your WireGuard system right to this. So you can WireGuard right in and capture all the packets directly and do your own filtering on there so you don't have to use any storage on the PF Sense, but having this built in so you can download the captures and the standard pcap file also really handy. Now let's talk about status monitoring. This also has its own dedicated video because there's a lot you can do, use a lot, you can edit on this. And it's a great way to look over time at the packet loss or quality of service you've been having with any particular connection. We have a wide open West connection. We have a Comcast connection. My Comcast connection as of late has been very troublesome, as you can see by the regular packet loss on here. Now you can go here, click the wrench and we'll change it from instead of a day, let's go to one hour, update the graphs. And we can look at the packet loss by itself, delay averages by themselves, any one of these. This is already built into PF Sense, so it's nothing you have to install as a plugin and it starts tracking immediately. You just have to change these settings. For example, to left access quality, you can actually overlay this with other pieces of information if you're trying to correlate that. I break that down in my video on this topic. But it's just a really simple way to go look, all right, has this connection had some packet loss 6% here, or look at it over different times, like an eight hour, one year, well, we'll have a year's worth data, we can do a month on here. So we'll update the graphs to a month. But I like the breakdown to be an hour, we can change it to a bar graph, update it. And we can see it based on these different pivot points. So this is a great tool for tracking over time, things such as packet loss or quality problems you may be having with a connection. I think worth mentioning are routes. This is something where sometimes there can be a lot of confusion when you're setting up routes because this is how VPNs know where things go. And if you do not have when you have separate networks, for example, this 172 1669 network is at my office, not here at my studio. So it is routed over this particular gateway. And this gateway is called ton WG one because yes, it is a wire guard tunnel, not understanding where your routes are going are all those things that can really lead to a lot of confusion as to why the VPN isn't working. So we want to look at the 3.0 network. For example, let's go ahead and update that. And now we see this one also is routed over that. This is one of those simple things. But it's one of the first things I check when someone tells me either their IP sack or wire guard tunnels aren't working to get something over to the other side of a VPN. Just go in here to the diagnostic routes and take a look at it package IPERF is great. We have it either as a server or a client setting up as a server, you would simply go here to server and hit start and run IPERF. I've actually got IPERF running on my other PF sense, which is at 105555.1 across the VPN. What I want to do here is see what type of bandwidth I'm going to get if I send this data across. So let's go ahead and run the IPERF client. And here it gives me the results. Now the advantage of running IPERF right on PF sense means I don't have any other factors that I'm trying to sort out. So when you have PF sense connecting directly to IPERF on a machine, whether that be a Windows or Linux machine itself, or in this case over VPN talking to another PF sense, I'm eliminating any other factors that may slow down this connection or be more things to troubleshoot. So as you're troubleshooting each leg of the network, this is just really helpful to do kind of a raw speed test. I didn't think they needed to be covered in this video, but there's plenty of other little details you can look at on the diagnostics page, you can ping things kind of an obvious if you can ping something to see if it's online. There's different tools just to give you statuses of everything. I mean, those are nice because I'll look at them on the dashboard and see if the service is actually running. But for the most part, I don't go to those too often once we establish that the service is running. Most of the time when people contact us for consulting, it has a lot more to do with the troubleshooting related to why is this data going here, why doesn't this router or VPN, my VPN didn't start, etc. So hopefully these diagnostics help you on your journey to figuring out why something's not working. Leave your thoughts and comments down below as to you know, what more you'd like me to cover around this topic. But I think I covered the ones I primarily would use in any one of my troubleshooting scenarios. And of course, the more in depth videos on things like end top, which is just such a cool plugin is linked down below. And head over my forum for a more in depth discussion. Thank you. And thank you for making it all the way to the end of this video. If you've enjoyed the content, please give us a thumbs up. If you would like to see more content from this channel, hit the subscribe button and the bell icon. If you'd like to hire a short project, head over to laurancesystems.com and click the hires button right at the top. To help this channel out in other ways, there's a join button here for YouTube and a Patreon page where your support is greatly appreciated. For deals, discounts, and offers, check out our affiliate links in the description of all of our videos, including a link to our shirt store where we have a wide variety of shirts that we sell and designs come out well randomly. So check back frequently. And finally, our forums, forums.laurancesystems.com is where you can have a more in depth discussion about this video and other tech topics covered on this channel. Thanks again for watching and look forward to hearing from you.