 PF Sense supports high availability. PF Sense is capable of having multiple nodes act as a cluster for high availability. This article is a brief overview of the hardware redundancy chapter of the PF Sense book. So I'll leave links to all this. This is just the publicly available documents from NetGate. And I highly recommend reading through all of it if you really want to get a deep understanding of this. They also, and I'll leave a link to it, have a Hangouts they did. I think it was about a year ago. And they offer a also in depth guide where they walk you through high availability and talk about scenarios related to it. Now, PF Sense does high availability with the Common Address Redundancy Protocol along with a few other things. So it's actually kind of a collection of services that they have. So they have the CARP protocol, the Common Address Redundancy Protocol to facilitate seamless switchover when the master fails and goes right over to the backup. It also uses XMLRPCSync and it uses PFSync. The PFSync is so not only are they sharing an IP address so the IP address to stay up when one of the servers goes down, it's also syncing all the state tables between the firewalls. And that's very important if you want seamless failover because if you fail over a firewall and then switch over to another one, even if it has the same IPs, there's a pause because there is systems with those states and are expecting those states to be there to continue the connection from one side of the firewall to the other. If you don't have the state tables synced, it's very much not seamless. All the routes have to rebuild. And that's what they have is that PFSync. So the backup firewall really isn't doing much. This is not like load balancing. That's completely different. This is just a hot spare ready to go and all the configuration is synchronized, the shared IPs are synchronized and the state tables are synchronized. Read through all of this and they have a lot of details on troubleshooting it and all the little nitty gritty if you have specialized setups and things like that. We're gonna do a pretty basic setup on this. We're doing it in my lab. Because we're doing it in my lab, we're gonna have to make a few assumptions. I do not have a bunch of public IP addresses available. So this is our lab setup and we're just gonna pretend, use our imagination here that 172.16, 69.25 and the 172.69 network that's gonna act as our WAN. I know it's a private IP range. Someone likes to point that out in the comments. Yes, I'm aware. This is for testing. So here is the first PF sense configured with this 172.16. I'll fix that. It's actually 69.112. And type this one wrong as well. This one is gonna be 69.242. And then this is the common IP between them for the other side of the network. So we'll get these details in a second, but this is gonna give you an overview of it. Now, couple of the requirements for high availability. That's a really important part of this. For a cluster to function, a few things are required. Minimum of three IP addresses per subnet. And I brought that up and that's why I went and talked about that. There's a couple different IPs on this and this is where it can be a problem for some people because even the WAN has a minimum requirement of a slash 29 network because you have to have a public available IP address for each one of the cluster. So the master and the backup have to both have an IP address and it has to be public. It has to be on the WAN side. Then they have a shared IP address. That's the third IP address they need public. So if you are running a single instance like a single WAN IP and you're going, but I wanna do this, you're going to have to get two more IPs all in the same subnet in order for this to work. Now the same rule applies to the private IP side, the LAN side and every network you create subsequently in between. They all required as three IPs, one for each of the PF sense in the cluster and then the third one is the one that all of your devices will use to do all the routing. So that is an important aspect that you have to do that and repeat it as you create different networks that you want to be part of the high availability. And if you have them not as part of the high availability and one of them in the firewall goes down, those networks will fail that you didn't set up this way. Layer two equipment has to have properly handled multicast, a dedicated interface for state and configuration and synchronization. So we have those configured as well and make sure you're upstream, ISP, other involves routers that properly respect the addresses used by CARP. So when CARP creates an address, it's a unique address shared between two computers at the same time which you're going, hey, that doesn't work well. They've got a special way that it does work. I'm not going to dig into the details, you can read about it, but the CARP virtual IP address is shared between two devices because that's what you want to give your devices to use and the two machines figure out who actually handles that IP address based on the master handles it unless the master fails and then it switches over to the secondary system, the backup system to use that IP address. So it is important that your ISP doesn't interfere with that and it kind of goes back into the layer two equipment multicast. Now you may have problems running this in a virtualized environment. If you're running it virtualized, sometimes you have to enable promiscuous mode and there's some more troubleshooting related to problems you have running this in a virtual environment. Ideally this is going to be on hardware. And speaking of hardware, they do sell HA configured systems from EF Sense. Now when you're building an HA system, the other requirement not listed there but is that the best way to do this. I know there are other ways and some will point out but you can do this. Yeah, there's kind of workarounds but you ideally want the machines to be identical. If they're not identical, there are sometimes problems because it wants to sync the state table to, let's say when is IGB zero, it will want to sync IGB zero states the other way. There you talk about in their hangouts, ways to work around that but obviously the more ideal situation if you have two firewalls that are identical, bought at the same time and put together in this HA cluster, you're just going to have less problems and less troubleshooting and less workarounds that you have to do. So, and generally if you're planning an HA system, you're not going, I'll buy it later, just get them all at once and configure it from the get go. So let's go back over to our configuration. Here is the virtual IP we're going to use for WAN. Here's the LAN and like I said, here's the master. One, one, two is going to be the master. Two, four, two is going to be the backup. These are both slash 24, that's, they're all in there even though they're not close in range just because it's a slash 24. Obviously you usually have, if you're doing a slash 29, a nice little narrow range, but you get the idea. Matter of fact, all these networks are going to be slash 24. Now, physically the way they're plugged in, we have a switch that will, it's all virtual because this is my lab but you'll plug the two WAN sides into a switch and the two LAN sides into a switch that handles multicast and there's some troubleshooting if you have a managed switch, which is not required if you do have one. You may have to make some adjustments like turn off, I think IGMP snooping has to be off for this to work but they have, once again, read through the documentation but this does not require a managed switch. It just requires a switch to work. Now, the sync doesn't require a switch at all. This would be a network cable just between the two boxes and you dedicate a port to it, whatever port you do and just make it the same on both of them and say this is the sync and what the sync is, is a system by which all the data transfers back and forth. The configurations are pushed this way, the PFs sync as in all the state tables get pushed this way and it has nothing to do with the carp side at all that's not tied to this interface but if you have it as a dedicated one you don't have to worry about anything slowing it down. Technically, you could tell them to sync between the LAN side but now you have all that traffic and all that sync traffic also clogging up your LAN just take two ports on it, one for each firewall and dedicate them to sync away you go. Now, because there's not really data traversing this as much as there is like it's not like the feed goes through the sync to get it over to here it just synchronizes all the state tables but that can still end up being a lot of traffic and you don't want it interfering or causing problems with so you want these two things to be clean and I just bring this up because it's kind of an important aspect to make sure that they're syncing in time because if they're not the failovers wouldn't be seamless. And if you use something like the 7100 that I showed from NetGate that has extra ports just take a couple of them and dedicate to it and like I said, no switch needed it's just a direct cable into the firewall. Now for clarity, this is the 112 server this is the 242 server. The theme I set on this one to the light colored theme this is the backup we will be using this one less other than setting it up with a couple of settings we have to put in here most will be done in here this is easy way to distinguish between them so whenever you see me in the dark theme versus the light theme this is going to be the backup server. All changes should start here at this server and then that way they're pushed over to the other one that's an important aspect of it even when the other one becomes the master because if this one fails you still want to make sure all your changes start always in the master and if the master fails get it back up and running and fixed and then still push changes that way because any changes done in the backup server are overwritten if they're part of what's synced from the master it'll just keep overwriting them for example firewall rules. All right so let's get started on this as I said the prerequisites are too cleanly loaded don't put a bunch of stuff on it just clean load a couple of PF sense boxes or buy them buy a pair from neck gate come up with an IP range that works for you so you know where your IPs are going to go we have the LAN 40.2 and then the other one is 40.3 because we want our LAN to be 40.1 that's going to be our carp address so we have to not use the same address on there because each of these boxes still has to maintain their own IPs so this is the three IPs for the LAN and then here's the way on 242 and 112 on this one and the sync addresses just can't be the same but as long as we're in the same network I just made a simple slash 24 network with a 10 call this 110.1.10.2 and this one's 10.3 I just did it for naming consistency because these have a two in it it's really your own preference however you want to do it and I named the interface sync so I'm never mistaken about what's going on over there and on both systems we have admin both systems I have the same password this is for convenience for when we set this up and the firewall rules are really simple in both systems I have this because I want to operate from in front of them so I have the LAN opened up on the firewall and allow remote access and then on the LAN just a generic open rule sync a generic open rule as well there's really not much going to be happening here because only thing this is going to be do is plugged in and dedicated to that sync port so nothing real special about any of that like I said they're identical on both because we want to create all the rules once we have this set as master that way they just push over to the other side let's get to the first step on the system that we want to be master we're going to go to system high availability sync turn it on choose the synchronize interface then put the IP for the secondary sync interface so put this here synchronize config to this one and we'll just toggle all you can have special use cases where you may not want certain things configured but for the most part I'm going to assume you're going to use all because you want this to be master and keep them identical but it does have granular options if you don't for some reason admin is the other system's name but in the password save and we can go over here to diagnostics oops our status system logs no errors if you were to put the wrong password you'll get a system notice at work but that seemed to work perfectly fine I didn't get any errors so now we have high availability enabled on here but when you go to like status carp because we haven't created any addresses nothing's defined so there's not much to look at yet but the next thing is in case someone's wondering and I'll do this the user managers are in sync so here is the admin and a time user admin and a time user let's make a sync user maybe we want to change the admin now these are synchronized between here but you maybe want to disable admin and then create users, et cetera, et cetera so we're going to hit save and please note no I did not give that person any permissions come and go back in there add system HA node sync save so the only permission this sync user has is that so we'll do this hit save go ahead and refresh this page and there's our sync user shows right up over here so now we can go to system high availability sync leave everything else the same sync there we go and now this system synchronized using that user now the advantage of that is and why I did this one minimum permissions don't need to add any more than we need to through a particular user and my general rule is I disable admin and create a different user less guessing that way people don't know what user you know not that I think securities are great but at least they can try admin all they want when we're done and we'll disable that so they have to know what username is assigned there just a little housekeeping for doing that okay next thing we're gonna go ahead and do is let's create some carp addresses so we go to firewall virtual IPs add carp now where are we gonna attach them to these are well let's do our WAN side first and we'll do this IP it's a slash 24 it's attached to the WAN now the good news is the virtual password set a good password but you don't have to remember it once you typed it here let's see WAN IP once you've done this hit save apply we go over here firewall virtual firewall virtual IPs away we go it already came over here so you don't have to remember the password and put it back in this as we add the virtual IPs over here they're gonna show up over there so let's go ahead and add now our LAN side choose carp address make sure this is a slash 24 and we try this one to LAN save apply in the you're a lucky person who has a lot of IPs and you want to attach more of them you can I mean if you have more WAN IPs and we'll just add one real quick here we'll add 24 second WAN so on and so forth so if you have a big block of them this is where you would assign all of them on there uh... that you're gonna be using so when you apply the changes and we go over here and we'll just they're showing up over here in the same way I said you don't really have to do much with the backup once they're in sync with each other everything just kind of flows over once you have them synced now let's go over here and we gotta fix the NAT and what we're doing here is fixing we're gonna go to hybrid save and add and this is an important aspect to add right here you want to say a particular address is going to use the carp if not when it's going to the master it'll use the master uh... when address when it's going to be back up to use the backup IP address but this would not create a seamless failover uh... because of the system fell over it would be stuck on over there and all the states wouldn't really be in sync what we're gonna do here is we're gonna choose our WAN IP the carp WAN IP and we're gonna choose the network source zero slash twenty four there's a couple different choices here uh... for this particular demonstration because we're using some of the rfc 1918 private IP ranges for our WAN we're not going to choose that but you can choose like an alias for an rfc uh... network and say all those or you could just create individual groupings or maybe you have a series of translations where you have to have different things going out different IPs there's a lot of different options there I'm skipping IPv6 just I don't have this setup on these particular firewalls so it's WAN IPv4 network and we're doing uh... the 40 slash 24 we only have one network for demonstration but we want to make sure the address translation goes out that set outbound to carp and we'll go ahead and apply the changes I didn't duplicate but you can if you want this rule here if someone's using a vpn that requires ISAKMP that does need what they call static port translation which means don't assign a random port uh... come in and out of in this case port 500 so if you have a firewall need for that you'll have to duplicate that rule I'm just skipping it for now when you do this here for this outbound nat change we're gonna go over here firewall outbound once again part of the rules so it flows right over to the secondary system as well next thing we gotta do go to DHCP servers and this is an important aspect as well we have to fix the DHCP server by default DHCP servers is going to want to hand out the address of the uh... whatever server's handing it out the master in this case so we have to put the gateway in as the carp address this means hand out to the machines this particular carp address the other thing we need to put in there is what's the fail over up here dot three because but here this is forty dot three and we can have two DHCP servers trying to hand things out so failover is failover ip of forty dot three so we go here save hand out the carp there's the failover and now here's the cool thing if you go over here DHCP server it's smart enough to know that the failover IP for this is forty dot two we didn't have to put or change anything but by pushing a setting here of dot three it says hey you're dot two so you're the failover and basically they go ahead and configure that so that way the DHCP works on both and that's something that is synchronized as well is the DHCP leases so there's no lease changing uh... they're in sync on that so when new addresses come in it doesn't say hey this is reserved or not reserved or this is available in the pool because that's kept all in sync so it won't accidentally hand out IP addresses if the master fails that are already in use they keep all that uh... information in there and of course uh... having the failover with the DHCP server means the DHCP server can't can't run on both so this keeps them from trying to fight with each other at all the master hands it out the backup does if the master is not available to hand out IP addresses so now we're gonna go over here to status and carp failover and we see that this system is set up as the master status carp failover and this is set up as the backup and we're gonna go over here and this computer is uh... 192.168.40.107 so if we look over here just in the uh... DHCP tables we can see that computers right here it has an IP so away we go it's good to go it's sitting there booted up and uh... working through our virtual firewall environment so let's create a rule and we're gonna go here to nat this is add a rule so i want SSH into that 107 part 22 now here's the part that's important normally you're used to if you create a lot of rules you're like oh yeah i use no problem in uh... ssh to devian and create the rule but you don't want to create it on the wan address if not you only create the rule for just the master firewall and that's not the ideal thing so you have to make sure whenever you're doing this you choose which one you want in the virtual ip show up right here so we're gonna choose dot 25 so the virtual wan ip we assigned this and hit save apply and now we can ssh into that machine and we'll go here and just to show you again nat same thing rules fly right over now in case you're wondering and i'll do this as a quick demonstration test on backup you can create rules on the backup server and they don't show up here but what's interesting if we just edit this rule at all and i have to create a other role is put x on here apply refresh the rules get overwritten so anything you create in here in case you're wondering will get overwritten if it's part of what's getting synced uh... just be wary of that that's one of the reason we say always do everything in here so let me show you how this works now so we'll ssh root at sixteen sixty nine twenty five and we're in we're into that machine that's behind there though one i two six forty one seven so your firewalls work pretty much the same you just gotta make sure when you're creating them that you create them with the virtual ip as a destination like i said we could create the ip address here we'll just do it real quick to be the wan address but if you did so we'll just do this real quick apply fails to go in there but if we put in one one two in a failover case this would do just no good because well now we're doing it so those ip's are so usable that you've assigned on the wan side uh... for these two systems but they're technically not going to be part of a failover if you create any rules on them alright now failovers are cool but of course what you're probably really looking for is a demonstration of how this works how seamless is it so we'll log back into our debian system and uh... we'll ping something like cloud flares one one one one so here we are pinging away it's working no problems well diagnostics pf top and we can then go uh... host forty dot one oh seven and there we see it pinging out the cloud flare and it's you know everything's working so let's go ahead and drop this firewall let's just uh... forcibly shut this one down so we'll leave this pinging right here so we have plenty of packets h.a. master and we can do a graceful shutdown but what fun is that so let's go ahead and force shut down the system and it's down so we'll go over here status we're doing all this in real time so cart failover it's instantly uh... the master so let's go back over here council it's still pinging and ninety nine packets sent ninety six received looks like there was a slight glitch and we lost three packets but the machine itself is it you know up and running it's fast losing three packets it happens sometimes on a good day uh... so it's pretty seamless on the failover and the system didn't like do a long pause and along you know wait for states to rebuild so it's paying something else i think we can pay in google or you have google pinged uh... forums i think i have ping disabled on that i do anyways you get the idea though it's not pausing it's not like it lost all the state tables in the machine really slow this see it's failover was pretty seamless now even though this is master this is still not the head ends wonder how availability sink this is still not checked it doesn't switch over it's temporarily the master until the master comes back so we're never going to put that machine back up and we'll start this back up and as soon as this machine starts back up it will become the master again and the system will be back up and running so while the system's down while the master's down unless you plan to turn the back of one into being the master don't make any changes to it uh... because those changes will be overwritten once the system boots back up and the process itself is pretty seamless uh... for the switch over as soon as this is up and running and ready to route packets it will switch right back over to this one handling everything and those carp addresses those virtual IP addresses will then be handled again by the master until the backup is needed again and that's it it's as simple as that important things to think about when you're doing this is going to be always make changes on the master make sure you have all your hardware in sync uh... make sure you have notifications sent because it's so seamless uh... people won't know maybe went down so make sure you have some way to monitor these servers and you can monitor them individually because they do each have their own IP addresses uh... so that way you can know if something is down uh... in set you can also have notices sent to you via email via pf sense for any changes or uh... things that are happening so now this one's back up and running status it's in backup mode over here press the page here and of course this one is right back in master where you go so pretty straightforward to do works really really well even i'm using a virtual environment was may not be as ideal uh... but i did virtualize these using this is xcp ng and i virtualize it in the methods that i've done before on the channel uh... but that's all you have to do to make the failover work and set it up configured it's not too hard to do please read through all the documentation because uh... people who need h.a. systems are your uh... once you go outside of lab play right here usually need lots of networks lots of lands and a lot of settings so make sure you read through and understand this uh... this is you know fairly straightforward to get started but gets complicated really quickly and does require some thought and planning especially because for every network you create that you want as part of the failover uh... whether that's a v-land or a regular land you gotta remember the three i p addresses one for each of this the master server the backup server and then the one that you hand out to all the clients that carpet uh... that is going to be that one in between so that has to be repeated for everything you want to be part of the failover and i know i'm us a little bit repetitive saying that but it is a important aspect of when you come to planning and when you're thinking about okay i have to have all this setup sit down and sketch out the whole process uh... to make sure you have it all right and put it together alright thanks and i'll leave links to all this and do watch the uh... high availability on that gate hangout that they have i'll drag it over here i'll leave a link to this uh... this was in march twenty seventeen hangout it they cover it in a little bit about an hour twenty six minute depth of uh... getting it going so i went through the basics they go through the more in depth and i guess in our tfm all day it's uh... important to make sure you understand all the concepts thanks thanks for watching if you like this video give it a thumbs up if you want to subscribe to this channel to see more content hit that subscribe button and the bell icon and maybe youtube will send you a notice when we post if you want to hire us for a project that you've seen or discussed in this video head over to laurancesystems.com where we offer both business IT services and consulting services and are excited to help you with whatever project you want to throw at us also if you want to carry on the discussion further head over to forums.laurancesystems.com where we can keep the conversation going and if you want to help the channel out in other ways we offer affiliate links below which offer discounts for you in a small cut for us that does help fund this channel and once again thanks again for watching this video and see you next time