 Time here from Lawrence Systems and we're going to talk about HAProxy, PF Sense, the ACME protocol lets encrypt So there's a lot to talk about here It's gonna be a little bit longer of a video. If you want to learn more about me and my company head over to LawrenceSystems.com There's a higher spot up at the top. If you want to support the channel on other ways There's affiliate links down below for deals and products and services that we talked about on this channel and This has been a popular requested subject and it's actually a pretty awesome feature built into PF Sense. Now HAProxy if you haven't heard of it is very very diverse. We're going to talk about a specific use case of HAProxy and which is SSL, TLS, offloading mode and working as a reverse proxy because a lot of people get sick of those You know self-signed certificates and the click-throughs and things like that but not all devices have easy ways to get certificates installed and Some devices may not have ways behind your server to get the certificate installed at all Therefore one things you can do is put HAProxy in front of it combined with let's encrypt search and have automatic free renewing certificates While also having security for the outside of the network now Let's talk about the layout real quick here and a couple things to get out of the way One this is running on a net gate PF Sense SG 1100. Yes, this tiny little box So we're going from the internet to the SG 1100 We are going to be setting up my NAS right here Which has a self-signed cert which is running free NAS and it's called Purple NAS sync thing also running a self-signed cert Now both of these are HTTPS and to make the demo diverse I set up a couple other web servers that are not running any type of certification or encryption So we have a couple jails set up inside of free NAS here One's called as command and one's called Nova prospect and both of these are just running port 80 No, HTTPS. No SSL TLS. No type of cert of it certificate But what that means so I can get this out of the way is that you'll be able to see The traffic if you were inside the network. So all these are on the same flat network. This is really boring setup I had configured at my house just for this demo and You're able to see traffic on the inside here But all the traffic that gets out here will be encrypted And this is a popular use case for this not only because of the certificate problem But let's talk about real quick. There are some jails in free NAS that have this problem. They don't come currently With their plugins next cloud being one of them set up for SSL or TLS So if you were to build a SSS or TLS jail and this is part of one of the examples I just didn't feel like setting it up at home Um You could load next cloud here now granted internal of your network would not be encrypted But once it gets outside, you'd have a self signed or not self signed but a proper certificate from let's encrypt Issued here. So the TLS encryption is happening here and there's no encryption on the local network So yes, someone can sniff the local traffic here It's a self sign search. So at least it's encrypted but it is self signed And we'll be addressing that when we set up the HA proxy So you'll notice there's an option for HA proxy because it can actually act as a proxy for fully signed Certificates and do offloading. There's a lot more features than I won't go into complete depth on here But it can handle it But one of the things we're going to be doing is because these certs are self signed We're going to be telling you to ignore those self signed certs and I will comment real quickly This is impressive that the SG 1100 handled this as well as it does But if you want to scale up and you want to do this in production With high volume, you're going to need a faster box HA proxy Will do wonderful for the all the testing we're doing right here on an SG 1100 in its small footprint, but I see if you're putting this in production You're going to need a Faster heart piece of hardware in order to do the offloading because well It does take a little bit of processing power and memory to start running a more elaborate setup Now it does do I will note as well load balancing, but we're not doing any of the load balancing today That's going to be a little bit more of an advanced video Maybe I'll do that later depending on how much demand there is for it, but that's more into the corporate side The other thing I'll note is yes, you will be able to see my public ip address at home for this setup Someone always likes to know when I've Accidentally so to speak leaked that out, but not a big deal This is designed to show you all of this matter of fact one of the things we're going to start with is Right here in digital ocean We have this domain we use called the detroit yodelingcompany.com Well, just detroit yodelingcompany.com and I have some records set up and these records right here Are the a records we need to get lets and crypt working And we're going to start there and show how acme works inside a pf sense the automatic certificate management engine How it gets certificates, but it does start with before you can even think about certificates having your proper a records for validation now I like digital ocean. They have a really solid platform By the way, there's an affiliate link down below It is much appreciated if you want to sign up that you use my affiliate link It does give me a few extra they don't give me money. They just give me Credits to use for hosting projects and things like that. So I don't have to pay money So it's kind of like money. Anyway, it's just making sure i'm clear on the disclosure here So this is my public ip address and I need these a records pointing. So all of these domains work But if you're doing this and you don't really want things publicly exposed You don't necessarily have to We're going to be using dns domain validation and the dns validation with the api key and digital ocean makes this really really simple So let's look at that real quick. So here's all the different ones. I have in here purple nas home To Detroit Yodeling company calm sink thing to Detroit Yodeling company calm Ask and these are all dot Detroit Yodeling company calm right here all these certs And they have all been recently issued because it's been setting it up for this video And we're going to go ahead and add one more in the list here So test two is not in use right now So test two we can add and we'll just walk you through these are all set up But I'll walk you through how you add one. So start here go add and we'll just call it test two for youtube youtube demo count name homestead now Quickly here not a big deal to set this up. You can Decide to set these up when you add Name let's encrypt testing you can do the test and then you can move it over to The actual setup put your email address in your account key create new account key. This part's pretty straightforward. So if we Do this right here. It's going to pull a key and then we register with it This just registers an email address and there. Yes, this is a You know partially exposed private key and I'm not worried about it We're going to not save this particular one, but you do have to have an account with let's encrypt It's free to set up. You just do it right here. You don't got to do anything else and I can register it So here's the one I already have it registered right here. I just called it my house home cert key Really straightforward to do that that not a big deal Once you have that then you can start adding certificates And the final thing is making sure you turn on the cron entry so it automatically renews these when they Expire will get close to before expiration. It'll automatically kick off a job inside of pfSense that will renew them So now let's go ahead and add that youtube demo one youtube test to demo oops youtube demo status active private key no No worries here Mode enabled domain name. Let's go back over to digital ocean domain name is going to be Test to I like that they put a little copy button just makes nice and easy doing this So domain name we put that here How do you want to validate it? Well, this is awesome because they have so many different things in here Web root web fdp standalone hdp standalone tls dns manual This is cool too if you do a dns manual because what dns manual allows you to is actually just put in and I I didn't go through the whole demo on this but that's what this record is here This is what a dns manual looks like it puts a txt a text record in To your dns for the challenge response So there's a lot of different ways to verify it and there's a huge number of providers in here So we go through all the providers And you can see there's just a ton of different companies if you're a cloud flare user cloud dns If you're an azure user an amazon user really easy to put any of these in there And by the way, yes, you can run pfSense in azure or in amazon behind or in front of your infrastructure So everything behind it can get certificates and that is sometimes how this is used So if you were to spin this up and then you were hosting all your stuff in amazon or azure And you wanted this to handle the ha proxy ssa offloading This is still the same process you'd follow I happen to be doing it on an sg1100 at home But the concept is exactly the same So let's get over here to digital ocean now if Well one last note here if you just do like the local folder standalone It does require that you open some ports because it will do the standard type of verifications But I prefer the dns verification right here. So we're going to choose Digital ocean We have to put in my digital ocean api key this i'm going to blur out Uh, I will not have throw this out there in the public even though when i'm all done This is all getting destroyed But the digital ocean api key is pretty easy to create They've got plenty of work instructions on digital ocean But what these do is when you're using any of these other providers They're are going to give you a specific api key So it's like even though there's a lot of different options. They all kind of work the same sometimes It's api keys a secret But what this does is api key keeps this private Um In terms of don't put it out on the internet don't paste it somewhere public But you put this api key in there and it will reach out and Confirm that you are the owner so pf sense using the api key reaches out to the ocean reach out to So it's your krypton is going to sign the certificate here Based on that information based on the digital ocean api key. So i'm going to copy the key real quick and Save it in here, but i'll blur that out. All right It's saved but not issued Quick note. I did have to do this. Uh, you can't have spaces in there when it I hit save I had to take the space out for test 2 demo in case you're wondering what the difference is between the way It's saved and what you've seen it now all's we have to do is issue renew So currently you can see the last renew date for these and we're going to go ahead and issue renew this one And it takes a minute It'll spin and it'll do the certificate renewal for the test 2 demo Now once it's all done It does take a minute you get this right here and it just lets you know that the whole key is installed And it's all set up configured inside of here in the acme service So we can go back over here services acme certificates And now we have this one and it's issued today wednesday march 11th 2020 at 904, which is the right time So that is now done now The plugins and everything that are loaded loaded here. I'm going to show you real quick on the package manager I didn't mention this at the beginning, but make sure real quick. I did load This as a plugin in case you didn't see under services the Acme plugin. I did load that and the other one we did load is ha proxy So though that's the two we're using here now. Let's go over here to system Certificates or certificate manager under system And we can see all these certificates and where they're set up and where they're installed now by default web configuration default does have the service certificate of Self signed so that is from when you set up pf sense It'll have it and I do have another shirt. I have in here for my open vpn server But you may have noticed at the top it says home.detroityodelingcompany.com colon 10443 How does that work really simple you go over here and we go to advanced And instead of using the self signed cert, I'm using the home.detroityodelingcompany.com I can use any of the certs that are in here. There's even the test to demo if we wanted to change it what this allows us to do is Not deal with self signed certs on my System here. So I'm accessing it via the public internet home.detroityodelingcompany.com colon 10443 As you can see here one more thing that's important or pfSense will not let you log in Is the alternate name so home.detroityodelingcompany.com is the alternate name when you're setting this up for a self signed cert for The system itself it goes right here. I don't recommend opening your system To the general public like this. It's generally not a good idea I did this in the way my rules are and let's go over to my rules real quick So firewall rules you can see I have a rule right here at the bottom for lts remote web access And the source list is lts office This is filtered and actually this entire demo because this isn't you know, obviously live on the internet so to speak I have it all filtered. So it specifically only is allowing My office to communicate with it. So it's not open to the greater public so to speak Any of these connections matter of x we're going to be using 443 for ha proxy These are all not available to the greater public. They're all filtered So the only source is the public ip address of my offices is allowed to access this for this demo So if you were to go from let's say somewhere else to this ip address, it won't work So we have The firewall rules all set and ready. So i'm able to access things From my office here to my house for this particular demo But one other thing of note if you're doing this and you don't want public access You only want to have your own local servers for your own purposes and this is something we use as well You can set this up. So you have it as a reverse proxy for local things without any public access We're doing it with public access because I wanted to show the entire process for it But if you want you can delete these and if you're local it will automatically wrap them around provided you have net reflection Turned on inside of pf sense, which is pretty easy to do under system advanced Firewall and nat And that reflection mode is on pure net what this will do is allow the system to have no problems with um Working so when you put the public ip address in even as far as like, you know home That's just right. You're linked up in com But you're inside your own network with nothing open to the external world It goes. Oh, you must want that internally and you'll see that when we get to ha proxy What it's doing is reflecting back in because it goes. Oh, we can return this internally So just a note on if you didn't want to public expose thing It's not necessary As a matter of fact, you can build like your own if you had free nas dot your domain dot com And you just didn't want to expose it to the world Which is a really good idea don't publicly expose free nas Unless you really really want to but then you could just reflect it internally without any actual public exposure using ha proxy But then you could still have the not dealing with the self signed cert You'd have a properly signed lesson crypt cert. All right Moving on to the ha proxy part now now we get to the fun stuff But that's pretty much all the acme cert works. Like I said, it's easy enough to set your certificate here But how do we set it for all the servers? So let's go back over here to ha proxy and start working on that part of it So we have two pieces that need to be set up with ha proxy You have the front end and the back end and can only give me imply you actually got to set the back end first So here are those servers purple nas syncing ask man and nova prospect Back over here. Here they are again purple nas syncing nova prospect etc These have no encryption no encryption self sign search self sign cert. So how does that work? go back over here And the reason this one's grayed out and these ones are here is because when they're in use They are showing like this. So let's go back over to the back end and take a look at it So active nova prospects nova prospect address plus port so address 192 168 1.40 port 80 not encrypted. Don't check ssl And what certificate should we use? Well, we're going to tell it to use the certificate that matches this So when you're adding the certificates, we're going to add another one just to show you but it's pretty simple You would go here to the back end when you add a server You just choose whichever certificate you wanted to use who's the ca? Well, let's encrypt is going to be the ca we're using and here's all the options So when you add new servers really simple to do you choose the one that matches which one you want to use And to keep the naming simple we named each one the same Now there are more advanced ways you can actually reuse the same cert and create really specific rules So based on certain things it gives people different search based on Rules in h.a proxy we'll get into that But we're keeping this on the simple side just to get you started with it So when you create a back end and we'll look at this one here too So here's purple nas so active purple nas address plus port is the forward two options Which pretty straightforward, but you can do you know different ways to do it like I said Here's the server address port 443 And this is left unchecked because it has a self-signed cert We don't want to checking the ssl cert because we was going to find it's invalid. We it's a self-signed cert So but we are saying talk to this in an encrypted fashion So when it talks up the purple nas I already have this set up and I have encryption So if we go and look at my free nas, which I'm logged into right here See it how invalid Organization LTS cert it's just a self-signed cert. I created when I turned on the encryption for this So that's why I get that error when you go to it and you get the self-signed for your connection The site is not secure because well I self-signed the cert So you don't that's why you'd want to checking go back over here to the back end the same thing with sync thing I have sync things set up Same exact thing sync thing happens to run on port a 384 and I turned on certificates With turned on tls ssl with sync thing, but it generates its own certificate It just does that automatically when you turn it on. I've got other reviews of it So the same answer we want yes to talk to this encrypted, but don't Check the cert because well it's invalid Then we go here and just to confirm once again. No ssl checks No encrypted ssl and we're talking on port 80 on these. So if I go to these there's Go all the way over here HTTP colon slash slash not secure no certificate as you can see Site settings nothing in there. So it works it works perfectly fine, but no certificate completely unencrypted normal traffic. So Back over to how this actually works. So once you have the back end all set up There's really not anything you have to do down here Pretty much you can get into like I said a million options for load balancing and everything else You want to put this in there? They've done an amazing job over at pfSense of building in every feature access control list and everything else So you can get really fine grained and create expressions and rules of what works. What can connect It has health checking in there. So you can actually constantly check the servers and use that health checking information to Define the system in different ways for how it does it for example, if you're load balancing different servers Just you can get notifications for mail level mail to who it sends information to Statistics for each one of the front ends etc. Etc. And if for some reason you found a parameter that wasn't In here, yes, you can actually pass through per server parameters for each one of these But for the most part we're going to leave all this at none. So when you Go in here, it's basic. It's just this server Don't leave all these at the fault. Don't need a health check method. It's not necessary for what we're doing here Um, not worrying about any of this Nope, unless you want to get email notifications per server pass you all these are just left empty. So it's really really basic setup to get Get you started on this Now the front end I made it a little bit more complicated because I have these all set up So what we did here and we're going to add one more to show you what it looks like I built the first piece of the front end is going to be the Azkaban Now what that means is we're going to tie it to the way on addresses a little support port 443 Check the SSL offloading because we're offloading SSL because this is handling it and what type of Offloading HTTP, HTTPS offloading just like it shows right here. So the client talks to HA proxy and then this Talks the server and this is an example of SSLTLS offloading It's like they have in green here where it says clear traffic That is clear traffic here, but ciphered encrypted traffic here And that's actually what's going on with this particular server So by default it grabs this Then we go down here And this is where we're going to skip over and ignore this part first because I want to show you how this works So the first one we added was the Azkaban. So I go over here And if we pull it You see what this is is open SSL client server name azkaban.detroit you're only coming down. That's the sni name We sent what's the host 69 14 6153 port 443 And then I just wanted to show the subject line. It pulls the subject and shows assert. What does that look like in action? We have a fully find valid certificate at azkaban to try to destroy your link coming to comp So completely working certificate right here and valet The next one is this one now We are both pulling exactly the same ip address But when we send the sni here of this in the browser nova prospect dot detroit your link coming dot com We yield a different search. So let's go ahead and change that. So if we go here nova pros nova prospect dot detroit your link coming dot com We see we get a different cert sent to us and What this does and now we'll go back over to ha proxy Is it goes through here and it says hey, how do we how do we get these going? so we get the Nova prospect when the host matches Nova prospect at detroit link dot com it says send this so it's going to go here host matches This is the sni sent so the the server name information That's sent by the browser and it says when this is sent and it matches this rule send this certificate Now there's two pieces to this first rule access control list nova prospect host matches nova prospect Link on com use back end. This is an important part condition name nova prospect use back end nova prospect So what these rules are doing is all saying when it matches this use this back end the defined back end over prospect Send it over there and the same thing with the next rule down It says sync thing that the host name matches sync thing that detroit your link company dot com So look at that one real quick do the same thing We'll put sync thing right here We pull sync thing dot detroit your link company dot com and we have the valid certificate there Now sync thing if you remember is on port 8384, but we're redirecting it to port 443 That's defined right here. It says no matter which one of these rules we have always Send everything over port 443. So there's sync thing. There's the nova prospect. So the back end use sync thing Now the last little piece that's really important Is down here. We had to add nova prospect and sync thing as additional certificates That's important because you want to make sure these certificates are available So the front end needs to know what certificates need to be pulled based on the sni information And it's pretty simple because the default box is checked and we're going to add one more to walk you through the process one more time Also, we have to have a default certificate the as command one So if we don't send an sni information that matches any of these acl rules, what certificate are you going to send? Well, that's what that that's an important part too. So we go over here And we up here again So sync thing and nova prospect are both valid. So if we pull them But we put in gibberish Which doesn't match anything It pulls the as command dot detroit your link company dot com because that's the default service So if I can't match any of these other acl rules or conditions, just go ahead and send the default cert and if someone were to Go to this website and we'll just do it real quick And you head over here and we put https colon slash slash Not secure Proceed service unavailable because nothing matched. So it says I can't send you anything Because you don't match any of my acl rules. So even though the common name was this it still doesn't send me the actual This right here because it didn't match dsail rule So people who don't know are just probing at port 443 on this particular server If it was publicly accessible They would just get service unavailable because they didn't send an sni name So you could build some other catchall rule if you wanted to but for most part if they're hitting 443 You don't need to tell them a certificate or anything I mean tell them certificates so they can figure it out But it doesn't have to actually feel out fill at the back end because well nothing matches So One of the other things that doesn't match right now either and the next one we're going to add is Go back over here purple nas Does not pull it. So purple nas doesn't exist yet. It does exist Because we have it here. It's pointed there, but if we went there, it's not giving us the right certificate so what we want to do is publicly expose and back over here and we're going to go ahead and add one more to this So let's Walk back over real quick. Look at the back end And we see the purple nas back end already set up, but it has no front end tie in so let's tie it to a front end We're going to an edit Leave this the same up here at the top and let's add one more What's called purple? Make sure we got the right name hit the copy Typos are the killer copy and paste whenever you can Now when you're saying host starts with ends with matches regs host contains There's a ton of little things you could do for custom acl rules Matter of fact, you can Fine-grain this where if they hit certain sections of the website it can direct a different service Like I said, this has an amazing amount of options on it But we just want host matches because we want to say when the host SNI matches purple nas. Detroit loading company.com We're going to create this rule access control is purple Now it doesn't really matter. I can call this really anything But whatever I call it and it won't auto fill this. So if I go down here Use back end back ends going to be purple nas I have to make that whatever I put here match right here. That's the important part So it doesn't auto fill like I've already done sync thing before and it doesn't auto fill it So you just gotta once you get a copy and paste So purple so now we're saying first part of the acl rule And then the action from that rule. So when this rule is matched purple host matches purple nas. Detroit loading company.com use back end sync thing See below use back end back end purple nas awesome, but we're not done yet We don't have that certificate added. So we got to go down here and say Purple nas we set to make the certificate available. That's all we had to do and this box is checked It says add acl certificate for subject alternative names. So now this is just about The bottom part here and it's not always that you're doing this unless you are trying to do something with a different certificate name So, uh, but for what we're doing right here. Yes, this is exactly what we want. Let me hit save and apply changes All right, so we have the same public ip 443 And it's ss off loading and purple nas right now as we know has this little self sign certificate But now what we should be able to do Is go HTTPS to the purple nas And there we go. We have a proper self sign certificate Mail insert for purple nas going right here. So I can log into my free nas Now, what if you wanted to put it on another port? I'll show that real quick and We go here I'm going to add And if you've seen when my firewall rules, I have another port open 12443 purple nas HTTPS offloading no need an access control list just run through Basic options choose the certificate purple nas There's no additional certificates, but if there were you check this save apply And what we did was we just reuse the same back end and now we're pointing it at 12443 with the same one. So what happens if we do that? Oh, I did it wrong See what let's see what mistake I made on here. Okay. I see what I missed This is an important one. I missed this a couple times. You have to define a default back end right here Or it doesn't work. So define the default back end as purple nas. I did miss that going through it I think it's the most common one that I keep missing is that particular one And it works really simple But I I think that's the most mistake I made when I'm setting these up And sometimes I forget that it doesn't default to a back end You do have to choose a back end to make this work So kind of gives you an idea You can run these on multiple servers multiple addresses all at once We've bound these two here and they're going to work internally or externally and you don't have to have the way and rules Unless you're doing like I'm doing I'm at my office connecting these to my house to get these working So kind of gives you an idea of how the system works But pretty straightforward to do lots of rules lots of options There is so much more you can do with this like I said I'm touching the tip of the iceberg on terms of this a lot of people just want to do A basic self-sign certificate and this will definitely get you started for that or get rid of the self-sign certificate I should say and do a let's encrypt sign certificate for local servers But by doing it with dns It's one of my favorite ways do it because you don't need to expose any of your infrastructure You can keep all the ports closed You can have it validate just like I showed you with digital ocean or a long list of other places do this And of course if you are building infrastructure in the cloud There's the options like I mentioned beginning of doing this with both azure Or amazon and building some cloud infrastructure with pfSense in front of it I'm doing the same thing handling all the let's encrypts Uh certificates there and it's just a really nice service because as long as you have this set to auto renew It will work perfectly fine It also has and I will mention this write acme certificates to conf acme in various formats Used by other scripts or daemons integrate with severe manager if you wanted to go beyond this or not even run The ha proxy directly in here, but somehow pull those there is the ability with pfSense to I'll have it write out those certificates to grab them and pull them and put them elsewhere also when renewing the certificates Something you may have to run into here Is when they renew On they use to restart web server processes after certificates been renewed add enable command shell command You can tell it to restart services as well if you need to on that way if there's any Problem you would for example restart ha proxy. So we'll do this one real quick here Because until it rereads the certificate you don't have it So I want to mention that for some uh housekeeping that we're going to head and enable And add that we restart ha proxy after we do the renew This is an important thing because you'll go all of a sudden my certificates seem to be renewed But the services are still showing the old ones just restart the service Well, obviously the goal is to keep this as automated as possible But use the restart web server process I will mention this that if you are going to do this in production and you don't want to have a Phone call from everyone going hey all my certificates are showing expired You will have to set these up. So restart a local captive portal instance restart the GUI ha peer I'll restart the firewall shell command There's different ones you can do because you do want to renew the cert for example where it says home Detroit only coming at common like we showed in the beginning you may want to restart that as well If you're using this certificate because it's just not starting starting the whole pfSense system It's just restarting the specific web Web GUI here or the ha proxy one and I will might do a further video on this later because The next question comes up about one of the things it has in here Is the captive portals you can use these when you're building captive portals To use a certificate for those because sometimes you want a captive portal and if you're capturing username password on there You probably want that encrypted even if it is on the local side because you're using it for some validation Yes, you can use these same search for the captive portal may be a future video I do but the same answer about once you do that so it doesn't expire You do want to have that service to start so restart it so i'll leave um A link to if you obviously the pfSense documentation is excellent on this topic where it shows you even more options There's also a couple hangouts videos That from neck eight themselves that i'll link to as well that go through in some of the different topics in depth on on this So it's a it's a pretty fun thing to play with it's definitely an amazing feature built in the pfSense and it Really does get rid of all those problems You can run to with having all the self science certificates for things whether it's a local service You're running that you don't want exposed or one that you do want exposed But it's definitely a great way to handle all of this and keep it all nice in one place and makes life a little bit easier All right, thanks