 Thanks for coming. What we're going to talk today is the Willady demon that runs on OpenBSD, which We'll use as a specific use case, which is application layer getaway So known as reverse proxy in most of the times So quickly who am I? been Ciss admin since the last Well, since the late 90s In 2015 I started working as a freelance as a technical architect, which in France means Doing mostly diagrams and throwing them at operators so that they implement this I tend to quite Self-host every Services that I can I mean like email or whatever. I can take it at home or on a VPS and Most of the time I do it using OpenBSD There's a few things that I run on Linux like master them for the moment, but nearly 99% of what a run runs on OpenBSD and If you ever stumbled to that URL That probably if you're not French who won't be able to pronounce This is where I Write articles about mostly fuss and most of the time it's about OpenBSD. Like how do you do whatever we're using? How do you do this? so What is really these so it's? multi-purpose Demon it's available on OpenBSD since version 4.3 Although it was named host I can't pronounce this thing host stated in OpenBSD 4.1 It can do several things can do a lot balancing could do application like a getaway It can do transparent proxy That is for example, if you want to connect to a remote SSH server you can do it via relay day We're not going to do this today What we're going to do is the application layer get away which basically looks like the diagram on the right What in general and simplified what a networks looks like is that? The user is somewhere on the internet or in the internet or whatever and you've got several zones in your IT There's probably one zone up with the firewall do the all the filtering what's allowed to go in what's allowed to go out Then most of the time. There's another zone someone someone sometimes called DMZ Where you put your hardware like wolf and things like that and this is probably where you would put your relay server and then Far in the in the land zones. You will have your let's call them back-end server whatever Even if those are like web servers and considered front Application server in the application layer, but we'll call them back-end servers and those are where your services run and This is what is exposed by the relay D server So first of all a few things about relay D what you have to do Should do must do is read a few main pages The main page for the demon itself is really D The configuration file has its own main page With loads of things inside. I probably need a few readings to get all around it It's called really D.com and there's a specific utility which calls relay control which Allows you to well control the the demon and that consists of Getting informations on what what's happening on the relay decision and sometimes well sometimes you can also provoke actions like Telling relay D to stop really things switch to other Back-end server, etc. We'll have a look later on The configuration File is named ETC relay D.com. There's an example file that you can get from the ETC example directory And Then when that's all done you can control the demon using those kind of Commands relay D dash DVN Basically will check the configuration of your relay D server and will yell at you if there's errors So that allows you to correct them before restarting the demand Then it stops and then you got no service at all When the configuration is good you can start The D demon so you got the RC control commands enable start stop like any demons in in open VSD and The relay control commands is I've not written all the commands that are all the available comments If there's loads of them, but it's basically relay control give a comment to run and then arguments to that comment We'll have an example at the end of the the slides About the words used in the Configuration one thing to note is that if you or if you're already used to pf syntax and words and stuff like that It's mostly the same with the relay D. So you won't be Surprised with the words they say basically what we're gonna use here is Micro which has which are variables that you can use several times in the in the configuration The tables are like objects where you put your reference server or group of servers and then you decide what to do with those groups The protocol is where you will say You will define what happens to what kind of protocol you have you can have basically TLS protocol HTTP protocol There's probably DNS protocol There's probably one more I forgot Anyway, we are only going to look at the HTTP protocol in this one talk and then the relays are the sections that says Where you listen to Request what you do with it and who are they gonna send it Referencing to the backend servers. So that's the simplest HTTP relic configuration that you can get It's a HTTP reverse proxy. It's there's a really few few lines What you do is you define a protocol of the type HTTP So that's the HTTP first word protocol is just telling you that it's a protocol and WWW is the name you can use you can do whatever you want with the names There may be shoes if you're trying to use underscore or dash there was something sometime that there was weird things happening So just try not to use them and when you define the the protocol is here is Just saying Take something as a request and pass it to the server do nothing or more Then what you define is the realization Where do you tell the lady where to listen to so that's An IP example with a port you tell them to use that protocol that we've just defined up In the configuration and then you tell him to forward all the request to that particular back-end server using the the particular port of the of the server in this configuration We are all using 80, but you can barely use whatever for you once What you call name species? Oh The names don't have to match you can have relay name foo and the protocol name bar The only thing is that your protocol has to have that particle name I've used WWW to just to get an idea of what we're doing But we could we could define HTTP protocol John and say really is like Tom or whatever There's no no problem with this What I call a better simple HTTP really We're using macro here as in pf We got external address which represents the public IP of our servers We define a variable called web host and we get the private IP for this The idea is that it will be usable inside the rest of the configuration We have Basic logings information in this log state changes means that the log will tell you every time A server a back-end server goes up and down For example the table here you define Table name web host and this is the name of the object. I could have also named it John or whatever And you say this table contains one single object which is the web host Server that has the IP defined up there and the only thing that change From the previous example is that now we're listening on the variable named External address Rather than the setting up the the IP directly in the relic configuration and same thing happens for the forward directive it may not make Lots of sense right here because we're only using the variable once But if you want to define a more relays than this and keep using the external address for example Then you will only reference the macro in the configuration and we'll have to change only the Definition of the macro on the top of the file as for now. We've been using only HTTP So what happens if you want to activate TLS the idea is that You activate the TLS on the relay day and these back-end server can still be using HTTP if that's okay for you The thing to do is first of all is to get TLS Certificate which is out of the scope of this talk because this is done using the ACME client and the HTPD server So if you want to get a certificate just look at the main page It's quite in the example files a configuration files. It's quite straightforward What you will get is files like Like the pink names You will get ETC SSL and certificate name dot CRT and the same thing in the private with a key extension The thing is that in the protocol section, you'll have to reference those files and this is what the TLS keeper Function does and what you'll see is that what I'm referencing is relay D dot example Which is the exact name of the certificate without the extension and by default What really do is that it will look for a certificate that has this particular name and adds the extension to locate the service. There are some others Possibilities for naming the certificate. I didn't went into this because could be big Complicated sometimes there's a whole section in the main page that explain what you can do basically you can have You can have IPs or ports in the hole in the finance Then the other change is that in the realization you're not listening to HTP, but you're now using the TLS option and Hopefully changing the port and this is just what is necessary to go from the HTTP from to the HTTPS configuration now Most of the time you will have on Specifically if you self host things but in real environments, you probably have several backend servers And what you want is that if you have three application servers and one goes down you want to stop using it to stop sending requests to them and Concentrate on the two that are still working. So for this we're gonna create Specific table, which has the same name as previously, but we're referencing three hosts, which means by default Really they will will use those three hosts to send information to or HTTP request to and get answer from from them and then in the So there's nothing to change though in the protocol section for yet In the realization. This is where you are going to tell that Related to forward all the request to that particular table and you have options to Let's set different scaling algorithm to tell him Do you want to use like source IP? Which means the same? Client will get will get through the same back-end server or do you want to use round robin? algorithm or There's there's quite a few Algorithm they're all detailed in the configuration But they're they are classical ones and there's another thing that you can do or not is Use the checks that that is Related can Regularly check if the server the back-end server is up and running and this is how it decides if it continues to use it So for example here, we're sending a request with slash health check to the server every Every five five seconds, which is the value of the interval value and If related gets an HTTP Response of to oh, and they will consider that the servers is alive and continue to use it if for some reason He gets like HTTP error file. Oh, oh, then the server will be dropped and will never be used until he gets another To oh response Another thing you can do is use full box servers full box servers is Either a server or a group of server that will be used in case the other group is not working think of it as like Like like when you get that that particular error message that your application is as dying you and you get Get HTTP errors and you don't want to get This you you don't want your user to get this message what you want your user to to get is the we're coming soon Web page, so you can use fall back this way you define a first Table with their production server you define another table Which here we've called fall back that points to another server another URL and in the In the real a section you get two forwards directives You got the first which is the same which you say by default forward every request to the servers reference in the web hosts table using in this case the round ribbon algorithm and if the both server stops replying HTTP 200 To relay day, then you can switch back to the server reference in the full back table Which could probably be a simple HTPD server with a single page that says we're on trouble where we're working on it come later In this configuration the full back will be automatic this means that The first production servers there are two so they'll get They'll get all the request in a round robin mode if the first server So let's say we're post two Fails all the request will go to the web server one And if for no reasons or for some reasons Host one also dies Then relay they will notice them and forward all the request to that third server And this is when as a user you will see that static page or whatever you decide to do There are sometimes you don't want that automatic behavior to happen for example mostly in business continuity plan you may want to Check things before redirecting your users to another server for example if those are remote servers So that implies latency and you don't want this until you're sure that the primary servers are dead or When those are mutualized server because they're not supposed to to get all the charge from the user, but you want You want to be sure that that's the right thing to do in this case, and that's the second yellow line You can set the disable options to the to the table and this will Tell relay day to not do the automatic switch. So what it will do is Send the request to host one and two if one of those hosts fail the all the requests will get to the Remaining server and if those two servers die then really they will stop for Sending request to any any other servers, and you will have to do a manual operation so that it the new Request will go to the full box servers so here's Here's an example of what I was describing You can see how you can use the relay control Comment here. We say relay control show summary and what it's what this does it that it tells you basically what happens so we can hear that we can see that the The relay that we've named WWTS for the moment is active all the two hosts that we've configured are working and The fallback Table has been disabled by default So in this case which what this means is that the two primary servers are working and our services working properly now When failure happens in this case, we love we lose the two servers in one shot If you use the relay serial command what it will tell you is that the both the hosts are now down The first we relay is still active, but there's nothing going through there So in this particular case, there's absolutely no service for your users. So this is where you get the HTTP error 500 things like that What you will have to do To re-enable the service and switch back to the year fallback servers is run a command like relates it to a control table unable to The table 2 is the fallback table and what happens now is that the table becomes active the two server gets checked and if they are up and running now the The request that goes to those servers And what this means that now your server is back again your services back again so There's one thing to note is that if you keep it this way as Soon as one of the primary servers will go up again The service will fall back to that particular server which you may want or not depending on the the error depending on you have to Start the server but make some checks on it So if you don't want the fallback the the failbacks are the failback procedure to happen What you have to do is disable the first the first table Then what will happen is that is that all the requests will still go to the secondary server and where when you're ready you just have to enable the the first table The previous examples were quite simple in the sense that it was One IP one of QDN one one host name. Let's say What you may want to do is have your relay day serve multiple host names like Like in Apache you say you can talk about a virtual host engine X might be server blocks I'm not that good at engine X or it may be an error But I think that's how it's named in in engine X. The idea is that you want your relay day to manage let's say WWU my website and my my cloud my net my website So how you do it is first you create your tables for the servers So here I've create created a table that's named blog Which points to two servers and I've created one another one that says that's named cloud and points to another server In the protocol section, I have to manage the Certificate in this particular case. I have one certificate for QDN You don't have to do is to do this you might use the as equal server alias or thing as that in the in the server in the Certificate, but in my case are mostly every time use one certificate per QDN So here you reference them this means that relay day will look for a file named blog example CRT and Another file named in this example next cloud dot example dot CRT It will load it and then you add you add information to tell Really do how to manage this so first I do what I do in PF Which is block everything and start telling him what to allow So in first thing, I'm gonna check the host header of the the HTTP request and basically which What this does is says if the host is block dot example Then you allow it and you forward to the server that that our reference in the block table And then if the host is cloud example, then you forward to the other table, which is cloud and With the block with the with the initial block sequence If you try to come to that relay day and say, I don't know let's say Europe is the dot com It will just drop the the section when you've done that you have to modify also the relay section and tell it tell it that the blog table will have to forward to a specific port in that particular case the the The block table will forward to the to the two servers the Web host one and two will doing the check the round robin check as we saw previously and if it's going to If the if the header was cloud example, then it will forward to the table. That's named cloud So this is how you do virtual hosting in relay day Another thing you might want to do is play on the path. I Usually because let's encrypt is free. I usually get Fully-quantified domain name and a certificate, but some people like to do it with a path. Let's say WW my website slash goes to the blog slash next I'll go to your next long term. So whatever and This is how you can do it this way so the configuration is Briefly same regarding to take the table what we will mostly change is the protocol section Rather than using the host header. What we say is that we want to to pass or block Request for which the path looks like this one. So it's in this example I'm telling relay day that anyone trying to do CGI bin a request We'll get popped off because I don't care Anyone that's trying to do wordpress authentication will be will be dropped also because I don't care don't use wordpress and The tool asked just says that if the pass looks like Slash next cloud slash something each the the request should be forwarded to the cloud tables Which are the cloud servers that are running my next instance and If it's not the case then the request will go to the other table, which is my blog Like in pf the directive are passed in in the order, which means the last matches Defines what happens? Okay? The problem if if I do pass request forward blog before the past quick path next cloud then if I do Slash next cloud it will go to the blog because it's the first that goes Now another thing you can do with relay day is well solve problems in this case As I said the my backend server usually don't use TLS because I don't need to and I trust my line enough To not do it and when you run things like Bakel master don't search Angie The thing is that the back-end server will yell at you if you are running HTTP and say you're not secure I don't I won't do anything and you can't get content out of those servers If you want to solve this with really do what you have to do is tell really D to send a particular HTTP header to the back-end server so that he knows that somewhere TLS happens and that he can gives you outputs and this is how you do it you take the protocol section and you do a match on the request which is the thing that comes into real ID and You set a particular HTTP header, which in this case is x forwarded proto You set the value to HTTPS and this header will be added and sent to the remote server so that my master don't Server will get for example the whole request that has been done by the user plus that particular HTTP request and this will let him know that we're doing TLS. We're secured so he can reply and That's what he has to do another problem that you can find is that some application are talking way too much and What you see is that from the user perspective if you look at the at the header Get lots of information that you may not want to be outside if you're a real company and you don't want to have Information like what's the server version you are using what the application framework you're using Maybe there's some or some information that leaks inside the HTTP header. So what we can do is you can tell really need to check Their response the HTTP response from the back-end server and if they find something they just Either drop it replace it or whatever in this example What I'm doing is that if relay they sees a response header that is Exposed by which basically probably is done with WordPress or things like that This header is just removed which means from the outside of my network. Nobody knows well there There are ways to know it, but you can't just use the HTTP header to see what I'm using There's another use case. You can play with the server HTTP header in this case I was just playing so what I'm telling relay day is that every response that goes out of relay day We'll have the server HTTP header set to Microsoft is This may Me this may be either stupid or just fun The thing is that if you got script kitties that are just looking at your HTTP headers Then they will try to do stupid things So that's the one point the other thing is that if you have old software client software like I don't know Lotus Notes or whatever that expects HTTP header for whatever reason you can force them to get the proper HTTP header that they That they are waiting for and this may help working Last thing you can do is Some some of the software you get on GitHub or whatever don't really bother about security and privacy of the user and What this means is that some HTTP headers that can be used by the web navigator Are not published by the back-end server. So in this case you can do it using relay day in the in this example, I'm setting three Particular HTTP headers, which are the X access export action The X consent type options and the permission policies what those basically says is that I Know that my back-end server is not publishing those headers, but I want the The user navigator to have this information to know what he can do and what he can't So using the match response header. I just tell that Related to create and to add those HTTP headers to the whole HTTP response that the user gets And if you're like me and you have no idea what you're doing, you can go to the Mozilla security Observatory, it's a website. You just enter the URL of your website It will parse it and it will tell you you're missing this you have this information It sucks to change it with something else. It's it's quite really really well done There's explanation for every HTTP headers. There's use case explained so you can guess Do you need it or not? Will you change it or not? What what's the value to use if you're using software like next cloud? They have a specific Website which calls Next-class security scan which does the exact same thing he scans your riders and he says, okay We've noticed this this this is good. This is good This sucks and this is when you can take those information and add them to your relay day server if you have to correct things now log management By default what happens that? relay day logs using syslog all the logs are happened to the demon and messages logs and they have those looks which Are to my taste not that friendly when you're trying to debug things Basically, what you get is that you get information about what's up and what's down and you get that kind of Let's say weird information, but really basic information of what's the request like and I find it sometimes a bit complicated to understand you got the the public IP and you get where it's Going to but you don't know much about What's happening? So sometimes you need more information to get to correct things. So what you can do? well first First what I like to is have my relay day log into a specific directory file So I'm just modifying the syslog the configuration No, shouldn't 90% happens if I disconnect and reconnect will it fail? No, so it's okay. That thing is missing logs. Anyway So what I do is configure syslog to get this his own log file for relay day, so I won't go into this. This is a simple example of how we do it with his syslog to get the specific logs from relay day into a particular fine name What's what's interesting here is the configuration of relay day to add things to the log file Here what we do is that we Ask related to log the URL that he received from the user We ask to log the host the user agent HTTP header And what we also do is log the response Here the content type and content length HTTP header so that we get those Informations in the log file and as you can see we get something that looks a bit more like Apache logs or whatever you can you know what the user is doing, you know where he's trying to Which path is trying to get you get? The user agent if it's important for you and you also get the size of the things going out So you can check a bit more things So this is how we add those informations now Don't check the time Okay So this is kind of Let's say complex things that you want you want to do so on the left you got the user side and all the the right side is the relay day configuration and What we're saying here is that? does the Thing right here, so you you enter the relay day here And what we want to do is we first want to check if the fqdn that we're using are Authorized or not and if it's not authorized and just drop the the request Then if we succeed we have three cases that we want to apply differently on the top What we want to do is that if one of the two of the two fqdn that we've defined is Those one we want to apply logins Specific logings and then we want to improve the security and privacy as we've seen before The middle case is that if the fqdn is something else, I'll call it number two number three But anyway, so we're going to tag this particular fqdn and take action specific to these Fqdn and in this case, I'm going to check if the source IP is authorized or not If it's not authorized, then I'll just drop the session if it is then I can proceed and do something else in this particular case I add HTTP cache control headers to get PNG and CSS cache for example because my Stupid web server doesn't set this particular header and the third The third options is that if the request matches another fqdn Then we're going to check if the user agent is acceptable or not If it's not we drop if it's acceptable we can go forward and then We can check if the path is okay. We can do things or drop the section and then If it's if it's all acceptable Oh Yeah, and in the last thing is that we want to drop Some specific headers because we decided that this those headers shouldn't be thrown to the to the user and when this is all done We're going to the forward section of relay day and the replies comes from the server and goes to the user the whole dotted sections I've separated in different Files just to be easy to To read on the slides. You don't have to do it But it may be a good idea if you want to do lots of things for different for lots of FQDN difference in this case you can use the Include function to get configuration from external files So get includes in your real idea comm and then you set up different files in setup You you set up all your configuration in and when you reload it, it will parse the whole set of files. So The first use case I'm creating a file. It's name a TC relay DSLat dash SSG comm for example, and what we're doing is that what we're Saying just previously. We're checking for the headers. So we will only authorize something like that is Ww.example or something that is blog.example if the FQDN is not those any of the other Lines won't be passed. Why because I've used the tag section the tag keyword and This says that if the host name is Ww.example I put a tag which is named SSG on that HTTP section and I'm going to parse it Later on and this is what happens in the last in the last line I can match the header to The header host I set it to log that particular Variable in the in the log file only if the HTTP request has been tagged with us with that SSG for example words that I've used previously So using tag and tag it you can do those kind of directions So in this case I'm just logging a bunch of things and then adding a few headers to the to the to the replies Because only in the case that the initial case that the FQDN was one of those two Names that I've configured The middle the middle fork was for example, I create another Another file and then I want to match something in the host data that is cloud example and if The header is cloud example die Then I use I set up another tag which I named next cloud and then I do something with it That is if the request or tag with the next cloud I add some Some some user agent Checks and what I do is die block them because for example, I don't want Google Google bot to come to my next cloud instance and do things I Also may want only a few people to come to the admin Section of the next law instance. So what I'm doing is that I'm checking if the request is from a certain network address Then I'm authorizing it If not, I'm just dropping the the request in Practice it's a bit difference. What I'm saying is that Mapping every week request that goes to slash admin to and I'm setting a tag that is forbidden and If I match the request and it comes from those IPs then I target next cloud and the Magic inside is is that the forbidden tag is never used elsewhere in the configuration. So what happens is that they Really, they won't know what to do with those Request and what happens that they don't gain access to the to the path. So this is how I Block access for an section. Yep. No in this particular case I did the dirty way What you can do is that you can set a tag forbidden and this will forward to a page that said get out on my way For example, that would be the proper way to do it probably And then in the next cloud example for a case for example, I don't want the HTTP server header to be published. So I just drop it Last example kind of the same same way For for example, Grafana, I want the whole Website to only be accessible from a subset of network Ranges, so I'm not working on the host HTTP header. I'm using the URL which means that anything that looks like Matrix dot example slash whatever Will be parsed and only allowed if the source IP is one of those two lungs and I'm doing a bit of some more magic here What I want is I want to apply Configuration only to Only two objects that are already tagged. The problem is that I can't check. I don't think I came You can get a header that is already tagged take the action and re-tag the same way It won't work really they will bright will brag at you. So what I'm doing is that I'm looking for already tagged headers Applying things and tag it with and other things like we've done for the with the forbidden tag and In the configuration of the relay D what we're using is that that particular Second second tag. This is why the cache control header is applied to the HTTP Requests that are tagged gcache and not the one that a word tagged a graphana So when you've done this with all your fires What you can do is go back to your standard relay decomp and then you just got the includes which are the in the middle So what you get is kind of same table definitions You've got your in the protocol section. You define the whole set of TLS certificate that you have Then you include the configuration that you've set and then you do the the past request and In this case what this said is that any request that has been tagged SSG for example, which was done in the sg.com file will be forwarded to the Table that is nine named blog all the next cloud will go to the blog and all the graphana and gcache will go to the Table that is named graphana Before we go to the questions, there will be a little change in schedule There was a cancel talk following up this in here and we are filling it that in with a Talk or panel discussion from Tom's mouth About bringing together our small or medium-sized Companies with BSD developers to fund little projects that solve specific little problems This will start at the top of the hour and now back to the questions or I do have a question to the audience who's going to Asia BSD con and Did not yet have graph We are looking for a person to take graph home and bring him to Asia BSD con Not everybody at once Yes, about that size Is it you? No, yes, I couldn't hear that Okay, thank you Right any questions you mean having several Related server Not really because I'm using it on my personal web things So this there's not much traffic I've never reached there was sometimes I think for Mastodon's I had to raise a bit the number of folks Yeah, folks. Yeah, that was No, I haven't tested it the the only thing I tested is running several instances of relay day for example here I've got a configuration that in that includes three or four FQ then in reality I have a server where I have three or four a different configuration for relay day and three or four instance about that are running In the money never checked no idea all the whole questions So there was a question about Reloading the configuration that barely Stops all request until the all checks checks of our have been done by relay day Which implies that the the whole web service is not available Until really these finished to check all the all the servers other questions you mean Yeah, you mean do I do I maintain a list of acceptable user agent? No Well, they each time I've checked the the HTTP user agent that either I I didn't have that much big things with login Etc. So I must admit I've never Don't any check on this way because well even in next cloud I think that the The user information is shipped in on you using other HTTP adders So I've never checked this. Yeah, sorry So, yeah, the question was how can you manage the? The user agent when you get both the user agent and more information inside And how do you how do you split it to make your decisions on all the questions? No, okay. Thanks a lot