 So my name is Pierre Yves and I'm going to be talking about load balancing in OpenBSD and especially using the HostedD daemon. So let me give a little history about the daemon. It actually switched names a bit so it was a bit confusing at first for people. I started by calling it SLBD for server load balancing daemon but didn't realize somebody else already used that name. So there was a little confusion. Then it changed names again and again until we reached the decision that HostedD was good and meant the HostedD daemon for load balancing. I worked on this between OpenBSD 4.0 and OpenBSD 4.1 and it was introduced in 4.1 stable and release and that's why when people were able to try it out. Reg Floater worked with me on this daemon and helped me implement layer 7 load balancing features which will be available in HostedD 4.2. So I'm going to give a little overview of this talk. I'm going to try and introduce existing tools and concepts that I'm going to use throughout the talk. Then explain the design we want to reach with the HostedD. A little bit of the features, the tools that are provided give and we'll talk about example configurations of what people really want to do with HostedD right now and at last we'll talk about what's coming next in HostedD. So first, what is load balancing? I'm sure many of you already know that but just to give a small definition, the goal of load balancing is providing a way to spend a service across multiple machines. The typical situation is you have a web server that provides a dynamic service with PHP or Perl or whatever and this application hogs your system and you're not able to service user requests with only one web server so you're going to set up four or five web servers. Maybe you'll do the same with your databases and then you'll need something to spend a service and to have a single IP servicing all these requests. So basically it functions either as an application gateway or proxy which is the difference between layer 3, load balancing and layer 7 as we'll see later and usually you use that in conjunction with high availability which means providing a reliable service and having the option to fail over to a backup service if everything goes wrong. So HostedD didn't implement everything and relies on some of the features that were already present in OpenBSD and one of these is obviously PF which already had load balancing features. First PF provides fast access address tables that are stored in the kernel and a user-learn interface to modify those tables either through PF CTL or through a set of IO controls that you can use. PF also provides different load balancing methods to access these tables and as an example here you have the role that says let's redirect any incoming packets to this public IP on the HTTP port to these web servers that are stored in a table. Sorry. Another set of tools in OpenBSDR, so Carp and PF Sync that I use for high availability which help you share a public IP knowing which is pretty much like VRRP as some of you may know. So Carp handles the virtual addresses and PF Sync handles state synchronization across firewalls to provide a secure and transparent failover between firewalls. Some of the features we used in OpenBSDR the fact that Carp interfaces can be grouped together and that a Carp group once it is created can be manipulated from user-learn which is especially useful if you detect a software failure and you decide that the machine isn't able to provide a service anymore and then you failover to the backup posts forcibly. I'm just going to give an overview of what the real difference between layer 3 load balancing and layer 7 load balancing is since we're going to talk a lot about that later. Layer 3 happens at the packet level which means when a packet comes in on a gateway its IP is inspected and then the destination IP is rewritten so basically it's an extension of NAT and in PF it is just that it is just another form of writing NAT rules so it is pretty simple in essence and it is really fast. Layer 7 happens at the application level so it is just a proxy basically. You are in the middle of a connection you receive a connection, you terminate it and you initiate a new connection to your destination hosts. This allows packet processing and this is especially useful in environments where SSL is required because in order to provide a single certificate you need to terminate the SSL connection on the inbound. So with all these tools there still was a need for a unified load balancing solution in OpenBSD. When you load balance a service let's say you have four web servers you want to know which ones are currently running and not direct IP packets at them if they are not up so knowing when a service fails was something we needed. We needed a tool that is able to maintain address tables because when a service fails on a server I wanted to be able to remove it from a PF table and we needed a tool that can provide a solution to general failure which means I have no hosts anymore I want to switch over to a server that says sorry we are not available right now which is useful in many cases and I wanted to be able to create PF rules on the fly without relying on a set of scripts. So similar applications already exist commercial vendors of course provide such solutions F5 with big IP, Nordel with Altium and Cisco with many different types of services of course I wasn't really happy with that because I love BSD and OpenSource and there were also few Linux projects that provide load balancing solutions but which I found limited in many ways for instance Kipple-ID all of these rely on a similar thing as PF tables which is IPvS which provides more tools but in a different way. So the OpenSource offer wasn't really wide and I decided we needed another tool So these were the initial design goals of OCD First security of course because I was going to write a tool that will mess with the kernel and the firewall a tool that will check hosts at a very high rate and privilege separation was going to be needed to make sure each task runs in a CH route and doesn't do anything else that it's supposed to Efficiency since load balancer is here for efficiency and spanning your service across multiple servers and improving your request rate so efficiency was needed, simplicity because that's a concept we love in OpenBSD I wanted to provide the same kind of configuration file syntax that is found in other demons in OpenBSD and a consistent syntax too because even though in the design Layer 3 and Layer 7 are really different things from the administrator's point of view it is basically the same so I wanted a consistent syntax and last, something which I found really missing in other tools is an administrator friendly tool which would let you check what's going on and take action and we'll see later what we can do with the tools provided The actual design of Host8D is based around four types of processes There is a parent process which is used to start the demon as a whole This process handles configuration loading and reloading it doesn't do much because it runs as a route and it is thus the most important part of the security wise it's the most important attack vector so it handles only configuration file loading and reloading it handles external script execution as we'll see later what this is used for and corp demotion requests when some other part of the application thinks a corp interface should fail over the parent process handles it The other type of process is HCE which is only one process actually which is the host check engine which is a fast model process scheduler for host checks it handles a lot of different types of checks to see if a host is up we'll see exactly what type later it is fully asynchronous and it is able to schedule checks and notify another process of straight transitions There is also a PF engine which is only here to listen to incoming requests from the host check engine and take action as far as PF is concerned that is modifying PF tables creating PF rules, RDR rules, mainly it is... this is a little mishap here we have a fourth type of process which is the only one that is forked more than once and which is a really engine used for layer 7 load balancing it creates listening sockets for all services it loads SSL certificates whenever needed and it filter protocols before relaying connections and we'll see how that happens a little bit later I'm just going to go into a little bit more detail about how things happened I didn't invent anything when doing that since I took a lot of code from the existing daemons in OpenBSD, especially SPFD actually which was my main basis I took safe buffer routines found in BGPD which are also found in SPFD actually and I used the IMSH protocol that is becoming very common in OpenBSD daemons right now this protocol is just a method of communicating between different search-rooted processes all my four processes run a specific task but sometimes they need to communicate with another one and that's when the IMSH protocol comes into play so whenever HCE needs a PF role to be changed it sends a message to PFE, the PF engine and it then says with a specific type of message and a content and action will be taken by the PF engine since we were going to do a lot of SSL type stuff in Host8D, specifically checking hosts, datum, SSL services we needed a set of asynchronous SSL functions which were created to be able to check many SSL servers at once and last I used like OSPFD actually LibEvent, which was developed by Nier's provost to facilitate asynchronous socket programming LibEvent is useful in the way that it hinders the actual type of asynchronous socket routines that are used in our case it is KQ like in FreeBSD but it can also use pull or select so it was a pretty easy way to get started before we talk about the actual configuration of the daemon I'm just going to go through the different types of elements that were created for Host8D our basic element is a host which means a service provider a host is something running a service on a certain port a table will be mapped to a PF table and means a group of hosts providing a common service on a common port and then we have the concept used for layer 3 load balancing and layer 7 load balancing a service means a virtual service at layer 3 and it defines an IP to listen on and a table to use as we'll see later a relay is a layer 7 load balancing declaration and it does just the same only that the table use will be used at layer 7 and a protocol is a helper for relays to manipulate connection as they go through the layer 7 load balancer is this clear for everyone? so the typical layer 3 setup when you're running a dynamic web service is having one internet reachable host which is going to be the load balancer listening on a public IP and usually on a different network either through VLANs or different network interfaces having other web servers listening on the same broadcast domain as the load balancer and so the incoming connection will go through the load balancer across a private broadcast domain and go through the load balancer again to reach back the internet so this is what it looks like as far as configuration goes there's a simple statement to add to PF to let host ID manipulate it and then a simple configuration file hostID.conf as other demos openVSD we can have macros here we only need two concepts we saw earlier the table which we call web hosts we define the ports it listens on a check method and two hosts the check method just says for each of these hosts connect through HTTP and check that when we request slash the code is 200 we'll see the other methods that are available later and then we define a service FreeW which listens on this address and redirects requests coming in for port HTTP to this table so it can specify the interface this also runs on so once this is done and hostID is started an RDR rule will be created and a PF table will be created containing these two hosts provided their service is reachable and this is what it looks like from the administrator's point of view HostID CTL allows us to show what is going on inside and we see our free concepts here our two hosts that are marked as up since the two hosts are up the table is active and since the table is active the service is active as well so the rule is in PF and the table is populated with the two addresses before we go on this is what we can do HostID CTL we can do forcible hosts disabling maybe because we want to take a web server out of a pool to bring Apache up to this or to change something in the application and we don't want requests going to that host at this moment maybe we want to do this on a full table a full set of hosts so we can take a whole pool of hosts down with table disabled and re-enable it later with enable and of course this can also happen at the service level so this is the three types of command we can use reload reloads the configuration file re-reads the configuration file and starts up the demo again right now reload only works for layer three configurations a layer seven configuration cannot be reloaded at that time because it won't have to reload SSL certificates and right now this cannot be achieved of course we can, as we just saw we can display the status of just about everything either by specifying what kind of elements we want to show or by asking for a summary display and the last feature of Hustler CTL is the monitor mode which dumps incoming events and especially HCE the host check engine events this mode either by calling host state CTL or by directly asking for the AF UNIX sockets it runs on can allow different types of programs that might not need a lot balancing features but that want to know the status of a set of hosts they can connect through the monitor mode and use that data typically I have run several little monitoring tools that rely on this and we can talk about it later and this are the table options we can use in host state D especially the checking methods most of the time people run HTTP or HTTPS services through load balancers so there are two types of checks that are available first a HTTP code status check which is just going to ask for a just going to do a head request on the URL and see what the status code is but sometimes that's not enough because in many places you cannot rely on the status code and you want to see what a page actually outputs especially in places where you're afraid of defacement so what you can do is take a show-on digest of a page and rely on it to not change and you can ask for that and we can see later that it's really simple to implement ICMP checks can be useful if you're not able to know what data to ask for and if you just want a simple check for other types of protocols we implemented a send-expect feature that just allows you to send a string of data and expect something back on a socket for instance let's say you're running a SMTP service and you want to load balance it what you're going to do is send nothing because it is possible and expect an SMTP banner with a hostname and if you get that you can it's more likely that the service is running and the host will be in a table for more complicated scenario we didn't want to put a lot of stuff into host ID for all types of protocol and we can rely on external scripts to be executed and given the output of the script or the return code of the script we will take a host down or up this is useful if you want to have SMTP requests for instance decide whether an host is up or down and that's another set of simple tests simple TCP connections if I can connect to a specific port for instance if I want to check my SQL 4306 port I can use TCP which is also available for SSL ports other options that are available are specifying a real port specifying a timeout for a table because some pools can be allowed to response slower than other pools and the ability to start a table disabled and this is a simple use case of a modified configuration from what I showed earlier where we want to take action if we have no hosts in our pool anymore so we're just going to set up a table service server with a simple check running on the load balancer itself we are going to set up a small HTTP server on the load balancer either with Apache or another high speed web server and in the service definition we'll just have to add a backup table which will be used only if this table is empty and in that case that is what host state CTL shows it shows here our two hosts being down our table being thus empty since it's empty this table which is active is used and the service line shows it at the end so this is useful for displaying a story page but it's also useful as a celebrity when you're working on your bringing your PHP application up to date you want to take your pool down bring it up to date and bring the service back up and you can do that with host state CTL which is how we're going to do it by simply saying bring this table down by its name and then we'll see that the table has simply disappeared and its marked disabled and our service is still active using the backup table so this is the administrator friendly part of host state D and these are the layer 7 features Generic TCP relaying if we want to relay a service we don't know about yet in host state D of course HTTP and HTTPS relaying which means that it's possible to forward from HTTPS to HTTP real hosts which is useful when you want to have a fast load balancer and you have an expensive PHP application resource wise you spend it across 405 servers that are running HTTP plain HTTP and you handle the SSL termination on the load balancer side and right now well you can also do HTTPS from end to end which is required in banking environments and such and then comes the DNS relaying which is really new which won't even be available in 4.2 OpenVSY 4.2 because it came after which provides which spans across multiple DNS servers and inserts randomization in the DNS IDs to protect some DNS servers that aren't available as far as IDs go of course we cannot achieve Generic UDP relaying for other protocols and this one happens it's a stateless protocol and we wouldn't know which hosts well which session a packet is for So you mean this could be used I'm talking about HTTPS on the load balancer and sort of clear text web servers behind it's been tested with some kind of some kind of crypto accelerators so we can have a front end that perhaps like the VSE7 that are very weak in CPU but can do crypto very fast then you have sort of heavy machines behind but that doesn't do crypto Yeah, well that's what it's used for actually by some commercial products right now and I actually use that specific configuration at work where we stack a crypto accelerator in the load balancer and only do HTTPS on this machine and then have HTTPS hosts doing clear tests on the back end and yeah the high event cards work good for that for medium sized applications and certainly bring the cost down because otherwise you have to go with really high-end servers to do that So let's look at a small HTTPS relay config I left out the service parts so it's a layer 7 only configuration and we still have our tables with our two web hosts here we have the relay which looks a lot like the service definition only that here we specify a protocol and say here I'm going to relay and while I'm relaying I will open two editors that I can use in my logs to know who otherwise in Apache logs I'm not able to know who the connections come from I will also shut down things like keep alive connections to ensure my traffic is well load balanced because if if requests coming or come back with keep alive specified of course I will stick to one host and this is not the behavior I want in that case I assume you want to listen to port HTTPS and relay the listen like a relay you're right that's what I wanted to do exactly, yeah I messed up because of course the table here is composed of HTTP hosts and yes here I should be listening on 443 we have three relaying methods available right now I think the most important one is the table forwarding because it is the most consistent with the layer free configuration I just I can even use the same table actually it is also possible to use fixed forwarding so a one-on-one mapping which is a pure reverse proxy configuration and use the address specified in the service too that is also possible I'm going to try and make it fast to fill in the schedule as far as actions that can be taken while HTTP traffic goes through layer 7 load balancer I can I can work on a connection direction and take action on a specific set of things change headers I can open headers I can remove a header filter connections whether a header is present or not and I can also feed a header's value into a hash that will be used to select a host in a table when I want fixed host addressing I mean every given I'm sorry I'm going to make a clear sentence I can use headers to make sure every type specific type of HTTP traffic goes to a specific host and this is useful in one for instance you don't want a type of request to go to one host and another type of request to go to a different host I can also work on get values in the URL and look for key and value pairs in there so this is what we generally want to do with the HTTP there are some things missing right now that will be added later and we'll talk about that afterwards this is the variables you can manipulate it's pretty simple and the remote address and remote port is useful to add to a header for Apache logs I'm going to go quickly through that we have other protocol options we can specify the types of SSL ciphers we'll use the session cache the types of SSL we want to do same goes with TCP and setting options at the IP level the size of a socket buffer and this TCP backlog I'm just going to talk quickly about some things people usually want to do with load balancers like load balancing across different broadcast domains because sometimes you have a load balancer in a location and you want to move to another location and that is HostedD can cope with that by tagging traffic as it comes in on an interface and you will just have to net the traffic that is tagged by HostedD once it comes out by default HostedD configuration files already includes the tag for service definitions HostedD can also be combined with Carb because usually you want load balancing but you also want high availability for the load balancer itself the only thing that you have to do is run HostedD on all the hosts that are load balancers and that share a single Carb address which means you'll have to not use addressless Carbs which means you have a Carb address and you also need an address in the network on the underlying interface Carb uses well, I'll talk about development later and we'll just see what the future holds of course layer 7 reloading because it is a big showstopper now adding more layer 7 protocols and more importantly conditional tables conditional tables means if I see a cookie, if I see a session cookie if I see a specific type of header maybe I want to use a different kind of table all together to for instance say if I request a URL of slash static I want to go to my static pool otherwise everything that ends in .php goes to my php pool and this will be possible hopefully pretty soon and to stay consistent between layer 3 and layer 7 there are other things that I hope will soon end up being available and the more important one is weighted hosts because sometimes you have different kinds of machines in a pool and one machine cannot service as many requests as other ones so you want to weight it down so it won't be you won't get as many incoming requests as the other ones and this is also needed to implement list connections which if you want to to pull and weight down hosts dynamically based on the number of connections they get you need this weighting feature the future is these are the more complicated things we're thinking about GSLB means global global server load balancing I'm not sure it's a I think it's a commercial vendor on the world right now but basically it means when you have load balancers in different locations in different physical locations you might want to say I'm going to take that site down definitely because I don't have any service to provide anymore so you can do that through GNS and that would mean having a GNS server that accepts dynamic updates and sending requests to it when a service is completely down and for instance let's say you have two locations so you put in your GNS configuration you have a GNS load balancing address with the two IPs of your two locations and when one of the sites comes down a request to the GNS server to remove one of the addresses from the GNS configuration and this can also be done with BGPD and it's actually going to be simpler to do I guess with OpenBSD since we already have a in-house BGPD implementation and this would mean if I cannot service a request anymore I will stop announcing my addresses through BGPD and traffic won't come to my data center anymore one of the other features we're thinking about but it's actually I'm not sure it will ever end up in other state the world see about that if we can find a clean way to do it it's direct server return this feature is a little tricky it's a feature that allows you to load balance incoming requests but requests coming out of the real service go through don't go through the load balancer so what happens is you load balance at the ARP level and not the packet level and it's a little tricky and it's a little it's a clear violation of OZ layers so I'm not sure we'll do it and finally one of the things we want to see ending up in Hosted in OpenBSD as a whole is TCP splicing which means for the layer 7 load balancer once you finish treating headers and once you have decided what hosts a session goes to you just bind two sockets together and you don't go through userland again to read from a socket and write to another one you just let it come in and go out through the kernel so that's about it for us AD I hope I didn't push the schedule too much thank you and I hope you can visit OpenBSD.org again if you have any questions feel free to ask me there's nothing specific to carp in Hosted here right now if you're at layer 3 it's going to be transparent if you're at layer 7 it's not going to be obviously because since you're terminating connections even for standard HTTP you're going to have to to break connections when carp fails over because that we won't be able to recover right now I guess that's about it you're welcome thanks a lot