 Thank you for inviting me to Paris, if I start to speak too quickly, say something, throw rocks, whatever. I'm a little groggy because I got to spend last night at the police station. Your French police are wonderful, I highly recommend them. So, I was not there for anything that I may or may not have done wrong. So, I'm here to talk about the OpenBSD WebStack. So, who am I? I've been a sysabin for 20 plus years now. I helped found the Southeast MichiganBSD user group, which is this little tiny dot on the other side of the planet from here. I write books, fiction and nonfiction, and my most recent tech book is on RelayD and HTTPD from OpenBSD. So, let's talk about the web. The web is a sewer, it's awful, it's horrible, it is full of crap. The web is the worst thing to happen to the internet in the whole history of the internet. Well, okay, we had to use that, that was pretty bad. There was a lot of junk and crap on that, but, you know, there's just all kinds of terrible things. Okay, mailing lists, they're bad too. But really, in the good old days, the internet was just so much better and cleaner and easier. Okay, fine, never mind. So, let's talk about the software lifecycle here. There's a whole bunch of open source developers in here, and you've seen this many times. Once you've been around a few years, you realize that someone writes a simple tool to solve a problem, and it's a great tool, and people love your tool and they start to use it, and you get a patch. And the patch looks nice, it looks good, it adds something on, this is kind of cool. And then other people send features, and then you get, you know, some hypercaffeinated squirrel that has nothing better to do than add really awesome functionality to this thing you made, and it's wonderful, and then you get a patchy. So, and then someone comes along and says, hey, I'm going to write a simple tool to solve this problem, and the existing tools are just so damn huge, and someone sends them a patch. So, OpenBSD does things a little bit differently. Have you seen the blog post features our fault by Tedu? If you have not, if you take nothing else from this, go read that article. It is written in clear and accessible language. I'm going to turn this so I can paste properly and still see my notes. It's written in clear and accessible language. There's no make files or code or anything in it, but it explains what the problem is with software. The more you add to it, the more complexity you get, and the more ways there are for it to break. OpenBSD has one thing that they do really, really well, and that is saying no. No, you can't have that. No, we're not going to do that. No, this is what this tool does, and no more. And you can pipe it through Setanock and write your own Perl script around it. That's fine. We don't care, but we're not adding that function. And most of us for web servers, we need simple web servers. 90% of the pages on the internet, not the popular sites, but just the raw pages, are pretty simple sites that do pretty simple things. They don't have complicated backends. They're just providing a document or presenting the company phase, or some idiot's website that they publish in an article, and suddenly 50 million people come read it, and their web server bill goes through the roof. It's really simple stuff. And that is the simple stuff is where the OpenBSD web stack really excels. It's composed of the HTTP web server. If I have a criticism of their project, it's that they called the web server HTTP. Go search for that on Google. RelayD is a general purpose proxy and redirector. They use Carp for IP layer redundancy. All of this plugs into PF, and it's built on top of LibreSSL. The LibreSSL point here is important because if your system doesn't have LibreSSL, you have to either build your own packages or run a different web server. But they use LibreSSL specific APIs. Because why would you use OpenSSL if you had a choice? So it runs on OpenBSD, of course. TrueOS, HardinBSD are built with LibreSSL. I've built custom packages on FreeBSD. And the farther you vary from OpenBSD, the less luck you will have running it. I'm sure someone will port HTTPD to Linux and plug it into IP tables. So, HTTPD, simple web server. The config syntax is reminiscent, and there goes someone who doesn't use LibreSSL. The config syntax looks like nginx, looks like pretty much every other OpenBSD server. It runs privilege separation, ch-rooted, and divar-dub-dub-dub. You configure it in atch-tptd.conf, and dynamic content comes through fast CGI integration. And here's what a config looks like. You set a macro. You define a server. All the servers are virtual servers. You can include a file. You can have web aliases. You listen on the macro on port 80 and set a root directory inside the ch-root. If you have used any OpenBSD software, this should look real familiar. You can also have per-location rules in there. Here we've set, well, we want the whole site automatically indexed. If you're in the files directory, there's no automatic index. You have to know where a file is to be able to grab it. And here we've set up some password authentication. And even put a little message that will appear in the authentication box. So there is a default server that listens on all IPs of the host by default. If a client just does a netcat to port 80 and a get, they get the default server, which is in the htdocs directory. htptd makes heavy use of blocks and redirects. There is no rewrite. There is no, like, Perl-style mod rewrite. This is a feature. There are ways around it, and we'll talk about it. But this, with blocks and redirects, you can do things like here we're listening on port 80, but we transparently redirect to the TLS version of the website. You can use globs. Standard shell globs. You have servers, you know, web servers zero through nine on your web farm. Well, they can all share a config. I'll share a root directory. You can use wild cards again, just like the shell. Here we have any host name under mallard.info. And we have an alias for just plain mallard.info comes here. You can do really complicated, annoying things like this. I don't know why you would want to, but it's an option. Perl-regexes. Does anybody, how many of you feel you know Perl-regexes? Okay, the fact... I normally tell people that they're wrong when they raise their hand. However, the fact that every person raised their hand kind of hesitantly and... Yeah, you're right. From the openBSD perspective, as I understand it, one of the big problems with Perl-regexes is verifying the code is just ghastly. And most people only use a tiny subset. So why add these features that nobody really exercises and with code that is gigantic, nasty, and unavoidable? So instead they grab Lua patterns, which have a lot of regex-style functions. They're much smaller, much simpler. They are not Lovecraftian, and they look a lot like globs. They look so much like globs that they've added this match keyword to say, no, this is a pattern. So, and I'm going to go a little quickly through this. I'm just trying to let you be aware of what's in it rather than teach you how to build this because we have a bunch to get through. Classes of all digits, all printable characters, all alphanumerics. Put classes and characters in brackets. You have all seen this before. You already know how to configure these. Anchoring to the front and back of a string, one or more, zero or more. So, you can use all of this to set server names and directory names and locations. So, HTTPD defaults to the traditional Apache log style. Does anybody here actually use traditional Apache logs? And a few of the OpenBSD people raise their hands in the back. OpenBSD is written for the developers. They like Apache logs. That answers the question of why the hell did they decide on that format. So, it also supports combined logs which most other people use. And you just set that there. Set in the access log, error log. These are all under the CH root. Debugging is pretty straightforward. Test the configuration before you start it. Test an alternate config. If you're really having trouble, you can run the web server in the foreground with verbose debugging and watch as it answers replies. And this is where you discover that you used an asterisk instead of a plus in your Lua pattern. And that's why people are getting the wrong server. Now, dynamic content. PHP rules the web. I'm sorry. There are people staging the revolution, but right now they are the empire. OpenBSD uses slow CGI to support the fast CGI interface. Slow CGI is actually pretty fast, but if you're running dynamic content and you get a 500 error, slow CGI is not running. That little chip will save you about three hours of screaming at your keyboard. And basically you just set a location. You say where the scripts are and tell it fast CGI. And it feeds it to a socket inside the CH root. I'm talking about the CH root. The plus side is if your web server is broken into only stuff inside the CH root is accessible. The downside is, as a sysadmin, only stuff inside the CH root is accessible. Remember, you can't sim link out of a CH root. So if you're running WordPress or something, you will want to add stuff like resolve.con, local time. These critical system files you may need an Etsy host. And if you're running something more complicated, say you have an actual Perl CGI. You'll want to use LDD to find the shared libraries you need and get them in there. But let's look at... If your web server can run WordPress, it can run anything. So let's run some WordPress. Pardon me just a moment. Some important details here. If you go searching for the MySQL package on OpenBSD, you will become very frustrated. They're using MariahDB. Since I discovered that, I'm now using MariahDB everywhere because Oracle. OpenBSD will let you install multiple versions of PHP simultaneously. I strongly recommend you pick one and stick with it. And go into RCConf. You don't need any special flags for HTTPD. But you do want to start your chosen PHP FPM processor, which is fast CGI for PHP. You'll also need a database socket inside the CH route. And you'll need to tell MariahDB to put the socket inside the CH route. OpenBSD also ships all kinds of modules. But the configuration for the PHP module initiation scripts go in this sample directory. You'll want to copy your INI files into the actual PHP configuration. And then start the PHP FPM module. So all you really need to run PHP is tell it an index.php is an index file. And then specify, instead of the default fast CGI socket, tell it no, look at the PHP socket. And that's really it for configuring your dynamic web server. There's no long config file. There's nothing else. And the nice thing is I'm pretty confident, given the people developing it, that it's not going to get... You may get an extra line or two of configuration somewhere sometime, but it's not going to be much. I'm sorry, I moved my head a lot. I'm going to play Dalek a bit. TLS, how many of you run SSL? You do know that TLS is the successor protocol and you should not be running SSL. So having said that, how many of you run SSL? Yeah, a bunch of hands still go up. I know. I'm sorry we're stuck. Remember, TLS, transport layer security is a much better name because it only supports data in transit. TLS does not secure your web server. And really, despite all that happens with these certificate authorities and them sending, you know, you have to send your driver's license and a DNA print and all of these things to prove your identity. All that these certificates do is verify who controls the DNS of that website. So I'll sum this up as saying certificate authorities suck. They charge large sums of money to run shell scripts. And when I say that I suck, what I really mean is that I'm envious. So, let's encrypt. How many of you have used let's encrypt? I love BSD cons. I deeply love BSD cons. Okay, so free SSL scripts for everyone, for every purpose, for anything. And let's encrypt works. There are two ways you can verify that you control a domain. Either through an entry on a web page or an entry in a DNS server. OpenBSD includes this ACME client for the automated certificate management environment. I can't hear that name without thinking of Wiley Coyote, by the way. And OpenBSD has the best simple ACME client I have ever seen. I love it. And the certs are good for 90 days. You can manually renew them if you do it that way. You're an idiot. That slide shouldn't have been there. I'm sorry. I've given this talk two times and each time I think I need to pull that slide out. Okay. How ACME works. At the first time you run the client, it creates an authentication key pair with let's encrypt. The client asks the ACME certificate authority what proof it will accept. Let's encrypt accepts HTTP and DNS. Client gets to pick between the options. ACME client only uses HTTP. ACME client creates this certificate signing request, sends it in. Let's encrypt says if you're really this, create a file on the server and sign it. ACME client creates that file, the CA checks the file, verifies the signature, issues a certificate. And OpenBSD has a directory in the CH root just for ACME operations. And basically you can just set this location on your website. Tell it that the root directory is there. This root strip two looks a little odd, but all it means is strip these first two entries off the directory, off the URL before putting it in that root. And don't automatically index your ACME directory. And here is an Etsy ACME client dot com for the domain. You set the domain name, set some alternate domain names, define where each of the files go, and then say sign it with let's encrypt. So with ACME client, start off with ACME client minus A, create that key pair, get that cert. You need the minus D for the domain. The first time you run it, add a couple minus Vs to see what's really happening behind the scene. And then turn on TLS in HTTP dot com. Here I've put in an automatic redirect. And here is the actual TLS website. We're listening on all addresses. We add TLS on port 443. We defined the certificate with the full chain file. We have the path to the TLS key. You can also use HSTS, high security trans. I forget the acronym. Strict transport security. Thank you. Which sounds really cool. But what it means is that if you ever break your SSL, nobody can talk to your site, including let's encrypt. So be real confident in your setup and that you've renewed certs a few times before you do that. Not that I've ever locked my own web server renewal out or anything. There's this fiendishly complicated process to check and renew your script, your TLS certificates. And if you get a, you know, you run ACME client and the domain name. So do you need to renew? Yes or no? If you get a certificate back, reload the web server. So how many of you have used OCSP before? Okay, OCSP is a, if your website is small and you don't care how fast it is, OCSP is traditionally not been worth your time. Every time you contact a TLS website, your web browser contacts the certificate authority to make sure that the cert is still valid, that it has not been revoked. So this slows down your website. OCSP basically staples a recently generated certificate for your certificate that says this certificate has not been revoked in the last seven days. And it provides that to the client and it provides a bit of a speed up and it doesn't tell the CA when someone is actually using the certificate. So there is a command, OCSP check, set the file, set the key, plug it into httpd.com. It is so easy, it is suddenly worth doing for everyone. OCSP certs expire weekly. Use this complicated shell script. The worst part is the path to your key files and reload the web server if you have a new one. Okay, some things you can do, you shouldn't necessarily do these, but the internet is stupid and sometimes you have to. You can affect the TCP behavior of the web server in httpd.com. You can set a time to live on your web server packets. So if you don't want this website accessible outside the company, even if the firewall team screws up again, you put a time to live of two on your website packets and they can't leave the company. You can change TLS versions and ciphers, don't do that. The ciphers and versions were chosen by people smarter than you. If you have users with their own websites, put their home directories inside the CH route. Okay, Relady. Relady is an arbitrary traffic redirector. OpenBSD comes with PF. It can redirect traffic to hosts inside your firewall. And it includes a load balancer that is basically fault oblivious. It sends traffic to hosts whether they are alive or not. And Relady is basically, it just tests the host to see if they should receive traffic and it adjusts the tables in the firewall appropriately. So Relady can also act as a layer 7 proxy. You can terminate a TCP connection at it, muck with the connection and create a new connection. Always start with a PF firewall, a working firewall. You have to have packet forwarding, make sure that the inside can't connect to the firewall. The usual lock down your host. So Relady has four components, a parent process, thank you privilege separation, a host check engine that pokes at all of your servers and sees if they're still alive, a PF engine that talks to the firewall, and a relay engine that accepts and creates connection. Relay control manages all of this. Relady's, the part I usually use is the redirection, which is just pure TCP IP forwarding. I have 10 more minutes, oh crap. Okay, you have tables, you can say listen on the outside address and forward to these servers, forward to servers in this table, and this is how you check those hosts. You can set macros like every other open BSD program. You can use relay control to check and show what redirections you have, what hosts are there, are they up and working, how many hosts are in a current table. Host checks, you can check by ICMP ping. You can do TCP port tests, HTTP response code tests. Are you getting a 200 error? Do you suddenly have 500 errors? You can check and see if TLS is working. You can do HTTP response codes over TLS. And you can check the integrity of files, download a file from the web server, SHA-256 check it, if it matches, great. You can also do things like, here we're forwarding to a mail server and we send this string, nothing, and we expect 220, mail.MichaelWLukas.com is my mail server working. Keep it in the pool or not. Here I have a script to check a DNS server, which we dig a domain name, we grep for the string that means that there is an authoritative answer here, and we invert the return code and send it back. If the DNS server is serving authoritative data, keep it in the pool. Relays accept TCP connections, accept traffic, create new connections and forward it on and can filter the traffic. I am hurrying up a little here because we had camera trouble in the beginning. I'm sorry I did not anticipate that. Here's a common thing you'll see where we relay traffic to SSH servers inside your firewall so your developers can get access. And we've defined a protocol here down at the bottom, this protocol fix-up where we apply a change to the TCP stack. When you create this connection, use TCP no delay. I encourage all of you at some time to run an SSH connection without TCP no delay. It is an education. Relays let you do really strange things. Here we've defined this HTTP protocol called get only and it passes anything with an HTTP method of get and then it returns an error for anything else that says forbidden method. My developer swears to me that his application only uses HTTP gets. He's promised this. Let's find out. You can block requests. Bill has also promised that he will never again use PHP MyAdmin. So we're blocking that particular URL and we're returning an error message for Bill. In writing this book, I discovered there's this thing called WebSockets that basically reimplement TCP IP inside HTTP. You can set something that says I can't request this through this upgrade method. Say no, no, we don't allow WebSockets. I also encourage you to try this because it is an education on just what Web developers will do to get around the fact that we have blocked everything but port 80. And if you really, really hate your Web developer, you can strip the user agent header. And I'm giving all of these not just to be mean but just these are examples of things that you can do because, again, the Internet is stupid. So here's a config straight from the example that you can put your SSL on the load balancer, take it off the web server. These days, a TLS acceleration is not as desperate and neat as it once was. However, if you find yourself stuck and the new hardware hasn't arrived and you've got to get some load off the web servers, it is nice to have the option to move that load elsewhere. Other things I realize, they do not do SNI yet. There is a rewrite in progress to work on that. You can enable and disable client-side renegotiation, session tickets, et cetera, and you can do high availability with KERP. So I have probably about 30 seconds. A little more, five minutes. So we have a few minutes to talk real lady. I'm going to take care of something now as a U.S. citizen. Trips like this are deductible on my taxes if they are for business. So I need to say, I'm a writer. Buy books, please. Thank you. So and if this should show up in an IRS audit, this session tape, it's nothing against you personally, Mr. Auditor. I love you. You are a nice man. So are there any questions? We'll start in the front. Yes, sir. The Beaches WebStack with the library for first CGI to write it in C. It's from the same guy who made the Acme Client, if you heard of it. I haven't worked with that. Question. First thing to answer to that question, the author of both Acme Client and Beaches and Mando, Chris Depps Johnson's explicitly told me to greet everybody at this conference he stuck in LA and couldn't come. Second thing, I'm currently doing an audit on that library on behalf of Chris Depps company, K-Pam. And I must say reading the code is fun. It is very small. It is very clean. It is idiomatically written. So if you have a choice, then writing a web application in pure NCC with the Beaches stack is probably superior to writing it in PHP. But of course, you have a nice... And it's also quite true to the OpenBSD way. For example, myself, I have also written my main web application, my own OpenBSD org in NCC without any scripting language that works and it gives quite nice and fast applications. But sorry for... Thank you. I love BSD cons. Other people answer the questions. Next question. Let's say I have two hosts, load balance with Relady. Can I tell Relady, hey, I'm working on that one host. I want to do a software upgrade. Please keep that out of the rotation. And don't fiddle with the PF tables in the next few minutes. Of course you can. Perfect. Thank you. I was wondering, you said that the HTTP server is shrewd, but is shrewd. C-H shrewd. C-H shrewd. Oh, yes. But is the PHP FastG server C-H shrewd as well? Yes. Okay. Yes, you put the socket into the HTTP C-H root and the PHP itself runs elsewhere. I trust them to have locked it down. I shouldn't, but I do. Hi. I'm pretty sure you template for your configuration for HTTP, at least for virtual hosts. Do you have a specific template for HTTP only before using Acme Client to generate your first certificate? Or do you have some kind of automation? That's sort of chicken and egg problem I see here. Yes. What I usually do to bootstrap TLS on the web server is I run a simple site that only has the virtual server and the Acme entry and I generate the cert there and then I switch everything over to TLS and let's encrypt, we'll go to the TLS site to check for the new Acme updates. You will probably kill me. What about HTTP 2? Yes. Yes. What are the penalties for violent crimes in France? Asking for a friend? I'm sorry? Yeah. Oh, call it a protest. Okay. The web server is what it is. If it does not meet your needs, feel free to use a different one. Having said that, you know, there are people working on the OpenBSD folks watch what's happening. One of them will need HTTP 2 and they will. Oh, and just for the record for the camera, since the camera's pointing it up here, can we show that Peter Hessler has raised his hand? Oh, no, you raised your hand. We know who's responsible. So first off, HTTP 2 solves the problem in the wrong way. The correct answer is don't load 700 JavaScript things in your web app. Two, the other answer is use a different web server. Don't care. Can I add another web header to HTTP like HPKP? Relady will let you add all kinds of web headers. I do not recall off the top of my head if HTTP will let you add a header. No, thank you. Just curious, do you use this for your own websites? I use this for some of my websites. I'm switching the rest over. I'm running my, my database stuff goes on a ZFS backed web server. However, I'm planning to do a book on free BSD packaging next year. So I'll be building Libre SSL packages and going that way. Okay, thank you. I have a question concerning Acme client in an enterprise environment. So only for internal use, how do you get the certificate there? Well, there's a, how do you get a certificate in private enterprise that if it's a private site. What I would do is there is a back in the day, we called it split horizon DNS. Where you had a public DNS entry for something and then a private DNS entry with a different IP. And depending on where you were, you got different answers. So, and you can have multiple names on one Acme client cert. So I would put something that faces the public that has just a whole slew of names on it. Renew the cert out there and use some whatever scripting mojo you have to pull that cert inside the enterprise. There's also Dan Langell is working on something called anvil, which is this. If you have complicated let's encrypt needs. It may solve those problems if it but solve that problem as simply as you can. You mentioned or I think it was Ingo who mentioned the C applications that you want to run on your dynamic website. Yes. Do you have any suggestions for running simple, small Python or Lua scripts? It's perfectly, it should be perfectly doable. Get a you're going to need something to run fast CGI protocol. You point the HTTP fast CGI socket configuration at the socket for your fast CGI interpreter. And it should just work. I have not run Python or Lua websites. So I'm sure this is a wonderful group of people to ask during the break though. I learned that the fast CGI argument in HDBD takes a port number as an argument. So instead of a socket, you do colon. That's one way to do Python applications. Okay, I did not know that. Thank you all.