 All right, thanks for making it back after lunch Right now right floater will talk to us about modernizing relay D and the there is a rocky road to HTTP 2 Please welcome right Thank you. Yeah, okay. It's It's great to see you here, especially after lunch. I think I had a plate with that much meat and fish and all that and No, I have to talk so We see I hope it's worth it so Let me talk about modernizing relay D and a little bit about HTTP 2 and why The focus of this talk shifted a little bit to to other things We did in relay D over the last few months By the way this People rarely know that relay D has its own Puffy mascot version that I once Did myself. It's a plumber the pluffer plumber puffy That you see in the corner so a little history lesson about relay D and In the in the latest context Then I will talk about some new features and enhancements or things that have been requested many times over the years and I finally unslacked and fixed or finished them About HTTP 2 and what is being worked on there and some some other future Stuff and then I hope you You have some questions Afterwards, so we lady got imported in 2006 So it's almost ancient now And it was in interesting piece of code because I Wrote a little demon that was able to do health checks on on different web servers like using ICMP checks or TCP checks or a little HTTP check and then based on the health check results Configure a pf table to so in the back in the days We had these rdrs in pf and the rdrs code only Forward to a round robin table so pf got the request and uses the round robin table To send this to a backend host and this was static So if one of the hosts in this table was down pf did it anyway So the little idea was just to health checks and based on that populate the table and Then about the same time PR Eve showed up and He has Had implemented something very similar this SLBD simple load balancing demon that he did And it was the same idea and it had a few few more things That I didn't have in my implementation and and he liked others. So it's like oh great. Let's import This this load balancing demon I convinced you and We can also get a new developer because we also imported PR Eve in the project basically so Based on that we we we worked on the code made it ready for the tree and this Because I didn't like the name SLBD We named we renamed it to even worse thing to oh My gosh, but what oh it was host state D in the beginning and And this yeah, that was the famous first comment with many to do's so It got in in 2006 in 2007 So in the beginning it was basically just running health checks and then configuring pf4 this as I say redirects or the layer three forward But then I got asked Myself by somebody who who wanted to to run an a load balancer in a big web application and was a platform where you could buy a screams from German soccer games and all that and This magenta company was involved there and they asked oh, can you do load balancing and do your support and SSL termination I said sure and then I looked it up and Implemented it actually and this is how relay D got support for the the the relays a layer seven Part so the difference is one thing I can terminate here TLS connection on the front and talk HTTP to the back end so this requires me to to go on all on the Application layer now right so this is basically SSL team termination in this Particular our product as well as good because it was it was a free low test basically because thousands of people at the same time wanted to Order their soccer game that they could watch online and all that and I Was sitting in the data center with with some people watching the game and at the same time see oh really D crashed Okay, let's fix it run it again and so it was kind of live Testing the good thing is we had relay D redundant. So if one crashed the other one took over and nobody noticed any problem So this is how relay D got the relay part if you're now the syntax it has redirects and relays So the relay parts came a few months later You can do a little bit more so for example instead of just doing the load balancing based off on source IP What pf basically does you can look at at the cookie value? so that you Maintain some persistence. So if a client provides a specific cookie you send it to Does the same back-end or you hash in some other values to make it less obvious for for some attackers basically so This is all part of the layer seven Over the time it got renamed multiple times SLBD in the beginning as I said host state D, which I never really liked it was a compromise Host state D. We thought it is clever until some Americans told me that the first part is not Really a good idea and the last thing I came up with a name Relay D which because it is more than just something that monitors the state of servers it is relaying from a to b and I even like a relay the name More than like proxy because a proxy is usually very application specific and a relay is very generic Relay D can do generic TCP, but it can also do TLS and HTTP so Getting stuff from a to b in the middle. Now, that's what relay D does and a few years later Relay D got an offspring basically when we removed engine X from open BSD and We we needed and Another web server. I said, okay, I can we turn relay D into a web server because it already has Part of the implementation, right? So HTTP is its own piece of software, but it's based on relay D still and it tried to sync code back and forth Okay, long history but meanwhile in 2019 The web has changed a bit, right? So everything is distributed and everything is like Cloud and apps and whatever and people don't even use real web servers anymore You have some Node.js and whatever is fancy right now So We we see okay, we need to clean it a little bit up Technically like like cloud just changed to use lip TLS instead of open SSL That's just the detail, but this enables us to use modern TLS versions basically Or SNI later and But still what relay D Lacks or lacked partially is do some Filtering for the requests blocking based on source IP things like that or Assume forward to a back-end where rather nodes have different ports This is one thing that in the in the in the past you had like a pool of web servers and all of them had their own IP and And but all of them were running on port 80 and now this is not the case anymore If you have some whatever Docker farm and and things like that. Yeah, so this already existed in a way in 2007 that there were some I think ruby Frameworks where you run the services on on some weird ports and some front and forwards it But I always said that's stupid who does it right? So we never supported it Yeah, so it's about small features that we added and then there are available now and then The biggest thing to add is just we're running on an old version of of the HTTP protocol right relay D is still HTTP 1 1 only It works in most of the cases. I mean there's no regression yet, but It can be better. It can be maybe faster and all that with HTTP 2 so In May I decided to to have a hackathon with myself So I went to the craft work Which is kind of a cafe co-working kind of thing and sit on my chair and was hacking there and people were looking at me What is he doing there every day as I had my one big hackathon where I implemented for some of the obvious things and Yeah, some some others So one thing is that was actually a bug and you about this bug It was a little bit hard to fix, but I finally did it. So relay D has So you have the relay statement Usually a listen on and a forward to so one side the other side and then you can attach a protocol to it for well for the in this case the HTTP filters So in the past The problem was So I added this possibility based on a back end or on a URL path to switch the back end Which is very common, right? You have a website and it has static content like slash images or slash static and you want to serve this from the Whatever small static web server and then you have all the dynamic stuff you want to serve from your Node.js go rust whatever wrap application so in order to do this you have to write a Such a filter and then you can switch the the back end It only contains one IP here right now, but it can be a pool of multiple hosts as well so the problem is This didn't really work Because relay D took in a keep alive session like HTTP Keep alive a persistent connection it it selected the back end for the first request and Then This was sticky right then it kept on using the selected back end for all connections of the same of all requests of the same connections now I have it So the work around was really terrible basically enforce the connection to close For each request So like in the old time like one request per TCP connection This work, but it was not a nice way So I fixed this finally to be able to to remove this and now you can Really use use this feature to have multiple server back ends Um in order to use it it only works with tables right now, but as I said you can put one IP address in there and There's there's some special implementation details. So you have one listen on and one or more forward two rules So the first one is always the default That gets selected The other ones are kind of ignored unless you you're selected with a rule But you have to define them here as well You can also add a Health check statement there if not, then their hosts are always online So this is working now also like since May and The from two filter rules and something that I added Actually, the code was all partially there and we lady and I never finished it somehow and I forget about it but now you can you can write these rules and specify source destination matching syntax so that you you give a specific URL only to to Internal hosts and think like that. Yeah, so you can so it's Yeah, I know it looks a little bit like pf, right? It is not pf it People said is it really wise to use the same syntax and I said well it is an application layer Firewall In a way. Yeah, so this is possible now. You can also block Based on IPs or network rangers and things like that and it's evaluated and last match in order of course SNI so If you ever looked at open SSL itself and how SNI is a server name indication is Implemented if you if you if you use the library it is kind of a mess, right? You you have multiple SSL objects and you somehow need to to deal with this with some callbacks and all that and so In Libre SSL we have lip TLS, which is a really nice TLS library That is sits on top of the lip SSL library But it's really easy to use it is it's safe. It's Nicely designed. I wonder myself. Why is it? Such a rare thing not so many people are using lip TLS. Yeah, I think lip TLS is one of the main Advantages of Libre SSL. It's a really nice library. Just just try to use it You can make a TLS server with just a few lines. It's very easy to use and there are some We used it with rust as well and all that because also very easy to integrate lip TLS in other languages And you don't have to deal with all this op-nesses elements So lip TLS made it easier to add SNI support and SNI basically means instead of having one server certificate So server has a default certificate once again, but it can select a different Certificate based on the server name that is sent as part of the TLS header or it's an extension So now you you can By default relay do you don't have to configure your TLS it looks up there the when you do SSL it looks up the TLS in a magic pass yeah, etc SSL then The relay dot key and and things like that. It's documented in the man page But now you can configure multiple months and it was it still uses this pass etc SSL and so on as documented But then with the specified Basename file name so you add multiple ones and then if they match it uses a different key Very easy to configure actually just drop in the new new keys and And add a line for each and then your SNI works And then in the back end your web servers can actually deal with the different host and and all that I wonder who has looked up this IP address yet, but anyway OCSP another thing HTTP got OCSP support like some time ago. It's even mentioned in the Httpd and relay D mastery book Shameless plug from Michael W. Lucas So there he Mentioned and describes how to use OCSP with Httpd. This is called that what back at it And now I finally got this over to relay D with a little different syntax because relay D has this you see the magic pass again So when there's a file dot OCSP Either for name and port or just name then it will use it. So the OCSP is basically I'm not a TLS person per se But it basically means you you ask the West web server and the web server can give you a signed OCSP response from from the CA that says Tells you about the validity of the certificate OCSP stapling and So when you do that it it really you don't see a functional difference But it makes it easier for your clients to look up the validity of the certificate and then fasten things up because you don't have to do an external request to a OCSP server or use a Revocation list or something like this So the route to HTTP to it started really nicely Took this picture from a motorcycle trip this year So yeah, you have some Stones and it's rocky and all that but on the other side you have the beautiful view on this is fear much that are they right? so it looked like that and Okay, it's like okay I did these features on my list and then I started well We do need HTTP to support after sometime like open VST very often is and We get asked about in a new thing. We say no, that's terrible. We don't do this Like a CPI move terrible thing and APM works great, right? So we didn't get a CPI until somebody implemented a CPI and With a lot of work. We have a really nicely working a CPI stack the good thing was a CPI boss We didn't drink the Kool-Aid and we didn't get the reference code. We have like a really nice and Working implementation now the same Herpenbeb with VMD, right? We hyper-visors and VMs are bad but now we have a really nice implementation that Contains VMs much better than QM or does and and so on and with HTTP to we said this is HTTP HTTP to and these What was the previous name excuse? I just lost it skippy. Yeah skip Speedy speedy. Yeah, why do I think skippy as speedy speedy? So speedy was yeah was basically this in the beginning speedy had one major difference how it compressed the headers But you say oh, okay all these additional states and multiplexing that looks very dangerous, right? So I think it it's very easy to make it wrong So we were skeptical about this But I said, okay now it's time actually 2019 as the time since HTTP 3 is is it already standardized or they plan to do it this year I'm not sure if the RC is out yet But so since we get HTTP 3 now HTTP 2 is vintage computing. So let's implement it, right? So I started working on it But but because it needs so many changes in in relay D I started something very nasty instead of sending millions of patches around on an open BST list I started just moving ahead doing stuff and then later when I feel confident enough I I will share it and incrementally not in one block But instead of doing this just silently on my own I said, okay, I shared this github you can follow it But this is like work in progress code, right? So HTTP 2 is multiplex multi-stream protocol. That's a major difference It is using header compression and then it's a small binary protocol So instead of like opening a telnet and typing get HTTP 2 whatever or NC You know you you basically have the same you have the headers But they're all encoded. So it's binary protocol for the headers and for the communication itself One thing that they did all headers are lowercase now And They define some standard headers that the most common ones with these colon syntax So method is now a header not something of the String anymore and authority used to be the host header But you can still add arbitrary headers below that like user agent or your own x organization and all that So I'm not going too much into the HTTP to Detail but one thing for examples relay D was written in a way that it expects request response Or actually it does support pipelining in a way that the client sends multiple requests and then you answer these reclass crests in the same order So it's Possibility and HTTP one It's broken in many cases, but it's still They try to solve some issues with it and that never really work But usually it's still the same request response scheme even if you just make it in chunks But HTTP to like one client connection like a single TCP TLS connection Can I can open multiple streams? And this totally asynchronously so you open a connection for your web server and the web server can push Responses to you basically because there's some picture like the logo of the company you're accessing website that you might need So it's just sending you the logo without you ever asking, right? That's one feature or you you can have this one connection and then start multiple streams Connection Requesting different things like the CSS and the JavaScript and the page all at the same time over one connection And then the server sends the responses in in frames and in packets With their stream ID so you know where they belong what they arrive totally out of order It doesn't matter anymore So you just need one Connection and you can also send your DNS requests over it, right? And so it's all going there So really he was not designed for this so one word This is actually a branch in my my code because it's I Need to merge it in at some point, but it's it's a lot of Mechanical work in the tree. I had to split like one main structure that was used for the relaying request response I had to split it into like a peer like one side and Then multiple streams to it so that like one peer can have multiple streams and So the way so you can dial it directly access some fields like the peer address anymore And then you you look look it up and see so this is a large diff actually And I'm really scared about getting this in but I get an okay banner and then it's it's all fine Okay, I still have time H pack so I talked about header compression. So this whole thing these HTTP header is doing header compression actually in in speedy day, I think they use deflate or something and that was vulnerable to what was a crime attack so and so an HTTP to they use this HPEC algorithm and actually I After looking at it. It's really nice idea Let's just think about it. You have an persistent connection where you send multiple requests so some headers like past slash is Something that repeats all the time. So HPEC allows to reduce it to one byte Just in a static table and an index and you have other common these Colon headers as very common combinations where they just index and then you encode it like this and Then other things that might be your your string like user agent the user agent is usually a very large thing But it never changes So when you have one connection where you do 100 requests, you only need to send us user agent once then HPEC adds it to a dynamic table and Then the next request only sends you an index in this table So instead of the huge string for the second subsequent request in any of the streams in this You only send like the idea 127 or something like this. Yeah, that's a Very simplified Description of how HPEC works. Yeah, so your index well-known headers and you'll learn Other headers dynamically during a connection. So the request size of the headers can be stripped down to just a few bytes So I implemented HPEC It's not an open missy yet because there's no use case for it But the plan is to to use this in relay D and later HTTP D for the HTTP to code So it's it's secret. I looked at existing implementations But usually it's like, yeah, it doesn't really fit and actually it was a nice experience implementing this stuff and I looked at the I mean, it's always with algorithms, right? People learn algorithms are saying you have to do very smart and you have to invent your own algorithms to make them even better But all these tricks are really Scary once you do multiple loops and pointer Mathematics and all that so I said, okay my implementation. I think I want it to be fast, but The goal is not to optimize it as much as possible. The goal is to make the code somehow Readable and obvious So even if I allocate more data or my my whatever my space complexity is a little bit more than it could be The idea is to to have this a really solid piece of software so smart tricks are all always dangerous and I also want to utilize some like like Otto's Maloch tricks for example Like free zero and things like that so So This is the year also online the thing is I spent some days running AFL housing on my code And I have like hundreds of test cases to verify it actually AFL after running for like 10 hours found one really hard to find back and I was able to to fix it so I'm thankful for for that Yeah, so it evolved into a little API it's part of the man page So you see I even wrote a man page and all that so it sits there. Nobody uses it, but It intends to be clean so that we can integrate it later Yeah, but this is basically it we have h-pack we have the streams something that works with it and I can parse maybe the first HTTP to request but then After I submitted this talk actually the last month I Yeah, had some Changes so I couldn't really much focus on on continuing this work So is it way poor there now it is not because there's already some code and we have time I mean really the existence 2006 so we have time until 2026 or whatever to add HTTP to support, but I'm still working on this. Yeah By the way, this is a tunnel in Glarros in in Switzerland as well So I have some initial code. You see these magic HTTP to preface That's what they send over often you negotiated HTTP to with this TLS LPN you can switch to it So it is compatible that you support both protocols and then the first packet is this Weird preface. I don't really know why they're doing this because it's static But it's part of the protocol and then you have you have the basic header some kind of TLV, right? the stream IDs and then each packet so they're Different ones like data, but headers as well the the h-pack compressed headers And then some some other Packets that that you don't have and then HTTP one well world. I like go away. I think that's still from the speedy Google days But also use the scary ones like window update and all that so that reminds me a bit of TCP, right? And that's a whole goal of like Removing control from the OS and moving it to the browser by by doing things like that And ping is a heartbeat for example I'm not sure How dangerous this is but then in just a month ago There was this this list of CVEs, right? You've probably saw on all web servers ten minutes, okay, yeah and Because all HTTP to implementations out there are vulnerable to to denial of service attacks and all that because they didn't get the Multiplexing right to have resource exhaustion attacks and all that all the things we were concerned about back in the days, right? so we were right and they were wrong and So we we can try to make it right now learn from these mistakes and Implemented properly the thing is some of these issues are really related to the protocol itself So you really need to Work around this, but it's possible. So I'm really glad that this happened before we finished the implementation so quick now just quickly is HTTP to over TLS over UDP or DTLS and this is HTTP 3. Yeah TCP is that we need this it is a UDP base protocol that you cannot really firewall anymore But the standard now, so Of course you can still block it Yeah, you have UDP port Is it 443 I think it's something you so you can block the quick port and I'm doing this on my firewall So so they they go to to the HTTP ports But this is basically the same protocol with the main difference that The HPEC algorithm got a little bit changed And this makes sense because HPEC has some state and It expects the individual request to arrive in order What happens on a TCP connection? but now with was quick they can arrive out of order and then it The HPEC might cause head of line blocking where you wait for a specific state to to advance But otherwise, I have to admit once we have HTTP to it might not be too difficult to add quick support But no, we're not doing this Okay, that's about almost the last slide Real AD so let's rewrite it and rust because that's what everyone doing is doing now Actually, I do started to like rust, but yeah, I'm kidding really seriously I did implement MG in rust RMG after they announced Emacs and rust are Emacs a serious project. I made my not so serious project RMG But this is Basically on my roadmap. There's not so much. I know right now. I think HTTP 2 is the most important thing We did get a few other features like new binary health checks I know and some web sockets compatibility So we got some user contributed patches into real AD as well. Yeah, just keep on sending them So for the questions if we still have a minute. Yeah, we do I Would also like to ask you right now. What do you think is needed these days? For load balancer or what can you do with another load balancer like engines? That you cannot do with real AD I Will utter the evil words like full rooting domain support So you can run multiple real ADs in multiple different rooting domains. It's still not there Yes, I Have a patch I Have a patch now for about half a year or maybe yeah like this That allows one real AD to to to run in multiple our domains and relay between them and all that And I think I have to upstream this patch Somehow It's not too long Yeah, I have to really this this is a good thing I should remember just Digging out this the patch and to do this which I really wanted as well I actually used it, but then I didn't have time to polish it up and release it I had I was using VMD were multiple VMs were running in different routing domains And then I had a real AD in the front end to to do some like TLS termination per routing domain for that I had exactly that's problem. Yeah, I Didn't really understand how the lady works with HTTP to on the back end So I get it correctly that the age really the support is supposed to support the HTTP to in the front end as in terminating the HTTP to Yeah, but can I also use it to like establish new HTTP connection to downstream? To the upstream that's that's a good question. So he asked what I do with HTTP to support in the back end and Talking to the server Not in the in the front end part, right? Well, the basic idea of HTTP to is more or less if you look at it it it's bi-directional so you open a connection and Of course, you want data from from the server but every side can make push or Request so it's it's not such a big difference Talking from a client to the server then from a server to the client because the handshake is almost the same and then all that so my plan is first on working on the Reladies server side to the outside and Then yeah in the beginning it would have to Most probably that's in my poc that does one request. I'm talking HTTP one to the inside But in the end it doesn't really matter. I mean relay D for example long before we had a of two in pf Supports not six four it can do six IPv6 on the outside before on the inside So just generally it doesn't really matter you can use different protocols and then the relay can translate it But the plan is first accepting HTTP to and doing HTTP one and the inside I wanted to ask about the tooling for debugging HTTP to connections on open BSD What would you use? What would you probably need to implement for instance that you I mean nowadays? You just netcat to a server and you know writes in it in the text But it that's probably isn't really possible anymore So are you thinking about writing some tool that you could like plug between netcat and you and that would like Create this binary Stuff well, maybe even the streams and stuff so you can actually test it with something that is yeah scriptable Well one thing When Google did speedy that did one thing right because they define speedy to be TLS only So there is no plain text version of speedy And then the committees came in and they said all we need HTTP to over a TCP as well And I'm not sure if I think the browser's didn't want to implement it I'm not sure if they do it now or not, but in theory at least that was the design You cannot netcat it because it's TLS always and I think it's a good idea I don't plan to support HTTP to over TCP at this point because well it could but That's not the idea. The idea was that it's always TLS. So then you cannot netcat it So how do I debug it? Well, I don't know printfs and all that you could probably I Mean in I D we had something like this or I think can be The demon could put it into a PCARP format that you can use TCP dump on it, but Yeah Sorry, just a plug if you if you extended netcat with lip TLS because it's so nice and easy and What not then? You could basically establish a TLS connection using netcat and then Then you would have a basis for debugging Maybe or you something you can try to write a pet to modularize. It's better Yeah, yeah, well, maybe it would be interesting thing to do this SSL men in the middle in netcat Okay Other questions. Do we still have a minute one more question or something like this? Yeah one last question Okay, otherwise. Thank you very much panel Yeah But but the the God talked trade the train tunnel is is much longer Yeah, but I live in Switzerland, so I like the tunnel Okay, thank you