 Hello, and welcome to HTTP2, the sequel is always worse. Have you ever seen something that was so complex, it just had to be hackable, if only you had time to understand it? HTTP2 is a beautiful beast, but it is complex and where there's complexity people take shortcuts and things go wrong. In this session, I'll show you how you can use new features in HTTP2 for a range of high impact attacks, and also how those shed light on a type of request smuggling that has always existed but never been noticed. Complexity causes trouble for me too. I first looked at the HTTP2 spec back in 2019 when I was doing research for HTTP decency attacks. I loaded up the spec, looked at the size of the browser scroll bar, and then proceeded to skim read it so fast that I didn't even bother reading the security considerations. My second encounter with HTTP2 was after I presented this research at Black Hat USA, an audience member asked me, did those decency attacks work against HTTP2? And my answer was no, it's completely secure. My third encounter was the same day I was at a party after a few drinks and someone else asked me the same question, and I gave them the same answer. And then they proceeded to explain to me exactly how you can use request smuggling to exploit HTTP2. And it's tricky to explain this next bit without coming off slightly badly, but at this point in time, I just spent nine months exploiting request smuggling. So the prospect of immediately diving into more request smuggling work didn't exactly fill me with joy, and instead I just put that idea on the shelf and went off and spent a year researching web cache poisoning instead. One year later, I came back, tried the idea out, and it really worked. I could hack loads of interesting servers with it. There was only one fly in the ointment and that was Bitbucket. Every heuristic I used said that Bitbucket should be vulnerable, but every exploit that I tried reliably failed. And normally when I encounter something like this during research, I spend a bit of time on it and then I just abandoned it and move on or I'd get stuck forever. But I'd previously seen this type of behavior when I was doing the original decent catech research and I was determined not to let it escape me a second time. So month after month, in between exploiting other systems successfully, I kept coming back to Bitbucket. And eventually I had a breakthrough. Thanks to something that was a complete fluke, I found proper evidence that there was definitely a vulnerability here. However, what this didn't do was actually give me something exploitable. So once again, I had to put it on the shelf and move on. But other than that, things went really well until around March when Emil published the research that he'd been doing on the same topic as me at the same time. And this left me in a bit of a tricky situation because when I present some research it has to contain some novel content. But all the novel content that I was planning on presenting had just been presented publicly by Emil instead. So I really needed something new and therefore naturally I went back to Bitbucket again. And this time I finally cracked it. And this led to a cascade of findings including a new more powerful type of desynchronization core type. An entire class of issue that was previously useless becoming practically exploitable. Atlassian having to log everybody out of JIRA worldwide and contacting the computer emergency response team leading to further findings and Atlassian awarding me triple their maximum bounty. Out of that mess I've managed to extract some information that I hope you'll find really quite useful regardless of whether you've seen Emil's presentation already or not. I'm not going to be approaching these topics in chronological order because looking back at it it doesn't make sense even to me. Instead, first of all I'm going to show you how to use HP2 for request muggling just like what I was told in the party in Vegas. After that I'll focus on request tunneling and show you how to confirm and exploit this largely overlooked type of vulnerability. Then I'll share four or five new HP2 exploit primitives and finally I'll cover some practical considerations such as common pitfalls and share some tooling and talk about how to prevent these attacks. If you're watching this live at Defcon then feel free to hit me up with questions on Discord or on Twitter as you prefer. Although HP2 is complex, there's only actually four key things that you need to understand in order to wield it as a weapon. Here we can see an identical request represented in HP1 and HP2. The first key difference is the request line. HP1 has the request line containing the method and the path whereas HP2 has removed this concept and replaced it with pseudo headers which look just like normal headers but they begin with a colon. These are used to transmit more or less the information that you would expect in the request line. The second difference is that HP1 is a plain text protocol so when a server parses a request it has to do things like look for a new line in order to know when one header ends and the next header or the body starts. HP2 meanwhile is a binary protocol and the headers are represented using key value pairs. So what that means is when I show you a HP2 request in this presentation on a slide what you're seeing is an abstraction. It's not the actual bytes being sent on the wire because they're binary and not readable. For example these pseudo headers aren't actually represented as header names starting with a colon on the wire. They're just mapped to fixed bytes which are predefined in the RFC. The third key difference is that with HP1 the length of the message has to be communicated via either the content length header or the transfer encoding chunk header. HP2 meanwhile has requests being composed out of frames and every frame has its own built-in length measurement that can be thought of sort of like TCP. And what that means is that in HP2 you don't need to specify the length of the message in a header and there's also no real potential for ambiguity about the length of a HP2 message. And that's why I thought that request smuggling against HP2 wasn't possible. The final key difference is to do with the way that the protocols handle sending multiple requests over a single connection. HP2 does support this but it's not very well understood. All you do is you open your socket, you send your request down it, you read the response back down the socket and you send the next request straight down the same socket. Just concatenate it on the end and read the response back like so. So you're relying on the responses coming back in the order that you sent the request to. HP2 meanwhile decides they don't like this because it is pretty sketchy and instead they introduce this concept of streams. Every frame therefore every request and response has a label which tells you it's stream ID. A stream is simply a request response pair. So that means that the server can send the responses back out of order but thanks to the stream ID the client can associate the responses with the appropriate requests like so. Now that we understand HP2 let's see what damage we can do. As usual all the case studies presented in this section are based on real targets to have bug bounty programs. All these vulnerabilities were found using an open source tool that I am releasing as part of this presentation and of any bounties earned over half have been donated to local charities and the other half will be spent on beer when that's once again permitted. The section will be travelling quite fast so if you're not already familiar with request smuggling and you struggle to keep track I suggest pausing it and checking out my prior presentation, HTTP de-sync attacks. So why is request smuggling possible against servers running HP2? It's because the vast majority of servers that speak HP2 actually only speak HP2 with the client and then they rewrite the requests as HP1 in order to talk to the backend server. This behaviour which I've dubbed HP2 downgrading is ridiculously common. For example Amazon's application load balancer does it by default and you can't disable it, you can't tell it to talk HP2 end to end. What this approach does is it basically dodges all the security benefits that should be brought by using HP2. In fact it arguably makes things worse. When you get a classic request smuggling vulnerability it's typically because the front end and the back end disagree about whether they should use the content length or the transfer encoding header. However if the front end server is speaking HP2 then there's basically no possibility for the front end and the back end to directly agree because the front end is going to use the HP2 message length and then the back end doesn't have access to that data so it has to use the content length or the transfer encoding header. If that sounds like a mess that's because it is. If you take a server that supports HP1 and you turn on HP2 support you've just doubled the number of ways that it may be vulnerable to request smuggling. We're going to start with an extremely simple case study. The RFC for HP2 says that even though the content length header is not required you're still allowed to send it provided that it's correct. What could possibly go wrong with that? Well the front end used by Netflix on their main website where you watch videos and stuff forgot to verify that the content length was correct. So if I send the HP2 request you can see here then it would get translated and downgraded into a H1 request that you can see on the right and that thanks to the incorrect content length will get treated as 1.5 requests on the back end and the malicious prefix shown in orange would get stuck on the start of the next user's request. I crafted that prefix so it would redirect them to my server and by running that in a loop I could hijack other people's JavaScript imports in real time, compromise their account, still plain text passwords and credit card details and so on. That was traced back to the Netty library and Netflix awarded me a $20,000 bounty for it. After that motivating start let's move on to something slightly more complex. The H2RFC also says any message containing connection-specific header fields must be treated as malformed but it's a bit vague about what happens if you fail to obey this line so I'm going to fill in the blanks. One server that failed to obey this was Amazon's application load balancer and it meant I could exploit pretty much every single website using application load balancer. For our first case study we will take a look at Verizon's law enforcement portal thing. So here I'm sending a HB2 request containing the connection-specific header field transfer encoding chunked and because the server has failed to reject this it's been forwarded on to the back end and the back end has prioritised the transfer encoding header over the content length which is the one that's actually correct and once again I can cause a desynchronisation and append arbitrary content to other people's requests and once again I'm using this to trigger a redirect. When I reported this the triage just said well you haven't really proven that this is dangerous so I decided to try and redirect some live users and I found very quickly I ended up redirecting people who were in the middle of an OAuth login flow and thereby they ended up leaking their OAuth secret codes to me via the refer header. That worked on application load balancer as I mentioned and also on everything using Incapsular's web application firewall which is supposed to make your website more secure. That led to a $7000 bounty from Verizon. Notably this didn't get me any bounty from Amazon or from Incapsular. There's a bit of a disconnect between the vendors that write the vulnerable software and the end users that end up paying for it. On another server with the same vulnerability I found I could once again redirect arbitrary users and I was once again told you need to prove that this is really dangerous so I once again started redirecting real users. However this time something really quite cool happened. I started getting requests that landed on my server that said something along the lines of hey there I'd like to have permission to send you my credentials. So I hastily made some tweaks to my server to make it say yeah absolutely go ahead send me your password and sure enough they did. I've got a brilliant video taken with TCP dump showing a stream of creds landing on my server in real time from their live traffic but unfortunately it was near impossible to redact so I can't share it. This led to a $10,000 bounty from Verizon. Right that was the basic stuff now things are going to get a bit more interesting. One cool thing about H2 is that being a binary protocol it lets you put arbitrary characters wherever you like and then it relies on an extra layer of server logic to say actually no you shouldn't be putting that character in that position. Firefox's start page at start.mzilla.org was powered by the Netlify CDN and they forget to say you shouldn't put new lines in header values. This led to the request header injection vulnerability that I'm using here to smuggle the transfer encoding chunked header and cause a desynchronization. Here I've grafted the malicious prefix to fetch data from my own site on Netlify and thanks to Netlify's cache that harmful data would then get saved and basically persistently give me full control over every page on every site using Netlify. I reported that to Mzilla and Netlify and got 2k off each taking the total earned so far to 41k. When I tried the same technique on Atlassian's JIRA something quite unexpected happened. I expected to see two responses coming back to me the normal response and the poison response but what I actually saw and you can see here is a huge range of different responses clearly coming from different JIRA deployments intended for different people and containing a huge amount of sensitive information. How did this happen? Well I wasn't sure initially but I think I eventually figured it out. The problem was I realised that by putting new lines in headers I could actually place my entire malicious prefix inside a header instead of putting it in the body which meant I didn't need to use a body and therefore didn't need to use the post method and so it might work on more targets. That was fantastic reasoning on my part but the thing I forgot to figure out is I didn't account for the front end needing to terminate my request by adding on a couple of new lines. So my smuggled request looked like this and I thought I was sending 1.5 requests but when it got downgraded thanks to the front end sticking the new lines on the end I was actually sending exactly two requests. So what happened then? Well the first response went to me as usual and the second response went to the next user but because it was exactly two requests their response then went to the next user and so on and basically the front end completely lost track of which responses should be going to who so random responses to everybody and thanks to the set cookie header ended up persistently logging random users into random accounts leading to Atlassian having to expire all sessions and log everybody out. So if you find a request smuggling vulnerability and someone demands to see the impact of it smuggling exactly two requests should get them the evidence that they're looking for. The root cause of this was Atlassian was using the Pulse Secure virtual traffic manager front end which shouldn't be confused with Pulse Secure's notorious VPN we also saw that this worked on Netlify and as usual with most of my techniques it also worked on the ImpervaCloud WAF thing. Thanks to the work of the computer emergency response team there are more advisories from other vendors vulnerable to the same technique on their way if you check out the white paper on our blog you should find that it's their reference there by the time that you're watching this. While Atlassian was waiting for Pulse Secure to release a proper patch to fix this vulnerability they tried deploying a few hot fixes themselves and these issued some of the other potential bypasses that you might fight. The first issue was they were filtering header values but they weren't filtering header names exploiting that directly isn't super easy because it results in an invalid request hitting the back end and the request being rejected but HP2 on some servers lets you put colons in header names making exploitation actually quite straightforward. Another issue was that even after they patched that they forgot to filter pseudo headers so we could get injection inside the request line and exploiting pseudo headers is pretty much just the same as exploiting other headers but you just need to think about what the resulting downrated request will look like as a valid request line so that your request doesn't get rejected. The final issue was that in the path pseudo header only they were only rejecting the Slash R Slash N sequence so if you could send Slash N by itself you could once again exploit them. In summary we've seen a range of techniques that you can use to exploit HP2 downgrades and achieve request smuggling. Now I'm going to take a look at something less flashy, less obvious but still really quite dangerous. When you find a request smuggling vulnerability the possible attacks are heavily affected by the approach that the front-end server use to decide whether to reuse an existing connection to the back end or to create a new one. Normally you can exploit everybody fine because there's no particular restrictions on this but sometimes you'll find that when you're trying to exploit other people you can only poison the response to requests coming from your own IP and the reason that that happens is because the front-end has decided it's going to establish one connection to the back end per client IP. However this is still exploitable via cache poisoning and you can still use the regular old techniques to leak internal headers and thereby use that for request tunneling. So it's not that bad. The most extreme scenario is what I'm going to focus on. This happens when the front-end refuses to ever reuse a connection to the back end server. Some servers are just set up like this and also Amazon's application load balancer has a feature called HP Decent Guardian. It doesn't work against HP2 so it didn't prevent any of the attacks shown earlier but against H1 if they see a suspicious request they will try to mitigate the damage it might do by putting that on its own connection. So I'm going to show you how to establish that this vulnerability really exists how to confirm that you found it and also some new exploit paths that will let you turn it into something really quite high severity in the right conditions. Let's visualize what's happening here. Here we've successfully smuggled 1.5 requests to the back end and then sent a follow-up request. But because the front-end is only sending one request down each socket the follow-up is being sent down a different socket and the poison socket is effectively being discarded. So the victim, the follow-up is completely unaffected. This, as I mentioned, is going to cause us numerous practical problems. The first key problem is that although the server will flag up as being vulnerable using the regular time-out-based request smuggling detection technique the usual confirmation technique of sending follow-ups will, as we've seen, just seen fail spectacularly so it's easy to mistake this for a false positive. We need a different way of confirming it. Now you might think it's quite easy just smuggle exactly two requests and see if you get two responses or maybe smuggle 2.1 requests if you don't want to cause a disaster that we saw on JIRA. But unfortunately this response that you can see here containing two responses doesn't actually show us that this target is vulnerable because this is just how HB1.1 works. When you're looking at this response you can't answer the question does the front end think it's sending us one response or two and thereby you don't know if it's vulnerable. HB2 fixes this problem for us with absolutely no effort on our part. If you see HB1.1 headers showing up in the body of HB2 response and I promise you if you try these techniques out you will then you know that this server is really vulnerable to a quest tunneling. The second problem is that request tunneling is often blight because the front end looks at the content length header coming from the back end in order to know how many bytes it should read off the socket and pass on to the user. So we can smuggle two or three requests to the back end and the back end will generate responses and it will try and send them to us but the front end will only show us the first one which is the one that we don't have much control over and isn't particularly interesting. This makes life extremely difficult and this is the behaviour that was present on Bitbucket that drove me completely insane. Eventually as I was testing this endpoint on Bitbucket I started to get frustrated for another reason which was that the file being returned was so large it was making the whole UI lag inside Burp repeater and I thought you know what I actually don't particularly care about the response body I'm only looking at the headers so to stop the repeater from lagging why don't I just change the request method from post to hit thereby only asking the server for the response headers and that worked the server only served up the response headers but it included the content length header even though there was actually no content and that meant that the front end read into the next response coming from the back end and it proved that this target was really vulnerable to request smuggling so if you find blind requests smuggling you may be able to use the head method or potentially the options method to make it non-blind. I think that's a brilliant lesson on how sometimes you might not have any particular spark or insight but actually if you just spend long enough trying to hack something you'll basically find the answer if it exists just to regroup false. So let's say you've just confirmed your tunneling vulnerability and hopefully made it non-blind how can you exploit it? Well as we've seen you can't attack other users but what you can do is you can tunnel internal headers front ends often append secret headers to incoming requests that are trusted by the back end for critical functions such as knowing who you're logged in as but to exploit these you need to know what they're called and the normal header leaking techniques that you can do with regular request smuggling don't work. However if you can inject new lines and headers you can actually cause a different type of desynchronization that lets you disclose these headers. Take a look at what I've done here I've crafted the HP2 request such that both the front end and the back end think I'm sending one request so that means even if this is a blind request smuggling vulnerability I'm still going to see the response coming back where the two systems are desynchronized is they're not desynchronized about the length of the body which is the normal thing they get confused about instead they're desynchronized about where the headers end and the body starts so that's meant that when the front end appended its internal headers it actually from the back end's perspective appended them into this S parameter which is part of the body being used to do a search on the WordPress back end on bit bucket thereby leading to it reflecting a whole bunch of internal headers back to me. I also found that by heading different paths I could get the request routed to different back ends with different internal headers including certain secret keys. Finally if the stars are aligned you might be able to use tunneling for cache poisoning. Cache poisoning is one of the highest severity attacks that you can do with request smuggling and the head technique when combined with the ability to put new lines and headers enables a unique and extra powerful variety of it whereby you can mix and match response headers and bodies. So here I've chosen a set of response headers that just have content type text HTML and then I've chosen a different response that reflects my user input without encoding it in the location header. By itself that's completely safe you don't need to encode user input that's going in the location header that's going in the location header. I can poison the cache and get full control over every page on bit bucket and triple their maximum bounty for this plus the jury as you mentioned earlier taking the total earned in this research to 56k. Now I'm going to take you on a tour of some HP2 exploit primitives. Each of these use a H2 feature to give you some kind of foothold on the target and although it's a bit like on full case studies and bounties these are all based on real behaviors on real live systems. In H1 duplicate headers cause all kinds of problems but due to the way the request line is designed you can't send a request with the duplicate path. However because H2 supports pseudo headers you can now send a request that has two paths. Admittedly the server should reject it but plenty of them don't and plenty of them differ about which path they give priority to which is sure to lead to some unpleasant exploits in the future. Even better H2 introduces this authority pseudo header which basically replaces the host but it actually says the host is allowed as well and then it says that they're both optional. This means you'll find plenty of servers that do support both and if they treat them inconsistently you may let you perform host header attacks. Another cool thing about H2 is the scheme. It's basically just meant to be HTTP or HTTPS just that ASCII string but the cool thing is because this is H2 we can put arbitrary binary data in there and it doesn't have a direct equivalent in H1 so that means it's new attack surface and people thereby forget to sanitize it. I found that on Netlify we could put an entire URL inside the scheme and that was used to construct a URL on the back end and thereby let them get extremely confused although there was no clear security impact with this it's potentially going to be more usable on other systems and on another target they once again use the scheme to build a URL but they then try to route the request to that URL so I got a server side request forgery using the scheme pseudo header. Some H2 servers don't let you put new lines in HTTP header names but they do let you put colons in there so on some servers you can use this to get full request smuggling on the back end provided that the back end tolerates this irritating colon which ends up getting stuck on the end of header value however often the back end will be fussy and they won't like that so a better option if you find the front end has this gadget called the host header because the host header is expected to contain a colon and often servers just ignore everything that occurs after the current I did find one server that was vulnerable to request smuggling using this technique and I got halfway through exploitation had to take a break got distracted for a couple of weeks and then I found when I came back that the vulnerability had disappeared looking at my logs the server banner was reporting that they patched their Apache front end so I went looking to see if Apache had any security advisories related to this issue and I couldn't find one so I thought you know what I'll install a vulnerable version of Apache locally and use it for my demo I installed Apache in the correct vulnerable version and I couldn't replicate the vulnerability so maybe it's there if it's a request line or something else I found that Apache's mod proxy when it's doing HTTP2 downgrading lets you put spaces in the method pseudo header leading to a request line injection vulnerability when a vulnerable version of Apache and this works on the latest version of Apache it's a zero day at the time of presenting hopefully it's been patched by the time you're watching this like in the request line you can use this to bypass block rules implemented on the front end and also to escape any folders that they might try to trap you in finally a few practicalities firstly because HTTP1 and 2 share the same port when they're spoken over TLS then the client is relying on the server advertising HTTP2 support via the ALPN field in order to know that it's allowed to speak HTTP2 otherwise it will just default to HTTP1 and there's a whole bunch of servers out there that actually support HTTP2 but they forget to advertise this fact in the handshake so if you use a regular client like a browser or curl it won't speak HTTP2 to them and you'll think they don't support it and you'll miss out on loads of awesome attack surface fortunately this behaviour is extremely easy to detect so HTTP request smuggler the open source extension I'm releasing a major update to for this research will detect and report this but Pscanner will also detect and report it and you can test for it simply using curl with the following command line flags I found a real server that was vulnerable to HTTP2 powered request smuggling and had this hidden HTTP2 configuration the only catch there was that I could only exploit other users that were using HTTP2 which was nobody because they weren't advertising it hopefully you'll have more success than me on other sites you'll find that although HTTP2 promises amazing encapsulation between your requests sometimes you'll find that one request sent down a connection will break all subsequent requests sent down that connection even though it doesn't actually lead to the server telling you that it's got corrupted or closed in the connection so that's something to watch out for and another thing to be wary of is some servers treat first request down any given connection differently you can manage these two bits of annoying behaviour using the requests per connection configuration setting in Turbo Intruder which I've updated with a full open source HTTP2 stack that I coded myself and you can also deal with it and I'm going to be doing more research on this because I think there's some cool stuff behind the scenes there the tooling situation in H2 is a bit of a mess because H2 is a binary protocol you can't simply use something like Netcat or OpenSSL to speak you need to code your own client but coding your own client is a huge amount of work and you need to use a library but actually libraries will refuse to send the kinds of malformed requests that we need to send in order to trigger these vulnerabilities to get started with this research myself I coded an open source HTTP2 stack from scratch and integrated that into Turbo Intruder so you can use that library also there's a HTTP2 stack built into Burp Suite which is more battle tested and can also be accessed under API and via Turbo Intruder and it may be worth checking out HP2 smuggle which was released by Emil as part of his research and I believe works by patching Golang's HP2 stack as far as detection and exploitation goes as of today I'm releasing a massive update to the HTTP request smuggler open source birth extension which you can use to detect and also aid exploitation of these vulnerabilities in addition to supporting the normal timeout based technique that we've used in H1 and H2 it also supports using the head method that I showed you earlier in order to find request tunneling in a reliable manner that basically doesn't have false positives which is pretty cool I highly recommend HP request smuggler that's what provided oh it's tricky if you're a network architect then please if possible just you speak HP2 end to end don't do HP downgrading rewriting something from one protocol to another protocol is not a good idea and I'm sure plenty of other things go wrong that I didn't spot and put in this presentation so don't think just because you've defended it against this stuff means you're safe in HP 1.1 limitations about what characters you can put in what positions and as a developer I think probably you just need to drop assumptions that used to be safe to make in HP 1.1 like don't assume that the request method doesn't have spaces in it don't assume that the scheme is at all trustworthy why don't you just look at the actual scheme the request is using if possible there's a further reading available I recommend checking out the white paper I've published which accompanies this research and will contain slightly more up-to-date information on certain topics also we're releasing a bunch of online interactive labs so as part of our web security academy which is completely free you can try applying these techniques to real systems that we spit up for you on demand just so that you can hack them and gain some practical experience also it's probably worth checking out Emil Lerner's presentation on HP 2 for a different perspective on the same topic and I highly recommend also watching response smuggling pining HP 1.1 connections which is another DEF CON presentation and it introduces some new request smuggling exploitation techniques which should be completely compatible with HP 2 downgrade based desynchronization as far as primary sources go this is mostly based on my prior work HP desync attacks but if you want a significantly better explanation of response cue poisoning check out DEF PRAM's presentation on YouTube the three key things to take away are that HP 2 breaks critical assumptions at multiple layers of the stack HP 2 downgrades are extremely hazardous and should be avoided and request tunneling is a real threat if you have any questions hit me up on Discord or Twitter or chuck me an email don't forget to follow me on Twitter thank you for listening