 Uh, Luis here. This is his first talk at DEF CON, and give him a round of applause! Cheers! And, uh, he wants to talk to you about exploiting cache services, cache servers. Um, so, uh, I'm gonna let him go, and, uh, enjoy. Alright. So, thanks everyone for coming. Last day of DEF CON. Let's do this. So, this is edge-side-include injection, abusing caching servers into server-side-request forgery and transparent session hijacking. So, I know that's a very long title, uh, meant to mention some of the many things that you can do with edge-side-includes, especially when you're injecting it. So, for the rest of this talk, we'll be referring to edge-side-includes as ESI, for convenience, because that's a mouthful. Um, so during this talk, we'll learn about edge-side-includes. Uh, we'll talk about the problem that it was created to solve. Uh, we'll then talk about the problems that it created by introducing it in a unsafe manner, and then we'll talk about mitigation and migration. So, my name is Luizio Marcil. I work at GoSecure in Montreal, and to give some context on what ESI injection is, basically, it's a new class of attacks. Of course, it targets ESI enabled caching servers, so it's not a widespread attack, uh, unlike what James Kettle presented at Black Hat two days ago. So, this is really targeting ESI engines. So, this was discovered by mistake, basically. Uh, one of my colleague, uh, Laurent Lézoni at GoSecure, was tasked with reviewing the caching configuration for one of our clients. So, our client, which is a large ISP, basically wanted to cache overview of the security features. And we kept seeing references to ESI kept coming up. Edge-side-includes. So, we're a bunch at GoSecure, and we never heard about that. None of us ever heard about that, so we started looking for documentation. And we saw that the first and final specification from ESI was in 2001. And I don't know if you were doing web development 17 years ago, but security was not invented yet. So, we started looking into, basically, vendor documentation because the specification was so old, we thought, this can be right. And we kept seeing stuff like that. WordArt. WordArt in documentation is always a sign from an attacker's perspective because it tells you, well, I wasn't even in high school when this was done. So, okay. So, the documentation isn't going to help us, so I'm going to explain it to you. So, basically, let's look at this very primitive web page example. So, you have a weather web page. And to the end user, this is a single HTTP response, right? But to the end, to the ESI server, the ESI caching server, this is multiple fragments. And these fragments were invented to do one thing, which is invalidate individual elements of a web page instead of invalidating the whole document. So, when you think of caching, usually you'll cache a static file, and ESI is invented to cache dynamic files. So, for the forecast, Monday, Tuesday labels, you can keep that as a static fragment, but the forecast, for example, 27 degrees, you can invalidate that within the next hour. So, with this knowledge, we know that there has to be a way from the application server to tell the caching server where fragments stop, where they start and where they begin. So, this is done through fragment markers. Those fragment markers are in the HTTP response. And they look like this. So, you have an ESI tag followed by an action. It's basically an XML tag, right? Instead of having a secure layout, you just have tags inside of the HTTP response, which is going to get stripped once it's evaluated by the ESI engine. So, these tags are parsed when a specific HTTP header is sent by the application server. So, you're not able to inject ESI everywhere, but usually when they use ESI, it's implemented everywhere. So, let's look at our first feature. We have ESI includes. ESI includes are, in my opinion, the most relevant ESI tags. So, you have two files to show how this works. You have page one.html and page two. Page one is sitting on ESI server. The second one is on another server called API. And you can see in page one you have an ESI tag, which is an include tag pointing to page two. Now two things can happen here. You can have a cache hit or a cache miss. If you have a cache hit, it means that the ESI engine can just replace the tag with the content of page two. Easy enough. If it's not there or if the caching tree is invalidated because it's been too long, well, you have this file. So, the ESI engine is going to do a side request for this file to fill in the blank. So, whatever happens, this is what your end user is going to have. You're going to have the content filled by the engine. So, to illustrate how this works, let's look at a very example of a cache miss. So, you have your clients, your low balancers and your servers. So, your client is going to request one.html and the caching server has a cache miss, meaning the file is no longer valid. It has to go and get it. So, this is done through the upstream server. So, the low balancer requests this file to the cache, to the application server. This is sent back with the ESI tag saying, hey, please fill in the blanks with two.html. So, this tag is parsed on the caching server and the side request is sent to the API server. Now, the API server responds with the content of page two and the ESI engine is able to fill in the blanks. Our second feature before delving into exploitation is ESI variables. So, they're a very simple feature. It's a very simple feature that has no attributes, so no XML attributes, and basically the content of the tags gets expended to access metadata about the current HTTP transaction. So, you're going to be able to access stuff like this. So, the HTTP user agent, cookies, query strings, basically anything relating to the current HTTP transaction. So, now we know about ESI includes, we know about ESI variables. We also know that the tags are sent by the application server and they go through the caching server and this is where they are parsed. But there's a very important question that we have to ask ourselves, which is how can the ESI engine know which tags are legitimate and which tags are injected by malicious user? Think about cross-type scripting. It's basically the same thing except we're not exploiting browsers. So, that's a very important question and that's the problem. It's that it can't. And you're able to inject ESI tags and do basically whatever you want with the caching server. So, to illustrate this, let's look at a very basic example of ESI injection. So, you have the content of the city get parameter that's echoed back in the HTTP response. Now, the caching server is going to parse anything that is sent there. So, you can put ESI var and they're pointing to the user's PHP session ID. If you know PHP, you know that this is HTTP only cookie, meaning that JavaScript is not able to access this. So, if I'm able to access this from a caching server perspective, I can effectively leak a session cookie and effectively take over the account. And this works as expected. So, let's try to build a payload in order to do this. So, we're going to look at two ESI engine implementation. First of all is we're going to look at Apache traffic server. So, I looked at this one for two reasons. First of all because it's used by high-profile organizations. So, showdown tells us that it's used by Apple, Yahoo and Comcast. The second feature, the second reason is because they have the initial ESI tag implemented, but they added bonus features, some of which are security features. So, our first security feature is cookie whitelisting. So, even if you're able to inject ESI tags, sometimes you're not going to be able to access the cookies because they're not whitelisted. If you want to access the cookies, you have to programmatively configure the traffic server to say you can access this cookie. But by reading the documentation you also find out about another ESI variable called HTTP header. It allows you to refer to any header smart. Meaning that you can access the cookies. So, the whitelisting doesn't work. It's so easy to bypass. So, that was fixed when I reported it two months ago. Pretty fixed, quick fix. So, good for them. So, let's build a proof of concept to do HTTP only session hijacking with our JavaScript. So, I built an image source tag. You can do basically any tag that requests HTTP header, but this is fine. So, it's pointing to evil the local which is an attacker-enabled server for which I have a web server pointing there. So, the file name that is going to be requested by the victim's browser is an ESI tag pointing to their own header cookie. So, when this is going to go through the traffic server, it's going to expand the value with the session cookie and then the browser is going to request this URL. I'm going to access this in my HTTP logs. So, now let's look at how this would look in a real world example. Wow, it's working. All right. So, I built a very simple message board. So, you have your victim on the left, you have your attacker on the right. It's two different browser. So, there's no cookie contamination. On the middle, you have basically everything that is stored in the database. So, you have hi, hey. So, you can see what it looks like before being sent to the application server. So, now our attacker is going to put the evil the local, well, basically the pedal that we just looked at. So, the file name has the header for the cookie. And it's going to hit send afterwards. So, this is going to pollute the database of the application server. Now, when they send it, you can see in the database, it's sent properly, it's stored, everything is working fine. Then the attacker, by refreshing the page after sending it, basically attack themself. So, they're going to leak their own cookies because the browser is going to send a side request for that image, which is not really an image. Now, we're going to wait for the victim to refresh the page, and we should effectively steal their session. And as you can see, the session cookie appears, so we just stole their HTTP-only cookie. We're able to take over their session completely and become that user. So, we're going to look at the session cookie. We confirm that it's HTTP-only. We're going to replace our own value with the one that we just stole through ESI injection. We're going to save that cookie and save it. And then once we're refreshed, we should become the victim. There you go. So, we have HTTP-only. So, this is HTTP-only cookie hijacking without JavaScript through ESI injection. So, that's nice, but you need to inject a page for which a user is going to travel to. Sometimes, that's not always easy. Sometimes, think about self-excess. The impact could be great, but you're only attacking yourself. So, let's try to crack the impact up a notch. So, I looked at another ESI implementation, which is Oracle WebCache. So, Oracle WebCache is usually sitting on top of WebLogic application servers. It's not necessarily sitting on top of that, but we've seen it sitting on top of that. So, I looked at it because it's usually high-scale application and also because the initial ESI specification is implemented and they also have bonus features. Unlike ATS, they went with the least secure option, which is they added the ESI inline tag. This tag is pretty easy to understand. It allows you to override the engine, the ESI, it allows the ESI engine to override any cache entry with arbitrary data. So, here we're going to override jQuery.js with arbitrary content. What a great idea. So, jQuery.js is going to be filled with the content that you see here, which is basically a Ajax request. So, once the user is going to refresh, the file should trigger Ajax request to our Evil the local server and this is going to get expended, meaning that I'm going to get their cookie again because there's no cookie whitelisting in Oracle WebCache. So, now we know where it can override file and we can make the browser do anything. So, the browser is going to request this file because we took over jQuery.js. So, let's look at the demonstration of this, which is already running for some reason. What if I do this? All right. So, we have the same application server, but now it's super safe because the sys admin is they notice basically that HTML was not being stripped, so ESI was also being injected. Well, there was a possibility of ESI injection. Now, we can see that the victim is confirming that jQuery.js exists and the content is valid. So, we have jQuery 3.3.1. Now, our victim is going to refresh. The attacker is going to put, sorry, payload just to see if HTML encoding or escaping is working. And as you can see, the attacker is no longer able to put HTML char set, meaning that ESI injection is effectively mitigated. So, that's a problem for an attacker's perspective. Now, the attacker sees a feature which is a user list. This user list will reflect anything that is in the search box. So, that's a pretty good vector for either self-excess or ESI injection. And we can see here that HTML is not escaped. So, we're able to put our ESI payload in there, which is going to override jQuery.js. So, the attacker puts it there, summits it, it's echoed back in the HTTP response, and now if everything works properly, jQuery.js should be overwritten. Now, the attacker has effectively polluted jQuery.js. Nice. Now, the victim can just refresh the page and once anyone refreshes any page on this website, they're going to send us their cookie. So, there you go. We just overwrote an arbitrary file with arbitrary content, meaning I can either deface the website and steal anyone's session using ESI. So, that was basically a proof of concept to override arbitrary caching trees with HTTP-only cookies. You can use JavaScript, but as you can see, it's not really necessary. So, now let's talk about mitigation. So, if you like web application firewalls, you have ModSecurity. It's a pretty popular product. It's gotten way better in the last years, and if you use the OWASP core rule set, you're good for ESI. We talked with one of the developers of the ModSecurity team and they basically said, we already stripped anything that is XML-like, which includes ESI, so you're good with that. If you don't want to use a WAF or if you don't use Apache, you can use proactive escaping. And what I mean by proactive escaping is you might think that since you're okay, like you're escaping HTML everywhere, so ESI is the same chart set I should be good, well, not necessarily, because when you think about it, contextualized effort of escaping will often ignore HTML and JSON, meaning that HTML is never escaping JSON because the content type is already telling the browser don't interpret this as HTML, but we're not exploiting browsers, we're exploiting cache servers. So here, I can put an ESI include tag and a JSON response and it should work fine, right? There's just one small problem. We have an invalid ESI tag because of those backslashes because of the double quotes, but ESI engines have a very flexible syntax which is nice and I can just drop them. So this will allow me to do server-style and JSON response. Let's look at a brief example of this. So you have effective REST API sitting on slash api, slash me, which is going to respond to basically a small JSON payload. So you see that my full name here is Luizio Marcel and I can overwrite this with an ESI include tag. This ESI include tag is going to say please fetch REST server slash server status, which is just some server sitting on the local area network of the cache server. And you can see in each area, it went ahead and fetched that file for me, meaning that I can do server-style request forgery with ESI includes. Now most ESI engines will not allow you to do server-style request forgery on arbitrary host, so you have to whitelist them prior to doing ESI include of them, but some implementations will just allow you to do server-style request forgery through anything. For example, squid cache just allows you to do ESI include whatever you want, like I just changed the content type to text HTML so that you can see that the JSON response is the greenish area, and then the redish area is basically the content of the server-style request forgery through ESI injection. So to illustrate how that worked, I used the same imagery that I did before, so you have your slash me which responds with an ESI tag saying please get that file for me. The file is fetched and then the content of step five and three gets concatenated together and I get server-style request forgery, I get it content by ESI includes. Okay, so if you want to do ESI injection, you need to first identify if you're messing with an ESI enabled caching server. Someone on Twitter called Alex Bircin came up with a pretty smart solution which is leveraging ESI comments. ESI comments basically are tags that are going to get stripped by your ESI enabled server, so if you have this HTML comment-looking tag which is removed from the HTTP response, then your ESI engine has removed it. If you do the same thing but instead of ESI in the comment, you put something completely different, for example, foobar or just foo, and this one does come back, then you're probably messing with an ESI engine. If you don't want to mess with manual detection because it's pretty painful, you can just use automatic detection. So you have burp active scan plus plus, burp upload scanner and sure how Ikinetics does it, but I know that burp active scan plus plus and burp upload scanner are using the aforementioned heuristic, so it's really reliable. If you think that ESI injection, well, ESI as a feature is a good example of a robust caching mechanism and you want to implement that, I personally wouldn't go with ESI because basically it's pretty broken, but you can use cloud fair workers which are basically JavaScript files that allow you to do basically fragmentation of your HTTP response. So you can see on the bottom screen you have a fragment pointing to footer and that footer is specified in the HTTP header which is so much more secure than just pointing to the resource in the HTTP response because if you've done any web pentesting in the past ten years, you'll know that HTTP response splitting or just injecting HTTP headers is so much harder than it was ten years ago, so most frameworks will just not allow you to do that, so if you have to inject in two places instead of just one, it's of course much smarter and it's probably a lot faster too. So this is basically ESI injection, there's a lot of research to be done with this, we documented this I think in April at this URL, so basically you have our prior research, we applied it I think half a dozen ESI engines, some of which are pretty famous if you may, so you have Akamai, WebSphere, Varnish, Fastly and Squid, we found a whole bunch of bugs like Denial of Service with Squid, we found basically XSS, Filter Bypass for Chrome, so a whole bunch of interesting stuff, so if you want to go ahead and search for more ESI bugs, I think you'll find a lot of them. I think this is my one minute and a half for questions.