 We're about to get started. I'd like to welcome Justin and Samuel. Welcome to DEF CON, first time speakers. They will be giving a speech on Crossing the Chasm about an exploit that they performed. And as a reminder, this is a public service announcement. Watch your schedules online and use HackerTracker to be able to see the latest information and what's going on. There's a lot that goes on at the con. So once again, please give a warm welcome to our speakers. Welcome, guys. Hello. Hello. So I'm Sam. So this is Crossing the Chasm, a web app pentesting story. So this talk will get a little bit technical, but it's worth it in the end, I promise. So who am I? I am Sam, aka Irby Sam. I'm a hacker, and I also have two DEF CON black badges from a few years back. And most importantly, I got a puppy about six months ago. And so there'll be more puppy pictures later, I promise. Nice. And I'm Justin Gardner, aka Rhino Raider. I'm a full-time bug bounty hunter, and here are some additional facts about me. You can find me on Twitter at my handle Rhino Raider, and my DMs are open if you guys have any questions about the presentation or have any similar interests. So when we're dealing with a product called Chasm, we had to do something punny, right? So I figured I'd make this little presentation for you guys here, representing what it's like to be a bug bounty hunter on a pretty hardened target. So here, here's Sam, and I am in the middle of the desert. And on the other side of this chasm, we can see a mirage of RCE in the distance, but we have to go for it. We have no choice. And unfortunately, this mirage is separated from us by a technology called Chasm. And represented here by the deep chasm. This is a VDI software enterprise for essentially managing what applications your user can run and in what context they do that. So we threw a bunch of exploits at it. Nothing was working. Nothing was sticking. But in the end, we did find the bridge over this chasm, and this is that bridge. And we'll explain that a little bit more, but all you need to know for now is we just hooked this up and hopped right on over the chasm, and we made our way to the RCE. So I'm going to give it back to Sam to give you a little bit more context on what the heck we're talking about right now, and hopefully it'll make a little bit more sense in a moment. Thanks. So for some important background before we get into hacking, so Chasm Workspaces is streaming containerized apps and desktops and users. So it's really intended for the enterprise world where you want to administer your end users running Ubuntu or Chrome or Windows inside of Docker containers, and it will deploy them for you and then allow end users to connect to them, and then you can monitor them, do various things, enterprise-y things with them. And it's built on top of Docker and BNC. And so I'd give you an idea of just how dockerized this world is. It's turtles all the way down. It's Docker containers all the way down. And I apologize for putting so much information on a slide, but I'll give a visualization in a minute of what all this looks like. And so at the bottom of this list, you can actually see the Ubuntu Docker container that I deployed. And at the top of this list, you can actually see the Nginx container that I deployed, or sorry, the Nginx Chasm web, Chasm Proxy. And so the Chasm Proxy is actually listening remotely, and then every other Docker container is listening on local ports. And if you're not familiar with Docker, you can kind of think of it as a virtual machine. But instead of running on bare metal, it runs on top of the operating system. So it's really just kind of one layer up. And then it has a bunch of access to the underlying operating system, depending on what permission to give it. And so diving in a little bit into a little more detail into that Chasm Proxy, when you actually spin up a new container, a new Nginx configuration actually gets pushed to the Chasm Proxy. And so this containers.defolder will be updated with a new file. And so when you push a new container, this folder gets updated. And then Nginx actually gets restarted. And then the end user can connect to the newly formed Docker container. And throughout this talk, we'll try and share at the bottom the commands we ran, as well as references. If you want to repeat this or learn more about this. And so at a high level, this is what, if you deploy Chasm workspaces onto a single server, this is what it'll actually look like. Your end users will connect through, as well as you, will connect through Chasm Proxy. And they'll be able to connect to their end user containers. So every blue square here actually represents a Docker container. And the bottom row are the command and control containers that are used to manage the rest of the system. And they're all built on top of Python. It's either going to be Tornado or Cherry PY. And so these are the purpose of all the containers. The most important one being Chasm Proxy, which routes all your traffic. And then Chasm API, which is actually the API you're interacting with when you're using the application. Chasm agent is also important, because that actually has the ability and permission to deploy new containers. So it's a privileged Docker container in the Docker parlance. With multiple servers, you actually end up with a series of secondary, or when you want to deploy this into a real enterprise environment. When you have multiple users, you want to deploy it across multiple machines. And so in order to accomplish this, it actually will, on the primary server, it'll keep Chasm API. And then Chasm Proxy will proxy through to the secondary servers to run Chasm agent, which will then deploy your end user containers. And this allows you to scale incidentally. And so in a very high level hand wavy way, this is roughly what credentials look like in the system. Your end user session tokens will get passed to Chasm API as a cookie. It'll also get passed through to their containers. And then Chasm API will interact with the rest of the system using secret keys, which are all internal to the Chasm system. So at this point, we were given user credentials to a bug bounty program. And we were asked to look at this without source code access. We didn't find any vulnerabilities. This was actually really secure software. And this talk wouldn't be here today if this was in secure software. It would be a very boring talk. So at this point, we decided we were kind of given up looking at this without source code, and we wanted to start testing this with source code. And so before we did that, we actually stopped at their issue tracker and just made sure that nobody else reported any security issues first, not really finding anything we moved on. So we went to their website to see if we could download their software. And we came across this page. And this page is the most exciting thing you can see as a bug bounty hunter. I'll add some arrows to show why and a few more arrows. This button means that you don't have to call sales. You don't have to put your credit card down. For some hopes of finding some bug in the future, you can just download software and see how it works. And as you can see in the Community Edition, it has all features in Enterprise, which is exactly what you want to see. So at this point, we went through and installed this onto a server. And when we looked at the installation process, we saw it was actually downloading from Docker Hub and installing the Docker containers that way. We then played the game of find the API that you're calling. And in this case, we picked one called API Get User Attributes. And we looked through all of the servers that we're running, and we were able to find it in the Kazan API binary. So we copied the binary out of that Docker container. This is the proper way to reverse it. The actual way that we went about it was running strings on the binary and then googling a whole bunch of stuff. But this is the actual way that you're supposed to identify that it's running as a Python binary. You can then identify the version, drop into a Python environment, extract the Python files, compile the Python files, and then decompile them using Uncompile 6. The steps are listed here in case you want to repeat this yourself. We then dove into client-8-under-square-api.py, and we were able to find our Get User Attributes function. And this actually gave, well, this concluded our game of find the endpoint. And now we can actually do the real security work of looking at all the endpoints for security vulnerabilities. And so at this point, I'd actually given up almost entirely. And then Justin came along and said, hey, there's, when you create a new chasm, a workspace, there's a host header that gets passed through as one of the inputs. And so we were a little bit curious to see how that worked. So like a good bug bounty hunter, we spun up Burp Collaborator. And we got callbacks. And so we immediately thought this was a crit. And game over, we found the coolest vulnerability ever. And unfortunately, we were only getting our own session token back. So it's not really a crit. It's more like a blind SSR, a really limited blind SSRF from a security standpoint. It's not super interesting. And so we obviously wanted to maximize payout as bug bounty hunters, so we decided to dive deeper. So now's hacking time. All righty, so let's look a little bit more into this. We've seen so far how we were able to reverse engineer the software, get access to the code. And now we're going to start finding some vulnerabilities and figuring out why we're getting that callback. So as we mentioned at the beginning, the configuration for Nginx for this software is a little bit strange. Whenever a container is spun up, there is a new configuration file injected into this containers.d directory. And the callbacks that we were seeing from this was other parts of the system hitting these internal routes that they created to communicate with our new API. We can see that here in the proxy pass definition for both of the locations defined here. So we had something interesting here. After playing around with it a little bit more, we realized that it would actually leak the session token of anybody who visited the page where the thumbnail of this container was displayed. So if an admin came and they viewed that thumbnail, then their session would get leaked. So that made us feel a little bit better, right? Because we have a vulnerability that we've definitely got secured. But like Sam said, and like my lovely desert graphic in the beginning showed, we saw RCE in the distance and we wanted to follow it. So we continued. So looking back at this configuration right here, we thought about what other things we could do. And of course, when you're injecting into a proxy pass, most people would think of SSRF. We already had blind SSRF. Let's see if we can get it to be full read SSRF. So the way the proxy pass works is when we passed this host header that we were able to inject from the command that we, the HTTP request we did before, we just changed the host header. That got injected right into the proxy pass here. And if we could hit these routes, then we could hit any arbitrary host from the perspective of the server, right? And in today's modern cloud environments, this normally results in at least something a little fun if not full RCE or takeover of the AWS account that this is hosted on. So we thought this would be a good target to go for. But we ran into some issues here. Yeah, so the goal would be full read SSRF. We ran into some issues here, hitting these NGINX configurations. The first one, as you can probably see, is the first entry here has the internal directive. The internal directive in NGINX only allows this route to be routed by internal redirects or other internal requests. So there was no way that we could possibly hit that endpoint. The second one, we also thought looked great, but that GUID is not leaked to the user in any way, shape, or form. It just gets pulled straight out of the database and used by other pieces of the API. And we couldn't leak it to ourselves because it's being overwritten by a moderate right there. So that resulted in a big sad. So we had to think a little bit more about what we were gonna do. So next we noticed that there are some variables that we can access within the NGINX configuration file. The remote address is one of them. And actually NGINX provides a really nice alphabetized list of variables that we could go through. So we went through, I don't know if you can see it in the screenshot there, but all 222 of those variables. And there's nothing interesting in there at all, not even one. So we went through all of that and couldn't find anything fun to do with that. But then we started thinking, okay, variables, what about environmental variables? Could we leak environmental variables from the host machine? So we started, like any good hackers, we Googled how to get access to environmental variables in NGINX configuration and this lovely Stack Overflow article popped up. And it mentioned running arbitrary Lua and Pearl in the context of an NGINX configuration file. So don't mind if we do. So we decided to quickly pivot and go down that path and yeah, no, it's not gonna be that easy for us, unfortunately. There was no Pearl or I mean, there's no Lua or JS modules loaded and even though they did tease us a little bit with defining a Pearl's module path in the NGINX command, that path did not exist. So there was no Pearl modules being loaded and we couldn't use the Pearl modules. So we're back to the drawing board and we're trying to come up and we're like, okay, well maybe we can modify this NGINX configuration file to do something funky. So we start, this is the vulnerable request. We start injecting valid or what we thought was valid NGINX syntax into the host header and big surprise NGINX fussed at us and said the front-end proxy from Sam's explanation earlier was not accepting that host header because it had spaces and slashes in it. So this is getting more and more, this is getting more and more difficult, right? So what's the sitch here just to summarize? We have a blind SSRF via NGINX configuration file injection. We've got callbacks that contain the victim's cookies but we kind of wanted to avoid any social engineering because this is bug bounty and that's how it is. And we can't get full read SSRF because of the internal directive and we couldn't find the GUID. There were no interesting variables to leak, no access to code modules and that we couldn't use any spaces or slashes in the host header. So this was looking really rough. We were feeling down at this point. And then Sam has this lovely idea. I'm gonna read. I'm going to try and call the API directly from within the container and see if Python is okay with the spaces in the host header, which was a lovely idea. So essentially what he's doing here is he's using those end user containers that we're able to spin up and he's trying to communicate directly with the chasm API from within the containers. So he spun up a nice Ubuntu box here and boom, we could reach the internal API without Nginx getting in our way and lo and behold, Cherrypy is a little bit less finicky about their host headers and will accept pretty much whatever you throw at it and a little extra, which we'll find out later. So that was great. We thought we had a way to inject into Nginx configurations and we proceeded to write up an exploit and take it to prod and nope. The prod was running the multiple server configuration and we cannot access the chasm API. So at this point, Sam and I are about to lose our minds and we've been going back and forth on this vulnerability for a while and we were stuck. We were really stuck. So we tried a bunch of things here. We tried HTTP request smuggling. We dove in. I don't know why we were so obsessed with this vulnerability, but we just really were and we started reading Nginx, when you start reading Nginx source code, you know you're in trouble. So we started going reading Nginx source code and we couldn't find anything in the host header, parsing logic or anything like that there. But then we repeated this process for Cherrypy and we found something really interesting regarding ISO 8859-1 headers which I had never heard of before. So upon finding this, I sent a frantic message to Sam and I think you can see above me saying, man, this bug and then 15 minutes later, dude, I've got it. So you can kind of feel the bug bounty hunter drain right there and the lows and the highs, right? And so let me explain a little bit about ISO 8859-1. If that looks familiar to you for some reason, that's because it's used often in SMTP headers. So if you've ever clicked C original in your emails and inside of Gmail, then you'd be looking at something like this. And one of the things that neither of us knew and that I think is not very widely known, that headers are actually supposed, according to the original RFC, are supposed to be able to be parsed via 8859-1, ISO 8859-1 encoding. So that was exactly what we needed and the Cherry PY code you can see up at the top right says, yeah, if there's an equal sign and then a question mark in the beginning, let's go ahead and try to decode this value as ISO 8859-1. So the format of this is actually very similar to URL encoding, which is super convenient for encoding web vulnerability or web payloads and we can do something similar here. We have to define this little header in the beginning, the 8859-1, question mark, Q, question mark part, and then essentially we can base 60 or URL encoding code our payload and then just replace the percentage signs with equal signs and you get a fairly close to valid ISO 8859-1 format. So I think this is one of the takeaways from this talk is that if you're having trouble smuggling headers through some sort of intermediary proxy, whether it be a WAF or a proxy in our scenario, this could be some way to bypass it if any of your web servers are in the stack or accepting and decoding or normalizing this ISO 8859 value. So you can see right here just the character to character translation of what it looks like for that to be decoded and the value that you see in the red is what is gonna be passed to Cherry PY in the end when the header is parsed. So at this point, you know, we have a way to get through it, the chasm proxy and we have a way to inject host headers. So we're almost there, but we first have to get through Nginx configuration. And so here, what we're actually injecting into is we're not, we're injecting to multiple places with the same string. So we're injecting it to two proxy pass directives and then four add header directives where the string is wrapped in single quote characters. And so this was something that we had to get bypass. Our end goal here really was to add our own location so that we could add arbitrary Nginx configuration. And so we tried adding to a new location point to the top two injection points. And we used, we didn't use any single quote characters so that it would be okay for the remaining four injection spots. Unfortunately, this doesn't work because when Nginx sees a duplicate location, it rejects the configuration. And so that didn't work. We got this error duplicate location slash new. So we then tried something a little bit more creative where we actually used some quote magic as how I'll describe it to inject into the top two positions using quotes and then a comment to break out successfully. And then on the last four injection points because the single quotes would take priority over the double quotes. We won't escape from that string. This should have worked except for there's one evil dollar sign in a regular expression right in the middle of where we're trying to inject into. And Nginx rejected this because a single dollar sign is an invalid variable name in Nginx. As an aside, there is actually a solution to this where you can put this geo dollar default dollar sign into your Nginx configuration, but it has to be done at a higher level than we had access to. We only had access to the location block and then we could also break out of that into the server block. This had to be placed in the HTTP block, which we didn't have access to. So this didn't work. So at this point, you've reached the most boring part of our presentation. I promise you'll get better from here. So now is the time to wake up if you're sleeping. So this is really exciting. So it turns out you can't replace, you can't repeat static locations in Nginx configuration, but you can have repeat regular expression. And this is due to the way that Nginx parses the configuration file. It will first compile a list of prefix paths looking to see quickly find a match. But then if it falls back to regular expression, it will actually go through them one at a time and in the first match it gets, it will send it through. And this allows you to actually send through multiple identical regular expression location paths. And so this was a huge bypass. So as I say, so you can repeat regular expression just for fun, that's big. It was big. So at this point, you know, we built gadgets. We weren't quite sure exactly how we were gonna exploit the system yet. So we built a full read SSRF finally. We built a local file inclusion where we could actually pull files off the system. As an aside, they were very secure and they built the Nginx proxy as a read-only file system. So we couldn't write any files to it, which also made this more difficult than it should have been. And we built a cross-site scripting payload as well. And so we spent a lot of time looking for other interesting tags that we could use to either achieve remote code execution or somehow cause interesting things to happen here. And there really weren't very many, but there's one other one and this is a little bit obscure. So client body and file only. And so what this will do is when Nginx receives a post payload, Nginx will save the post payload to a file on disk. And the expectation is you then have some sidecar, Python or Perl process or Lua that would then pick it up and do something with that post payload and then delete the file. So it's more of a functionality than it is like a security feature, tag, directive, sorry. And so why does this matter? So we can actually combine this with our, so if we could find a secret key that we could save, we could combine this with our LFI payload to steal some secret key that's somewhere in a post payload. So that's exactly what we did. So if you remember this where we had these secondary servers where we had connections going to chasm agent, one of the requests that went through to chasm agent was a health check and this went through about every 30 seconds and in that health check post body, there was a secret key that could be used to communicate with that chasm agent. So we asked, what if we get access to that secret key? And so we tried to get it. So putting everything together, ignoring what's on the right side for right now, I'll get to that in a second. So we carefully intercepted a single chasm agent request using that client body and file only directive. We then forwarded along so that the system wouldn't think that the secondary server was unhealthy. If it thought the secondary server was unhealthy, it would take it offline and break the whole attack chain. Well then use our LFI payload to steal that secret. So breaking this out a little bit better what we're actually doing here, we're putting this into a host header, we're using our ISO 859-1 encoding. So the first in the yellow, we're using that to keep off working for this workspace. We're then for the top two injection points, we're using, oh sorry for the bottom four injection points, sorry, we're putting this into a comment so that everything is commented off correctly. That's in blue. In white is really the magic here where we're intercepting the hull of the health check and then we're stealing the secret that contains in there. Then we're using a regular expression location to create an LFI backdoor using the doc secret end. So any path that ends in doc secret will be interpreted by this regular expression and we're using that to steal the file and then we're gonna put an unused location at the end of all this to close it off nicely. So we were also hit by a 255 character limit which is why this is a little bit code golfed. So I do apologize, that's not super clear. So running this, we, as you can see here, you end up with a series of files and we get our host token in each one of these files which is super exciting. So what can you do with chasm agent at this point? So with chasm agent, it actually has a well documented API where you can either spin up new containers or you can call, you can make privileged calls to other containers to run commands. So in this case, what we actually wanted to do was we wanted to have chasm agent make a Docker API call to run a command on itself. And so because it was a privileged container, that would allow us to access the underlying operating system. So using our host token, you know, we sent the request on the right side here to the chasm agent to run a command. This command mounted the underlying operating system as root and we simply touched a file to demonstrate that we had access and as you can see here, we were able to achieve file write as root which would have led to RCE if we had actually exploited this for real. At this point, we were asked to stop. The response we got back from the bug bounty program was, well, that's worrying and we got our crit, we got our RCE. So we were very excited. So no good slide deck is complete without a vendor response slide. So Nginx doesn't consider this to be a vulnerability but they might introduce some checks in the future to prevent certain characters that are obviously invalid for host headers from going through. Cherry Puy actually addresses for the host header so other headers are still vulnerable, however. Chasm Workspaces has a configuration fix for this and if you're running this, we would recommend you apply it. Additionally, in the latest version, it's been fixed as well. And in terms of a timeline here, this is just to give you an idea of roughly what this looks like. We submitted this on 3-1, on 3-6, we reported the additional ways that we could exploit the system and then took us another 22 days to demonstrate the Docker container breakout. So some takeaways. So from the blue team side, you have to ask yourself how your organization is handling running a server that has multiple domain names. In general, you'll have a QAM prod. How do you handle the host header there? Are you trusting the end user to supply that because that could be a little bit dangerous if they're doing something with it? You also have to be a little bit careful with local Docker networking. I mean, that's true in any networking environment, but that's especially true with Docker when mixed low and hyper-illegible environments. And then on the red team side, we know your sprained payloads everywhere, so why not just add ISO 8.59-1 to that list? And then similarly, build gadgets and then constantly reevaluate what you have and what you can do. What we presented here, we tried to follow a logical plot. We jumped around quite a bit when we were actually exploiting this. And then similarly collaborate. I think there were multiple times where we both wanted to throw in the towel and completely give up on this. And then one of us found something that just pushed us that much further, which is really why we worked on this for a month, I think, which is overkill. So that's it. Questions? Hey, so you found the Cherry PY, you were able to use that different encoding. So was that also affecting Engine X for the host header encoding when you did the... Could you say it one more time, we can't... Yep, oh, sorry, Tess, can you hear me better? Okay, so when you did the actual exploit, it was on the Cherry PY server. Did the actual encoding trick bypass on Engine X as well? Can you hear him at all? Oh, sorry. I can't hear him at all. Slow it down a little bit, go ahead. You can stand up. Oh, yeah. So when you're testing the actual proof of concept, you were on the Cherry PY, correct? Yeah. I think the... Yeah, so when you actually exploited it on a Cherry PY server, but the actual exploit was done on, like is it, you're able to do the encoding trick on Engine X as well? Sorry, I think it's right here, sir. Yeah, no, sorry about the having a hard time hearing. Yeah, so the question was, I think, when we actually did the exploit, were we able to do it through Engine X as well? And I think that, yeah, so we were. And so the cool thing about the ISO8859-1 encoding is that all of those characters are valid characters to be in a host header on Engine X. So it just, what ended up happening was, all of those characters that were causing the problem in the Engine X configuration just were encoded and then flew right through the Engine X part and then were decoded by Cherry PY on the backend and that's what allowed us to bypass that little mismatch between the parsing of the headers on the two, on the reverse proxy there, so. Yeah, I think I said that as well. Go for it. So within Engine X itself, the configuration parsing logic is heavily dependent on certain control characters which were blocked by trying to traverse through Engine X. So that's why we had to do this. Any other questions? All right, well you can find us on Twitter afterwards. If you have any other questions, our handles are down at the bottom and thank you guys so much for listening.