 Good afternoon. It's good to see everybody here. My name is Ed Schaller and I'm here to talk about WebStreet application servers, JSP Engine. And some work I did with that and it turned out to be a very interesting exploit. Out of curiosity, how many people here actually run or are familiar with WebStreet application server? So a fair number. Okay, so maybe I can move forward to some of this fairly quickly. This is a rather complex exploit but it's pretty simple when it comes down to it. And it's a little unfortunate when your screen server comes on. First of all, here's our outline. The slides are going to be on the website as soon as I put them up there with the exploit and such. So we're going to talk about what WebStreet application server is, talk about JSP's and NOLs, the WebStreet plugin and NOLs, WebInf. We're going to have a demo and talk about fixes and extras and should be good. So first of all, WebStreet application servers architecture. WebStreet application server is IBM's JEE application server. It's one of the more popular JEE application servers, at least according to Gartner. It's up there with Oracle and Microsoft as the top three. It's heavily used in the financial services industry and in insurance sectors. And it's not very cheap. So probably I can go buy it for you on the website. But there is a time bomb free version online at trial that you can go and you can download. So I have to warn you that I have the best authority that the initials WAS are fast WAS, but I always get it wrong to say WAS. So you can laugh at me whenever time I do that, whatever. So anyway, here we have the common network architecture that you usually see in a WAS deployment. You have the client browsers obviously on the internet that connect through firewall to your web servers. And those web servers redirect through the HTTP WAS web server plug-in into the actual web servers. This is done with a HTTP connection that is handled by the web server plug-in. The web server plug, first of all, this is the common architecture IBM shows it. If you look at this, it's hard to see, I realize, but the main features here is that the big grey box is WAS itself. And you have the web server plug-in in between it and the internet. So realistically being that the web server plug-in forwards everything into WAS, if you compromise a web app on WAS or WAS itself through the web container, basically whatever you compromised has full access to most of the internal resources that WAS does with some exceptions depending on the network deployment that's done. So plug-in is what's in between it and the internet, I'm getting ahead of myself. And for a while in case we do all that. So the web server plug-in is what you need to describe a little bit first to understand. I'm going to refer back to this architecture because we're going to jump around between the different areas to understand it. This is basically an extension module that sits inside the web server. Whether that be for Apache, IBM's Apache, which is an IBM HTTP server, IHS, IIS, several other ones including Lotus Domino and things like that. So basically it just connects straight via HTTP to the actual WAS instances web container. It adds special headers to keep track of where the client came from and such. So it's mostly used for load balancing and fail-over so that if one application server goes down it detects that and then it forwards to a different application server in turn. One thing that is important to realize about the plug-in if you're working with a WAS installation is that its primary purpose is not for security. In fact it's really not to do anything with security. In some cases people, I see people actually doing trying to do blocking rules on IHS and such to try to protect things on WAS and this can cause issues as we'll talk about a little more later here. So let's talk about the URL handling and the plug-in itself. Not all of the requests that go into the URL plug-in on the web server actually get forwarded back to WAS. What it does is it bases the URL configuration. It bases on effectively the same sort of things that are done for the web container itself, from the web.xml and from IBM's extensions, configurations inside the WAS file that you're running on. So as we know these are simple file blobs that get pushed in. They are matched in sequence. The web server plug-in is actually very simple although it's frequently misunderstood. So if you're asking for a servlet it forwards it in. If you're asking, you know, if you have a simple blob it forwards and what we're going to be talking about is start a JSP, anything is JSP forwards and if you have the web server, the WAS feature of file serving enabled, this is a kind of an IBM specific function that if you don't have an enabled static content inside your war does not get served. If you have it enabled it does get served. So if you have it enabled all the connections get passed from the web server into WAS. This is one I see misunderstood a lot of times. If a connection is forwarded in, much like a reverse proxy, otherwise it requests to handle for the web server. The reason for this is it's usually used to handle static content from the web server is a speed optimization so that the static content can serve by IHS or Apache instead of going all the way to WAS and serve from Java. So this is just frequently misunderstood. It's amazing how much I see it. If you're interested in more there's actually a paper about it that IBM has that's referenced in the websites I recommend looking at and it's actually for an older version of WAS but the plugin actually hasn't changed much in quite a while actually. So now let's talk about JSP's and how they handle it all. Now we're going to talk or move from the plugin here onto the WAS instance, the web server application server and let's talk about strings. I assume that most people know about strings under Java. The operating system sitting underneath Java is written in C which of course has null terminating strings and which means you cannot put null characters into a string and have them used later on because it terminates a string that's been used whereas Java is a counted string that can have nulls in it. So the big question is of course that we're going to start with here is what happens if you try to open a file in Java like test.txt backslash zero. So with a null character in Java, well actually you get test.txt. This has been known about for a very long time, formally just a specific instance has been talked about some people on the web but it's been known and used in a contact split for quite a while. So what about the JSP engine inside of WAS? How does it deal with a null character coming into WAS? JSP handling is very simple, it locates the file, translates its JSP into Java and then it compiles it into a servlet and runs it. So what happens with WAS or at least was before it was patched if we request a percent zero zero dot JSP for a arbitrary file inside the war? Well the answer is it actually compiled it as test.txt as a servlet and runs it as JSP. This is similar to what I'm not even trying to pronounce it but those are the characters on the web I didn't make a mistake. It's found in Tomcat. If you're here I really apologize. So this allows us to read files through the JSP engine. This may seem not too useful but we'll see how this can be added to in a series to actually make something useful happen. So basically if we have our context root for a web application we just append percent zero zero dot JSP to the end of it and we'll get the file. Obviously instructions on what files we can get. We can't just get any file from the web server we want. It must be in the war file which is usually expanded and we can't read files and meta in for web in and it must compile and run as a JSP. Which actually isn't too useful unless file serving is turned off and you're trying to get files that aren't put on the IHS instance that's serving static contents. So the question then comes as you're looking at these files is what's the valid JSP? Turns out that basically anything that doesn't include a less than percent sign is valid JSP as far as the WAS JSP engine is concerned. HTML usually qualifies under this XML. Most text files then you get some really weird ones like you can download zip files and jar files. I didn't know these compiled as JSP's but they do. Most class files also do. So you can get a lot of interesting things through the JSP engine in this manner. Still not doing anything too interesting but what about directories? For some reason and I really don't know what. If you open up a directory in the same manner in WAS JSP engine it gives you a list of all the contents of that directory and I'm really not sure why because if you open up a normal file in Java from like you can just do a file, a new file object and open a file input stream it doesn't happen. So they must be doing something special here and I haven't been able to track that down in the time I've had. So basically you can request your context route with an old character after it.JSP and you can certainly get a directory listing which is obviously a lot more interesting than just getting the static files because they have to have directory listing enabled and most sites are going to be doing that we hope. There are times when you actually have to use a period in front of the directory. This is usually the case with the context route where you have to do .00. Interesting enough in some cases you can actually do . but it has the way that WAS handles the . normalization and checks to make sure you're not getting outside of the war. It's pretty specific except it only checks for . so you can actually use this to read the contents of the exploded war. I mean it's exploded here. It's kind of one of these things where you see it and you're like oh I can almost get those files but I can't. So it's rather frustrating honestly. But you can do it. So this is a lot better than just serving up static files to the JSP. We can list innumerate internals of your war and what files that you may have accidentally left in there in terms of JSP's or other files that are being added to the war in the process of this application running. Which sometimes in and of itself can be a really a lot of interesting information. I dealt with one application that put all its logs inside the way that the exploded war. So once you found out the directories you could get them all. So we have to circle back here and go talk about the Webster plugin. And most instances of wars you're not going to see direct connections to wars that aren't going through the plugin or some sort of proxy in front of it. They're good reasons for this primarily dealing with speed and static contact especially. So the plugin wasn't really designed for security but in some cases it gets in the way of insecurity too. So I don't know if that really means it's a security device or not. But the problem is that going for an old byte in wars is easy. But the plugin is written in C and so it really doesn't like the null character. It terminates your string and instead of getting your test.txt you get test.txt which wars then just tries to serve out as a normal static file and if you're not serving static files then you can't get it or if you're trying to do a directory. So the next challenge in this puzzle in terms of how to actually do something useful with this vulnerability is to figure out how to get nulls past the plugin. So character encodings. This also qualifies as one of those things that if you think something is generally known by the industry, check. Most of the character codings we deal with are either ASCII, ISO Latin in the United States, internal Java you have UTF-16, UTF-16, UTF-16. But what may not be known is that UTF-8 is actually how Java reads strings natively. So even though it keeps it in memory UTF-16 it actually reads them as UTF-8. And so UTF-8 is a multi byte byte character encoding. It's actually a rather elegant character encoding. If you follow the character encoding improperly, if you just do it a default implementation and don't really think about it, single byte characters and actually any smaller than the full maximum bytes characters can be actually encoded as multiple bytes. This is explicitly forbidden by the spec. If you follow specs, come on. Actually there's a lot that don't. Let me remember there have been a lot of Microsoft IIS exploits that have utilized this. And in terms of the Java also used to do this as well. It decodes early long ones. This was made known as part of another tomcap vulnerability by Simon and William. And I'm not going to trade for that. Sorry. It was fixed in, fixed by son and their JVM quite some time ago. And it was also fixed by IBM. They share a lot of common code and so they share a lot of security fixes. The problem is that for some reason IBM's fix didn't work. It actually has to deal with the fact that son's JVM now by default uses NIO. And so when IBM took that patch and applied it to their JVM, they didn't think about the fact that their JVM doesn't use NIO by default to read strings and apparently they never tested this. So one of the morals of the story is test your security patches. This has really been fixed in these new releases of the JVM. Very thankfully unfortunately this is not any was fixed patch yet. So you're still looking at having this particular encoding issues in was unless you actually installed JVM by hand. I don't know if they're going to do an iFix of this. The IBM folks have been very nice about all this actually. So let's talk about UTF-8 then and what we can do with that with a web server plugin. Well the plugin is in C and it handles single byte characters. So ASCII and ISO Latin and it doesn't know anything about UTF-8 and especially long UTF-8 characters. So if was is in Java using a JVM that accepts over long UTF-8 characters then we can just instead of doing a %00 we can do a c080.jsp and suddenly we bypass the plugins problem with null and we get our null to the back in Java in was and we can actually get these files. So plug is no longer an obstacle to doing anything fun with this. This gets a little bit better. So we mentioned that web-ins and meta-ins, again back here on was, are not accessible. If you look at the spec the service specification explicitly says that you're supposed to return a 404 for anything inside meta-ins for web-ins. This has actually been a source of major problems for a long time. If you actually go and grab all the class files in was you'll find that it's checked a ridiculous number of times and they still missed one. There may be other ones unfortunately. It turns out that their file-serving servlet that handles when file-serving is enabled actually talks about, actually allows a slash dot slash web-ins. So you can actually grab files outside of the web-ins directory. If you're not familiar with Java web applications this is where all your configuration is. This is where a lot of your code actually is too that you can then download and decompile. So this was fixed in PK a potential security exposure of the file serving feature enabled. There was no details released about this so there may be other things that were fixed in that same fix pack I'm not sure. I had never really analyzed it but I know it didn't work afterwards. So surely they didn't just check this in the file server surely. Well yeah. So if you do a dot slash in web-ins you can actually get it through the JSP engine as well which is obviously a problem. So we have to play our encoding games again to get around that but in which case we can still go in and get the web.xml configuration for your web application by using %0, %ae which is an overly long period in our path which the plugin does not recognize and then our %0, %8, 0 which is an overly long null byte and suddenly we actually get the web.xml. This was fixed along with the %0, 0.jsp issue in pk81387 which I was surprised talking to one of the IBM architects, he said this was the only number he can remember for fix packs so it worked really hard on it. Possible application source file exposure which hopefully everybody is applied at this point. This also works for meta-end in case you want to go look at things that are left in there. So the big question is this the whole truth? Nothing but the truth? You can actually make possible application source file actually be remote code execution in many instances. So, again back here in WAS instead of the plugin, we look at .jsp striking back. If you're a Ruber, we locate a file, translate it to .jsp Java, we compile it and then we run it as a servlet. So, if we can get a file into the war we can get remote code execution through this method. So, I was looking at a web application and I had an interesting discovery playing this game with webmaps. Here's a listing of the directory of webmaps and there's this attachment directory. Does anybody know what that attachment directory is? I'm just curious about it. So what it turns out to be is that Apache access one is implement soap with attachments. This is not access two, we're talking about access one, it's a little older. For those of you who are not familiar with how email is used as mime headers and actually goes through and attaches things as mime entries, mind bodies to the main request so that you have a soap, your soap envelope with all this goodies and then you have your attachments that are referenced from your soap service. When large files actually go through access, it caches them to disk so that you're not running your application server out of memory in the process of handling the file that these get put into the webmaps attachment directory. So this directory is created if not present at start-up and guess what it was as the Jax RPC implementation is based on. But not there are Jax WS implementation which is of course based on access two instead. So, Apache access actually has some interesting bugs that make this less truly it would think. It's very good at optimizing how it reads in attachments from the disk. Your service actually doesn't handle attachments, it doesn't even bother reading them. So it doesn't get to the point where it actually caches at the disk. It's rather frustrating because we have to find a soap service that handles attachments. And at least in my where I work, we don't have very many of those if any. So it's a lot rarer to have this which makes our attack a little harder. So it occurred to me, well what happens if we just you have a series of mime attachments that are attachment for the soap body and so that our attachment has to be read before it reads the actual soap message. Well it turns out that that actually works but there's this other little bug in access the way it handles local variables that if you do that it actually saves it to another file system directory specifically where your temp directory is for java.io which is usually slash temp on Unix machines. So great we can dost two file systems that doesn't really help us very much. So let's talk about soap encoding. Soap encoding actually allows you to reference attachments. We just need to reference that attachment and it'll read them in. It allows this href attribute to be applied to parameters and such inside our soap message. So when you use Apache 1 Apache Axis 1 it'll actually parse the attachments and pass the large ones to our disk where we want them inside our web directory. So this of course requires that we have a web service that's actually using soap encoding so we're talking here about RPC encoded messages as opposed to document literal or RPC literal which are different methods of using soap. I swear this is not my fault. Axis 1 provides an interesting feature which unfortunately is not present in the WAZ implementation the WAZ extension of Apache Axis 1 so we can't use it directly with WAZ and what happens is that the client for whatever strange reason can actually send a fault to the server as an initial request. Usually this isn't something that should be happening so not surprisingly the server says what? It said me fault. I'm supposed to say you fault, not vice-versa. But the nice thing is that it parses it anyway first and just for fun soap fuses soap encoding. So we can send attachments with them too so not only are we sending a fault we're sending a fault with an attachment and it also doesn't require any web services to be actually configured in Axis 1 it just requires the exposure to Axis 1 server itself which is plenty enough to actually get this to work. So let's talk about actually how to use this we have to put together a lot of things here to make this actually work first. The attachment file names are fairly random and the first thing we have to do to actually exploit this is we have to connect the walls use the JSP bug to figure out what's currently in the attachment's directory baseline that then we send up our attachment as a so fault then we get a directory listing after that and we go through and find out the JSP that we put in there the file we put in there we request it as JSP at which points the file is transferred to the Java which means we've achieved remote co-execution on walls with all of it so demonstrate this we have a little cute shell over JSP which baselines the attachment directly uploads the JSP with the so fault finds new attachments and then checks that it actually has this attachment in case the service is already handling attachments in which case you have to find your own there is a race condition here in some on some servers depending on how they're configured and how the timeout is especially in a walls instance it's deployed for development instead of production so in our case we need to copy our JSP quickly out of that directory so it doesn't get removed while we're using it which is kind of frustrating and then it brought the standard and put the standard output from command line to always be to the server so if you're going to use this and finally when it's done it removes itself this is exceptionally noisy in the log so it's not exactly very useful for anything but a demo but it's great for demo so let's do this here I have a walls 7.0 instance 7.0 fix pack 1 which is the last fix pack that was vulnerable inside of this I have the demo application which is going to be on the website this afternoon that is running and it actually is being accessed through the plugin which gives us our plugin page and the access servlet is actually mapped to the slash access one at this point we can we have a shell script that is running our java exploit that takes the context root directory and then the relative path the access one servlet and gives a command name that we want to execute and with this case we want to execute the shell so in this case we see it updating and baselining it and then we're hitting it very quickly in this case because this instance is running in dev development mode and therefore it removes our attachment very quickly it then finds it baselines it and it gives us a shell of the server on the server and we have full access to the server it's running walls this tends to make wasm as really a happy incident when they start executing commands it's actually much more efficient if you actually want to exploit walls do you actually run stuff and have stuff natively stored inside of walls instead of creating a shell because most of us get a little fishy when they see a ps list that includes a shell being launched from walls because it really shouldn't happen so afterwards we just remove the jsp and we're done so we clean up after ourselves and hopefully nobody noticed but with that exploit it's very noisy so people will notice hopefully not that you couldn't do it a different way so let's talk about fixes this has been fixed first we have affected platforms walls runs on an unbelievable number of platforms I counted 15 and I'm pretty sure I'm wrong I'm pretty sure it's more than that when you consider everything AX and Linux were the ones I dealt with explicitly because that's what we run case insensitive file systems actually are interesting because they're not vulnerable to %00.JSP bug and this actually I believe has to do with an earlier fix for security problems with the case insensitive file systems the only one I know of is of course Windows that walls runs on so if you're running Windows you don't have to worry about this one you still have to worry about encoding the over the long UTF-8 encoding obviously depends on the JVM that's in use on the application server in non-IBM JVMs hopefully this will be fixed before the IBM one at this point those two that I know of are going to be HPUX which runs on HPVM and Solaris which runs on some one obviously in terms of versions 6.0 was fixed in 6.035 which is the same as 6.023 which is fixed back 35 6.1 was fixed back 23 7.0 fixed back 3 and other versions you can apply the iFIX there's also fixed packs for 5.1 and 5.2 but I don't have access to those to try them so hopefully you have these done let's talk about IBM for a second 15 platform variants that I count I'm on the low side I'm pretty sure because I didn't count things like IA64 for Windows and such 4 maintained versions of about 60 different variants can you guess how long it took them to fix this ballpark 8 months how about 2 weeks I'm impressed how do you do 60 different variants in 2 weeks kind of a public fix from the time I tell it very, very surprise if you are dealing with IBM for security vulnerabilities I would recommend that when possible you always submit them as a PMR that is a service request because that is how they handle everything related to their software and if you don't do it that way you're kind of in the la la land where they're not sure what they're doing and you may get forgotten about but if you do as a PMR they will get it done and sometimes they can do absolutely amazing things like 2 weeks to fix this sort of thing they also ticked off a lot of people too their fix is actually very elegant this is my pseudocode for it basically they look at the file they're trying to open and they ask the operating system to canonicalize it and they compare the two and they say this is really the file I'm opening and if they don't match it well there's a 404 which is nice this actually has problems it causes a lot of problems with people running on units systems that for whatever reason we're putting symbolic links in their that will fail that test right there because the canonicalization of a file path in units will actually return the absolute path not the symbolic link so that was a problem there's some fixes for that I haven't looked at specifically the WebM fix is not quite as elegant they just check to make sure you don't have WebM if you weren't in the URL path which solves this problem it does work very well it won't work but hopefully that shouldn't happen too frequently so before we talk about workarounds we're going to talk about some workarounds that you can do but first you're more than a year behind on patches at this point if you're working in workarounds it's a really good thing you're not running in other servers so the best workaround that I like the best is actually disable the runtime compilation and reloading of JSP's this is a little frustrating the container will actually tell you you can disable it at the application level this is done with putting some specific features into the IBM WebEHC.XMI there used to be a way to do this but container level is still in the information center and I can't get it to work so I don't believe that it's supported anymore this makes the fix a little hard because you're dealing with it at an application level instead of actually doing it on your entire server or your cell you're going to want to make sure that you pre-compile your JSP's either at deployment time or batch compile them before you go in there doing that works great the other way to do this is to block JSP's which we have to talk about a little bit of course if you actually need direct access to JSP's this is not going to work it's going to break your application but for a lot of people these days are using all of your controller whether that's Struss or Spring MVC or some other web framework usually your JSP's are only in views and only 4 or 2 internally so you actually can do this without breaking stuff there's a lot more issues here that we can talk about before blocking JSP's so I got a question what are the following byte sequences having a column if the title is a giveaway so 80 through C1BF I'm not going to read these they're all invalid byte sequences for UCF8 guess what the IBM JVM does with them this is the fixed JVM do you think this is a problem it's an empty string this actually in the unfixed VVM it only does it for 80 through FF because they didn't consider over long and valid anymore this is obviously a very useful variety of ES invasions because you can throw as many invalid UCF8 characters into your URL as you want and the IBM JVM will ignore them when it parses them so suddenly you can be requesting a whole path and replace every other new character a new valid UCF8 code anywhere you want and get around it of course this also makes it very difficult to write JSP blocking rules so if you're going to write a JSP blocking rule you have to worry about this and there's lots of fun games you can play with even coding issues honestly the other thing you need to remember if you're going to try to block JSP's is that there's more than .JSP you can start at .JSW or also handle those JSP's and in some cases also start at .JSPX of course if you're using newer versions so if you're actually going to go and try to do blocking JSP rules you need to actually use the UCF8 over a long decoding you've got to keep that in mind you've got to keep the decoding invalid ones with a replacement string and of course multiple JSP connections extensions but this is not trivial to do one of my friends tried he did a good job on it actually but it was not very pretty so another browser is it really goes home and starts playing with this don't forget that your browser will normalize your dots and such it also does the same thing with URL encoding in some cases Firefox and Windows actually for some reason will translate a %2e into a slash which it won't do in links interestingly enough so if you're bouncing you need to hit the same server over every time obviously otherwise you're going to be bouncing between servers with the JSP that you upload at the end of the server you're hitting so server affinity with the server plugin is handled by the JSashman cookie so JSSP is by default when their first run actually give you a cookie so as soon as you baseline the attachment structure you just need to make sure that you keep your cookie unless that server goes down they never go down so on the Defcon CD there's one text file with the URL to a website because I missed the CD but on the website and I'll be uploading this there are there's a tool to do the mapping of your JVM which actually will list out all the different byte combinations for UTF-8 go through them and then show you what they're being converted to what they do with the file system actually reveals some other issues with JVM if you're interested there's also a sample application that has nice URLs to try this on your own obviously you'll need to download the trial if you don't have laws yourself there's also an exploit for the access one faults that I showed you, the same one with the source code for all of the above so there's a lot in there and you're more than welcome to send me email on this and tell me that my code sucks in conclusion anytime you're dealing with a barrier between one implementation of a specification another implementation of a special specification whether it's UTF-8, whether it's HTTP whatever it is, this is a really great place to look for bugs in this case we did between the plugin and Java and between Java and OS sometimes we have to deal with multiple cases some people I talked to people about Java and I had to do Java for my job for a couple of years not a lot of interest in Java but these same sort of things actually happen in managed environments like Java and .NET and Java is of course not immune to that and what I really like about this is that you have a small series of these little vulnerabilities that you write down your reports as low or medium and never get fixed but sometimes when you can weave these all together you can actually make something that really makes a big problem so how about this