 Cool, so the last year or so, it was working as a part-time security analyst at WordFence. I've now moved over into a full-time senior developer role there, but I'm hoping for the next half hour or so to tell you some stories, some of the things I learned, some things that I saw while doing site cleaning at WordFence. And just on a side note, I tried to get them to sponsor, but we couldn't make it happen. But I do have a pile of swag, and so I've got some swag quiz questions throughout the talk. First answer can come back and get some swag, and I've got some extras as well. We'll talk about that later. Okay, so it's the first question that gets asked is why site cleaning? You know, why look at malware, why clean-infected websites when there are other things to be doing? And the answer's pretty simple, it's curiosity. I've been doing security for a number of years now, usually in the defending-use site, sorry, installing, setting up screen headers and worrying about cross-oscripting or that sort of thing, but I hadn't done any actual malware analysis or cleaning-infected websites or looking at any of that sort of thing. And when I was looking for a part-time job, I stumbled across the job and thought, that sounds like fun, you know, looking at malware, that sounds like a great idea. I'd never seen it before, I thought it could be interesting. And so I signed up for the job. Now the job application started off as a standard application, you answer some questions and you send in your resume. But there was also a second component, which was, they gave an SSH and credentials, so the IP address, username and password, and an infected website was there and had to clean it. And so I had seven days to clean an infected WordPress website as part of the job application. And I'd never actually done it before in my life. My WordPress experience was, you know, copying the files to a server and running the install script and maybe installing some plugins from the repo. I hadn't done any WordPress coding. I hadn't looked at any actual PHP malware or any malware before, so I really had no idea what I was doing. So like any good IT person, the first thing I did was Google. So it's a good place to start. There are heaps and heaps of things online about cleaning up hacked WordPress sites. So a lot of the big vendors have heaps of documentation. Word fans have got heaps, security has got heaps. Some things are more marketing-based than others. A lot of the vendors will try and sell you into their site cleaning or into their site products, but there's still heaps up there and there's a bunch of independent researchers that have got information as well and videos as you can see. So that's a great place to start. So I spent a couple of hours just looking through that, figuring out what I'm looking for, where to start, what to do. And so when I thought I had enough information, I moved on. And the first thing I did when I logged into the server was back up everything. And the biggest reason for this was I didn't want to break something and have to go, oh, guys, I accidentally deleted the wrong file and it's all broken. Can you fix it so I can go and clean the malware again? That would be really embarrassing and instant fail. But also, if I failed the application, I wanted to keep the code so I could look at it again because I referred back to curiosity. I wanted to see if I missed something, what did I miss? And so I backed up all the files, all the databases, everything I could find that I could back up on that server I backed up. And once I'd done that, then I went looking for malware. So step three, and the theme in a lot of these is look for anything strange or anything that's unusual. And if you're very familiar with the WordPress code base, you'll know what's familiar and what's not. So if we look at this file listing, for example, something stands out, strange, standing. Thank you. So up here, I see the next slide better, WLO. So that one is unusual for three reasons. First of all, the name. That's not a call file. You don't find that on a call install. Secondly, the date. Everything else is 30th of May. That one is what, the 25th of September. Why has it got a different date? And that file size, this one here is the file size. It's tiny. Everything else is bigger. So that file from memory, PHP Open Tag, eval, getparameter. So it's just a remote code execution. And that's far too small. So that's instant red flag. That's something you've got to look at. And so I spent a decent amount of time looking through the plug-ins and the things. Look for anything unusual. Anything that shouldn't exist. Given my limited knowledge of WordPress, I downloaded fresh copies of everything to have a look to see what that was. Once I'd done that and found what I could, I thought, why not? I'll install WordFence. So I installed WordFence on the WordPress site. And I ran the scanner. It found a whole bunch of stuff to delete, a whole bunch of modified files, and went through all of its recommendations and cleaned up the site through that. And I kept the backup. I didn't need it. I didn't break anything, so that was quite good. So after using WordFence and cleaning up that, the next thing was to look at the database. So the databases have, you know, if they've infected the site and infected the database, you generally find a bunch of suspicious keywords. So big thing for database is SEO injection when you want SEO spam. And so you'll have the standard information that they're trying to put in the site content. But you'll also have hidden links hiding inside it. So if you can look in that one, it's in the highlighted bits, are hidden links that won't show up when you're reading the content, but they're there, and this one's linking to a cheap pharmacy website to try and get its rankings to raise up. And there's a whole bunch of words that you'll generally find when you're looking for this sort of thing. So, you know, Viagra, born, callous, weight loss, casino, et cetera, et cetera, goes on. And unless you're cleaning, say, a pharmacy website, you won't find these terms on a normal site. And I know from experience cleaning a pharmacy website when it's been infected with SEO spam is really, really hard, because you don't know what's actual legitimate content and what's not. Luckily, you know, you can find like the position app, so it's throwing it far off to the left of the screen. That's a dead giveaway. But sometimes it's not so obvious, and you've got to make a judgment call, you know, is this site actually selling Viagra, or is it a fake spam site set up? So that's always a fun one. Yeah, so they're the basic steps, that's been true, to clean the site. And it turns out that they're, you know, the steps you kind of need to follow. I actually managed to get 100% on this application, which was quite nice, because you know, I'd never cleaned a site before. So I was very happy with that one. And throughout my years as cleaning websites, this is basically the steps that we always follow. So, you know, that's something to remember if you're interested in cleaning sites or have to clean a site, is those steps. Now, I glossed over it a little bit, but all they gave me were the SSH credentials, so I could log into the SSH account on the server. But how did I get into the WordPress admin in order to install the plugin and use the scanner? No, use something else. No, close, I didn't create a user, but I did something similar. Yes, change the password. Yeah, so WordPress passwords. By default, they're assaulted. You can see at the top there, those three. At the top, that's the same password had times. And the salt in the hash function means the output is different. I'm not gonna go into the reasons why if you're interested, come find me later. But the bottom three is MD5 hash of the same password. And as you can see, the output is the same. And so what you'll often find, and sorry, WordPress, if the password is stored as an MD5, it'll understand it. So if you try to log in, it'll let you log in with that password, then it'll save it as a salted hash version. So if you've got access to the database, which I did, then you can simply go into the user's table, change the password. And we'll quite often find, when cleaning a site, that the user passwords have been changed to MD5 hashes. And so when the attacker can get into the database, they'll change all the passwords to their own passwords, use the MD5 hash, and they can log in as soon as, you know, to the actual admin account and do whatever they want. Which makes it very easy to gain access to WordPress if you have access to the database. Does anyone know what that MD5 is? Yeah, password. You need to get in of all the swag over there. All good. Okay, so the first thing that I encountered when I started working as a site cleaner was this vulnerability. It was the site of last year. I don't know if anyone remembers it, but it was a vulnerability in the API where it was incorrectly unauthorizing the user using a non-straight comparison. And so it allowed anyone who had access to the site regardless of, you know, their user privileges, whatever else just a guest could modify post content, page content by the API. So they WordPress quietly fixed it on the 26th of January last year, and then six days later after they decided it'd be in enough updates, enough sites had updated latest version, and they disclosed the vulnerability. I assume the idea was they wanted to announce it, let people know about it so that it could be protected, rather than just let it linger and wait for someone to discover it. The problem was it was trivial to automate because it's an API call, and heaps and heaps of sites had disabled or broken updates. And so heaps of sites weren't updated and were vulnerable to this. And so we saw a huge influx of site cleanings come in from this because of all the sites that, as I said, it had disabled updates or broken updates. Luckily it's trivial to clean because it's only a database and all it gave access to was editing page and post content. So what we had to do was run a script and we had a couple of internal scripts to run, just upload it and then it would clear out the post because the attackers were using very similar sorts of cylinders or what they're injecting in, so easy to clean, but it was still hitting a lot of sites. And so the takeaway here really is keep updates happening and don't disable them, whatever you do. If you have to modify call, then rethink what you're trying to do because if you have to disable updates, you've got a problem. And so accessing database can be incredibly trivial. We discovered one time that a number of, we were skidding a number of sites and they're all had a very similar injection and they're all on the same three hosts. And we looked into it a bit more and we realized that the shared hosting providers had dodgy permission set up on their network share, which meant that this account here could read the files in every single other account across their network file share across their entire fleet. Which meant that all you had to do is compromise one of the accounts and then iterate through every account, read all the database credentials and you got database access. And full database access, you could easily modify the user's table, create new users, change the passwords, do whatever you want. And so that was quite painful. We actually implanted, so when we discovered this one and we discovered the host that had this problem, we reached out to them and they're all really responsive. And so they're fixing something. There's four or five that I can remember that we've actually reached out and worked with to fix it. And it's part of our standard site cleaning process now. So when we analyze the site to clean it, well I see check for this vulnerability and if we find it, we'll reach out to the host directly and get them to fix it. And as I said, they've all been really responsive when we've told them about it and explained why it's a problem. It's a bit hard to read on screen, but I discovered this one day, yeah, it's really hard, one day I was cleaning the site and in one of the directories, these are all sim links for wpconfig.hp pointing to all of the accounts on that server. So this is the patient zero account. And so they ran a script that iterated through all of the different accounts on the server and it created sim links to all of the account, the wpconfig files in that one directory. And then they used hdaccess to enable directory listing. So the control server running in their own infrastructure just has to hit the directory listing and go through every single page, harvest all the database credentials and they can connect to all the databases and do whatever they want, gives them full access. And if you can see down here if it's visible there's a configuration.php. So it wasn't just targeting WordPress. There was something that's Joomla, maybe you can't remember exactly which one that is, but they were targeting multiple CMSs or common ones running on shared hosting. It was a very profitable attack for the last Joom, but all they did was database injection, which was quite easy to clean. So they couldn't have done a lot worse. So I mentioned that the backups, backups are essentially doesn't, you could accidentally delete some files or something could happen, but they're also really useful in cleaning sites. And what we try and do with cleaning sites is take a full backup before we touch anything. One day I discovered a site that had about 80 gig of uploads and the FTP was really slow, the file manager was really slow and painful. And I decided I couldn't see any malware in the file itself, so I figured I just move it out and then delete the infected files, put the clean files out there and not have to worry about backing up 80 gig. And then the file manager timed out and I deleted a bunch of customer files. So yeah, the takeaway from that one is always do backups and even if it's gonna take forever, just do it anyway. I think you can explain to a customer, I'm trying to do a backup, it's gonna take a couple of hours, but the danger is if I don't do it, you might lose files. So there's a trade-off there, but you've got to take the backups. It's a fun contenting instrument saying I've accidentally five gig of your 80 gig of uploads. So the next question, the fifth thing is why access logs are also very important? And can anyone tell me why access logs? Yep, there's nothing a bit more than that. Can you expand on why that's helpful? Yeah, yeah, so just for the recording, it's so you can see who's doing what and what they're doing. And so here we've got some logs from a fake theme upload that we discovered. So when I was cleaning the site, I found that picture such DB.php file, which was infected. And so I was looking in the logs to figure out what was going on. And we can see at the top there that the attacker has logged in. So there wasn't any sign of brute force in here at all. So they used maybe reuse the password somewhere. And so the attacker tried to password that, had been in a password dump, or it was a really simple password or some other method of compromise. But so the attackers logged in to the admin section and they've gone into the theme uploader, uploaded their theme, and then there's their files. So now the attacker has this site which allows them to do whatever they want. It also happens quite a lot with plugins. So again, the attacker is logged in via the login form at the top and they go to the plugin install, they upload their own plugin and they activate the plugin so they can use it. And then at the bottom, they're testing to see if the plugins work. Notice in here we've got a Kismet. So this is a fake Kismet plugin and we set up all the time. It's the most commonly fake plugin that we see. You'll see a Kismet, Kismet 3.2, Kismet 2.1, whatever, Kismet some number. And because it's default, everyone just ignores it. They see a Kismet on their site and go, ah, it's supposed to be there. And so you often find fake plugins called a Kismet. And there's a couple of other things I'll come with that called, but yeah, so it's quite common. And so this one, they uploaded their fake plugin and then they checked to make sure it worked and then they kept going. So what they're doing here is they're getting it to upload, they're updating their own malware essentially. So what's installed by the plugin uploader was a very, very basic entry point. And so they've kept it nice and simple to try and get past any sort of scanners or firewalls that would check the upload. And then once they've got their code in, then they can get their code to go out and download the payloads they want. And so they're here, they're downloading the payload for a WP update file and they're saving it as WP update in the root directory. And as you can see in the bottom here, they kept updating their malware. So if you check out the dates here, they're updating their own malware multiple times a day to do whatever it is they want to do. Maybe they're changing source IPs or destinations or something. And they kept doing that until we got the right item and cleaned out the malware. So it's quite interesting that malware gets updated more frequently in the plugins. It's always funny. And you can see different iterations of the code as well. You know, some people are really good at writing very good, nice structured malware code. It's really weird. And then other people just don't seem to care about in-depth comments or anything. It's just mess. So, you know, different authors, different code, right? But yes, that was quite an interesting one. And here's something a bit different in that what they've done here is they look for a fresh install that hasn't actually been run yet. And so they've found the setup config page and it's waiting for a user to install WordPress. And so they've gone through the setup pumping and they've connected to their own database. They don't need to care about the database and the shared host you can't, they're not interested in that. What they want is to get a file on the server. So they've gone through the setup connecting to their, and then their login is there. I mean, they just created in their setup and now they're uploading their plugin. So they've uploaded and activated their plugin in there and they're running it. And so what we often see in this scenario is they'll get in by the setup. They'll set it up to their database, they'll upload their malware, and then they'll remove the config file and clean it up. And it will look like it's a fresh install waiting to be installed, but really there's malware already on the site. And so you'll eventually come along and install your WordPress and it'll be infected already. And you won't know having indicators of the fact that it's infected because they've hidden it. And if you can see the time here, I think it's about four minutes. So we've got here, or 1454 up there, down here is 1458. So a total of four minutes, it took them to find this, it set it up and saw their malware and they clean it out at the end. And so if they managed to find your site when you've uploaded the files and you're about to run the setup, say you're looking for the database credentials or something, this could happen while you're not even noticing. So it's really important to be ready when you're going to install WordPress to do it immediately, or even throw it behind an IP block or something. Some method of protecting it or event someone from getting in. And I mean, installing it before you get to it. Okay, so my favorite malware, this one's a fun one. So one day I was doing what I thought was a typical site clean. First step was install WordPress and run the high sensitivity scan, which checks everything it can find, looking for common things, some of the base 64 decoding functions, et cetera. And I found absolutely nothing, which happens occasionally. Usually that's because there's new malware on the site and it's one of the reasons why we downloaded the files to a cleaning server to run it on there separately and we harvest the new malware to improve our scanner. So I copied the files over to the cleaning server and I found three changed files. So these aren't new files, they're not random files, they're files that WordFence should have found that were changed, but WordFence thinks they weren't changed. But the cleaning server did find they were changed because it run separately. And so the question is, why did WordFence miss these changes? Close, close, what were you gonna say? Yeah, you're pretty close. So this is what was in the WP scanning file. As you can see at the top, they have removed the files from the known files list. So this is the list of files that WordFence knows about and it has just four. So the three files they changed, they've removed from the list and then they're hashing the class wpupgrader file and throwing the new hash in there. So essentially they're telling the events the hash for the file, it's the same as the hash we've got and so WordFence got, it's fine and move on. Which is very sneaky. So the question is, why weren't they doing that? So I looked in WP blog header and it's a bit hard to read. At the top we've got a bit of conditional but what it's doing is when a request comes in, it doesn't meet their conditions. So basically checking for a user agent and such. Then it grabs the full request and then it sends it to their server and their server will process that, return some output and the code will return what they've sent and it die and end there. And so what they've got is they've got a copy of the site cached on their server which has been modified slightly and any request you make to this account is gonna grab their copy and return it. It's not gonna execute any more PHP code, any more WordPress at all. It's gonna return their copy and on their server they were carefully modifying the site to add in, in this case it was, or the first case I found, it was adding a new drop menu and then that had some SEO spam links but it was carefully hidden and it was fully themed in the site so it took me a while to notice it. And so the idea here is that if you're looking at the site and you find a spam link and you search for that in the source code it's not that because none of it's there on the server itself, it's all running on their server. And they must have a fairly impressive setup because basically they're turning all of their infected sites into proxies, pointing towards their server which is serving on the sites. And so they're crawling the sites of their server, grabbing the content and modifying them. Which is quite sophisticated really. You can see why they're trying to hide from WordFence given the amount of effort they're putting into to infecting the site, returning infected and modified pages. And the third file they modified the ClassWP Upgrader which is quite nifty. So the top bit there, they're watching for any WordPress updates and they're removing the ClassWP Upgrader file and the WE Blog header file from the update. So those files won't be updated. And then the two files that they've modified in WordPress. And then at the bottom, they're looking for WordFence updates or uploads. And they're updating the WF Scanner Engine file with their own custom payload, the thing we looked at first. And so they're allowing WordFence to update but then they're modifying the scanner to ignore those files and they're just not letting the WordPress files update. So we can break sites if there's enough changes in those two WordPress files. But if WordFence changes, it'll just try and update the file anyway. So that was very clever. And it gets around the problem that some malware has where when you go to an automatic update or update a plugin or something, it can wipe out the malware because if you modified the core file and the files are updating the update, it's gone. So they're bypassing that which is quite nifty. So there's a lot of thought has gone into this malware. So you can see why I love it. No idea who wrote it. It'd be interesting to meet them one day. If anyone here wrote it, you know. Yeah, I don't know. Come find me afterwards or something and I promise I won't hit you. So this malware continues to evolve. So we update WordFence to detect it on block. And we've changed our known files list and the way we handle hashes and a few other things around to try and detect it. And so they update their malware to bypass our block and evade detection. And it keeps going round and round and round. Like so. And so, you know, because WordFence has running, has WordFence codes in the site, they have access to modify. And clearly they're running WordFence on their test site. Current status, as far as I know, one of our current analysts shared this with me a couple of days ago is that we're winning at this point where WordFence is detecting the latest iteration of their code. That may change now. It may change tomorrow, who knows? Okay, so something a bit different. Sometimes you'll find code that looks like this. Anyone see what's going on here? Was that sorry? Honing, no. Here's what's up, Jor. That's the next slide. Yeah. Code highlighting. Can you see it now? It's all commented except for a few bits. And to make it a bit more obvious, that's the code isn't commented. And if we condense it again, we get that. So what we've got here is they're in creative comments to try and hide those two functions. Base64 decode and assert are commonly used in malware. Base64 decode, because you can base64 encode, a payload, new PHP code, et cetera, into a nice string and then send it via a get parameter or something, or via any other method. Get a pass, basic scanners. And assert is used because it functions basically the same as eval. And for some reason malware authors seem to think that we don't know this. And so they'll always use assert instead of eval thinking that we won't look for assert, which is just stupid because we know both of them and we look for both. So why use assert when eval works? That's all right. Yeah, so that's kind of funny. But the interesting thing about this is once you find it, it's very easy to write signature for it because you've got to look for code, closing comment, more code, opening comment code. And if we look back here, there's a number of lines where that would match it. So once we find out once, it's very easy to find. But it's quite a creative, creative way of hiding malware in plain sight, essentially. And sometimes you just get malware that's just weird or funny. So in this one, we keep finding this one pop up, this malware author keeps quoting Harry Potter. And every time we find it, it's a different bit of Harry Potter. I'm pretty sure there's working their way through the series. And every new bit of malware, they put this got more Harry Potter on it, so. And they're also manually typing it out by hand because sometimes there were typos in it as well, which is funny too. Yeah, so that one's always fun to find. Okay, and we had the epic time of the persistent attacker that almost sorted us completely. So this was a, this was a fun one. It started off like normal, your customer site's infected. And so we clean up the site, access logs enabled, so we enabled access logs. And we sent the report to the customer, we advised them to change their passwords. We didn't know how they'd got in because we didn't have access logs, we couldn't find the information about that. Customer was happy, they changed their password. One week later, they were re-infected. Like great, okay. So we have a 90-day guarantee on our cleaning, so it's fine, we'll happily clean a re-infected site. So we clean it again. We looked at the new malware to see, maybe we missed it the first time, is it possible that we found it now because we've got some new signatures and we hadn't found it before, but it was a new malware that we hadn't seen before. So it was definitely a re-infection rather than simply un-syncing something. We checked the access logs that we had enabled from Lynch's activity, but there was nothing in there that showed how it was created. So we could see the malware was being accessed, so it was being used, but it lined up with the creation dates, the malware, nothing that made sense for how the malware could have actually got there again. And because we enabled access logs at the end of the cleaning, we had the full history of when it would have been created. We knew it was created in that window. So we had a very limited set of logs, which made that bit easy, but we still found no... Look in the database. There were no injections, no changed passwords, anything like that that could indicate that they're getting in by the database somehow. No one theory was that they could be changing user password and logging in or uploading something, but nothing in there. Next thing we did was look at plugins from Lynch's behaviors. Maybe there's a plugin that's carefully hiding a malware payload delivery system. Maybe it's running Kronz to replace malware every week or something strange like that, but there's nothing in the plugins that was suspicious at all. Now, they were running on cPanel. So cPanel has... It logs your logins to the cPanel interface and it also logs it to pActivity as well. And so check those. There was no fDpActivity and there was no suspicious logins. They just showed our IPs and it showed the customer's IPs. There was nothing else there. So they weren't getting in via cPanel and the customer had changed the cPanel password as well, so it limited that area. So we're out of ideas. We weren't sure how they could have, you know, what was going on here. So we cleaned the site, and we watched it, kept an eye on it to see what was going on. It was reinfected within minutes of putting the new file up there. It's like, great, what do we do now? So at this point, we were wondering if maybe the problem isn't the site code or the database, but there's something wrong with the server. Maybe the server's infected or compromised or there's some dodgy permission somewhere. So we deleted all of the files from the public HTML directory. So come and see what happens. Now it was created in the directory. So it's definitely not the site that's a problem. It's something to do with the server. So we deleted it again and we watched it very, very closely to see what was going on. See if we could notice anything about the activity of the malware or anything else. Saw that. This is the VIMS hot file. So the attacker is logged into the server via something. We realized, oh, this is how it supports SSH. So some hosts have SSH, but they don't tell you. And you've got to scroll through pages and pages and pages of documentation to find the obscure port to get in via SSH. And clearly the attacker knew about this. We learned it. And as you can see here, they've got a whole bunch of connections open by tour and they're keeping it connected. And so anytime one of them drops out, they'll just reconnect another one. They'll recreate the authorized keys password, authorized key for SSH key, et cetera, to get in. And because we were basic user, you know, what could we do? How could we get them out? Any ideas what we did? Yep. We asked those things, provided to kick everyone off the server via SSH. So we changed all the passwords and we removed the keys and they kicked everyone out of SSH and it got rid of them. Talks a method though, talking to the host was fun, having to coordinate. Yes, we do have access and go through all their process, but you know, we got there eventually. So that was quite a good feeling when we got them out and the site wasn't reinfected. And as far as I know, they've been fine ever since. So that was good. Okay, and just to finish on, often when you're doing, when you're cleaning up, you wanna look at the malware, you wanna figure out what's going on. And so, but you'll find malware authors will wrap their malware in say base 64 encoding to try and hide it to make it harder for scanners and harder for you to figure out what's happening. And sometimes it feels a bit like this, where you'll come across the malware and it'll be encoded, so you'll decode it. And it'll be encoded again. So you'll decode the next layer and the next layer and you kinda get a bit frustrated at some point in there. And it keeps going. And it keeps going. At the very end, it's something so trivial. It's why bother. Thank you.