 Hello, everybody. Nice to see so many familiar faces here. I'm Otto, and I believe many of you have seen me talking before. This is the 14th word camp I'm attending. I'm the CEO for Cerevo, and I'm a very technical CEO. I've been involved in doing plugins and done some patches for WordPress Core. In fact, to most of the stack that we're using, because I'm a die-hard Linux and open source advocate. I've spoken many times at word camps about this talk, WordPress Security 101. In that talk, I tell what you should focus on if you want to protect your site and what are the unnecessary things you can ignore, because there's so much false security advice out there. If you haven't seen that presentation before, I recommend you go and check it out. Today, I'm not going to talk about how to protect your WordPress site. Today, I'm going to talk about something different. I'm going to talk to you about a Friday in November, what happened then. Hopefully, you can learn something, maybe some technical techniques, but most of all, a mindset about what needs to be done when, despite all protections, somebody gets in and there's a security breach. I'm from a company that does premium hosting and upkeep for WordPress. That involves that we take care of security for our customers. We do upkeep, which means that if the site goes down, we take it back up. That covers also security incidents. If our client and your site gets hacked despite our protections, then we will fix it for free, remove the malware and make it back online. That's something. We have two and a half thousand sites that we are upkeeping. Security incidents are rare, but since we have so many sites, they are regular and we have lots of experience in investigating and cleaning up sites. Today, I'm going to tell you about one Friday, which was slightly unusual. It started like this, just an average Friday. No warning signs of anything. Then suddenly, at 11.37, one site started alerting that it doesn't work anymore. Then a few minutes later, another, and then immediately a third, and then two minutes later a fourth one. Something very suspicious was going on. It wasn't even that Friday 13, just an average Friday. We went to look what happened to the sites and immediately noticed that all of the four sites had the same site URL. This is not the real site URL for the site. We started wondering what's going on. Is this a mistake by the customer or the customer's admin? Did they change this on purpose? The interesting thing was that all of these four sites belonged to the same customer. They were all new sites and the same media conglomerate owns them. It was really weird that they all stopped working at the same time. This was the symptom that you could immediately see. We quickly figured out that that page doesn't exist or didn't work. This doesn't make any sense. It's not something the customer did. Since it was the same customer who owned all the sites, we immediately started suspecting some kind of targeted attack against that customer. What happened on the site didn't make sense. We didn't understand why somebody would do a targeted attack like this to a customer. Definitely, it was a security issue. Somebody had got in and done something weird. The first person in our staff who noticed this and started investigating it, immediately realized it's a security issue and then notified our security officer on call about that this needs attention. The first technical thing done is that we saved the process list, what was going on at those customer sites, and then we shut them down temporarily so that no further damage would happen. Then we notified the customer that there's a security incident and we're investigating it. Then since it was four sites and we realized this is something unusual and big, it was escalated so it was not just the security officer on call but the three-person team investigating all the sites in parallel. By 11.55, the security investigation had started full speed. These are the questions we ask when we start to investigate the security incident. First of all, what's happening? If it's still happening, what do we need to do to stop it? What happened before this? What led to this situation? When did it start? Is there, does the site currently have malicious code, some kind of backdoor or something hidden somewhere? What has in general changed on the site right now and recently? Something in the files, something in the database. What changes are valid, something that's supposed to change and what is an anomaly, something potentially done by an intruder? Then of course, who did it? We don't see any names and rarely the investigation leads to finding a real human suspect but at least what we can do is find out the IP addresses and where the malicious traffic is coming from. Then we can also figure out a few identifiers and fingerprints related to the attacker. Then after that, when you have the situation under control, you want to figure out how they got in, how they did it and what kind of level of access the intruder got, what data could potentially have been leaked because then we know what to tell the customer about what is the maximum kind of damage that could have happened in this case. Then maybe try to figure out something about the motive. Was it a targeted attack? Was it just somebody trying to plant back doors and sell them later in the black market? In general, what damage was caused? Then the next steps in practice, what we did and the steps I'm presenting here is kind of our standard steps or at least the last fall version of them. What we do, we constantly improve our process, what we do at security incidents and this is almost our latest process at the moment. The first thing to do is to make a backup. The backup stores the state the whole system was in at the onset of the security investigation and then also when you have a fresh backup it's easier to compare the fresh backup with the latest backup to see track down changes because sometimes the intruders try to mask their changes and modify file dates and stuff so it would be more difficult to find what happened. With the backups you can then compare the backups and we have a few custom tools built for that so that we get a quick list of which files changed and then just using basic command line diff tools you can compare two folders and all of the files in them to compare the previous backup and then the latest state. This will reveal changes and then after that we check who logged into the VP admin and who logged in using SSH and tried to detect if there was somebody, some IP addresses or user names that don't belong there and then the next step is that we use this VP CLI commands which are very handy. How many of you use VP CLI with your hand up? Those who don't do, please learn it. It's really convenient especially if you work with lots of WordPress sites. VP CLI has this built in tools that you can verify that your WordPress core files haven't been modified. It would compare the checksums of the original WordPress release and the release you have installed and you can also do it for plugins and teams but this of course works only for plugins that are in the WordPress.org directory. If there are custom plugins then there's nothing to compare against or if there are paid plugins that are not accessible publicly then VP CLI can't compare them. Then we also have one custom VP CLI package which we use, it does the same thing but it also can show you the diffs. If it detects that the file has been changed it will show you what is the difference between the plugin files of this version on your system and then the plugin files of the same release in WordPress.org. In this case we found several plugins that had code modified. The plugins didn't match the versions that are released on WordPress.org but unfortunately WordPress.org, the current subversion system they use and the tagging system they use doesn't enforce that one single release version would be exactly the same. The plugin developer can make changes for example to the readme file so it depends on at what time you install that plugin which version you will get. There are some false positives that this technique can yield and in this case we use the diffing tool to see what was the changes and came to the conclusion that this was false alerts. This was not related to the security incident. We went through all kinds of things and it takes some time and even if we had three people working on four sites in parallel it took some time and remember that the sites were shut down temporarily during the entire investigation. The next step we do is to check the WordPress user list if there are any recently added users or some user names or email addresses that don't look valid. We also check the database itself if there are any recent posts or something, recent changes injected into the database and this is the SQL, this is one example of the SQL queries you can do and by the way I will be posting my slides on Twitter after the talk so you can copy paste all of these commands later. At this step we noticed something fishy. There was on multiple, on many of the investigated sites there was this variation of Trollherden in the user database and some foreign addresses which our customers surely don't use. This was an anomaly and since these have the dates when the users have been registered then this was kind of bingo for us because this then opened up, gave us lots of data to find more clues about what had happened. We knew that somebody had managed to get additional user accounts registered on WordPress and we knew what email addresses they had been using and we knew the timestamp when it happened. Based on the timestamp we could then look in the logs what happened around that time, before it and slightly after it then we found the IP addresses and with the IP address we could find more what the same attacker had done and just those are things you can do with basic grep from logs assuming that you have access to all of the access logs of your server. With this clue we found out in the logs what happened and this is how the entry looks like, looked from an access point log from an access log. Can the people in the back of the room see it too small? Yeah, you can see. We saw this pattern. Can you based on looking at this request understand what's happening? There are some post requests to the VP admin and they return 200 okay as the result which means that those posts were accepted and they didn't fail. They did something that they shouldn't be able to do from the outside and then quickly after that somebody registered an account and then verified that account because when you register an account you get an email which you need to click on to verify that before WordPress activates your account and after that they continued doing something sending some requests to WordPress which were successful and that was the IP address. It was a Russian IP but you can't really make any conclusions of this. It can be from anywhere in the world. It just happened to be that the last hop was in Russia so there was probably some Windows XP machine or something that hasn't been updated for years and was used by others to attack and we also knew the user agent of those requests but that doesn't mean anything either. It can be spoofed to be anything and if you look how quickly this happened in just a few seconds all of this then you can know that this was completely automated. What was those weird post requests to the WordPress admin? We don't log post requests for obvious reasons. It would be a huge amount of data and it would violate privacy so we don't log post contents and hopefully nobody else logs post contents at least not posting providers but luckily we have lots of other logs, PHP logs and database logs and others so we found some anomalies around this time and for example there was lots of requests to a table in the database with this name and that looked weird so when you investigate a case you find lots of weird things and the method is to check all these weird things and then try to eliminate if this was actually normal after all or not and this looked weird so we then graphed from the files what plugin is talking to this table in the database and then we found it that it was about this VP GDPR compliance plugin it was doing something but we didn't know which line of code it was and what and more and this time it was slightly past two and the sites had been down for two and a half hours almost so then we started looking into this plugin is there something weird with this plugin and we looked at its changelog and since its open source and on wordpress.org we could go exactly look at what lines of code was changed in the recent release and then we found that they've done if you know what SQL injections are then if you see developers adding these signs around what they're passing to SQL then you know it's probably something they've been fixing they've probably been fixing SQL injection stuff so at that point we were pretty sure that it's related to this plugin and our first solution was to just remove that plugin it's kind of ironic that the customer had this GDPR plugin to improve security and then it was the point of entry for an attacker and then very quickly after that it was the afternoon in Finland so US woke up and people in the US started blogging and posting news and then we found out that other people had found similar attacks and posting about it and here are some links and as I said I will post my presentation on Twitter afterwards so you can go and read on these articles to see more details about what happened Sikuri has a very good blog post on this specific vulnerability in this plugin so what was it? so it was an SQL injection thing so that it was possible to edit WordPress options inject WordPress options without any authentication so what the attacker did with those post requests that first they changed the settings in WordPress that anybody can register and then they changed the settings that the default role when you register new users is administrator and then after that they registered a new account which had administrator privileges and after that they could access the site and do more and this was reported by Adrian Merschen so credit goes to him for figuring this out and this was fixed in GDPR compliance version 143 so when we figured this out we knew that this affects everybody running this plugin so in addition to the normal tested updates we do we also do security updates and we decided this is something that we need to update for all of our customers immediately and we entered this into our automation and started rolling updates for everybody to ensure that nobody else would get reached and then it was almost 3 o'clock in the afternoon so this was the investigation part then we started thinking which steps do we need to take before we can reopen the sites again so what you want to do is you want to ensure that whatever was the entry point is closed and removed so nobody can re-enter and then you want to make sure there are no planted code or backdoors that could be used to access it so obviously we removed those user accounts which were not authorized and then our rule is that we try to recover backups and get back to those because that's the only way you can be sure the site is clean but in this case it was a site with lots of traffic and lots of changes and going back to an old backup was not an option so we used our security scanners to find if there are any suspicious PHP calls which are usually typically used in backdoors to figure out and luckily we didn't find any malware or actually we did find one file but not more than that and then also in addition to removing these fake accounts we also as a precaution changed everybody's passwords and we have an automated tool for that which also our customers can do if they think that they want to change all of the passwords at once for all users and here is a screenshot from our custom tool so we scan for patterns of malicious code all the time from our customers and then we analyze when we have findings then we need to analyze if it's a false positive or is it actually malware or is it just like bad practice malpractice that somebody wrote the plugin and did something stupid but it's not intentionally a backdoor and then finally after several hours of work intense work we were able to open the sites again and notify our customer that they are public again and sites are clean and reopened and once the sites have been reopened we naturally need to continue to monitor it closely to see if there are new attack attempts or if there are calls if there was some backdoor that you didn't detect and there's some suspicious traffic going on and during the investigation of course since it took so long, unusually long we sent lots of status updates to the customer telling them how we're progressing and what's the state we sent eight emails all in all and then afterwards a longer report about what had happened also we found out afterwards that the customer did actually get the notification emails from these fake user accounts but they just didn't realize something fishy was going on and you can't really blame the customer from it because there's lots of emails anyway and then there's just a new registration email and nothing indicating that it's an alarm or something and our biggest fear was that it was a targeted attack because the same customer's site all went down at the same time but luckily it was just a coincidence that they were using the same plugin on all of their sites and they were all targeted during the same days so be prepared so most of the advice online when you Google about WordPress security is about how to protect your site but you also need to keep in mind that despite all possible protections someday there are zero day vulnerabilities that will get out and bad actors are going to start bombarding the net to find sites that have those vulnerabilities and someday somebody might breach the site anyway so you need to have some kind of plan what steps you're going to do if that happens so here's our steps, what we did in practice I hope you learned got some ideas and I hope you all think what's your strategy what will you do if you get the bad news and if you do lots of WordPress sites eventually one day you will get some bad news and you need to react thank you so I think we have a couple of minutes and this is my Twitter handle so keep an eye on it for the slides for some questions so I think Janne has a mic so raise your hand if you have a question thanks a question slightly related to this plugin or GDPR plugin if the site has been hacked there is a question if personal data had been leaked and if data protection agency needs to be notified about that have you ever with WordPress done also this kind of forensics to decide or maybe was it important with these sites to decide if something has also been taken off the site so in this case we knew that the intruder got administrator access so they could download everything if they want but we didn't find in the logs any signs that they would have downloaded everything and also the attack which they did that they changed the site URL to redirect all the traffic somewhere else that was kind of their motivation and it didn't work very well because that site they redirected it to it didn't work anymore probably got too much traffic so in this case we concluded that there was not any significant risk that all of the user data could have leaked but in other cases this is something you need to consider in Europe that you need to notify the data protection agency about this and first and foremost you need to notify the customer as quickly as possible I think we had another question over there yes my question is a little bit more on the human side communicating this to a customer is always difficult so did you communicate it was the technical team who communicated with the customer first of all and then was the customer reaction that of alarm and kind of made your work a little bit more difficult or was a comprehension and so we have already made templates about what kind of notifications we tell the customer so this is the part of the preparation you should do that in addition to having backups and everything you can do in advance and then knowing some tools how to investigate and clean up you should also include in your process that what kind of email templates you have and what you are notifying and where so we have a good text to send to the customer so that they immediately know what's going on but they don't get too much freaked and don't do anything stupid and in this case everything worked very well the customer quickly replied that they got the email and they started to do their own investigations on their own side and checking through the of course they didn't have access to the site during the time it was shut down but then afterwards when we reopened it we asked the customer to please log in and check continue and check if there's some anomalies because of course the customer knows the site more deeply than we do so they can detect if there's something more unusual on the site and yeah we sent out eight emails to the customer during this process so the communication is a very important part of this Thanks Two questions? I have one maybe thing that you could I also have a question to the audience How many of you have been involved that your site got hacked? Ouch Well you came to the right talk We have another question over here Hi Otto and thanks for the talk In general what do you think that how big part of these hacks are related to plugins? Yeah so that's in my security 101 talk I recommend you go and check it out on WordPress.tv So WordPress core is pretty secure at least for recent years and all of the security issues are related to teams and most of all plugins and the sad thing is that if you look at vpvuln.com which lists this known security vulnerabilities and in my security 101 talk I usually show this slide with statistics of which plugins have most security issues then in the top 10 there is ITEAM security and WordPress so it's kind of ironic that most security issues are in plugins and then when people install plugins to mitigate security issues they actually get more security issues or can get more Yes plugins is the problem Any other questions? Great questions by the way I had one thing about the checksum that you were showing how the different versions etc You just go through that again like how updating a readme and how the vpcli command can fail Yeah so the thing with the WordPress.org plugin director is that when you submit the new plugin to get it in it needs to pass review but once it's in you can update it as many times you want and there's no review anymore after that and luckily there's a project called vp tide coming to core so that there will be more automatic quality control in the WordPress plugin directory but currently the WordPress plugin directory is not very supportive in quality things and this thing I was talking about is that when you publish a new version of a plugin let's say 1.5 then you change the version string in the main plugin PHP file and push that to WordPress.org repositories and after that people will see that they have an update and they start updating that but you can also make other changes to the plugin and push that without changing the version number so there can exist in the repository like slightly different versions of the plugin but they still have the exact same version number and this makes it hard to verify that if you have version 1.5 installed on your system to verify in a completely automated fashion to intact and not modified in any way because it can be modified it can have different versions Thank you Any further questions? Then I'll think we end there with a big applaud for Otto