 Hi, I'm Akshath. I'm the founder of Blockvault. We are a WordPress backup service and we have been doing this for over six years. This past February was possibly the worst month of my life. Let's give it another shot. Sorry about that. Yeah, so this past February was possibly the worst month of my life. I just come back from my honeymoon and you guys must be thinking, oh, marriage is tough. Marriage is tough, but it's not that bad. So we are one dreadful morning. One morning I received this dreadful email from a big customer of ours who had many of his sites which had the Blockvault plugin installed on it getting hacked. Now if you are... If you build a plugin or if you deal with WordPress, you'd be like, you are used to thinking that maybe it's not really you who's causing the WordPress site to get hacked. Sites get hacked so often. So the first question we asked was, was it us? It must have been WordPress itself. That past week WordPress had a major vulnerability. I don't know if you remember, but WordPress 4.7.1 had a major content injection vulnerability and that had come out, that had been made public in the last week of January. So you're like, okay, maybe this is related to that. Investigating further, we saw a common pattern. We saw that many of our customer sites were getting hacked in a very similar manner. But how could it be us? We followed the best security practices. We did in-depth code reviews. It cannot be us. But it was. We had solved hacks earlier. We had helped our customers remove malware from their site. We had helped customers deal with really major WordPress vulnerabilities like Tim Thumb, Gravity Forms. These vulnerabilities had affected millions of sites across the world. We had helped many of our customers deal with this, but this was totally different. We were unprepared for this. We were the culprits here. But it was not as simple as that. Like there were questions to be asked. We knew that we were related to the problem. But was it a plugin which was the problem? Was it our servers? Did we get hacked ourselves? It was not clear. We did not know what to do. We did not know what to tell our customers. Even how to tell them, in fact. We had never, we would email our customers once in a blue moon and never really had done bulk emails to all our customers. So how do we even send bulk emails to such a large number of customers? Existing and old. We did not have anything in place. One thing we did know was that we are going to help every customer of ours. This was a complex situation. Every website is important and the customers by themselves have their everyday users of WordPress. Some of them are experts, but even then this is a difficult situation. And we knew that we had to help them. The other thing we knew was that there would be no hide-and-seek. We had to be truthful. The customers themselves had limited understanding of what was going on. And if we did not, if we were not truthful or if we withheld information, then that would make things only worse for customers. We knew these two things for sure that we had to do these two correctly. Beyond that, so once we had this, we were like, communications played a very major role. We had to come within the team to our partners, to all our customers. Communication, it was a stressful situation for each and every one of these within the team and outside. Communication we knew was extremely important. So once we reached out to our customers, everyone, the entire team, eight of us, jumped in and we started helping customers over different channels. Chat, email, even phone. We normally provided only email support, but now in this situation, every channel was being utilized for the customer support. But it's not that simple. We actually were just running around like headless chicken. There was utter chaos. Eight people doing eight different things. Emails were getting lost. Cleanups were being incomplete. Sometimes you would do the same task again and again. There was, at the beginning, it was just chaos all over. Fortunately, we soon realized that this situation cannot, the situation requires more thought. Cleaning up a few sites once in a while is easy. Something of this magnitude is not that straightforward. You need to set up plans. You need to set up processes. It can be as simple as creating a document which tells our customer how they can find the FTP details from the web host. To creating tools to identify malware. We were lucky in many ways that we had some extremely powerful tools within our company to help us deal with malware. But nonetheless, we had to build tools on the fly. So, communicating bad news itself is difficult. It is even worse when you don't know the exact reason. We knew that we were the problem, but we did not know exactly where the problem lied. And finding this is non-trivial. We did identify the problem, one of the problems early on. There's this function called unserialize. If you are developers, you would have definitely come across it. Don't stay away from that function. That's my one take away from this entire talk. We did patch it immediately, but we could not be sure. There were only 20,000-plus installs of LogVault. How could so many customers get in? So many sites get hacked if there are only 20,000-plus sites. Maybe we still had, our servers had been breached too. So we communicated the worst-case situation. In hindsight, this was a good thing and a bad thing. We did play it safe by communicating the worst-case situation. After our analysis, we realized that the cause was only the vulnerability in plug-in, but the message just stuck. So, like I mentioned, identifying whether your servers are hacked or not is non-trivial. It means going through terabyte of logs, non-trivial tasks. You have to build tools on the fly to identify this, but you have to be committed. We spent a month figuring this out, like going through and identifying whether something was wrong or not. You will get false positives. You will think, oh, something looks funny there, but fortunately there was nothing on our systems which were wrong. It was just the plug-in. You will end up fixing it. When you are combing through your systems, you will find other issues and you will end up fixing them. Some of these are just distractions. I wished you could prioritize correctly, but sometimes you just, as things come your way, you fix the issues as you see them. Mistakes were made, many, many mistakes. We ended up mailing customers whose sites had already been cleaned. Some of them got angry. Many were confused. It was a very, very painful process. So we are making changes on the fly. We are doing these. We are building tools as we think of them. We are cleaning sites. All of this are complex processes. Make detailed notes as you are making these changes. You are touching your customer's sites. You are removing malware. Make detailed notes because they will help with communication. So there were a few lessons we learned as to what not to do for sure. Some of these are lessons which are easier said than done. The very first one, I don't think this requires rocket science, but do not panic. Again, easier said than done. We would panic and we would make more mistakes. Fortunately for us, there were a couple of people in our team who acted as anchors. They ensured that the team was calm and that helped us weather the storm. Do not shy away from the truth. Both internally and externally. Integrity is the key here. There are many situations which are difficult. Sometimes you need to make difficult decisions. When you know that you are going to be truthful, those decisions are easy to be weighed because there is only one answer in that case. When you are dealing with a hack or malware or any of these situations, it is difficult. It is very, very time consuming. And there is a tendency to just look inwards, huddle together, huddle the team together and just try and figure out how do you fix the issue. There is a huge strong tendency to do that. But remember, business is about customers. You need to prioritize them. Finally, I think one of the biggest things that we learned from this situation was how helpful the entire WordPress community was in the process. They helped in small and big ways. We reached out to security experts who came in, dropped by our offices, gave us perspectives on an urgent basis. Others who jump into a Skype call and review the emails we would send to our customers. Because these are complex communications and sometimes the way we would frame it would again be too internally focused. And having the community members look into it really helped us. Finally, customers will be your biggest supporters. They will be understanding and they will help you. Occasionally you will find a customer who is very demanding. You would want to run away from them. But no, jump into the call and deal with them. Some of them might even abuse you. But you must understand, they are going through a difficult situation. The websites have been hacked. The web hosts have shut them down. It is stressful for them. But overall, majority of them will be extremely kind. At least that has been our understanding. Finally, you will survive. And you will come out stronger. As in our case. Thank you. Thank you very much, Akshad. Well, we are open to the Q&A session if you want to ask anything. Yes, the gentleman at the back, please. Hi, thanks for the talk. So can you go into detail about what the hack specifically was and how they were able to get in and what you guys did to plug the hole? No. All right. So we use this function called unserialize. So if you know, in WordPress also, but in PHP you have a serialization mechanism, a data serialization mechanism called serialize and unserialize. Those are the two functions. Now, the way unserialize works is it is an inherently dangerous function to use. When used over an unauthenticated data or unverified data, it can cause a person to execute any code. It gives remote code exploitation. It opens itself, opens a plugin or a site up to remote code exploitation. So that was the case with us. Unwittingly, we had added unserialize to one version of our plugin and that caused our systems to be exploited. We immediately moved it out to a place where we knew that the input was verified and that solved the issue. But yes, we are not the only ones. After that, we have come across multiple articles being written which talks about how unserialize should be avoided at all costs, especially on unverified input. At that time, while we identified the issue, we could not be sure if this is the only issue. Our plugin is, we are a paid premium plugin, so we were present on 20,000 plus sites. Now, why would someone, so we thought our systems are compromised to be able to access these 20,000 plus sites? But then we realized that a hacker, the hacker in this particular situation just attacked thousands and millions of sites around the world. And when you do the math, you realize that it doesn't take that many resources to attack millions sites for an exploit. All right. Thank you, Akshath. So, as Akshath already mentioned, so the first thing that you shouldn't do is panic when such a thing happens, right? Because it's like your worst nightmare come true. Your business is out there and yes. So if anybody else wants to ask any question because it's a very interesting topic. All right. You can catch him later maybe during the tea break later today if you want to ask anything more. Thank you, Akshath. Thank you very much.