 Okay, let's get going. Hi, everybody. Today we're going to talk about security and hacking Drupal. I'm Matt Korstaff. I lead a team of Drupal developers at FFW, formerly Blink Reaction and ProPeople, where the largest Drupal shop in the world, about 400 people now, and we built enterprise Drupal applications for large corporate clients. We're based all over, but I work from the Princeton, New Jersey office. Today, I'm going to talk to you about the Drupal Geddin SQL injection vulnerability. This was a major vulnerability in the Drupal database API that affected all Drupal 7 and 8 installations and a handful of Drupal 6 sites. This bug allowed a skilled attacker to inject arbitrary SQL into your Drupal database, and in many cases, take full control of your web server. This is widely regarded as the worst bug ever discovered in Drupal, though. Now that the storm has passed, I think we can reasonably assert that little real harm was done. One of my personal websites was affected by this attack. This is whaleocalypse.com. This is where I post my cartoons about whales in a post-apocalyptic future. This is actually why I got into software development originally, not because I specifically wanted to be a software developer. I did, but mostly because I needed some place to post my cartoons about talking whales and talking fruit and parody real estate ads. So Drupal was the natural choice. Anyway, this is a plaything for me, so it really didn't matter for me that it got hacked. I just kind of shrugged and restored from backups. There was actually a tremendous upside to being hacked, and that was that it allowed me to examine the artifacts left behind by the attackers. I spent a long, long time looking at these artifacts, and I was able to use them to reverse engineer the tools these attackers use to break into my site. You can view these tools on GitHub, along with a set of files to help you defend against this and similar attacks. In this talk today, I'm going to show you how these tools were used to attack my site. I'm going to show you in detail how to use them, and more importantly, I'm going to show you how to defend against this and similar attacks. And if you want more information on this, you can get effectively the same talk on my blog, which is matcorpestoff.com, and it's linked off of GitHub. I'll put a link to it on the DrupalCon website. A couple of disclaimers before I begin with the technical portion. First, I'm going to reveal how this hack was pulled off in very, very specific detail, including a large amount of fully functional exploit code. If at any point you feel yourself going, hey, wait, stop. You're telling people how to hack my site. I'd invite you to stamp that feeling down. If you were still vulnerable to any of the techniques described here today, I'm sorry to say you were already hacked months ago. You've probably been hacked several dozen times by different individuals, and there's a very good chance that your server is now part of a botnet being used to send spam. The simple fact is that we need to be able to have honest, frank, detailed discussions about security in public in order to educate ourselves and prevent the next Drupalgeton. Because if we don't have that conversation, someone else will, and that someone probably has malicious intent. That means releasing the exploit code, and it means letting people use that code from learning and penetration testing. By the way, this is not just an academic exercise. You can literally go, I'll show you right now. You can go to exploitdb.com, and you can find five versions of this that are way worse than anything that I'm going to show you. These tools exist, and they are freely available, and pretending they don't exist will not make them go away. We need to learn how they work so we can defend against them. The second disclaimer I want to offer, this one's kind of more of a joke, but I'm going to confess it's just some really terrible things that I did on whaleoclips.com that let people break in. It should go without saying that these are huge risks that I took on a plaything website. I would never take these same risks with client work, and neither should you. So let's explore this bug a little bit. The bug specifically lies in the Drupal database abstraction layer. That's the layer of Drupal that allows you to connect to a MySQL or some other type of database server, specifically in this expand arguments function. This is part of Drupal Core. This is database.inc. This function you would never have occasion to use unless you were actually doing Drupal Core development, and otherwise you probably wouldn't even have occasion to know that this exists. This is a normal boring utility method that does some pre-processing on database statements before they're sent to the MySQL server. I'm going to show you exactly what it does. But the sort of key thing to realize is that this is a thing that flew under the radar because no one ever uses it or touches it. A normal Drupal database query. This is about as simple as a Drupal database query gets. What we're doing in this query is we're getting the title of a node whose ID is 123. We don't pass 123 directly into MySQL. Instead, what we do is we supply a token that will be replaced in the Drupal database abstraction layer, and the reason we do that is ironically, as you'll see for security purposes, because these tokens get sanitized for us. It was the sanitization itself that had the bug in this case, which is what made it such a terrible bug. So this is a pretty simple database query. You can get slightly more complex with a database query. You can use a token as a part of an in statement, and you can replace it with an array of values. The problem is there's no straightforward way to pass a PHP array into MySQL. MySQL only understands strings. So at some point in the preparation of this database statement, this array has to be collapsed down into a comma-separated string. That's what expand arguments is for. So expand arguments is going to do one completely simple straightforward thing. It's going to take NIDs, and it's going to change it into this. NID 0, NIDs 1, NIDs 2. It does nothing else. The problem with this implementation, and this, by the way, makes total sense, this is a completely sensible way for PHP to interface. It's a completely sensible layer on top of PDO, which is itself pretty cumbersome to use. The problem is that these keys, NID 0, NIDs 1, these are not sequentially assigned. What these are are the array keys that we defined ourselves on the line before. So I would ask you to consider what would happen if you could get Drupal to run a query like this. Digest this for a moment. You can get Drupal to run a query like this. There's a number of ways to get Drupal to run a query like this, and I'm going to show you how to do that right now. So let's hack into a website. This is a Drupal website. It's running on my localhost. It's running on this computer right here. Version 7.31, the last vulnerable version of Drupal before the bug was discovered and patched in 7.32. This is incredibly simple to do. So I'm going to put a breakpoint in my code. I'm going to start listening for debug statements. I don't need any programming capability in order to do this. I'm going to do this completely from the WebKit Inspector. What I want to do is replace the normal user name input form with one that has some SQL in it. And because I've created two of them, I'm going to turn my input from a string into an array, the first of which has this delete statement in it. So if we take a look at the flood table, we can see I have a number of failed login attempts. That's what flood is for. It tracks failed login attempts, and then it's used to ban users if they go over. What I'm going to do is... So we've stopped the code. Oh, I meant to do a conditional breakpoint. So this might be a little hard to see if you're at the back of the room, but we've built a query that's simply a string that says select star from users where name equals name token. Name token is supposed to be replaced by admin. Admin is supposed to be a string. But by creating two user login forms, I've turned my input into an array. And I realize this is kind of small, but one of the array keys is some SQL. So if I step one, two, three, now here we are on line 755. This is where the substitution occurs. Boom. Injected. We've now injected some SQL, and this is going to be passed directly to my SQL with no further sanitization. And if I allow this to run to completion, we can see I have, in fact, injected some SQL into Drupal. I've deleted records from the flood table. Oh, no. Any questions about how that works? There's a lot more detail coming soon, but that basic premise? Okay, great. So that's cool to truncate the flood table. That allows you unlimited login attempts. So it enables a brute force attack, but there's a lot more that you can do with that. So using the same vector, I trivially worked up a little Drupal-getting SQL injection client. It's pretty much the same thing. It just sends a... Again, this is on GitHub. It sends a post request to the Drupal user login form and does a little character escaping for me that's difficult to do with just typing manually. So I'm going to use this tool. I call it the Injector. I'm going to use this tool to give myself additional permissions that I don't currently have. I'll visit this site as an anonymous user. And of course, of course, anonymous users do not have the ability to create content. If I navigate to Node-Add, I get an access denied. So what I want to do is I want to inject some content into the role permission table. This takes about four seconds to run. And now when I refresh this page, you can see I have the ability to create any content type. So let's put some spam on the home page. Boom, we have front page content created in two Unix commands. So that, in fact, we only... And we only need one of them. So that's about as bad as a hack gets, but we're going to take this a lot further. Another thing that I, as a spammer, want to do is I want to run JavaScript on this site. Now I'm an anonymous user, so I don't have the full HTML text format available to me. So let's... I might as well be an admin, so let's just flip that on. I'm going to insert a new row into the role permission table, giving myself the ability to use full HTML. Refresh this page, and you can see now that I have full HTML available to me. And of course, of course, what I'm going to do with that is I'm going to run. And you can see what happens, even before the page finishes reloading, is now I'm triggering for every user that views the home page or views this render node. In fact, it's like hard to get rid of, even as an administrator, because it comes up so quickly. So that sucks. Because the MySQL database is so central to the Drupal application, and sensibly so, there's nothing wrong with connecting to a MySQL database so long as your connection is secure. Because the MySQL database is so central to the Drupal application, and all configuration is stored in it, once you get access to that system, you pretty much rule the entire application. You can do anything with it. So it's all well and good to understand what sort of damage is theoretically possible with this power. But what I was mostly interested in when I started doing this research is how the real hackers who attacked my site in the real world actually use this power. And I can answer that in tremendous detail now, and I'm going to show you exactly what was done. And I should note, the variety of attack that I'm about to show you was, according to data released by AQUIA, the same variety that was experienced by 68% of users hosted on that platform in the first week after Drupal Geddin was discovered. The remaining were some mix of people putting PHP in blocks and people changing the user one password to be like 1, 2, 3 or something, or adding a user. Those are all pretty obvious and easy to understand. This one is a lot more exotic. What we want to do now, you and me, we're in a hacking collective together, and what we want to do in our hacking collective is three things. First of all, we want to inject some SQL. That part's easy. We know how to do that already. If there's time, I'll show you the code that allows us to do all this, but we want to get arbitrary code execution, which we want to be able to run any PHP that we want on this server, effectively making it into our slave. And finally, we want the ability to upload any file that we want. These three things together are pretty much all the things our server does. It holds files, executes files, and connects to the database. So the particular style of attack it took me a really long time to understand, but I get it now. The first step, the SQL injection, it happens in the menu router table. The menu router table is where Drupal stores routes or pages. So if you take a look at the menu router table, these are all... I know this is hard to see in the back, but these are all the... when you implement hook menu, a cache of that goes in here. So when you navigate to example.com.admin.app this access callback, user access, will be run. That's the name of a PHP function, user access, and if it returns true, then you get access to the page. So what this attacker has done is he's inserted a new row. It's just kind of randomly named. It's XJFXIA. But the callback, instead of returning an HTML string or a render array, the callback is file put contents. So when someone navigates... navigated, because this did happen in the real world, when someone navigated to whaleocalyps.com slash XJFXIA file put contents run ran, and we know what file was putting in where because those arguments are supplied in the access arguments column. So they wrote out a file to the pull module, random selection as far as I can tell, and the contents of that file was some PHP. I'm going to show you what that PHP does in a minute, and going forward, I'm going to refer to this as backdoor.php, because it's impossible to keep straight if you use these random letters. So once that file is written to disk, it becomes a backdoor that allows you to execute any code. So the hacker navigates to this URL, and the act of navigating to this URL creates this file, backdoor.php, and then they never need to go to that URL again. They can, but they have no reason to. Backdoor.php itself, and I can I'm going to run this exploit in a minute, so I'll show it to you, is a single line of code that's deliberately obfuscated, but what it does is it listens for a cookie. It waits for someone to send it a request of a particular format, that it needs to have three cookies in it. These are just randomly named KCQF3, these don't mean anything. The first two, Base64DCode and CHJ1, these are effectively just passwords to prevent an unauthorized user ironically from getting access to this backdoor. The sort of meat of what's going on is this long string here. This is a Base64 encoded PHP string. What the hacker does is he Base64 encodes the code that he wants to run and he packages it up in a cookie and he sends a GET request to this file, module slash pull slash backdoor dot PHP, and then the server will run that as though it was its own file. So that sucks also. It's nice that we know this, that we know that this hacker got the power of arbitrary code execution. I'll show you in more detail how I know that he did this later, but what I want to know is again, in the real world how did this hacker actually use the power of arbitrary code execution against my site and we can answer that in a really interesting way. This is one of the artifacts that I found left over on my web server. This was like some random set of letters dot PHP. This is not my 404 page. This is the attacker's 404 page. Now my best guess at what happened here is that the attacker sent a request to backdoor dot PHP to execute some code that requests out to another web server which holds a secondary exploit file. What this code does is download the bigger exploit package that we want and save it to disk somewhere. As near as I can tell, on one of the occasions, and he did this on six or seven times in different spots on my server, but on one of the occasions that he attempted this, his server was down I guess, and all I got was his 404 page. In fact, now not only do I know what he did with this backdoor file, I know the name of the village in Romania where he lives because his IP address is right on there. It might be a proxy, but here it is. You can still go to that if you want. That's his website. DDoSing him, right? I'm actually pretty impressed. Good on him. It didn't really hurt my life and it was a pretty exotic hack. So when we take this file and we compile it down to HTML which by the way the first time I did I was super scared and I disconnected from the internet before I did it. When we compile it down to HTML what we find is that it's a file upload dialogue. So he got arbitrary code execution power and he used that power to create a new tool on the system that allows him to upload files and now he owns the whole system. He can really do anything that he wants and I will show you this one unfortunately takes about 15 seconds to run so this is the full exploit package that we're going to run and while it runs I'll show you I'll show you the code that we're executing. This is my best guess at the actual exploit that was used against mine and tens of thousands of other sites. If you want to check it out on GitHub it's really super commented up even if you're not a programmer you can probably follow it pretty well but the gist is I give it a URL it injects the SQL we want it gets this arbitrary code execution and then it uses the arbitrary code execution right here so here we send our cookies to the server it uses that I'm actually going to grab from the internet his uploader form and write it to my server so that should be finished by now yep and if I now navigate to backdoor.php I can see that the end of backdoor.php is just PHP info so you can learn a little about the system that you're trying to break into and if I were to ping this with the proper set of cookies which is actually pretty easy to do I would get the server to run any code that I want and then the other backdoor is uploader.php this is in the Bartik theme which actually because I'm running the Bartik theme gives us the convenient ability to change the site logo and there we go we have a fully exploited website and just to be to be clear what this particular hacker almost certainly had in mind was not to change my logo and be like HACK! GOT YOU! what he's almost certainly doing here is compiling a database of thousands upon thousands of exploited sites so if you want to exploit example.com just look in my database row 10 million you can go to example.com slash qpdl47.php presumably he sells this along with a set of instructions because it's really hard to figure out how this works otherwise as far as I know and as far as I can tell it was not interested in changing any content on my site and I really can't imagine why he would it's pretty low traffic so before I move on to the defense procedures and how you can protect yourself from this and similar attacks does anyone have any questions about the attack procedure yes sir the question was what were the direct repercussions on my pull module I don't understand what you're asking, I'm sorry what were the directory permissions on the pull model oh the directory permissions I'm going to get to that, that's one of the defense procedures and that's also part of that disclaimer I offered at the beginning as like don't let yourself turn into me but they were they were poor they were either 775 or 777 or something like that they were right to it which it shouldn't that's coming up that's what this slide is about but yeah good thinking yeah sure yeah it's very hard for me to say what makes an exploit kit marketable on the black market presumably he knows his customers and they want that, I don't know why but you're right that would be to me more terrifying any other questions about the attack payloads okay let's move on to the defense procedures and these are kind of in priority order of how important they are so if you are running Drupal 7 version 7.31 or lower or Drupal 8 beta 1 or lower your site's already hacked so just kind of be zen about that be okay, it's been hacked for a while now so don't get all stressed right now you've been living your life and you've been happy for 7 months with a hacked website it hasn't been so bad you gotta upgrade because you're spamming a lot of people right now that's very important it won't undo the damage that was done for that the only solution is to restore it from backups if you don't have backups that's a very challenging situation of course but this isn't really so much I've been doing this presentation for 6 months now it's not even worth talking about anymore the first time I gave this maybe you could save yourself by upgrading but at this point if you haven't upgraded you're hacked so go and do it be snappy about it next time the second most important thing that you can do to secure your site against this and similar attacks is to set proper file permissions in your code base so Apache the Apache web server never needs to write to your PHP files never needs to add them to them it never needs to change them it only ever needs to read them and for this demonstration I'm running the most insecure file permission that I could which is anyone can do anything and in fact if we're on the same IPLN right now you can probably get into my computer right the second because Apache is accessible to the outside network so this is very simple to deal with there are tons of there are tons of scripts available for automatically setting proper Drupal file permissions that I like this is on github but I'd also encourage you to do some googling on that, google Drupal Unix file permissions there's lots of different schemas depending on your hosting situation what you're trying to do with Drupal exactly but the gist of this is this is going to iterate over every directory in Drupal and set it to a proper permission scheme the files directory which is like where user uploaded images go different treatment than executable code and the settings at PHP files going to also get slightly different treatment but there's a lot of different variations on this and they all work fine for different use cases so I'll simply run that code it runs really fast and now you can see that the only person with the ability to write to to write to my my code directories is me and then my Apache user is called staff and he can only read if I had taken this one simple step this 10 seconds to paste this script in and run it I would have been totally safe from the that step where you navigate to backdoor and it creates a new file for you would not have been possible so that's a very simple thing that you can do that will secure you from a large variety of attack vectors the third most important thing that you can do is host with professionals and I mean specifically one of these four organizations and I'm calling these four organizations out as being as having a high security game because all of them were patched at the system level or put some remediation in place before full public disclosure of the Drupal get-in vulnerability I know there's a lot of hosting vendors here so I'm always looking to add to this list if there's other groups that had a similar similar experience that's awesome you just do your due diligence make sure that you're dealing with people who understand Drupal if you try and host on like digital ocean or rack space or something that might be fine from a perspective of load and cost but you also have to balance that with like how bad would it be if this site got broken into and if the answer like me is no not that bad I'll go write a talk about it and pitch it at DrupalCon like then by all means host yourself but if it's something that you're doing for a big client who's giving you money and that you're making guarantees to host it with someone who knows what they're doing this is not exactly a defense procedure it's more of a disaster recovery procedure but you want to make sure that you're taking automated nightly backups and that is true by the way even if you're on some platform as a service system that makes backups for you because generally those backups on a PAS system will run not as often as you need them to this is incredibly simple if you don't know how to how to set up Cron it's literally one command I'll make this a little bigger clear that crontab-e and you can look up Cron syntax but basically basically what this says is zero zero minutes and two is two hours so every day at 2am I'm just going to make a full dump in reality I would like gzip this or something but make a full database dump from my web root to a non-web root directory my backup scheme is that I make a backup every night at 2am I keep that for 30 days I make a backup every week which I keep for six months and I make a backup every month that I keep forever really easy to do it's a thing a lot of people delay on like it would be hard but it's really not it's also very important to store your code in version control you don't have to post it on github you can have a locally versioned copy of it you can see here I just it was one command to do or two commands I made one single commit ever for this codebase what it allows me to do is I can see it's hard to read I can see very easily the files that were injected into my codebase so if you find yourself hacked it's actually pretty simple just copy down your codebase put it in the same directory as your .gith folder and diff it so this becomes for me really really easy to undo and I got rid of all the exploit files like it was nothing that doesn't help with the database the database you're still going to have to restore from backups but the code should be very easy to recover in a circumstance like this a matter of minutes and in fact even though I wasn't like preparing for disaster recovery just because I just version everything I do because I find that easier it was pretty trivial for me to just back this out another important security measure is to make sure that your Drupal I'm not talking about your Unix Cron your Unix Cron is running even if you don't know about it usually possible that it's not but it probably is I'm talking about Drush Cron or Drupal Cron which is the process that gets executed when you navigate to like yoursite.com cron.php you can execute it from Drush here I have it running twice an hour so every 30 minutes and the reason this is important is that even when you have proper file directory permissions set Apache still Apache still has the right to edit or delete any htaccess files in directories for which it has right access which blew my mind when I learned it because basically what I'm saying is there's no way really to deny Apache right access to your htaccess file which I didn't believe when someone told me about it so that's actually what this helloworld.php is me confirming that that's in fact the case so the way Drupal deals with having a writable directory inside of your codebase is even if you don't know how to read htaccess you can see we just turn off php inside htaccess someone who is able to drive Apache on your web server is going to be able to delete this file it gets restored on cron that's why it's important to have cron running because if you have cron running actively they'll have a pretty small exploit window in which they can run code out of your files directory it's important to take php security releases as seriously as you take Drupal security releases and the reason I say that is that anyone running php 5.5 or higher was immune to the most common type of exploit the one that I showed you here today and the reason they were immune this is backdoor.php or it's a version that I reverse engineer in dobfuscate the original was like all named weird things all on a single line but backdoor.php which just a reminder this is the file to which you hand off a cookie and it executes any code that you want it does that by taking advantage of a vulnerability or not a vulnerability but a feature which has since been removed in pregreplace which allows you to evaluate a search subject as php so I can pass php into into pregreplace and it will be evaluated as though I were passing it into the eval function that's gone in 5.5 so just make that upgrade it's pretty painless to do in most circumstances php is pretty good about reverse compatibility within major versions this one is a little more controversial and it kind of depends on your use case but if you have a site that is low enough value that you can't afford to staff it full time which would certainly be the case for my post apocalyptic whale cartoon it's probably low enough value to suffer the minor risk of breakage during security patches so you can apply security patches automatically with Drush it's pretty easy to do that's the first line if you're versioning it's a little more complicated you should be versioning so the trade off here is that you might pull down a security update which breaks the entirety of your site that's something that you have to have to weigh for yourself certainly if you're choosing between having a site that's maybe hacked and having a site that might go down from a bad security patch it makes more sense to apply the patches that's my opinion the best case scenario is to just have a staff of professionals who will monitor and apply security patches and then test that they do the thing they're supposed to do but if that's not an option for you you really have to think about how bad your day would be if you came in one day and you found that your site was now hosting a bunch of pornography and it had sent an email to everyone in your database by pornography from mysite.com if that's a risk that you're not willing to take then just take this other to me smaller risk of your site might go down but it probably won't I've never had a set of security updates break any feature on my site ever and this is the last I don't know that this in most cases this will make no sense to do but in my case where I'm the only person logging in or where you have a small group of maybe 15 or 20 people logging in maybe even they're all in the same office you don't really need to let post traffic into Apache most of us are probably running Apache behind the varnish caching layer so you can do some pre-processing on the request that you get before passing it off to Apache and therefore PHP so one thing that I like to do is if the request is a post request which it would really have to be to do any SQL injection of any complexity and it doesn't come from my IP address at my house just drop the request I'm just not letting it through I can demo that for you if you want but I think it's probably pretty clear the meaning there so you put these lines in your VCL file and these lines in your HT access file so if someone tries to navigate directly to Apache they just get they just get a 403 and if they try to navigate to your site like a normal way go to yoursite.com and then try and log in they get this 404 so the last thing that I want to I want to touch on and I open for questions is this event is very scary for our community it was really the first of its kind it's been the Drupal security team was founded 11 years ago and this is the first vulnerability of anyone close to this magnitude that they've uncovered so it's the name Drupal again which admittedly I've been a very big user of it's easy to imagine it was easy to imagine while this was occurring that the world was ending but as we look back on it we look at the real functional results of this exploit it seems rather clear to me that it wasn't actually that big of a deal I mean it was a really bad bug but like honest to God show of hands did anyone here like lose money for data as a product of Drupal again I'm sorry that happened to you and that really sucks but here we are at Drupal and there's two people and that sucks that's bad I don't know that it warrants an analogy to the literal end of the world so just when we consider these facts that anyone who had proper directory permissions configured which probably nearly all of us did was immune to the most common attack anyone who was running PHP 5.5 or higher is probably about a third to a quarter of us are was immune from the most common attack anyone hosted on the four biggest Drupal hosts was immune to the most common attack and probably all attacks to be honest the amount of real world damage I think was just not that big and the learning experience was really important so in answer to a question that I was asked on the last time I gave this presentation do you think this was really Drupal again, do you think this was really going to kind of destroy some aspect of our community my answer is no so that's all I have for you and I have about 15 minutes for questions if you have any yes sir, the question was how was this vulnerability discovered and so that's kind of a complicated history but it was it was first reported maybe a year or maybe a little more before it was acted upon and it was reported in like it was reported in some queue like a usability queue that not reported as a security issue and I don't know who on the Drupal security team ultimately found the issue and decided to act on it but there is an official group of in fact we have one of them here with us today if I thank you for coming but I think you saw this presentation before there's an official group of people that are have the task of identifying security fixes and releasing advisories and assigning them an urgency so I actually don't know how this got into their hands but there have been a number of discussions about sort of improving the security issue reporting process as a result of this because there was someone in the world who knew this was possible one guy for like a year before any action was taken but as far as I know he's just like he's just a Drupal user who was doing some penetration testing for an application that he was building for his employer or she I mean I should be gender neutral here yes sir I think that person didn't really know specifically that it was a sequel injection they were like this seems like a bug and it might be sequel injection but they weren't really sure but then it was later found by a company called Section Ainz and Stefan Horst is the name of the person who worked there penetration tests so they found it and reported it to the security team who then coordinated them yes sir you need to wipe your whole server and reinstall your operating system and I'll tell you why and this is exactly what I did which again is a hassle but there's really no way around it let me give you a nightmare scenario to sort of make this point let's imagine that your server is in such a way that the exploiter the hacker was able to gain access to change binary files in your root and let's imagine that the hacker replaces the Apache binary with a new version of the Apache binary that he's created which will do exactly one thing differently every 5,000 requests puts a little bit of JavaScript at the bottom of the response I don't know about the rest of you but I would never catch that and like Unix systems are complex and there are a lot of places for worms to hide like it's a little bit of a balancing act like is that that scenario I just described like a real world thing that we know happened no not as a result of this bug a real virus that hit Apache boxes about 3 years ago so you do have to measure it's going to cost me $100,000 to wipe this server and I really don't see anything wrong with it I don't know maybe you'll get away with it but if you're running an enterprise application if you find one day that your server is being used to send spam is that a risk that your business can tolerate and I think probably for most of us you don't know so you have to restore from scratch yes you sir I think it's an excellent idea and in fact a previous version of this talk included that strategy but I cut it for time and also it's kind of hard to explain without live demoing it yes sir so Ben Jevins from the Drupal Superior team and I saw your presentation in New York City so I just wanted to point out there's one thought around the idea of the hosts that you talked about so this is not a question but more just for the recording so the security team is working through some policy on what security team members can share and so you have the slide about different hosts and the idea about certain hosts being more professional than others so just the Drupal security team works for all Drupal users so we want to secure all Drupal sites because those are hosted so some security team members work for some hosting companies but certainly it's not you know in the attention to secure their customers only thank you for pointing that out I was actually a little worried about that implication so first of all thank you for your work on the security team you guys yeah you guys do you guys do amazing work and then that's actually your work is one of the core things that keeps me on Drupal as opposed to some like young hot technology because like that's the perfect example of a human institution that we've built over a long period of time no matter how good your code is you don't have that in MeteorJS I know that was a controversial that was a controversial thing that some hosts got early access to this information I do want to say and being a person who got late access to it and who had his site hacked I do want to say I think you made the right call in releasing this information early to the biggest hosts because the world in which we wake up and like Acquia got early access I understand the frustration there and I share it a little bit but what we wake up to find that Whale Acolypse was hacked is such a different world from the one in which we wake up to find that WhiteHouse.gov is hacked and even if you're not hosting on Acquia or Pantheon or BlackMesh or platform.sh even if you're not hosting on any of those you would still lose money if they all got hacked because no one is buying Drupal services for a little bit on the same day so I think it was the right call and I think you guys do awesome work and thank you for it I just want to add a little piece of nuance to that the security team didn't provide the hosts specifically with the advanced knowledge it's just that some of the hosts have people on the security team there wasn't a decision made I work at Acquia and I'm on the security team right yeah I yes yes sir just a quick question on kind of this was kind of a knee-jerk reaction that some clients we had worked with had after the attack what do you think the value is in hiding Drupal on that sort of thing obviously change log is something that always goes away but from the metadata other things is that something that is actually worthwhile well to judge whether it's worthwhile I would need to know your budget and how much it would cost to implement I don't think there's zero that's a thing that people kind of brush off as like that's security by obscurity I'll never do anything if I'm running a bot that scrapes 20,000 websites I might miss a thing or two based on some seemingly trivial obscurity measure I wouldn't rely on that that's obviously not that's obviously not the only thing you need to do but if you have the resources to get it done I don't see any reason not to yes sir in the situation where you might take over a site where you don't know the history of it and when it was patched and so forth back around the time this happened is there something you can look for in the database that would give you any clue that it had been yeah sure now just to be clear if you're taking over a Drupal 7.31 site or a low or that has not been patched it is hacked you don't really need to check well I'm saying it's up to date now right yeah but no I can answer that question for you there's two things in the database that you would really want to look for that would be the canaries in the coal mine the first would be bizarre entries in the menu router table specifically in the column name those should all be like human readable English words if you have an entry in the menu router table whose name column is set to a series of random letters that's a result of Drupal Gedenoma certainly the other the other thing to look for it is in the users table or the user roles table either a new role or a new user called mega admin or super admin or something like that that was a common exploit that a lot of people saw you might also look in the blocks table for if any blocks have PHP in them but those are kind of the three big ones and just to reiterate assume that you are hacked gentlemen did you have we found some PHP files in our sites files folder so you could actually go to that file from the front end and the server would run it so right so that's the that's why it's important to have cron running yes to follow up on the database question there is a module that it's called the Drupal Geden module I wouldn't rely on it but one of the things it can do things like breaks in the UIDs for users because the often attackers don't get the UIDs sorted properly so there are going to be gaps or users with high UID numbers with admin rights things like that so there is a module to do some of that scanning I wouldn't say if it's coming out clean that you're actually clean but it is one thing to give you some indications good suggestion any other questions gentlemen in the back wall is really clear if we already have been infected we see the sites folder it does have the PHP files we look in our database we see the weird users and we don't have backups are we screwed at that point it's determining whether to go fresh install there's no patching there's no upgrading and removing files correct no matter what so the official answer is yes the un-official answer is that's a business decision you have to decide how how much money is it going to cost you to attempt maybe 99% confidence that you've recovered and how much money would it cost you if you found three months from now that you have been attacked again like it'll just come back there might be a script that is running that you don't even know about that's outside of your web root maybe so from a technical perspective is there a way to know for sure that you've cleaned out all the dirt thank you awesome session thank you yeah I agree lots of great detail I just want to echo that point I would say there is a resource there was a frequently asked questions about Drupal get-in and there's a lot of great resources in there including a flow chart by Bevin Raj about how to recover your site that I think is a really valuable tool that people can use there's also linked from that is a guide that's called your Drupal site got hacked now what that provides step by step how to audit your site and try to recover from it and make a decision about what level of effort you're willing to put into it and whether or not to do that we have time for maybe one maybe two more questions if there are any okay well thank you guys very much for coming