 This is rescuing me, rescuing a sad broken Drupal about doing Drupal site audits. I hope, hopefully that's the talk you're here to see. About me, my name is Matt Corks. I work with Evolving Web in Montreal. I've been using Drupal since Pluto was a planet. Yeah, this is me on Drupal.org, IRC, Slack, and on Twitter I had to add a couple of numbers after my name, didn't get there quite soon enough. Just very quickly about Evolving Web. We have, we're a small boutique Drupal development agency in Montreal, and these are a few of our happy clients. We've done a lot of, in addition to site audits, we've done a lot of migration and multilingual projects since we live in a bilingual city. Great, so why do we do a site audit? The, when I'm walking into a new project that was built by somebody else, I need to know what state it's in. You can't go bolting something onto a site that's in an unknown state. You try to do that, you add something and then something falls down somewhere else and you've got a bigger problem. So you need to, your first step is to get your project to a known good state where you know that you can reliably promise to support it. You can, you know what features you can add, how expensive it will be to do those. You know if there's security issues, if the site's already been infiltrated or if it's likely to be infiltrated next week. You need to know the lay of the land before you start working on code that was built by someone else. Maybe it'll be great, maybe not. Either way, you need to know that. So here's a few examples of things that I've seen. Does anyone here remember Drupalgedan? Back in Drupalgedan seven days. Anyone here have to fix one of those sites? Yeah, I see a couple hands. In the question and answer period, this might turn into a bit of a group therapy thing, that's okay. Yeah, it was a pain. Did anyone have to fix one of those sites that didn't have proper backups? I see one hand, two. And me, I can have myself to that. Inexperienced prior vendors. Multiple levels of outsourcing. So each of the agencies themselves is probably okay, but once that number gets over three, forget it. The person at the end of the line, they've got no idea what the actual client wanted. They're not going to be able to build the site that you actually need. Yeah, and the idea for this talk came from the fact that I was thinking about how similar it was to recover from a bad previous vendor and to recover from a security infiltration. I mean, in both cases, there can just be weird things in any corner of the site and you just need to read everything thoroughly and check it out before you can move forward. So, I'll start by talking about intrusion recovery. So someone's been in your site and you don't know what they've done. So let's find out. Who here has done something like this in an audit of a site that you suspect has been infiltrated? See a few hands, okay? Maybe a little under half. The rest of you, I hope you don't get to join that club. Yeah. So why did somebody infiltrate your site? That's the first question. Like follow the money, as they say. So usually, not always, but usually it's about the money. They're trying to make money off of your site. So they might be trying to do a phishing attack and get into people's email addresses and then from there go into their bank accounts. They might be trying to paste exploit codes like cross-site scripting for other sites. They might just simply be trying to fill it up with spam for Viagra or something. Usually, in my personal experience, the most common type of attack is that someone just wants to use your Google page rank because your site has a higher ranking than theirs and so they want to fill it up with links to their spam site. Less commonly, they want to make money right off of your site and of all the times that I've had to clean up sites, I've only once seen a site, a time when somebody defaced a site just for the pleasure of defacing it and they changed the homepage to be an advertisement for their strange political group which I won't name because it's not relevant. Yeah, but it's usually not personal. If it is, you have a much bigger problem because you'll be targeted. There are entities with state-level resources who are trying to break into Drupal sites. I've had to do audits like this and those are much more complex and usually they've gotten in with phishing attacks where they've sent a fake password reset email. I'm assuming most of you followed Drupal security news at least a little bit or are aware that there's regular security announcements. A good thing to know about those is that the vast majority only affect sites with untrusted users. If your website has five editors and you know them all, you can vouch for them. They're in your office or they're people who are involved in your organization. That's the vast majority of Drupal security problems which don't apply to you as long as they're not subject to some sort of phishing attack and aren't clicking... These are people you can trust not to click on some well-crafted password reset email. So usually there's script kitties and the bad thing about script kitties is that there's a lot of them. These are very low effort attacks and they're probably not that smart or they're not putting a lot of their effort into this. So they're relatively easy to find out. How will you know that you were attacked? Several times I've done audits for someone and I've realized that their site was actually compromised many months ago and they're only realizing it now. Again, these are usually script kitties. They're usually making lots of silly mistakes and often the first sign is that you're going to start seeing PHP errors or other warnings because they're not Drupal experts and they don't know how to change your site in a way that won't start throwing errors. So you'll see blocks disappearing off the front page or something like that or the comments won't load anymore. So if you or whoever's responsible for the content on the site is just visiting it regularly and monitoring it, that's usually the first way that people find out. You can also use a monitoring utility. Nagios, for example, is something that will let you know if your server suddenly starts using much more CPU. Our company had a server once that was infiltrated for the purposes of mining bitcoins. So there's nothing visible on the website but all of a sudden the CPU usage started going way up on the spiritual machine because somebody was trying to mine bitcoins on our web server. I don't think they got any. We found them before too long but that way you would know. There's services. Pingdom is just one example. There are others that will watch your web, the front page of any page on your site and let you know if it's changed, let you know if it's gone down so they just try to visit it every five minutes or whatever. Another really useful service is Google Search Console. Who here is using Google Search Console already for something else? Yeah, I see most of you are putting your hands up. Somebody in your organization should be doing that. It will let you know if there's problems indexing your site, if there's a problem. I had once advised me of a problem with URLs in the sitemap.xml file. So they will proactively send out warnings if they notice security issues. It'll take them a couple days because they're only not indexing your site every five minutes but they'll eventually send you a note just to be good stewards of the internet, I suppose, and that can help. You or somebody in your organization should be doing all these things. How did they get in? Again, in my experience, this is other than the one client who was actually specifically targeted with spearfishing emails, this has always been a publicly disclosed vulnerability somewhere in the stack in Drupal Core, like, for example, Drupalgetin, a contrib module, a library, or sometimes even the web server itself, the SSH server on your web server. So someone's just going through publicly listed announcements and seeing who hasn't done their upgrades. If you haven't done your upgrades, that means they'll find you. Often you can just find these with Google. If you type in the right magic string into Google, you'll find a vulnerable site. The security team has recently become, a few months ago, started becoming much more proactive about this and their policies changed. It used to be that most Drupal developers I knew were not aware that the security team only provided support for stable contrib modules. So if it was RC 15 of something or a dev release or a beta and alpha, there's no security team support because they figure it's not ready and they only have so much time. Now this is very clear. In fact, you get a little green shield icon as you may be aware recently the process for putting a new module onto Drupal.org changed. So instead of a long, laborious process to become someone who's authorized to post new modules on Drupal.org, now anyone can put up a module, but the process has moved to the part where you become someone who's able to request security clearance for your module. So if you're using a module that has security clearance, watch out. I mean sometimes you have to do it, but you need to be aware of that. The security team has one of the best disclosure records in the industry. There's a security announcement list. You can get it by email, Twitter, RSS, whatever you want. Hook it into your Slack bot. They will only tell you though about things that are in Drupal. If you are using a contrib module which is using some external library, or PHP CAS, plugs into the CAS single sign-on system, there is no security announcement list for this library. So there was a major vulnerability in that at one point a year or two ago. And the module maintainers in Drupal, the best they can do is say go click this link and go look to see if there's been an advisory recently. So you just have to remember I've got to click on that link once a month or so. Yeah, so be aware what modules there are. There are a few things that are packaged with Composer in the vendor directory in Drupal 8. There will be security advisers for it. There's already been one. So this didn't touch any Drupal code. It was just running Composer to upgrade in that case of develop library. But the security team isn't going to take responsibility for every external JavaScript library that you might be using. And of course you or somebody in your organization also needs to make sure that PHP, OpenSSL, your web server, your kernel, and all of those things are up to date. Yeah. Be aware also that if you're some old versions of PHP are unsupported by PHP officially, there's no security support for, I think it might be 5.3 now. I can't remember the exact end of life. But yeah, be aware of the end of life dates for the version of PHP you're using, especially if you're using, if you still have a Drupal 6 somewhere, for example, there's no version of PHP that is supported for that Drupal version that has security support. So again, you've got a problem. Okay, so what did they do? So the first step is that you take your site offline because you've got no idea what's going on if your server is currently sending out spam, for example. Take a security snapshot of your site, keep a copy of the site, the logs, the database. The first thing that I usually like to do is figure out the timestamp, figure out if your site was infiltrated because you will need to let your users know. So find that timestamp and then find everything that has been modified since then. Look for PHP scripts that might be doing strange things. Look for things that look like JPEG or PDF files but are actually virus payloads of some sort. Look at any nodes or users or comments that have been added since then. That's a common one. Look at your server logs but be aware, depending on how far they got into your site, that many of these things could have been changed by the attacker including the timestamp, the M time on files in your file system. So now you've made a copy and now you need to recover what you can. So how do we do that? So one thing you need to recover is user trust. You do this by announcing that you had a problem. You tell them what was leaked and what was the time. Does your site have credit cards? Were those leaked? Was it just their email addresses and their passwords? Many, many users are going to be using the same password on your site as a whole lot of other sites. This is a bad practice but people I mean, no one can memorize a thousand passwords and so people do this unless they're using some sort of fancy password manager. So you need to let them know that you need to reset accounts, like set the password to some string that can't be produced by the hashing function and tell your users to log in and click the I forgot my password button and reset their passwords. The session table by clearing the session table will effectively log out anyone who's currently logged in. So that means that if your attacker had an account and you reset the password that they can't still be active on your site you may have PCI or HIPAA or other compliance issues to report. Does anyone have responsible for a site that needs to respect either of those? I see a couple hands. If you had a problem, then again you need to disclose it to for example your credit card merchant account and you need to make sure that you've got passwords and API keys in settings.php that might need to be reset or else you have to consider them compromised. So that's user trust. The database. Hopefully you have hourly or at least nightly snapshot safety database that you can roll back before that time stamp. Remember I said figuring out the time stamp is one of the first things you want to do when you're doing an audit. If you don't have backups then this is going to be more difficult and you can't promise that you did it before. Oh by the way I should mention all these slides are already online so feel free to take photos or notes but I'll give you the link at the end as well. If you don't have backups so you can look for nodes and comments that were created after the time stamp of the infiltration and look for spam. You can make sure the PHP filter hopefully no one here is using the PHP filter. I won't make you put up your hands but make sure it's still turned off in Drupal 7. In Drupal 8 is an extra contrib module you have to download so it's easier. You need to look for PHP and JavaScript in the database. Things that might be running somewhere. For example it's a good idea to look for the short tag for the less than question mark not just less than question mark PHP. If that's in your database it probably shouldn't be in your database if it is you should know about it and check that code. You should look for references to script so that someone can look for things where someone is putting in JavaScript that for example will watch someone type a password and send it to somebody. There are also many other DOM events like on click that could be hooked so you need to look for those and you should also start taking backups if you haven't already been doing that. Just like people often the way you test your backup strategy is that you have a catastrophic failure and this is an example of a catastrophic failure. After something like this people usually start going through and making sure they've got better backups so don't forget that step or you'll have to do this all again and it wasn't fun so you don't want to. That's the database. You also have user files you need to recover so these are JPEGs, PDFs, whatever again if you've got nightly backups then this isn't too hard because you just roll back to the last known good state and then manually inspect anything that was added since then. If you don't have backups then look at the timestamp to find anything recent. It's also handy I find to use the Unix utility file which will just tell you what file type something is JPEG file might not be a .JPEG file. Look for things in the files directory be aware of where your web server is able to write. I'll talk more about this later but ideally your web server was configured so that it can only write to the files directory and not all over your PHP source code so look anywhere where it might be writing. Look for hidden directories so anything beginning with a . One thing I've seen a couple attackers do is create a directory called . because if you're just typing ls and you're not paying attention and looking for that kind of thing you might miss that. I missed it the first time and one of my colleagues had to point it out. And also again like I said you start taking backups. There are many reasons to take backups and this is just one of them. So your Drupal site of course is three things it's the user contributed files it's your database and it's your code base so now you need to check your code base hopefully you're using get or something equivalent so then again you just type getdiff and you figure out what's changed and then it's usually pretty obvious. Again these are usually script kitties and they're not trying to be subtle they're just trying to hit as many sites as they can as quickly as they can so check if you don't have backups then this is more interesting so I would what I've done in this case is downloaded a whole new copy of Drupal in another directory downloaded a whole new copy of every contrib module the same versions as whatever I'm using reapplied any patches that might have been there and just start from scratch so copy over the user files after checking them and other than that just build your site again if you don't already have some automated tool to do that the theme of course you wrote yourself you'll need to just check thoroughly every single file in that directory because otherwise you've got no idea what's in there also there's a module called hacked it's very useful the hacked module will produce a report of it will produce a report of any change to core or contrib so if you've got a project where you think a bunch of local modifications have been made to the views module it will produce a report and tell you these lines and this file have been changed however it currently reports files that have been changed and files that have been removed but it does not report any files that have been added so I find the hacked module is not enough in the situation that link there where it says it doesn't report new files it's a link to the relevant issue in the queue for that module if you've got time to work on that that could help but be aware of the limitation of this tool so again if you don't have backups something I would find useful will be to check for any feature overrides this is a little script I use in triple 7 in my shell all that will do is make a list it will print to the screen any feature that has been overrided is in the overridden state and it will create a file with the feature name .diff which will be a list of the changes to that feature so if there are change features you should be aware of that this might be useful just if you think one of your colleagues was being lazy and changing things in the database so they're committing it to get to but it will also help in the security audit in triple 8 the equivalent would be to look for changes in the configuration between the what's actually running on the site and what was last exported and saved and presumably uploaded somewhere with Git and so then start using Git or some equivalent tool the server so again I mentioned seeing a site that was infiltrated in order to mine bitcoins somebody got rude on the server I have no idea what they were doing I don't want to be responsible for going through the entire server and figuring out every possible backdoor that they could have put in place just a lot of them it was a virtual machine so I just threw it away and rebuilt it or my colleague did rather if you're using virtualization or containers this is much easier if not I mean you just reinstalled the whole server I did once have a colleague who an associate in another company that I was talking to who literally had somebody come into their data center and pull the machine out of the rack and then come back the next week and put it back so do you think they ever trusted that machine again? yeah it was like literally somebody in a black suit wearing sunglasses I don't know what three letter agency they were working for but they didn't know either and they never turned that machine on again yeah but most of you probably don't have this problem if you think you're likely to then you can put in a little spy cam in your data center in the top of the rack like they did those were some pretty funny pictures anyways to go back to more common problems yeah the PHP process on your web server where can it write it only needs to write in the temporary private and public user directories configured in your Drupal site so don't let it write anywhere else you can do this using an NFS amount you can do this using Unix making sure that directories are owned by root and not by the same process that the web server Apache or PHPFPM is running is really your web server should never be able to write index.php why would you ever want to do that so make sure that it's contained yeah and if you're running a shared hosting environment which I realize is not very common anymore it's possible to run PHP as different users every user is running in their own every process is running as the user associated with one specific client so they can't go screw up the sites of other clients so that one client site might be attacked in a bad state but at least the damage is contained yeah how could you have avoided this well these are mostly public security update public security advisories so you need to apply security updates you need to have regular backups like at least a rolling nightly backup as a bare minimum that's kept for like a week or two so that's so that will cover your backups of user contributed files and of the database your code base I'm assuming is already in git as I was saying you probably have a git ignore file that tells git not to commit files with various passwords in them like a settings.local.php or something else with api keys so whatever is in your .gitignore file you need to make sure you can recover in some other way however that is make sure it's something that you can get back you could also reduce your PCI compliance exposure by doing something like switching to a payment gateway that means that you don't have to take the credit card numbers or they don't even pass through your site it's just for example an iframe this will make your life easier and make other people less likely to want to break into your site because they're going to get less data from you yeah you can do that you can do that you can use virtual machines or containers when you can so you can tear them down and rebuild them when you run into problems there are many other advantages to moving to Docker or something like that but it will help you in the case of a security problem as well yeah so now I'll list some other resources I've noticed I'm talking pretty quickly by the way the security team amazing people amazing team of volunteers they have a helpful page in the documentation site which this is a link to called your Drupal site got hacked now what so this is a very good starting place with lists of other resources there's a module called security review which looks for basic configuration problems last I checked it was still on the screen so that will look for common configuration problems there's also a module called site audit which checks for a lot of other things like performance issues for example and it also does a few other very basic security checks there's an issue in a patch that's in progress to make the security audit module just go and run the checks in the security review module if you have both enabled which would be great and then this can for example on the pantheon status page they just show you the results of the site audit module just to make sure you're aware of basic things you need to fix on your site so if this patch lands that coverage would be greater which would be good there's a Drupal get in module which looked for which was made to help you secure your site that was a few years ago so I mean if you haven't fixed that yet it's probably too late but that module is still useful since there's a list of resources and a description of a process you can go through to recover a site which you might find applicable in other situations there's also something I only discovered about a year ago a very useful PCI compliance white paper that was produced by some members of the Drupal community so this describes all the things you need to do to your Drupal site if you're trying to make it PCI compliant the main takeaway I got from this is that there are many many levels of PCI compliance and the less you touch the user information and the credit card information the easier it is so there's these various levels and you want to stay at as simple a level as possible so this is a very thorough document that will explain what those different levels are to help you figure out how to get your site to a state where you could say that you're compliant PCI compliance is funny it's kind of like car insurance no one checks that you've got car insurance until you get into an accident or get pulled over for some other reason and PCI compliance normally works the same way they don't come and check until there's been a problem and then if they find out after the fact that you weren't doing your due diligence then you have issues there are companies that will go and proactively do an audit of your site and they will promise that it's PCI compliant to a given level these services are expensive because then you can go and sue them if they said it was good and it wasn't of course that costs a lot of money so yeah try to try to reduce your vulnerability as much as possible like I don't know if your client is okay with having their website the e-commerce be on Shopify or something it's one less headache for you great yeah I'll just take questions all at the end and I'm going to move on now to previous vendor recovery has anyone had to do this yeah about a third of you that's a yeah this is this is a lot of fun previous vendors some of them are great I'm not going to name any of the guilty parties here not even going to say whether they're at this con but yeah this is this this can be interesting you see one thing that's a little fishy and you get that bad feeling and then you've got to go start looking at everything else so what do you do when you've got a site and it's in some unknown state and you've got no idea who worked on it or who it was outsourced to here I've used the hacked module to get a list of every patch that was applied the patches on your site should be well documented somewhere there was a you can do that in for example if you're using a composer based process there's a way to explicitly list every patch you can have it just a directory called patches where you throw in a copy of every patch you've got on your site some and then in the history you can give your reasons for applying it with a link to like the issue up on triple.org if you wrote the patch and no one has ever needed to fix that before then you should put it on Drupal.org to help other people and so that this will eventually get folded into the next stable release of your module because then you'll never have to patch it again and somebody else will go check your patch and make sure that you wrote it right yeah so you but if you suspect that somebody wasn't already documenting their patches then you need to make a list of those and so the hacked module can help you with that yeah to collect all the patches document them somewhere whatever works for your organization patch files there's a specific naming system naming scheme that includes among other things the name of the module very short like three or four word description of what the fix is the node ID of the issue on Drupal.org the number of the comment because usually there's like a whole series of attempts at a patch before one is accepted so this will tell you which one it is so if the patch wasn't named that way I would just go and rename it so that I can figure out later like which module it's for or where it is an issue queue if there's a patch and looks legitimate I mean you should always just post it upstream for other people because if only because this is going to say you work in the future then you go look back and realize you submitted it and somebody else has since committed it and you say thanks me of six months ago you saved me a lot of work today so that's contrib and core of course this person has probably written some custom code minimum in the theme so then you need to just sit down and read every single one of their modules figure out what they were trying to do whether that was done in a reasonable way in my experience and people when I'm working with when I'm doing an audit of a site by a third party built by a third party who didn't seem to know what they were doing usually that takes the form of somebody not really knowing the Drupal API and not doing things in the right Drupal way like they don't know that there's already an API in core for that or that they could have just downloaded some contrib module instead of rewriting that functionality in a crappy way so often I get to just delete modules and use something from contrib or an API function call but you need to read all of their code in Drupal 7 features the .module file you can put in custom code there I never do that but I realize I have to look there it doesn't always occur to me that other people might do that I don't know if that's a good practice or not I'm neutral on that but other people might be putting it there so look in those files too don't just assume that everything was generated by the features module you should check for features overrides of course as I was mentioning earlier if this isn't already the case I would strongly recommend that you organize the modules so you know which ones are custom which ones are contrib in Drupal 7 usually I do this with separate directories under modules or site saw modules yeah in Drupal 7 there was this weird anti-feature that if you disable a module but you don't uninstall it the configuration sticks around and sometimes certain hooks in the module might run you probably weren't expecting that so go make a list of those modules that are disabled because somebody just turn them on and then turn them off because they might be having side effects too so if you're not using it and you're pretty sure you're never going to use it just like uninstall it and get rid of the module so that there will be less code for someone else to read the next time someone else is doing an audit of the site an analogous problem in Drupal 8 is that somebody might not know that they need to run Composer install that was in the vendor directory managed by Composer and probably not checked in to get depending on your practices so check that those are up to date as well as I mentioned before there's been I think just one so far security advisory for Drupal 8 which just involved a version of a library that was in the vendor directory so if you hadn't run Composer install your site is still vulnerable so typical mistakes often somebody who just doesn't know anything about a site that was just built by somebody who doesn't know Drupal very well they will just try to rebuild things in the theme and template layer I'm thinking here of Drupal 7 where that was just PHP and so they could write in any logic they want they don't know they'll just try to insert code in places where it probably shouldn't have been done I'll give examples in a minute so in particular look at all the templates for logic that shouldn't be in the templates you shouldn't be programming in your templates you should be doing that somewhere else which is why in Drupal 8 the templates aren't even PHP anymore all you get is some very simple if statements which should be enough for you PHP module people love to use this so you have a node that's just full of you go to edit a node you can use this code doing random things I mean that text area in your web browser is the worst possible PHP editor you could ever use if you make one little syntax area your site is a white screen and you need to go fix that in the database this is a terrible practice but people do it if they because it's a shortcut and it's the fastest way to get something on the page and they're not thinking of maintainability I guess because it's my job now PHP is a very powerful module that just lets you take a view and fiddle with it before it's displayed to the user so people often put too much logic there too there's a powerful but overused module called computed field which just lets you run PHP code before a given field is rendered there are lots of reasons I might want to do that but again it tends to get overused all of your PHP should be somewhere in the file system it's like it's crazy to put PHP in the database it drives me, not some people do that you should have it in the file system because then you get to do revisioning you get to have a get history you get to use the opcode cache for performance you get to have useful warnings and errors that actually say the file name if something goes wrong none of these things are true if your code is actually right there in the database there are ways to do this, for example computed field module maintainers are aware of this so you can just write a function with that magic name and that will run instead and you don't need to type in into a little text area in your web browser the PHP that you want to run the only you can do block visibility you can control that in a custom module for example block hooks in Drupal 7 the only reason I personally know of in Drupal 7 when you have to put PHP directly into the database is for custom panel visibility because there's no way to do that if you need to run custom logic you just have to type in some PHP sometimes but in that case that PHP snippet should consist of nothing but one function call and that function should be defined in a custom module somewhere and then you get to use PHP Storm or VI or whatever your editor is you get to have versioning and get and you get to have the opcode cache yeah and then things fall apart and then you get into these classic problems which I'll give examples of that I have actually seen on sites so custom patches to contribute or even to core that were never posted upstream but once I found them in a vendor's own GitHub repo which fortunately was public for some reason so yeah take those changes put them on Drupal.org then other smart people will look at them and give feedback maybe roll them into a future release if they're useful I have seen somebody once hack core so that they could have multiple install profiles and then they're inheriting from them I mean I guess you might want to do that except that Drupal core doesn't support this so they had to write a lot of weird patches to Drupal core to make this possible and they didn't quite do it right so it was it wasn't a good idea but they meant that the install profiles the site was loading multiple install profiles each of which was containing their own custom modules so there were copies of modules in like three different places and it wasn't clear which one was taking precedence because each of them had different patches applied to them that was really messy there's a module yeah UUID features, does anyone use this module or heard of it one person yeah UUID features do you UUID features sounds like a great idea when you first see it it was a way in Drupal 7 to put the content of a page for example your about page to save that in a feature because really that's basically configuration I mean your content editors aren't changing it much it's perfectly reasonable to want to have that in the file system the problem is that that breaks various basic assumptions in Drupal 8 core for example that there is a UUID for everything and so when you install the UUID features modules it comes with a bunch of little patches that you have to apply to core or else it won't work and so that way lies madness if you need to stage content like for example you want to have a brochure site and you want to define the whole thing in code so you can distribute copies of it without a database which is a great idea if you've got a simple brochure site you should be using another method for staging content such as the deploy module you could use the migrate module and just define everything in like a csv file and load that in when you set up the site there are other ways to do it so please don't do that I've seen sites where all the custom logic was implemented in views PHP or in theme functions as I mentioned before or in the templates I mean if you're looking at a Drupal site and you just know nothing about Drupal you can change this one thing that's showing up on this one page the template file is the most obvious intuitive way to do that so people put way too much logic there to the extent that I saw a site once where somebody built sort of like a lightweight version of views in SQL queries directly in the theme layer that makes as little sense as it sounds and while they were at it they re-implemented the T function in the theme layer yeah that site was a mess and that same site this is something I've seen on other projects but that same site had references to specific node IDs, user IDs, term IDs and even the title of a page and so there'd be branching and if statements in the templates based on the title of a page so my client changed the title and the site stopped working this should not be something that happens on your site you should it should be very rare that you actually need to hard code a reference to a node ID maybe occasionally in a CSS file but really you're probably doing something wrong or you should probably rethink your approach if something only works for on lucky node ID 7 yeah and when you see these problems whenever I've seen a site with these problems it means that nobody's updated the contrib modules in years and they've got lots of vulnerabilities too which then brings you to the intrusion recovery part that I was talking about before yeah finally I'll mention that for anyone who's not working in-house for one company but is working for an agency doing sites for other people you have some client expectation management you need to let them know that even though they thought the site was fine last week it wasn't a technique I've found is often useful is to bid on doing an audit of the site before you sign something saying that you'll fix it tell them just give me a little bit of money and I'll tell you what's wrong with it and after that I'll bid on fixing it and other people can bid on fixing it but it will reduce your exposure so you're not promising to fix something before you even know how broken it is and you can offer to do that at a reduced hourly rate as an incentive you can this means that your client is probably well they're bad at choosing a vendor at least they were before they came to you they're probably bad at reviewing deliverables you're probably going to have to spend extra time on the phone with them so budget for that sometimes it's a usually people are more convinced of things they figured out themselves than that someone else told them that's just basic human nature so if your client has already figured out that there's a problem with their site then be cautious and if you can't convince them that there's a serious problem and let someone else take this problem yeah and be better prepared so take time to audit a new project that you're taking on before you promise that you can deliver something by next Tuesday my company for example is happy to do audits as I mentioned at a reduced rate and I think this is a fine way to make sure you're not over promising and you can also offer to send people on training courses to something that my company also offers and as do several of the people who are here at a triple con courses coming up in Washington Atlanta and Ottawa and online as well yeah so now we have about 15 so minutes for questions and as I say here I'm also happy to hear your humorous anecdotes or war stories or cautionary tales or other things you might think might be instructive for this group if you have a question please come to the mic so that it can be recorded however these slides you can find at this link here also if you go to this session description on the conference website there's a link right from there to these slides as well so you can share them with your coworkers or go look at the things I click and so read the parts I skimmed over quickly or you could also just contact me directly and here's how to get me thanks and I'll start with you Hi I was wondering what Drupal Geddon was okay Drupal Geddon yeah Drupal Geddon is the worst security vulnerability I've personally seen in Drupal since I started using it it was this exploit that was remotely you could exploit it remotely you could do SQL injection remotely so you basically got to run any process login as any user run any PHP process you want on the server and change anything you want in the database all without having an account or anything on the site it was pretty bad and it got leaked and there had to be a security advisory early and it was just a great big mess but the security team did all the disclosure they could about that they they did their job as best they could under the circumstances so just watch what they say that vulnerability by the way I'll just mention quickly was found by an unnamed client who hired someone an outside agency to do a thorough security review of their Drupal site before they launched it they just thought that someone else should take a look at Drupal and they found something that no one had discovered that had been in Drupal for many years so thanks to those people for doing an audit and the rest of us benefited from that yes I work in government contracting which it might as well just be called legacy code so as an aspect of that a lot of what we're dealing with is that the life cycles of these products is anywhere from like 3 to 7 years is insane for a web system so a lot of times we are like living with code that we know is bad but ultimately we only have so much budget we only have so much time when you're dealing with that largely we do make sure that the critical security updates are there and cross scripting isn't possible but at a certain point what would you say is the most important things to keep from to fix as you're going along because they're not necessarily going to allow you to fix that one go are you saying you can't upgrade core or you can't even upgrade contrib I mentioned more referring to like you would inherit a project and they'll have a custom module but it has features that you can't afford to rebuild so how do you prioritize which ones you're going to fix first that's a hard question I realized I was curious yeah so I haven't yeah I had to do that I mean I guess I would start with the things I think are most likely to make the site blow up for example security issues like blatant security issues I mean there are there are three I think agencies that are still offering long term support for Drupal 6 like your site is that old they don't have many clients left anymore but there's a few there are security agencies that put a place a proxy kind of like how you can have a CDN like Cloudflare like they'll put in a layer that checks for basic SQL injection and things like that so maybe a service like that would help you if you know you just can't upgrade anything making sure you know who the people are who have accounts on the site and that it's a small number and that you trust them and it's not someone who left angry last year I guess I would start with things like that okay yeah I was just curious we largely we find that even in the government space like the security aspects is not the problem it's more just as I've been calling it time capsule code which is you can look at it and say oh this is how we wrote PHP five years ago and we can't really fix it but we can make sure that at least it's not as far as somebody can destroy our site but I mean at a certain point what is an acceptable level of bad I guess is what I'm trying to get at that sounds like a decision for whoever writes the checks so I guess the best you can do would be to present an audit and say these things here's the things I think are important these ones will take a week to fix this one would take three months to fix if you don't fix them then these things will happen and say I told you so later yeah thank you sorry my question was a little circuitous but I wanted to bring it out there if you don't have the budget then there's only so much you can do nothing's going to change that thank you thank you for a thorough review of security of hardening Drupal I want to encourage everyone in the audience to subscribe to the security list that Drupal has they do a great job they really do so I'm in higher ed and WordPress doesn't offer anything quite as covered you have to go to a third party vendor to get something similar yeah the WordPress people will only tell you about like the core of WordPress there's nothing equivalent for like all the contrib modules sorry plugins I can't even remember what they called them that's a whole different Bailey work the other thing is when we're like Drupal again what was interesting was the user table so it may sound tedious to a site builder but you want to spend a little time and if you find non contiguous IDs littering your user table that's a good indicator of an infection one last thing I would add to your presentation is to try to offload any services for example what form might be better served we use Qualtrics right and you can reduce your vector for attack by offloading you can do all the stuff with PHP you can make Drupal home base for all these things but get it back to just serving pages and try to offload a third parties such and also authentication we use Shibboleth and everything has to go through the user table so yeah thank you thank you so some of us work internally for a company that you're basically the marketing website or whatever so you'll get ideas that come across and it's you know we're actually going to A-B test this and you can pretty much tell from the get go it's nothing that's ever going to be long lasting or ever be reproduced ever again other than this one instance because it's generally a bad idea to start with but somebody got it in their head so you're doing it so there's often times things where you know okay I could insert some PHP into a theme file or you know if you're on D7 or I could do some direct PHP injection into the database and through you know content and accomplish this in 20 minutes to a half hour I could build a custom module to do this one thing it's going to take me a week and then it's going to go away how do you balance when to do the right way versus when to do it the 20 minutes way so that it can just be done and then go away again so right so you want to do just a quick prototype and just show it to your boss and then so you realize it's a dumb idea and then you know you throw it on like a you throw it up publicly for a short amount of time and then they go yeah that didn't do what we were expecting to make it go away and then they but if they like the prototype then they expect you to just maintain that for the next five years and reproduce it 37 times right in 20 minutes each time yeah yeah yeah and if somebody else comes in and looks at you they go you did this wrong and you did this wrong and you look like the bad past vendor and you're like yeah yeah yeah that's a good question I guess you have to hope whoever your manager is understands the difference between a prototype and something that's maintainable yeah so try to explain explain that clearly but yeah I mean if they want to give you an answer then you're only going to be able to support it so well hey do you want to just come to the mic so everyone can hear you haha if you're reporting to 44 different people then you might have some other problems yeah yeah yes yeah thanks hi I just wondering what's the timeline it takes you to all the other side let's see I mean if I start looking if I can see right away that there's nothing weird I mean yeah I've done it in as little as a day if the site was very well built and I could see that there's nothing strange another time it took me like three weeks because the site made no sense and so I had to go read everything very carefully and figure out what on earth they were trying to do so it depends how broken it is I guess that's the short answer but I mean the time it took me three weeks like after one day I could already tell it would take me through after a day I knew that it was going to take me three weeks right so yeah I'd say like in half a day to a day well in a day I could probably have a general sense of at least how long the audit would take if core has been hacked uh well I mean as soon as I as soon as you see that there's been an infiltration then you stop and you deal with the infiltration like when I said when I just mentioned three weeks like that was in a case where there was like a previous vendor who had just made a total mess of the thing and they needed to go read everything they did but if there was yeah if I see if someone had made some change to core that was like that was done by an attacker then you immediately stop and you roll back to the last known good version oh oh oh like like weird patches like not an attacker um yeah I mean if if everything else on the site looked in one day like that would be enough time for me to say that everything else on the site made sense except there's this one weird patch in core and if it's like a short patch then like it doesn't take you long to figure out what it's doing and hopefully do it in a better way or if it's a great patch and they just never published it to publish it um yeah so yeah it depends how much code there is and how much it's been changed I guess yeah anything else? yeah do you have objective tools that you provide for like performance in your audits uh is it just typically like hey test my site with Google and you know you get a number or do you have some sort of standardized reporting to say um your site is bad or your site is good from a performance perspective yeah um yeah performance audit that's a whole other thing I wasn't uh talking about that at all here there are tools that will do performance audits for example there's a company I think it's got a booth here called Blackfire that does performance audits and we'll tell you you know this code like these lines in this file are being run many many times maybe you should go look at that but um I have an idea of tools and vendors who will to help with with that um but uh yeah that wasn't the kind of audit I was I was uh I was talking about in the stock okay uh any other questions no great well thanks everyone yeah I'll take uh that's actually a flyer for training um yeah um yeah um yeah um um um um uh um um um um um um um um um um