 Thanks for being on time. I appreciate that. This is the how to not get hacked talk, a security checklist for Drupal server administrators. My name is Mike Richardson. I am the co-founder and manager and director of Ionstar, which is a wholesale Drupal hosting provider. We take care of Drupal, Laravel, and Node.js websites and help accelerate and secure them for our customers. And if you've ever seen me talk before, you would know that I'm also the co-creator of Tokaido, a local development environment manager for Drupal. That is certainly a project that I'm very passionate about. And if you have any questions about Tokaido, then feel free to come and ask me. Whoops, I've scanned ahead. So I have been in hosting for about 20 years, which is a very long time to have one career. In that time, I've had the pleasure and misfortune of working on and protecting some very large corporate and government sites from some very targeted and also very unsophisticated, not very targeted attacks. The highlight for me in that career, who knows what this tool is? I mean, obviously, the name's on it, the low-orbit Ion Cannon, but is anyone familiar with this tool? It goes back about eight years. Oh. So this is the tool of choice in about 2011, 2012 of the Anonymous Hacker Group, who at the time were targeting a federal government department website that I was custodian for. And as cool as the low-orbit Ion Cannon name sounds, if you're on the receiving end of it, it is very unpleasant. It generates TCP flood traffic using a home computer or a desktop computer on a residential internet connection. Anonymous had tens of thousands of people load up this tool with no, you didn't need to know anything about how this would work. You didn't need to know anything about how a computer worked, but you could join the flood of people that were targeting this particular government website. And we were on the receiving end of it, up to about 40 gigabits per second of traffic for a relatively not very popular government website, not very well-trafficked government website. And I'm happy to say that we managed to survive that attack relatively unscathed. This is obviously a distributed denial-of-service attack. Thousands, tens of thousands of actors all getting involved in trying to overload a web server with a particular type of traffic. There are of course other common types of attacks. The remote code execution vulnerability that if you're a Drupal developer, you are probably very intimately familiar with, but maybe not, hold in, maybe not for the reasons that you think. There's obviously also the more common everyday things that people talk about, cross-site scripting, SQL injection, phishing, session hijacking. I've been involved in, not involved in performing, but receiving these different types of attacks. And today we're going to talk specifically about remote code execution and we're going to talk about the things that Drupal sites are vulnerable to and how you can help protect yourself and how you can help identify if the people you're paying to do your hosting, if it's not yourself, are protecting you. I am, however, not a security expert. I consider it to be a part of my job, but it's certainly not what I do. And I want to throw in the disclaimer that your requirements are unique and you're only going to be as secure as your least secure component. Some of the stuff that I'm going to show you here is very powerful, some of it's easier to deploy, some of it's a little bit more difficult to deploy. None of it is going to help you if your username is admin and your password is admin. So you need to take all of this in holistically. We're also deliberately skipping some basics because things like cross-site scripting and SQL injection are very well covered and there's a lot of really good literature and a lot of really good material out there if you want to have a look at that. So that's who I am. Now, if you're going to get the most out of this talk, I hope, you are the administrator or a backend developer for one or more Drupal sites. You're either hosting these yourselves or you're paying someone to host them for you. And as I mentioned before, you want to be able to double check their work. You want to be able to see if they're taking the sort of everyday ordinary and common and best practices steps to protect you and you're in the sort of beginner to intermediate range. I know some of you in the room, I know that you're not in the beginner to intermediate range and you're probably not gonna have that much fun with this. However, I also want to clarify that I think Drupal is secure and this talk is in no way meant to suggest otherwise. We are all familiar with Drupalgedon and Drupalgedon 2 and the remote code execution vulnerabilities that Drupal has been a victim of. And even people I know who don't have anything to do with Drupal have come to me when those events have occurred and said, hey, I've heard you're having a pretty crap day. I think that that perception is very, very unfair. If you Google remote code execution vulnerability and you put Drupal on the end of it, you're gonna find things, things that we're all gonna be pretty familiar with. If you put Sitecore on the end of it, you're gonna find a very similar material. If you put Adobe Experience Manager on the end of it, again, you're gonna find that these remote code execution vulnerabilities exist in pretty much every CMS platform in any platform of any kind. They're very scary vulnerabilities, but they are by no means unique to Drupal. They are by no means overrepresented in Drupal. They get more press because the Drupal security team, I think go to a lot of effort to make sure that everyone knows about it before the vulnerability itself is announced. Whereas on AAM, for example, you can have a minor update that doesn't receive a lot of press, that doesn't receive a lot of attention that fixes something as nasty as this and that's publicly available, but you don't have every AAM server administrator going, I definitely have to patch today, because I've got people who don't even have anything to do with AAM contacting me saying, I've heard you're having a shit day. So, I think Drupal is secure. I don't believe that this talk should suggest otherwise. And this talk is also about applying a defense in depth strategy. Who here has heard that term before? Defense in depth. For those who didn't raise your hands, this is about having multiple layers of protection. Anything that you deploy to protect your site, to protect your infrastructure, can fail or can become vulnerable or can be misconfigured. And you definitely don't wanna be relying on just one layer of defense. You don't wanna just have a firewall that protects against TCP traffic for your website without having something else that will protect you if for some reason somebody opens that firewall up. So, we'll be talking about some of these today. But first, I thought we might hack something. I thought we might show the remote code execution vulnerability and actually tempt fate and do a live demo about hacking a Drupal site. I wonder, is there any requests? Anything anybody would like? I, of course, have a prepared environment. This is Drupal 8.4.5 running in my local Tokaido environment, which was easy to set up and easy to use, as you might have thought I'd mentioned. So, I have this environment here. If I run TokPS, you can see all of my containers. So, this is running locally. This is a vulnerability that is at least 18 months old now. This is from Easter last year. So, if you have not patched since Easter last year, this should hopefully scare the crap out of you if you weren't already terrified. I'm gonna hack something, but I don't think, I don't think I could find, no, I could find a Drupal site that still isn't patched. Hey? Any Department of Defense in any country? That's great. Are we gonna go halves in a trees in charge? No? So, I have a script here, exploit.sh, that is simply gonna echo hello to success.txt. It's pretty innocuous. And this is on my Mac. It's local here. And I have two scripts that are going to in, whoops, going to do two very simple things. One is going to get my web server to download that script from an S3 bucket onto itself. And you can see that the command in question is this one here. And then the next one I've got is going to run that exploit. And again, after it's downloaded that file, it's just gonna run it. And that's it. Now, you might think that something like this is difficult to work out how to do. I had not done this previously before I started preparing for this talk. I am, again, not a security expert. It took me all of about 20 minutes to find working demonstration code. That is not my code. I have not written any of that except for maybe the exploit.sh, because that is the scale of my bash scripting capabilities echoing to files. And I am going to show you my FPM container. And you will see there's nothing in my slash TMP that shouldn't be there. If I run that upload exploit.sh, that is going to use the vulnerability in question. And then suddenly, I have that exploit.sh in my temp file system. If I run the runexploit.sh, then I have the success.txt. That was scary, easy to do. When I have been brought in to help clean up a Drupal site that has been hacked, not hosted by us and not hosted by anyone who I think is here, but when I have been brought in to help clean up a Drupal site that has been hacked, almost always this particular vulnerability has been used to deploy a Bitcoin miner or some other cryptocurrency miner, something irritating, something frustrating. And thankfully, you notice it because your web server that suddenly is using 100% CPU and it's impacting your page performance. And that's the only reason that you would know that your web server was hacked. So if someone chose to use that for some sort of nefarious reason, it is very, very easy to not have any idea that it's happened and to not have any idea that it's happened for months and months or perhaps even still. If you were slow to patch Drupal 8.45 and you're still running the same web server, you may still be hacked and you won't know. That's all probably gonna sound like a lot of fear mongering. It's not meant to. It is meant to clarify that remote code execution vulnerabilities are particularly scary. And if there's one thing you take away from this talk today, patch regularly and on time. I was involved in projects when Drupal Get-N-2 came out that had not been patched for months on end. Suddenly we had to apply six months or 12 months worth of patches to a project. That was painful. And we had to do that under the cloud of if we don't patch soon, these sites are out and alive and they're gonna get hacked. So what some of the stuff we can do to try and mitigate this stuff, that particular vulnerability is hard to stop. The easiest way is to make sure that you're patched. The next easiest way is to deploy a content delivery network. Who is familiar with what a content delivery network is? Cool, I like the distribution in this room. If you're not familiar with a content delivery network, it is as much a useful security tool as it is a useful performance enhancing tool. So if you imagine that I've got customers all over the world, Melbourne, Sydney, Brisbane, New York, London, everywhere in the world that matters, and I want to have my website available to those users. A Caching, a CDN service will trick web servers, sorry, trick visitors all over the world. When they go to my domain, it will return to them the address of a local Caching server. When they request the image or PDF or web page or whatever they're getting from me, that's going to say, right, well, I have a local copy of that, so here you go. It won't have to go all the way from New York back to Melbourne where my server is to collect the content and return it, but if it does because it doesn't have the cached copy, then it's a very simple case. You get that content, keep a copy for itself, give a copy to the visitor, and away you go. So it's a distributed Caching network. As I mentioned, it's primarily the attractiveness of a CDN is in improving page performance. If you're delivering heavy assets from a position, from a Caching server that's geographically close to the user, then they are going to have the benefit of getting that sooner. You're going to have the benefit of 80, 90, 95% of your traffic being served by Caching nodes and not by your web server. So your hosting bill, as much as I hate the idea of a cheaper hosting bill, your hosting bill is going to be less. From a security perspective, the CDN hides your web server. If you configure it correctly, your web server address becomes unknown to the general public, and as a result, it becomes very difficult for anyone who wants to attack you to do so directly. They must go through the CDN, and the CDN will inherently provide protection for things like network-based attacks. The low-orbit ion cannon cool-sounding tool that sucks when you're on the receiving end of it is a network-based attack tool. And if you have a CDN in front of it, your chances of surviving even the largest 40 gigabit per second attack, as we did, you can survive that, which is how we survived the 40 gigabit per second attack that I mentioned. Something that's newer about CDNs that didn't exist back in 2011 when we were dealing with that is that they're now relatively cheap, if not free. If you're not running a CDN and you are running a Drupal website and you want that to be more secure, more resilient, and load faster for your visitors, it has never been a better time to deploy a CDN. I don't sell CDNs. I'm not getting paid to mention these providers. However, if you want to go away and have a look at this stuff, AWS, Google Cloud, Azure all have their own, mostly turnkey solutions for deploying a CDN where you just pay per gig for what you send. So if you've got a low-traffic site that's doing 100 gig, 150 gig per month, it's going to cost you less than 10 bucks to run that thing every month, and it's going to give you all of that benefit. Fastly are another CDN provider who also charge per gig, and I mentioned them specifically because they have an outside of Akamai. They're the only CDN provider I know that has a New Zealand caching node. So if you've got customers in New Zealand, that may be a very attractive choice. Akamai are the best-in-class CDN provider. They charge accordingly, but Microsoft.com, Apple.com, some of the biggest websites you've ever been to are all protected by Akamai. So if you're in enterprise or government, you could be looking at that. And Cloudflare is the easiest CDN to deploy. It is completely and totally free for the general CDN service. There's no bandwidth charges on the free plan. There's no reason not to have at least Cloudflare deployed to protect your site. So I would highly recommend having a look at that. The next thing you can look at is a web application firewall. I promise I'm not going to do this for every point, but who here is familiar with the web application firewall? Cool. About everyone who knew what a CDN was put up their hand for a web application firewall. So I will go through this relatively quickly. A web application firewall takes the CDN to the next level and scans the requests that are going through it, the application requests, for known malicious traffic signatures and filters them out if it detects them. Within a few hours of the Drupalgedan 2 exploit becoming genuinely available, Cloudflare, Fastly, and a few other CDN providers had either made available or automatically installed rules for their web application firewalls that would detect that exploit and filter it out. Some hosting providers were able to do that as well. And what that effectively means is that by using that WAF, if you didn't get to patch your site in time, you still would have enjoyed the protection that that WAF afforded. Unfortunately, web application firewalls aren't as cheap and aren't as ubiquitous as CDNs. So this is one of those things that you may or may not deploy depending on your budget, your commercial requirements, your time frame, your technical skill level. However, all of those providers that I mentioned that offer a CDN, they all pretty much offer a web application firewall as well. Akamai, Cloudflare, by far and wide, the leading providers in this space. Cloudflare, US $200 a month and you can have a web application firewall with built-in Drupal protection and you can also modify your own rules if you've got, perhaps you're under some sort of particular attack that's unique to your application. The next thing you can look at is, and this is a good one, if you're paying someone to do your hosting for you, are they using the same user for Apache or Nginx and PHP? This is, when I've been brought in to have a look at a Drupal environment that has been hacked, these services are either running as root, which you should never do. There is no reason why it should be the case or they're running as the same user or they're running with similar or the same groups. Your web server is untrusted. Your web server is publicly exposed to the internet. It is listening to everything and anybody with a grudge or anybody playing with a tool or anybody who hates you is going to be able to get to your web server and there's not a lot you can do about it. So you want to trust it as much as you trust them. If you run your web server as the same user or group as you run PHP, your web server inherently is going to have access to do things like write to private files. I had a slide and I've lost it. So you want to have a look at that. You want to make sure that your PHP server, FPM generally, needs to be able to write to your private files and your public files folder. It doesn't need to write to everything else but your web server doesn't need to write to anything. And that's why you want to make sure you're running those as separate users and you want to make sure that you don't trust Apache or Nginx in any special way. If you're running in containerized environments, Lagoon, Tokaido, whatever, you can look at running those as different containers with different volume mounts so your web server can't even mount your private files folder and that's defense in depth. Even if your web server happens to have permissions to private files, if it's not mounted, you're at no risk. Only give right access when it's essential. This is a slide I thought I had. So this is, again, this is like cross-site scripting and SQL injection. There's so much literature here. I'm not going to bore you by going into too much detail into this. I have other and more exciting ways to bore you but you want to make sure that your PHP server can only write to the private and public files directories. You want to make sure that your web server can only read from the public files and the web route. Yeah, Google Drupal file permissions and it's relatively straightforward. But you definitely want to audit that and something that I think that is often overlooked is people will set up permissions when they build the web server. They'll know that that's safe. They'll do the hard work and then they'll go away and then over a year or two years these systems have entropy, different developers and system administrators come in, they decay, they age and then eventually someone introduces a vulnerability. What we do and what a lot of other providers do is that on each deploy, they reset file permissions over everything. We do that on every deploy. We also do that on a timer and we also have a system that monitors if those systems change to something that's insecure because yeah, these systems do have entropy. The next thing you want to have a look at is don't use HD access. Who likes HD access? Okay, that's, I wasn't expecting that. I wasn't expecting that. No, okay. All right, so it's okay to like HD access. If you don't know, HD access makes it really easy to reconfigure your web server on a directory level. If you use Apache and you say, I want this directory to be able to do certain things, I want to have sim links in this directory. You shouldn't, but if you did, you could do so with a HD access file without having to change your web server configuration. The fear in that is that you create decentralized security that is probably unordered. It is probably not going to be detected. And again, entropy. You may have a developer who comes on board in two years' time and says, I want to do that. I want to have a sim link in that directory. So I'm going to have a HD access file and I'm going to change my security configuration that server administrator's never going to know about that and never going to be aware that that vulnerability exists. And there it is, you've compromised yourself. There's also a performance reason to not have HD access files. Even the Apache Foundation suggests, don't use HD access files unless you absolutely have to and the only reason you would ever absolutely have to is if you're in a shared hosting environment. Hosting is so cheap now that you shouldn't be in a shared hosting environment anyway if you've got something that you want to protect. There's a whole variety of reasons why you can't protect shared hosting. Immutable infrastructure. I promise I wouldn't ask everybody if they know what these terms are when I say them. However, who's familiar with immutable infrastructure? Less of you, fantastic. Okay, immutable infrastructure is the idea that every time you do a deploy, you replace your stack. This is something that we do and it's something that I absolutely love doing. If you get brought into an environment where a site has been hacked, one of the first questions you're going to be asked is, is it gone? Am I clear? Now if someone breaks into your house, your first question's going to be, are they gone? Can I go inside my house? Is it safe in my own house? What if you couldn't answer that question? What if you just don't know? Personal anecdote, I worked with a gentleman very early in my career who was completely blind. He did not have retinas. He worked through a screen reader that would read out his Linux terminal at an incredible pace. Think of an auctioneer on speed. It was so fast that you couldn't understand it unless you'd trained to understand it. We had a server that had been hacked. The hacker had placed what they were trying to hide in a directory called dot dot. If you run Linux or Unix or Mac OS, you'll know that when you do an LS, you'll see dot dot dot means current directory, dot dot means the directory above this directory, and then you'll have all your files. But if you very carefully escape the dot dot, you can have a file called dot dot. That doesn't mean the directory above. It just is a file called dot dot. We all looked for this thing and couldn't see it because when we did LSLSA, we'd see dot dot dot dot files. And we'd go, that looks normal. I didn't notice anything unusual here. He heard it, so he was able to see this. Once, my point in that interesting anecdote, well, I thought it was interesting, my point in all of that is that you have no way of saying I haven't still been compromised. If you ran a Drupal 8.4.5 site and you got hacked, or you don't know if you got hacked, but you didn't patch right away, there is nothing you can do to say I'm safe. All you can do is start again. And that's why with a mutable infrastructure, every time that you deploy, you replace your infrastructure. Like most of these things, you have the benefit of security. You also have the benefit of performance and practice. If you use a mutable infrastructure and every time you deploy, you get a new stack, or you can deploy stage in the same way. You can deploy dev environments the same way. You can deploy your local environment the same way. So it's incredibly, incredibly powerful. It is harder to do if you've got a commercial Drupal website with a team and a budget it's easy to do. So what's the easiest way to do something like a mutable infrastructure? Docker containers. If you want to go out and have a look at, how am I going to build a mutable infrastructure on Azure, on AWS, using virtual machines, there is not a lot there that exists. Thankfully, on container-based environments, Lagoon, Takaito, you can go and look at the container specs for DDEV or Lando. There is so much there that's open source, whether it wants to be or not, that is easy to inspect. There's a lot of really great examples to borrow from. I will confess on this stage to having borrowed from certain other platforms to enhance our platform. But containers are ephemeral by nature. Every time you redeploy a container, you have a fresh environment. You can't save something to a container permanently unless it's going to a mounted disk. So Docker is the shortcut to a mutable infrastructure and well and truly worth having a look at. Getting towards the end here, I have nine of these and I'm ahead of schedule, so I hope you guys have questions. The next thing I wanted to emphasize, and again, when I've come in to have a look at Drupal environments that have been hacked, a lot of these slides are about things that could have been done to prevent that. And this one is encrypt everything. There is no good reason anymore to not run a CDN. There is no good reason anymore to not have encryption in your application. If you're on cloud, encryption is literally a tick box. It doesn't change the price. It doesn't change the time to deploy. It doesn't change the performance. You can just tick a box and say, I want that volume encrypted, and it will be. It's that simple. What is less simple is getting your database connections encrypted. If someone's able to inject code into your server and keep that running and hidden from you and your database traffic isn't encrypted, they can inspect it. They can relay it off to their own server. They can have it, and they can keep it. And that scares the crap out of me, and I hope it scares the crap out of you. Unfortunately, encryption for database connections and Redis and Solar and all the rest of it is uncommon. Most of the major providers still don't offer it. If you're paying someone to host your website, pressure them. We need to change that as a community, as an industry. If you're running your own stuff, check it out. It is not as easy as encrypting disks on cloud. However, it is not insurmountable either. The next thing is it's more advanced. It's something that we've started to do, and I find really exciting. And I wanted to throw it out there. And that is using file system mounts that prohibit execution and removing things like the PHP CLI and bash from the containers that run PHP, which I'm still not sure is actually going to be completely and totally feasible. However, the remote code execution vulnerability that we looked at injected code and executed it. What if that can't be executed? When you mount a file system, I'll fast forward one. If you use system D to mount a file system, you can mount that file system or that path within a file system using the no dev, no exec, and no SUID bits, which the no exec one most especially will prevent files in there from being executed directly. They can still be passed to PHP. They can still be passed to bash, so it's not perfect. However, it guarantees that if you use this to mount your temp folder or your Drupal public files or private files folder, it is not going to be able to directly execute files, even if they have the execute bit set. They have to be passed to something like bash. So something that we're working on at the moment is, is it going to be feasible to run a PHP FPM container that doesn't have the PHP CLI, which makes it a great deal more difficult to execute things if you're able to inject them into a web server. That's it. A couple of interesting links, if you like. Uncovering Drupal Get in 2 is a white paper at, I can't remember where. However, it's a very interesting white paper. If you follow that link and you want to understand more about how that remote code execution vulnerability worked, that is a really great read. These slides and the exploit code are available at the link there. If you want to take this back to your team, back wherever you've come from, and you want to share the same thing with them, then let me know if you do. I'd love to hear that. And finally, if you're interested in looking at how to muck around with security and how easy some of these things are and how scary some of these things are, hacking is not difficult. And Pentestalab have a lot of really great free samples where you can look at old vulnerabilities and see just how easy they are to use. I think that's it. Ah, Drupal Slack. If you're not on Drupal Slack, you should be on Drupal Slack. There is an Australia NZ channel. I don't know if we've gotten to the 300 users yet or not, but did we? No? OK. So if you're not there, then you should be there. If you use Takato, you want to know more about Takato. We have a channel in that Slack. And if you want to find me there, I am at Otaku Mike. 294. So close. So close. Thank you, Nige. All right. Does anyone have any questions? Does anyone else have any questions? Have you heard of Drupal Steward? Ah, I have heard of Drupal Steward, but I feel like you should answer that question. Do you know what Drupal Steward is? So Drupal Steward is a program. Now, why would I tell you what Drupal Steward is? OK, but I feel like you should tell the room. But I'll tell you what. We'll make this an interesting test. I'm going to tell the room, and you'll tell them where I've screwed up. Drupal Steward is a program that providers of pretty much any description can join in order to know about vulnerabilities before they're announced so that they can engineer their systems to protect against those vulnerabilities before the patch comes out. Is that fair? Is that mainstream now? Yeah, I just had a hole a little closer to my mouth. So just one additional thing is an important thing to note about it is that it has a sliding price scale. There's like, if you're an enterprise site, if you're an Ocuator or a Pantheon, then there's rates based on a bunch of different factors that you can negotiate with the Drupal Association. But if you have just a small website that meets certain criteria, you can get it basically at cost, which is $200 of cloud fare like you had on your slides, I think something like that. So if you just Google Drupal Steward, you can find all the details about that. OK, great. I highly recommend it. Thank you. I think you should add a bullet on your website for it. OK, I will. If I ever give this tour again. And I should add it's not launched yet. So that's sometime next year. OK, cool. All right. I'm just wondering if you can share that info about no exec and everything in your PHP containers, maybe on Drupal Slack, if you did manage to figure that out? I will. Yeah, I definitely will. Awesome. The Cloudflare CDN. I remember three years ago, there was this big issue with Cloudflare not peering with Telstra. It basically means that when you are in Australia, I have a website on, say, AWS. And the client is on, a government client is on Telstra because it's like a trans provider. The traffic goes to Singapore. And then from Singapore back to your website and back. So because there is no peering. Has that changed? I haven't seen it. It has. So I started working with Cloudflare quite closely in my last hosting company, which was primarily Sitecore and AM focused. And they're providing a CDN service for Australia's largest retail website, which is bunnings.com.au. When we started moving that service onto Cloudflare, they had one caching node in Sydney. And that was an issue. They now have caching nodes all over the country, Perth, Sydney, Melbourne. And they do have better network routing than they had previously. We moved them from, I won't mention who, because I'm not sure if I should, but we moved them from a very, very large CDN provider. We got a 5% increase in offload by moving to Cloudflare. And we got a, I think it was about a 10%, 15% reduction in page load time as a result. So we were really, really happy with that move. And it cost about 20% of what the previous provider was costing. So yeah. I'll add, oh, sorry. I was just wondering if you have a preference in terms of OS distribution, as far as security is concerned, or whether it doesn't really matter if you have a CDN in front of it? If you're in Enterprise, you will probably be asked to use Red Hat. And I think that that's fine, because there are assurances that Red Hat will give you that you can't get from any other provider, unfortunately, if you're in Enterprise. And that's going to make the people you work for feel better. Personally, for the tachyto stack we use Debian, I'm really, really happy with it. I think it doesn't matter that much, especially when you move into containers, because it will pretty much run anywhere. If anybody wants to learn more about containers and learn more about hosting Drupal with containers, I'm also very happily on the Cloud Native Drupal panel tomorrow with Nick from Skipper and Scott from Amazing. So if you want to come along to that, I think that's at 10 in Hoon. Check the board anyway, Cloud Native Drupal panel. All right, thank you.