 All right, thanks everybody for coming, picking this security talk over the other one, because at the same time. But, so this is going to be defense in depth, lessons learned from securing 100,000 Drupal sites, so. I'm Luke Probasco, I'm with Townsend Security. David Strauss with Pantheon, and Chris Dijzel with Silver Media. So we come at this from multiple perspectives, and that's why the three of us are up here. Myself, I'm a Drupal architect. I've been working with Drupal for just about six years now, or just over. And from a site builder, themeer, and Drupal architect, I'll give you some insights. And I come in from my earlier experience as a Drupal site architect and infrastructure architect, as well as much more recently, the architecture of Pantheon to prevent compromise of the platform. And I'm with Townsend Security, and so I bring a perspective of compliance and data security, and basically security from an agency perspective. All right, I like this one. So, this is the FBI in the States, basically says there are only two types of companies, those that have been hacked, and those that will be, even that is merging into one category, those that have been hacked and will be again. And working for a security company, this is something that we see. It's actually kind of interesting to see, we can almost predict breaches at our company in the States because you have a certain amount of time where you don't have to report it, but we can see people coming to us and wanting to learn about encryption technologies, and it's actually quite interesting when they come around twice, so. And I would further subdivide those who have been hacked into two categories, those that know that they've been hacked and those that don't. Yeah, so son of a breach. Ah, there we go. Ah, son of a breach. You cannot afford to be hacked. The average cost of a data breach, the Poneman Institute does studies every year. It was $3.79 million per breach, or $154 a record. That puts businesses out of business. It's just something that should be not taken lightly. And so far this year, there have been 533 breaches and millions of records exposed. So if you just even add that $154, or even cut that in half, that's a lot of money. And so also a lot of these breaches have been by increasingly sophisticated external actors, where everything from Russian federal officials all the way to dissident groups are able to now fund attacks that are based on either buying unknown exploits all the way through paying for access to botnets to take down sites because of load, all the way to employing those bot sets to exercise known vulnerabilities in software so that it's more than ever before, if a site has a vulnerability and it's known, the chance that it will actually get eventually tested for that vulnerability and exploited on that is actually pretty high now. So you will be hacked. Let's just all get that out there right now and move on from it. But we don't wanna just go through doom and gloom and have this be a session where you guys leave angry and upset and scared for what you're doing. We wanna give you some practical advice. And so unless your site is permanently offline or on your local computer, you're gonna get hacked and let's figure out how we can prevent that from happening. And if it does, what steps we're gonna take? Yeah, so just if you wanna rewind there real quick, I have an audience participation here. Just to show how often data breaches happen, raise your hand if you've ever had your banks send you a new credit card and get ahold of you to say, hey, you're gonna need a new credit card. Yeah, this guy. Yeah, that just goes to that point. But another point that I do wanna make on here, we often hear I'm too small for a hacker to care about. And that is you couldn't be farther from the truth. This falls into the category of wishful thinking. And in fact, in the States, the FBI reports that smaller private companies are increasingly the focus of cyber criminals and are experiencing significant financial losses. And here's what an FBI consultant says on this read here real quick. Time and again, I have heard small business owners say that they have nothing to worry about because they are too small to interest cyber criminals. Instead, small businesses are exactly who the criminals are targeting for two primary reasons. In the criminal's mind, why go after large companies directly when easier access can be attained through small business vendor relationships? Secondly, since small businesses have less financial and IT resources, criminals know they are less compromise ready and tend to be less resilient. And we've seen concrete evidence of this on Pantheon where we've actually seen agency accounts be targeted for compromise when they're actually trying to also compromise the sites that those agencies are working on. So how do we build a security consciousness? That's the first step that you have to do. Before you touch any code, before you touch your servers, you need to start thinking with a security mindset. So let's go over the anatomy of a typical breach. So it's really important when looking at how to prevent breaches, what the typical ones have as factors. And in a moment, we'll also go over some of the ones that are not typical of breaches, but I couldn't emphasize more that almost every breach involves some sort of human element. Some person's been tricked, some person has not taken care of their personal security well enough, or their organizational IT has not taken care of their security well. If you look at, say for example, Sony's hack, it didn't involve a direct line from a vulnerability to organization-wide compromise. They compromised someone placed highly in the IT organization who had access to password lists, and then they had that go on to compromise the systems overall for almost every computer at Sony Entertainment. And that emphasizes one of the other issues with attacks, which is they often are a sort of festering wound, which is that a foot kind of gets in the door in some place and then that is used to build bigger and bigger compromises of an organization as they maintain that foothold. Going back to the Sony attack, you'll notice that they didn't instantly go to full compromise. It took over a year of sitting in those systems, exploring those systems, compromising the right individuals and their access to be able to pull off the sort of magnum opus attack that they did against that organization. So shutting things down early and preventing human error are some of the most prudent things that can happen for preventing attacks. Also important is to recognize what is not important in terms of attacks, in terms of what we typically see. Modern hashes and encryption are extremely strong when they're employed correctly. Even following the Snowden breach data from the NSA, they've listed many of the current modern encryption methods and hash systems as well beyond their ability to compromise. And to emphasize how difficult it is to compromise some of this modern encryption, back in 1996, Bruce Schneier wrote about how much energy you would have to harness to do a brute force attack against modern he's. If you, how many people in here have heard of Dyson Spheres? Good, okay, so for the people who aren't familiar, a Dyson Spheres is this kind of idea of putting something around an entire sun and capturing almost all the energy that that sun is producing and then harnessing it as a sort of super advanced civilization. And if you put a Dyson Spheres around our sun and captured all of its energy for something like 30 or 100 years, you still would not be able to perform a brute force attack against a 256 bit key even if you built a theoretically perfect computer that harness every bit of energy you captured from that just to walk through the keys. It's that hard to do a true brute force attack. So when it's employed properly, this is typically not the genesis of a hack on systems. And also, typically unless you're looking at nation state level actors, you're not seeing a lot of secret vulnerabilities that don't have fixes available. Even in the case of the Drupal get an attack, which was even on Drupal.org for quite some time before it was publicly announced as a security release that had a fix. It still took hours after the public announcement of it before anyone was actually compromising it. And also like this should almost doesn't need to be said but you're not seeing Hollywood style attacks. You're not gonna hook a computer up to a system and then say crack the password and then a bunch of green text goes by and you've compromised the system. So it's not an insurmountable problem, especially if you take some of the strategies that we have in here. And it's also important to think about security not as a bolt on to what you're doing as your strategy for building sites and maintaining internal security in your organization. It needs to be a fundamental consideration of everything you do. When you build a module, you need to be thinking how do I make this secure? When you deploy a site, you need to be thinking how do I do this securely? When you access your email, you need to be thinking about how you do that securely. You can't just simply do all those things and then come back at the end of the month and say, well, how do I fix the security of these things? Because if it's not in your mindset at the beginning, it's almost unreconcilable to fix it later without spending an enormous amount of time. And it's also, when you do that kind of fundamental approach to it, it's important to balance exactly how deep that approach is. There is a middle ground between complete ignorance and complete paranoia. And there are sort of diminishing returns as you add more and more focus on security. For every dollar or euro that you spend on focusing on security, the first few, if spent properly, can have an enormous benefit to your infrastructure, even if you don't want to spend that much time or money. But it's important to sort the priorities and that's what we're going to be getting into in this presentation. Which actually, this kind of shows a graph of the risk-reward effect of your investment, that the first bit of investment is going to be the most effective if you target it correctly. So I'll just talk just a brief bit here about compliance. I'm just curious. We did this talk over in LA and probably 90% of the room raised their hand, but who here is working on a project that involves compliance? Yeah, okay, interesting. So for those that are doing a project that needs to meet compliance, compliance is not optional. And in fact, I often like to say that compliance is really the low bar for data security. You should actually be doing much, much better than what compliance regulations say. And there are several modules and services that can actually help you meet compliance. So I guess we'll just kind of cruise along here. And next up, the security triad. This is not something we made up. CIA is not the US CIA. It stands for confidentiality, integrity, and availability. And the best way to educate yourself is to be thinking about all of the aspects in the triad, not just one. Confidentiality is roughly the equivalent to privacy. Data encryption is a common method for ensuring confidentiality. User IDs and passwords constitute a standard procedure. Two-factor authentication is becoming the norm. And just always err on the side of more confidentiality. Integrity is all about the integrity of the data. So first off, if you're building a module, never ever trust a user. Just put that as your first tape it to your computer. Don't trust a user. Anything that they give you needs to be sanitized, needs to be cleaned, needs to be scraped before it goes anywhere else in the system. That is gonna be the first vector that somebody tries to get in. And also just read-write access on your servers. Make sure that you don't have triple sevens on your entire server and anyone can go put any file anywhere they want. Make sure that you have proper read-write on your servers and you'll have an integrity of your data. Oh, and also one of the things that compliance typically doesn't address is the availability of your data. Which is that if your website's not up, if your systems are not online, you're not getting the business value that those systems provide. And that means that all of the security you might have layered on top of it is mostly for not. Because it's important to balance out usability of those systems as well as keeping them defended against things like denial of service attacks. And you have to be able to run your website on the internet. You can't just keep it on your local machine. So what does hacked mean? Oh, yes. Availability. Oh, there we go. So what does hacked mean? A lot of the times you see a site like this and that's what you first think of when you see hacked. I've had somebody come to me with a site that looks like this before and it's kind of obvious that you've been hacked. That's defacement. That's the easiest one to see. But as David mentioned, there's also denial of service where they can just flood your server with so many requests that it goes offline. And that will have substantial ramifications in your company. But then also there's data breaches which could go silent. And somebody can come in and take all your data out, grab your user data and do whatever they want with it. So hack does not just mean I'm gonna go change something on your front page and put some scary text on there. So really what we have to do is we have to look at a defense and depth approach. We always say that there's nothing perfect. You will never have the perfect secure website because it would be on your local machine. But if you put up enough walls, if you put up enough gates that they have to hop through and get over, the goal is to have procedures and practices in place where you can shut off one of those gates and limit your exposure. So having a defense and depth means all the way from the device, the application, the computer, the network, physical to your policies and procedures around the company. These are all incorporated into security. It's not just your code. So how many people have had one of their clients email them a password for a server or anything like that? Come on. I have had a client say, oh, I just bought XYZ server and here's the root password for it. Thank you very much. I have a lot more work to do now. And so just the first piece is you can have the most secure Drupal. You can have the most secure hosting environment. But if you don't have good passwords, if you don't harden your SSH, it's completely worthless. So protect those passwords. And there can be surprising compromises of that as well. Many people think that once they're using something like SSH keys, that you're safe. But it hasn't actually been the case. Even in the last month, there was an exploit for Firefox that where something was found in the wild where invoking the PDF viewer in Firefox, which doesn't require any special permissions, could read keys and other files off your home directory. And one of the files they had listed as one of the ones they tried to harvest off machines were SSH keys. So put passwords on your SSH keys. Don't just encrypt your hard drive. Make sure that even consider using things like application sandboxing, which is pretty broadly deployed on Mac OS and Linux now, as well as consider using tools to isolate different security concerns like using smart cards like Yuba keys for SSH. And one of the most important things for vulnerability is knowing when you're vulnerable, because there are millions of applications out there, millions of points that could potentially be vulnerable, but you need to be aware of the right ones. So I've kind of tailored this list to what we watch primarily at Pantheon for our customers and our infrastructure. And this is tailored to a combination of if you're deploying to Linux and you're deploying Drupal. US CERT and CERT EU are very broad information sources. They aggregate information from almost any company or organization that wants to publish security announcements to them. They also publish individual announcements for particularly bad breach potential for certain things. Those are free to get on, I recommend it. LWN does a roll up of major security vulnerabilities every single day on primary Linux stack systems. So that includes everything from the kernel all the way up to Apache, Nginx, and PHP. If any major distribution is releasing a major security update for any of those key components, they're publishing it. That is, I think their security announcements don't even require a subscription, a paid subscription. Of course, the Drupal.org security team is publishing announcements. There's a standard cadence for that of happening on Wednesday for releases, so you don't have to watch every day. You just need to pull it up about once a week. And then a lot of people don't know this, but you can configure package managers on systems typically do the bare minimum for security updates only. So if you're worried about just scheduling a sort of like YUM or DNF update or an app to get update for all packages that have changes because you might break something, you can just run them with a flag that says, I just want to apply security patches. And any package you have installed that has an update that has been flagged for security and any of the ones since the package you have installed, they will get updated at that point. And that's a great way to balance the concerns. And also following some of the major vendors and security researchers on Twitter is going to have some interesting information. I recommend adding them to a list where you can basically have a feed that you can check periodically for new announcements for security. All right, I'll talk a little bit more about compliance regulations. Each country in the EU has its own version of PII or personally identifiable information and protected information. And there's an interesting piece of language here. A confidentiality breach on personal data that were encrypted with a state-of-the-art algorithm is still a personal data breach and has to be notified to the authority. Nevertheless, if the confidentiality of the key is intact, the data are in principle unintelligible to any person who is not authorized. Thus, the breach is unlikely to adversely affect the data subject and therefore does not need to be notified to the data subject. So essentially what this means is you need to notify the data protection authorities in your state, but you don't need to tell customers if you have good encryption and key management. So I think that's an important takeaway as we're talking about securing data. The next slide here is about PCI DSS. I guess of you guys that have compliance, how many of them, how many of your projects are around PCI? Couple of them. So. Real quick, how many people work on any sort of e-commerce site anywhere? You're all under PCI. Whether you know it or not, whether you like it or not, just because you have a client that's asking you for PCI compliance doesn't mean that, or if they're not asking you for that, doesn't mean that they don't fall under it. Every site, unless you're using Shopify, which you wouldn't be here, is going to fall under some sort of PCI compliance and you need to know about it and you need to be able to address that with your clients. Yeah, and one thing that I always like to point out because I have a lot of conversations with people is that just because your cloud service provider has a little PCI logo on it, that doesn't make you PCI compliant. And it's great for them. Oftentimes that means that the hosting provider has an AOC or attestation of compliance for their platform, but it's still what you do with it that puts you in or out of compliance. It's ultimately the responsibility of the merchant itself, which is typically the website owner. So they have to actually make sure that everything is taken care of either by themselves or by the developer or by the platform. And that requires synthesizing all the different kind of controls and security mechanisms that PCI requires. And then they basically can fill in the report. Okay, and since we're in the EU, you guys are doing some pretty cool things, in my opinion, on data protection and requirements. Borrowed these points from a Sophos blog, but basically the EU is currently finalizing the new data protection regulation and it will likely become a lot this year. The European Parliament voted in favor of the proposed regulation by an overwhelming majority in March of 2014 and the regulation, well it still needs to go through further steps before it becomes a law. However, based on the near unanimous support so far, is widely anticipated that it will be adopted this year. And everyone who holds data on European citizens is affected, even if you're not located in the EU. And there are fines for non-compliance and that could cost millions of dollars. I think this is actually a pretty interesting statement here. Under the proposed legislation, if you suffer a breach of personal data, you can incur fines of up to $100 million or, this is where I think it gets interesting, 5% of annual turnover. Plus you have to notify the affected customers of the breach with all the associated costs and the loss of your reputation. And the best way to avoid a data breach is not persist data. If you must use isolated persistence or encrypt it. We'll talk more about that whole point later. And then, so finally, I think this is interesting too. I do have a lot of conversations with people and it's not just a credit card number or a social security number that needs to be protected. These are just some examples of things like, you know, a date of birth, birthplace, IP address. A lot of this stuff, just your marketing teams probably are collecting. All right, so we're gonna do, it's after lunch, our smooth talking voices are probably lulling me to sleep a little bit here. So everyone stand on up. We're gonna do a little short demonstration here. So everyone together? Okay, let's start with an easy one. If you have a Mac, please sit down. This is information that's sent every single time you visit a website whenever you're browsing. Okay, next. Oh, okay. Yeah, both of my neighbors should be sitting down. Next, sit down if you celebrate your name day. This is, if you don't know what that is, then you don't need to sit down. Okay, we found a Spaniard. I'm sorry? Okay. That's information that Facebook collects on you. And next, sit down if you were not born in February. Again, this is information that Facebook collects. Not. Not. So if you were born in February and you don't use a Mac and you don't celebrate name day, then you should be standing up. And you can see how rapidly we've segmented this audience just based on some very broadly collected public, personally identifiable information that everyone from Facebook to regular websites are collecting. And that's part of why it's important to collect, or not to collect. That's why part of why it's important to either avoid collecting it or to encrypt that data because it can identify people so individually. When Netflix held a challenge to do mining on their dataset, they actually tried to anonymize the data and people were still able to figure out some of the individual users in the anonymized dataset based on some of the ratings that they had given and the trends and sort of aggregation of the rating values and the individual numbers of the users. So it's amazing how much individual identity you can harvest out of relatively anonymized or seemingly impersonal information. You guys can sit down now. Yeah, you can sit down now. We'll make you stand up the rest of the time. So let's go through some essential security steps. These are gonna be kind of the, now that we've talked about having a security consciousness, the defensive depth approach, now we're just gonna start getting into the nitty-gritty of it. And first things first, keep a backup. This should be going without having to be said, but more and more it has to be. Use a backup. If you have a service like Pantheon, set it for automated backups and you'll never have to worry again. Personally, I have had a site that I got an email from a client frantically saying, oh my gosh, I screwed something up. What do we do? And I'm sitting here holding my six month old son, feeding him a bottle at five o'clock in the morning. And from my phone, I was able to reset the backup and get the site back up and running within like five minutes. So a backup is a very, very powerful thing to have if you know how to use it. If you do roll your own backups, it's important to segment the access control so that a hack of the website doesn't result in someone being able to also delete your backups. If you use something like S3, for example, you can create a key that is restricted to only creating files so that you can have your web servers be able to push their backups without the ability to delete them. Next, again, these should be pretty basic, but if you're not already, use some sort of version control, gets the best one out there to do it. So that way you can know that your code's been changed. In the events of a hack or a breach, and all of a sudden you think something's going on, or you just hit get status at some point on your server and a file has been changed, you're not the one who's changed it. And so that's a good way of seeing who's changed, or not who, but what's been changed, where it's been changed, and being able to version control that and roll it back is an important step. And while we'll talk about passwords and two-factor authentication in a moment, the best defense against reuse of passwords and the people picking bad passwords and having those passwords shared among multiple sites is to not have people use passwords on your websites, within your organization, and to minimize the use to the absolutely minimum essential necessity for that. For your end users, that typically means social authentication. There are plugins for Drupal and almost every other content management system out there where you can have people log in with Facebook, or Twitter, or Google, or GitHub, or a litany of other services depending on the type of user base you have. Every user that signs up for your site that way is not only going to be able to sign up faster, but they're not even gonna set a password in your system. There is no password for you to leak if they sign in that way. If you're an organization or you're working with corporations, there are enterprise login tools and protocols that you can support where a user can go through a kind of confined gateway that enforces policy and doesn't require them to put in their password everywhere. Again, that makes a great addition to security because it means there is less to even steal from the site. But you do probably have to use passwords some places. Here's the kind of well-known XKCD comic on some of the password combinations that people can create. It's sort of advice of don't be one of those websites that has one of those needlessly strict password policies about which special characters you use and how you format it and what combination of capitals and not because you're not actually helping your users create great passwords. If you are training your users or working with your own agency, recommend people install many of these free password management options. Here we have one password, last pass, and on the open source side we have password store and key pass X. Almost all of these have mobile apps that work with them, desktop apps that work with them, good browser integration and password generation so that you can generate good passwords, keep them securely and not use the same one over and over again because as I was mentioning with the foothold, it's not the fact that your recipe collection site got hacked that causes the real damage to people or a site that you signed into like that that causes the real damage. It's the fact that they recover the password from the recipe site that people are using and it's the same one they're using at their bank. I always like to say the best password is one that you don't know. I use one password all the time. I don't know half my passwords. I just create them. I know my one master password and that's all I need. So definitely if you are looking for something to secure yourself, go grab one of these. And in combination with passwords, it's great to add two-factor authentication. When this is often gets added as part of a password-based login where the next step after logging in with the user and password is providing the second factor, it's also sometimes used where it's just before a security critical workflow where for example in a site you could set things up where creating a new node may not require someone to get their second factor, but deleting nodes might to do those kind of more dangerous operations. And two-factor, two passwords is not two-factor. It has to be a combination of something you know and something you have or something you are. And typically it's something you have and something you know. The password being something you know, something on your phone being something you have. And this would have been able to prevent a number of issues for pretty famous hacks like targets hack would have been totally preventable by adding two-factor authentication. And the Firefox SSH vulnerability if someone's keys got captured would not be an issue if you had a second factor when you're SSH-ing because merely possessing one thing is not enough. And one thing to think about in this entire as you're building your site is that you're not alone. Your site is not alone. It used to be. You used to just have flat HTML that sat on a server, but now your server, your site is talking to other servers, other sites, other services. And many a times the breaches or the hacks that occur occur from an API source or from some sort of external communication. So know that you're not alone. And so if you don't need it, keep your hands off of it. This goes for everyone who raised their hand that they're doing some sort of e-commerce site. The PCI regulations are getting stricter and stricter. And unless you have hundreds of thousands up to millions of dollars to spend on doing PCI audits, which I don't think all of us do, then we don't need to worry about it because Stripe, Recurly, Chargeify, PayPal, they all have options where all of the card data is sent directly to the provider. They provide you with an authentication token that you can then use, but that token is not identifiable to anything or anyone. And the credit card companies do not consider the identification token that one of these companies gives you to be credit card data. So, and those provide you the ability to do recurring charges to allow someone to recheck out at a later date with the same card that they put in before. And in the case of some vendors like Recurly, you can even request them in writing to give you all the card data if you want to move between companies. So it's not necessarily vendor lock-in to do this. Every one of these is PCI level one compliant and you can build off all of the investment they've made in sequestering that data to make your services less likely to be compromised in terms of logs or databases or caches. Also, we're finding more and more that key management doesn't necessarily just mean encryption key management. Most of us here use one of these services, if not multiple of them, on a site that you would build for just a regular small site. If somebody had the key to your MailChimp account, they could send emails out with your brand on it. That's not a good thing. And so more and more now we're seeing these API keys are going to be and already are points of attack. I do have to say though, within Drupal we do a very, very, very poor job of managing these keys. The form API gives you a password field. The only thing that that does is give you the pretty little dots when you type your letters in. Once the data goes to the backend, it's not encrypted, it's not hashed, it's not anything. A good example of that and we have a patch for it now but the SMTP module that you use to send email out stores that email password in the clear in your database. There are multiple, multiple versions of this happening. All the commerce modules stored in rules configs which are all in your database. These API keys and tokens are just sitting there for anyone to touch. And it also does very little to encrypt it with a key that is equally available with the data. So like if you have your Drupal salt which is in the same database, encrypting with that doesn't really help you prevent much of a breach because a breach of the database still gives them the key so they can decrypt it. So we're gonna go through kind of the steps on how to secure your stack. We're gonna go from the hosting layer all the way out to your team. So yes, and we're gonna walk through each of these layers in a way of at least how we approach some of it on the Pantheon side to prevent compromise of these parts of the stack as well as what some of the best practices that our customers do to protect their system or to protect their website when they're already on top of a stack that is designed that way. So the biggest thing you can do to prevent a kind of cascading, festering wound sort of attack is to isolate the website itself. We've seen all too often some cases in the past where the website has gotten compromised which provide, and then the server got compromised and that server had credentials or other things to access other servers and because they put them all behind the same firewall or didn't have them properly segmented with firewalls or even just had reusable credentials even if they weren't behind the same firewall it allowed a kind of foot in the door attack where the website has to be pretty much publicly available unless it's an intranet. So you really don't want to compromise the site to be a foothold for doing further compromise of an organization. The next thing to do is of course be securing your operating system. This is kind of the once you've gotten above the hardware and VM level. This is the kind of genuine kernel of all the security that's happening on these systems. You got to install the security updates. If you're not monitoring for that then there are all sorts of vulnerabilities that can crop up over time since the time you deploy the system. You want to automate configuration so that whenever you are deploying new servers all the things that you got right once you will continue to get right with those new systems you deploy or when you're replacing a server. You also want to make sure that you're doing quick updates for these servers so that when you have to respond quickly to a vulnerability it's not something where you have to wait a couple weeks until a sysadmin comes in and is able to do that for you. And you also need to be managing of course your firewalls, your remote shell access who has elevated privileges on the boxes. And also a lot of people don't think about this but being able to find compromise when it's already in that festering stage before it's become a full blown organizational problem is important too. And there are root kit detection tools you can use that are free as well as package verifications usually built into the operating system. And you can periodically run that and it will tell you about anomalies. It will tell you what configuration files are not the ones that shipped with the package. It will tell you which files didn't ship with the package. And when someone is starting to put their kind of clause into the OS this can be an early warning sign. Next we have the kind of web server itself which is running the application and providing the kind of HTTP access to that. You want to be sending headers down to the browser that prevent some of the most common forms of compromise. Most browsers support headers like X frame options or strict transport security or things that or cookie options that are HTTP only. And so when you're setting those headers properly it means that one type of attack on the content management system doesn't necessarily snowball into an attack of any other website that those people are logged into. Of course you want to be logging things. You want to prevent some forms of fingerprinting to see if things are vulnerable. You want to lock down those private directories that might in a, you know if you're not using key management have some secrets in them. And deploy something like a web application firewall or CDN where certainly when some attacks get detected or get found mitigation can go in there first sometimes before you're able to actually do full QA on an update for the content management system itself. And your database. Distros are getting pretty good about this. It's pretty hard to have a default password. Most distros actually force you to set one at install time or have something where they don't have one set at all. But if you do have a default or you're using some package from a source that might set one definitely change that firewall it so that password is not the only barrier to entry because often people set passwords that don't get rotated or will get compromised and secure your backups of the stuff where the database itself is not the only way to compromise the data in the database. If someone gets to your backups that's almost exactly the same data. So I'll talk a little bit about data encryption. Currently in Drupal there is no native way to encrypt that encrypt data without a bunch of modules. And many regulations say like the EU regulation that we looked at earlier that if you have properly encrypted the data you are not financially responsible for a breach notification. And all of these modules up there except for the encrypt user where we're happy to say I'll have D8 versions. So do you have anything you wanna add on that? Yeah, so the new one to this mix is key in doing the most recent update to encrypt myself and another developer RL Hawk or Rick Hawkins. We were going through the encrypt and he added a configuration management to encrypt and we started thinking more and more about it and said why don't we have kind of a central station for keys in Drupal? This will make it much easier for us to regulate how they're used, where they're used. And so we created the key module and it acts as a central hub for all your keys. We have patches out there for all the major modules for e-commerce to SMTP and such. So they'll all start using this as their central store and it gives you options that range from okay to secure. And so definitely go check out key. It is currently in dev slash beta and it should be released this week. There's just a couple of small things that we're tweaking on it. So yeah, even with all of those modules that were just up on the screen, they all basically suffer the same problem whereas the key for the encryption is in your settings file or your database file places that are not secure. Essentially the key is taped to the front door. You can have the strongest lock on the planet but if the key is right there, it will be able to be unlocked. There's compliance regulations that call for encryption and key management. And there are standards and best practices. In the United States we have NIST which is the National Institute of Standards and Technology and even over here in Europe they often point to that because they do basically a definition of what should be a best practice. And back to the title of this presentation, key management is fundamental to a defense in-depth approach to security. So talking more about API keys, using something like the key module allows you to keep all those API keys in one central location. But some best practices around that are don't share your API keys with developers that don't need access to them. I will openly admit I've done it once and I will only have to do it once but how many times has somebody pulled a site database from live back to development and forgot to change the emails or forgot to turn off that rule that sends out an email and all of a sudden you're blasting emails from a development server to production users. So make sure that you have systems in place where developers aren't using your production keys. Go get, most all of the systems will give you development keys or development environments. If they don't, go create a dummy one and use that for your development and then switch it to production when you need to. Use per developer and per key or per system keys. So don't share the same key so like sharing the same password. And then most off, offsite key management is the best way to manage your keys. If your key is not taped to your front door and in Drupal that means in your database or in your file system, then it becomes much harder for somebody to access those keys and use them against you. Also it gives you more ability to regulate who's using them and where they're using them. So with that, Cellar Door and Pantheon are announcing today that we are launching Locker. This is a new system that will be available to users of Pantheon exclusively right now. And it is a offsite key management that is built into a Drupal module that is built into key. So you enable the key module, you enable Locker and your keys will automatically start getting sent off and stored securely in an offsite database. It leverages Pantheon's public key infrastructure on the platform to ensure that there's secure integration for the key management. So it is actually aware of what Pantheon environment you're working with so that you can set up keys that work exclusively in the live environment as well as separate keys that are used for development and testing purposes and this isolates in conjunction with Pantheon's change management tools. It isolates the production keys away from developers who have restricted permissions. And along those lines, with that, those keys aren't available for a developer to take with them. If you are in a position where you have high developer turnover, you have developers that are external to your company. If you have a copy of your database that gets walked off with, they have all of your keys. Luckily Locker prevents that from happening and makes sure that the keys are used in the environment they're supposed to be used and only with the people that should be using them. And I'll just add one more thing about this. Just a quick plug for you guys. Totally free for developers. Yes, so that's the other thing. Right now, we're launching into beta and so during that time for developers, it's 100% free, it's available today. Go check it out, try it out, kick it around and definitely give us feedback if you have any feature requests. Users have also asked about sanitizing databases as they take it back from the live environment. And this also offers an alternative mechanism for doing that where if you encrypt the data in the live database using the key that is only available in the live site, then taking that encrypted data back to the test or development or another environment, that data will not be available. It'll just be in the encrypted form and the key won't be accessible. So next step up is Drupal, which we're all here for. Just keep it updated. Don't let something go festering for a long time without getting the updated. And don't get creative with permissions and whatnot. I always see sites I come to that I'll take over and there's 20 different roles and 30 different permissions that are set and it takes one checkbox out of those hundreds to all of a sudden give somebody access to something they shouldn't. So keep it simple, keep it updated. Like we mentioned earlier, there's security announcements that are regular on Wednesdays for that. So look for those. And then also contribute module security. Every time that you install somebody else's module, you are trusting their code to run on your site. So active and popular plugins are most likely to have higher security scrutiny. If there are issues, if you're a module maintainer, we've had this happen on the encrypt module. There was an unsafe cipher that we needed to remove. The security team will contact you and walk you through the process of putting out a security release. But do look at the encrypt modules. Just because somebody else made it and it's on D.O. does not make it secure. You need to audit everything you put on your site. And as I mentioned before, a lot of attacks exploit the human factor through confidence building, through social engineering, through people just trying to get their job done and not wanting to jump through security hoops. So it's important to have security that works with your team. One of them that I mentioned before is using single sign-on tools where you can create a policy bottleneck where you can make sure that everything people are signing into, including the website, including dashboards, is handled through single sign-on. On Pantheon, we support this through SAML on our enterprise product that allows you to set up that type of policy and deploy it to a team that allows enforcing things like two-factor authentication and strong passwords. It's important that you secure the devices of your team. The reason that passwords got compromised at Sony was because they had a file, that was a master file with all the passwords in it. And it's extremely easy with tools like Google Drive and Dropbox to have people synchronizing that down to their devices. So enforcing policy on what people can put on their devices, what they can sync down to them, how they encrypt them, how they put pin codes on them, when they lock is important for the security. You're all clearly here at a conference. What happens if you left your laptop around and it went missing? Not because it was stolen, but because you might have just left it somewhere. Building security consciousness among your team is important just as it is to create streamlined policy and implementations for this. All right, so we're short on time here and I wanna make sure that we give some time for question and answers before we're done. So there's a couple more slides here. We will post these slides. These are some interesting insights into the Drupal Geddin attacks and how Pantheon was able to monitor those and how it's consistently going on through various vectors and different sites are targeted constantly for whatever reasons. And so yeah, so we'll kind of skip through these. One really quick thing to note is if you get hacked, don't panic. The first thing you need to do is notify the authorities or follow the procedure if you're with a organization like a university or a government that has certain policies, but then roll back immediately or as fast as you can so that you can get back to a stable spot. Review and then make sure that that review has actionable steps to go forward with that you're going to be able to use next time. So sorry we kind of cruised through that last piece there, but I wanted to open up in case there was any questions or so you guys can get to your next session. And I think there's a microphone back there because they're recording these so. You need to go around. Can you guys hear me? So my question is simple. So last year we had probably everybody knows here Drupal 7 had that problem. Yes, Drupal Geddin. And I know that Pantheon had very interesting measures. Very interesting measures. Would you like to profound that a little bit more? I don't know if I was a little bit late on the session. Okay, so what we did on Pantheon is because we realized that it was a compromise, it was a vulnerability in Drupal specifically. We run certain PHP before every request on Pantheon. That's to set up the environment. It's to handle platform integration. And in the case of Drupal Geddin, we added code that we deployed to all the websites on the platform to analyze the incoming request to look for the telltale signs of something that was trying to exploit Drupal Geddin. And then to deny the request before Drupal actually started any processing on it so that before a site had been updated to actually avoid the attack, most of the cases would get mitigated. We can actually hop back real quick to the slide here. We did some analysis of when it was coming in and what we did is we have a kind of stats platform, a foundation for our systems. We typically use it for platform monitoring rather than specific things for sites, but we set up this mitigation measure to actually ping our stats collection with a specific event for the Drupal Geddin attacks that we detected and blocked. And almost exactly seven hours after the public announcement of the attack, we started seeing a massive uptick in the attempts of the attack. It was actually done alphabetically by domain. So it looks like someone pulled publicly registered domains and then they were iterating through those and they were fingerprinting them if they were Drupal and if they were Drupal they were trying a Drupal Geddin attack. Fortunately, it looked like for most of the major attacks that happened or most of the scripts that people wrote that they were effectively detected because we didn't have any reports of compromise even for sites that took a little bit more time than we would have liked to patch themselves. And additionally, we did some proactive outreach to customers where even after we did the mitigation we detected which customers had yet to apply the updates that were necessary and we contacted them to say, by the way, I know we did this mitigation but you should really patch this on your site. And this is from a Drupal Architect standpoint. This is why you go with a hosted provider whether it's Pantheon or any of the managed hosting providers is that you as a Drupal developer are not a CIS admin, you do Drupal development. Don't think that you can manage your stack properly. Let somebody else do it and in the long run it will save you and your clients more money than trying to host it yourself. I'll name names. I know Aquia and PlatformSH did preventative measures for this attack as well. Great. Sometimes I've had requests from clients with IT teams of clients to take steps to prevent fingerprinting of Drupal. Sorry? There we go. Start again. Sometimes get requests from clients or the IT teams of clients to take steps to prevent the fingerprintability of Drupal. And we tend to advise that it's a bit of a lost cause because A, there's about a million ways that you can fingerprint it and to stop them all would basically mean not using Drupal. And B, looking through the logs, you see attempted attacks on, in the average Drupal logs for WordPress sites, for PHP MyAdmin. So the suggestion is that people will try and attack a Drupal site with a net of Drupal exploit whether they know it's Drupal or not. That's what we tend to advise. Would you say this is good advice or? I would advise, so in terms of fingerprintability, I think it's probably more important to prevent version fingerprinting than software fingerprinting of what software you're running. Like it's way more important for an attacker to know exactly what version of Drupal you're running than that you're running Drupal because that determines what's patched, what's not patched. In a lot of cases, what these attackers are doing is they're hiring botnets and they're hiring those botnets pretty much by the hour and anything that makes their attack more efficient is something that allows them to compromise more sites more effectively. And so they'll often do things where they can just do the most, you don't have to be perfect, you just have to be harder than the other ones they're trying to target. And so they might write a compromise where it tries to look for a changelog.txt, looks for the version that you have and if it looks like you have a vulnerable version then they attempt the more complex dance of say, signing up for an account and then trying to create a node or whatever else is involved in actually orchestrating the attack because they want to know whether you're vulnerable as quickly as possible and if your changelog is not available they might not even bother proceeding with the rest of the attack. I'm not suggesting that they won't but like it's all defense and depth. Every measure matters. Also from a Themer's point of view trying to scrub your front end of everything that screams Drupal i.e. views, panels, all that stuff it's gonna take you a ton of time and I think you're gonna see diminished results as you more and more try to decrease that fingerprint. Yeah, what he was saying was that you could download other files that have to be publicly available and they have known changes to them and then you can just figure out what version it is. Hi guys. Not like a security person like don't have a great amount of knowledge. I was wondering if you could explain conceptually how offsite key management works like specifically how the service knows if it's a legitimate request for a key versus an attacker who's compromised the system and is trying to be okay. So in the event of locker the service that we're doing we're using the cert that is signed by Pantheon and so we trust and know their certificate authority and so when a request comes over we look at that request and make sure that it comes from a Pantheon server. If it doesn't it gets rejected out of the door right away. So there's some very simple steps that we put in place at the very beginning to authenticate that and then we have other methods of identifying the site, the key name and all of those pieces to go actually retrieve the key. But one of the most common questions I get is why does storing your key offsite but yet still an API call away make it more secure than if it's on my database? And to answer that it's not every hack means that they have PHP control of your server and they can just start running arbitrary code. They could get a copy of your database from somewhere. They could get a copy of your code. I was working with a Fortune 50 company and we had to sit down with the security team and the first thing they said is assume that the entire database and the entire code base is going to be read by somebody that shouldn't be reading it. And so not necessarily running the arbitrary code but just reading it you don't want those keys to be there. So it gives you that extra wall but then also we have in place flood measures and other countermeasures that say okay if somebody's trying to brute force this and they have attacked a site and now they're just trying to strip every key that they can find we can limit that and shut that off as well. And to provide a tiny bit more detail in every certificate that panty on issues to a container we embed the site identity and the environment. So they actually know on locker that it's coming from a live site stack for this particular website. And then that is how it retrieves the keys. Additionally even if someone does get application control it's all about that defense in depth and how much effort the attackers have to do. It might be that a script kitty which is basically something who downloads an attack kit is able to obtain the database and they may not be experienced or capable enough of discovering the fact that they need to trick the site into making an API request for the key to decrypt the information because that requires considerably more knowledge of the architecture and a much more targeted attack. You don't have to be perfect. You just have to be harder than everyone else to hack. And it's all about the defense in depth approach. How many gates and how many walls can you put up? And if those walls are taller and harder to climb then you can shut them off or kick people out of them. It also means that your backups are no longer vulnerable because the backups, if someone gets one of those files that's a whole separate system. That's a whole separate set of files that people are handing around. And key management means that that system itself is protected against compromise of it alone. And interestingly enough, as Luke was talking about earlier with compliance, a lot of the times we hear, oh, you have to encrypt, you have to encrypt, you have to encrypt X, Y, and Z data. But what you don't see in that, if you read a little bit lower, it says you have to encrypt it and manage your key properly. And to manage your key properly, they'll state it has to be kept off site. So just encrypting your data and keeping the key, even if it's on the exact server outside of the web root, still technically does not meet compliance in certain situations. So please be short because I guess everyone wants to leave. Locker, is it open source or not? So Locker, the module piece for it is open source. So that's gonna be available for everyone to download, but it's a hosted service that we're running. Yeah, and that's precisely my point. The backend is hosted service. Correct, correct. This is your business model, so you keep it there. I mean, my long-term problem is with the trust and with the choice. I am convinced now you guys seems to know your business. So I go for your service, but tomorrow you got recruited by one company and they got recruited by another and I don't have any idea who takes over. So is it possible that the backend is actually open sourced and we can implement it and we can actually examine the people that are taking care of our security? So there are open source key managers that you can deploy in your own stack, but the reason why we're hosting it and we're managing it is because we've been working in key management in Drupal now for a couple of years and talking to a lot of people in the community and the two things that come up most are cost and so Locker's gonna be introducing at $30 a month, whereas most key managers are on a magnitude larger than that and the other one is just time and effort. To run your own key manager, you have to put it on your own server. You're gonna be the one managing it and everything else. So for us to manage it, it's all there, but the actual functionality of that key and where it's being stored and whatnot is not gonna be open sourced. And from the Drupal perspective, from the model, how much does it look you to your service? No, one of the great things about the architecture of this is the key module allows you to use any backend you wanna write that follows keys API. So they've been great about this in the sense of it's sort of like backup and migrate where there are completely open source uses of it. People can write their own backends for it and while there may be one hosted backend with a provided integration, it's actually shipped as a separate module. You get the key module, which is totally open source and you can write any backend for it and then there is the Locker backend for the key module where anything that the key module stores goes into those APIs. And you can export your keys back out of Locker. So if you wanna take it to a different provider and the key module itself, we built it to be that central hub. So at any point you could say, I don't wanna use Locker anymore, I wanna change it to a different service provider or a different method. And as long as it's a C tools plugin, so as long as there's a C tools plugin for it, you can put any back into it. Thank you. You will be the Pantheon booth? Yeah, we'll be around at the Pantheon booth. I have several other questions, but I don't wanna say. Yeah, he'll be at the Pantheon booth and we're around there quite a bit too. I think we need to pack up. Yeah, I think if there's any more...