 All right, we'll go ahead and get started people will start trickling in that's totally fine You guys are hopefully all here to talk about security and defense in depth We're gonna be talking about best practices around how you can secure not only your Drupal site But the stack that it runs on as well and Some lessons learned along the way So I'm Nick Stilao. I'm director of engineering at Pantheon So we run about a hundred thousand Drupal and WordPress sites and I have learned along the way a lot of you know We've seen a lot of different attacks and different exploits And learned a lot along the way building a platform and hoping to share some of that today with you I'm Chris Ditzel the founder of cellar door media. We're a Development and consultancy firm based out of Seattle I've done everything from small sites all the way up to architected large-scale e-commerce for Actually, African Airlines and so when you deal with security on the level of Major e-commerce. It's it's a whole new game. So Yeah, and I'm a Lu Provasco. I am the GM of Drupal business up at Townsend security where a data security company have enterprise clients worldwide and are really excited to be working within the Drupal community So yeah, I'm excited to be presenting with two great co-presenters We each kind of bring a different perspective on the on the security world My perspective is more a little bit less kind of specific to Drupal, but more kind of on the technology On the platform side And so that's kind of where my expertise will help guide us today And I'm coming at it from the Drupal architecture side From the developer side if you're a dev shop if you own a shop if you're a freelancer What can you do to protect yourself? And I come from basically a security company You know, we we work with a lot of clients. So I understand their you know concerns A lot of people come to us for a couple different reasons primarily meeting compliance or Needing to manage their risk of a data breach And I love this slide Because it is very true There are only two types of companies those that have been hacked and those that will be even that is merging into one category Those that have been hacked and will be again. We are seeing that it just shows that it's Absolutely important to have security at the forefront of your Drupal projects If Drupal is going to be considered an enterprise class CMS businesses care about Is it's gonna meet my business needs and is it going to be secure? So you have to build security in from the ground up in order to prove that Drupal is the right CMS for the enterprise And does it keep your CEO out of the headlines? Yeah, yeah So I think we're throughout the talk will be kind of talking about from a couple different angles One is kind of about what you can do to secure your sites and not or you know secure your business But then part of it also is just kind of what does security mean to Drupal to the community in general as we want that community to grow? Son of a breach you cannot afford to be hacked These numbers are real the Ponymon Institute puts these out every year These are the latest numbers three and a half million dollars per breach or a hundred and forty five dollars per record I don't know too many Businesses that could go through one let alone two of those and just an astounding number There's a website you can go to As of 428, I mean it just blows my mind that there's been over a hundred million records exposed So times that by a hundred and forty five dollars per record. That's a lot of money getting spent on data breaches And we're gonna be doing a little audience participation just to get people going before lunch So if you would go ahead right now and raise your hand if you don't want to participate in any of the things we're gonna do Okay, that's looking good. I'm feeling good about this Just to start off like who's gotten a credit card a new credit card sent to them by their bank like I have and actually It just happened that like Sunday evening before I flew down for the for the record the people that are watching this online That was every hand in the room Yeah, so, you know this this stuff's really happening You can see some of it with your Drupal sites and the kind of exploits you hear about in the community But you can also, you know, see it through your daily life through your interactions with your bank or how your Online experiences are changing with the different organizations work with You know and to drive the point home a little bit more this is a public service announcement from the FBI and from about a month ago and I think they need a little work on their website because it was actually like a lot harder to download this Just load the page but this is This is something from the FBI about a month ago saying like in the same word basically like ISIS and word press Like like, you know, like so this stuff is going on, right? Like this isn't just like, oh, you know my you know, whatever, right? This is like the FBI and ISIS and wordpress all in the same sentence in the past month, right? So all of that to say you're gonna get hacked Unless your site is permanently offline and you're just using it on your local machine You have to assume that your site is gonna get hacked I work with a large company right now and we just did a security audit and they the security team came to us and said Plan on your PHP being exploited plan on your database being breached What are you gonna do about all of that? How can you protect it in the event of everything that you own is being given out? so Don't freak out. This is all the the the fun at the top there is fear uncertainty doubt This is what a lot of people will try to push at you and we want to make sure that you guys Walk away from today with with practical steps with real world scenarios that you're not just running around freaking out like oh my god I'm gonna get hacked the world is gonna end and I should just stop what I'm doing and crawl into a hole so Don't worry. We'll walk you through it So the first step is building a security consciousness For you and your team Yeah, so that's right like you can you know like there are people you can pay to help you with security There are like you know legal entities like from the government that help you handle security. There are compliance organizations There's a whole ecosystem, but the one thing that you really cannot buy is educating yourself And so you know when you think about security the thing your Responsibility to yourself and your colleagues and the business owners and the sites that you create is to educate yourself and be able to Think objectively about security. So really it's about a frame of mind, right? Security isn't something you implement or you do or you buy or you like you prevent this attack or that attack You know, it's really about a frame of mind I think great security people when they walk into a room are kind of like exits fire escape like no Security guard by the door, you know are just kind of like noticing stuff noticing how stuff could be broken how stuff could be breached and You know so you don't want to be paranoid, but you don't want to be like ignorant either about this stuff Well the real goal. Yeah, I also think as Drupal developers You guys have a responsibility to your clients to be setting them up for success And you know if they get a breach or when they get a breach they're gonna say what the hell guys I thought I paid you a lot of money to make sure that I didn't have to go through this and so And I just think it's you gotta do it Yeah from a developer mindset every time I walk into a bidding process or an RFP the first thing I think about is What data is there? How do I secure it? What do I have to do? And that should be on every developer's mind as they're going through this right so, you know another way you can look at this is kind of on Risk mitigation how much it like how much are you investing in security and what risk to those mitigate? It's also important to think of kind of what bucket you fall into if you're doing a You know hair salon brochure wear website that might not be such a big target if you're working with a kind of Activists or political organization that might be it you know could be a target and you're gonna want to invest more So part of it is just understanding kind of you know how in order to understand how much you want to invest in security it's also understanding what what risk profile you have and whether you'll get away with kind of best practices or Going a little more in-depth or a kind of a very locked down locked down Just to double-check can everyone in the back here us Yep, raise your hand awesome cool Just double-checking we have loud voices and I don't like microphones So I'll tend to walk away from it and come back But hopefully the folks that are gonna watch this later online don't mind that so Okay, more audience participation Who here is working on a project that involves compliance? And I bet there's probably a lot more hands that aren't raised because you might not really get that Simple things like an email address or a username are also fall under compliance regulations So I bet that that's gonna probably raise a few more hands The thing about compliance, it's not optional And compliance is also the low bar for data security You know, I I read a lot of the industry You know magazines and journals and whatnot and and the common theme is that Compliance is just the stuff that you should be doing at the absolute bare minimum It's not going to necessarily prevent you from a breach It's just like the low bar you've got to do more but for a lot of people that can be difficult But I just urge you to just look at a lot of the compliance regulations If you're just trying to get a feel for what you need to do security-wise At PCI DSS is a great one and just look what do these compliance requirements? Say I need to do and that's what you should be shooting for right out the game There's also some studies. I'll make up some statistics But that I think it's like 85 percent of companies are out of compliance within two weeks of their last audit Right, so if you're really thinking about it from a compliance angle You're probably not really thinking like holistically like you know, you're kind of thinking about the low bar You really want to be integrating this into the entire life cycle Okay, so we'll talk a little bit about the CIA security tribe not the governmental CIA Stands for confidentiality confidentiality integrity and availability and this is not something that we made up so there's opinion You can you can certainly go online and learn a little bit more about this, but just to kind of Briefly talk about it confidentiality is roughly equivalent to privacy and data encryption is actually a common method for ensuring confidentiality use your like I said user IDs and passwords constitute Things that you need to encrypt and also two-factor authentication is becoming the norm And and just one thing I like want to stress is always err on the side of more confidentiality You don't want to be the the you know when when the CEO comes down and says What's going on and you just say well, I thought it was secure You know, it's it's you get it's all all these items are a part of a defense in-depth approach to data security The next is integrity This talks about data integrity is the data that you're receiving The data that was intended for you to receive are you did somebody man in the middle attack it? Are you sure that you're securing your connections in and out? And then it also goes back to don't trust your users just because somebody says that they're you know That they're an admin don't give them the ability to write arbitrary PHP in your notes The admin password could get hacked and just because they say that they're the password or they're the admin Doesn't necessarily mean that you should just give them every right in the world to exploit your system and then culminating with availability so this gets to kind of denial of service attacks so if someone's Attacking your site or your service in a way that it's not able to provide the value that it's supposed to do to the customer If your site is down That's not that's not a secure site And so one of the one of the reasons I think this the CIA tried is important is that like security is a big Top right so it's really helpful just have this as a very simple framework for kind of breaking down the different types of threats what you know How they might be able to impact your system whether that's Updating data that you don't know about or getting data that should be private or taking your site down So if you can kind of decompose security you can start to talk about it a little in a little more detail The thing I really like about this image is that it's a full loop and that if you break any one of these Your security goes out the door. You can have the most confidential System you can have it be 100% uptime And somebody man in the middle attacks you and you've just lost everything Same thing with availability. You can do it all but if you can never access it or the site goes down then your SOL so What does hacked mean these are the when we think of hacked this is what we think of I Actually did have a client call me and say Not the site that I built them luckily But a site that they were their brand was associated with was hacked by Isis and sure enough there was this big You know we are Isis and and this is what we believe in all sorts of highly inflammatory things on their branded website So this is kind of what we we think of but it's also denial of service It's a data breach It can it doesn't necessarily have to be somebody getting in and changing your front page or changing data on your page It can be as simple as just pulling small bits of information over time that will that will slowly leak and in cause a bigger issue Yes, so defacement is actually in some ways that Go there and you know and from a sample size of three if your homepage is black And it's not supposed to be if there's a really creepy joker and then like a general like Guy Fox on your homepage Is like a general sign you chances are it's not your site So the next is defense in depth Security as you guys may well know is not just a single level approach It is a multi-level approach and you have to Go at it from many many different angles So this this spans the gamut from from kind of the you know everything from the your hosting solution and the network there to the Physical machines are using the OS a lot of the tools You're running on the OS your web server your database server up through the JavaScript Even to the team you work with to make sure they're aware of security Maybe you're using a CDN. So there's a lot there to cover and I think one quote I love or one way to think about this I love it's like the belt and suspenders approach You're like, you know, this belt works pretty good. These suspenders are pretty fly But I think with both of them, I'll be like very sure So, you know, you don't just want to get one control you want to have as many controls over the whole stack And as we were talking about earlier, it doesn't matter how strong everything is if you have a weak link More audience participation, how many of you have had somebody email you and say, hey, here's the route password to my server Yeah, a client that owns their own owns their own server says hey, I just spun it up on go daddy Here's the route to it You can be as as hardened as you want and that one email can just break everything apart So make sure that you you know, don't allow root access via password Don't even allow root access. Don't even use root use of your own user harden it with With keys and we'll talk about this later And and just go through some basic steps of and and part of this like that example is you have to teach The end users who aren't going to be the security conscious ones. All of us know, okay, we don't use root We don't pass it around but you really need to tell your client when they send you that email look You probably shouldn't do this. You've just caused me a lot more issues than you've solved So are you vulnerable? Are you vulnerable right now? That's a good question and it's actually kind of hard to answer There are some part of the you know, if you don't know if you're vulnerable It's going to be hard to really, you know, get an actionable plan for for mitigating those risks So there are a couple different ways you can that you can kind of keep up to date on security updates So what a great one is us cert, which is a federal entity which kind of helps aggregate and disseminate information about this if But you're also going to want to keep up to date on kind of like any mailing list for any software using maybe that's a The patchy web server engine X. Maybe that's my sequel Maria DB. Maybe that's varnish or squid or anything like that also Depending on your OS, right? You're going to want to follow your OS security mailing list So maybe that's what or whatever that is But there's like a you know, so there's like a couple great resources But really there's no one great place where you can hear if there is a if there's an exploit that might affect you and I think what one of the best tools I found for this is Twitter and It involved, you know, just like following, you know, it's the people I follow But that's often I found out find out about exploits on Twitter before, you know, US cert kind of gets to it There may be a couple days days behind maybe before You know, it's kind of gone to the mailing list or something like that So and one other good thing is follow your own Twitter handle from a different Twitter handle so that you can see if somebody's actually using yours so And if you're a Drupal developer, the security team is awesome They have a full list of a rundown of every security update to core. What was affected how it's affected the severity of it You'll see all the information you need there to make the decision on how does this affect you and what do you need to do? So we're just gonna switch gears a little bit and just talk briefly about compliance What what I what's interesting about compliance is oftentimes Organizations fall under multiple regulations. I've had a lot of conversations here at Drupal con with Universities and if you think about Universities actually, let's see how many people are here from or from my college or a university Wow, that's awesome You guys are like just many Villages you guys you take in student data. You have faculty data. You have often have donor information You know, you also Probably have health centers for the students. So you've got HIPAA you have your state privacy laws your federal privacy You're really lucky or maybe unlucky as a developer. You could kind of work on a site that might hit all of Yeah, if you work at a state school, you probably do so You know additionally I mentioned this earlier if you're your as a group of developers You should also be asking your clients if they fall under compliance regulations because sometimes they just don't volunteer that and then You know towards the end of the project. It's like oh by the way. Oh, yeah, we have to do this Can we get that done in a week? So a couple chuckles people have heard that So What what's interesting about compliance requirements is it's up to you or well It's up to you and ultimately up to your clients, but PCI pretty explicitly says That no hosting provider can actually make you compliant. I think that that's often a misunderstanding within developers a Hosting provider can Provide and it's called an away their AOC and attestation of compliance for their for their platform But that doesn't extend to you and I'll I just want to say that again because it's really important That does not extend to you you need to put the proper controls in place You need to be doing the encryption you need to be doing the key management You need to be doing all the security things your hosting provider They can set you up to just to succeed, but ultimately it's up to you And I liken it to you spend all this money on building an armored vehicle But you don't want to roll around with the windows down right just because you have this platform That is the most secure thing around it's it's not inherited that you're gonna be secure It's up to you to make sure that you're using it properly to be secure. Yeah, and I like the idea of Often it's raised as shared responsibility and you know, I think that comes back to that security is a responsibility It isn't something you can you can you know do at the end of the project or isn't something you can buy or isn't something you can You know like a module you can install a single module Yeah, just this further Regulations like PSS make it very very clear that in the event of the breach It is the enterprise not the hosting provider or development agency that's responsible. So It's also I mean you have reputational risk because you've developed that site, but ultimately it is your client So I always find this slide particularly interesting Because PII or personally identifiable information is Not just a credit card or social security number. I mean those are the obvious ones, right? But but like I mentioned earlier things like your login name your email address I mean gosh, even if you're collecting IP addresses as websites become a lot more personal They know who you are. They're collecting this information. That stuff needs to be protected and Yeah, well, we can just go in here. Cool. So Had a little another little exercise. So everybody raise their hand up and then we're gonna lower it Even you and then you as well everybody one of you. Okay, cool. So now lower your hand if you live east of the Mississippi River Okay, I ran the numbers and that this is gonna work out perfectly Lower you your hand if you do not own a dog Yeah, lower your hand if you are not born in February Yeah, that guy those two in the back back stand up So we went from like I don't know how many people are in this room Maybe 200 to like two people with like three totally like random random pieces of data That you wouldn't really worry about but in this group if we if we had that we could identify those guys kind of So that's even stuff like, you know, your zip code date month of birth the state You know, whether you like coke or Pepsi more can kind of help like like target you and so that's that's kind of what you know Good ad agencies are doing I guess but it's also what good, you know Good hackers are able to do with seemingly inconsequential pieces of information. There was a there's a high-profile instance of this where even Data that you think is secure or that you think your You know the services that you're using using are keeping secure. There was a guy who I believe he was an editor for Gadget or wired one of the major tech blogs had a very Short one word Twitter handle worth a lot of money and so somebody called up Amazon and Amazon said well Sorry, can you answer your security question? He kind of be asked around with them. They said, okay Can you give me the last four digits or the the security code on the card that ends in one two three four? And he goes and then hung up called Apple and said oh An apple said well Can you just verify the last four digits of your credit card which he just got from Amazon? So he's able to verify it hacked into the iCloud account deleted everything on the guys and computers all of his family photos Everything he had so this is something that is actively happening and it's something that you have to be aware of even in from from a As you're developing standards and practices within your companies What data are we gonna give out and how are we gonna give that out? And could that data be then used to build a social hack on somebody? sweet, so we're gonna just go into a couple things that you should absolutely be doing and And if you're not doing them, it's totally fine just to get up from the room Or just power is just on it But but these are the things if you do nothing else you have to be doing these the first one is back up your data I Can't I can't stress this enough backups are going to save you in the long run no matter how you do it the I mean an example that I give is I have a new born and I was up at you know 5 30 in the morning feeding him And I'm sitting there and he's sleeping in my arms and all of a sudden I get a ping from Pingdom that your site is down I go and I look at it and something catastrophic that the Content admins had added in caused all these issues and whatnot and rather than trying to deal with it with a sleeping child in my hand I I was using Pantheon at the time, but I logged in and within two or three clicks I had restored a backup the site was back up and running and my child was still sleeping so In that instance a backup saved me not only In the issue with the client, but it also saved me some personal time trying to get my son back to sleep So just back up your data store it somewhere store it somewhere safe Don't store it on the same place where all your other data is because if all your data is breached or you lose it Then your backup is worthless. So keep it off site. Keep it somewhere store it You know wherever you can and there's tons of services available that'll allow you to do that In conjunction with that you should be using some sort of source source code management and and Everyone here. Hopefully is using git if you're not, you know, get with the program unintended But git allows you to in the event of a breach one of the first things you can do and we'll talk about this later is is you react by Loading a backup and reverting your git repo. That's going to clean up a lot of your issues It's not going to clean up any everything But it's the first step that you can do and if you have get you know What files have been changed where they've been changed and how they've been changed if you can't If you can't identify that then you're gonna have to go line by line by line through the entire droopal core Or and figure out where that little exploit is Whereas if you can hit one button and get revert you're gonna be in a much better position Yeah, so another one is just use secure passwords, right? There are tools that can help with this whether it's one password or last pass or keep Sx if you're on Linux And This is kind of the xkcd joke about this which I don't even well It's kind of funny, but really I think just use you know one log in last pass late You just create random passwords, right? This is just something you should be doing it's something you should be doing for yourself It's something you should be doing for your family and it's something you should be doing for the team you work with, right? There are well you go into some slides about like the SSH attacks we see on Pantheon and like we know not like we use passwords We you know we use certificates, but like there are just thousands of attempts like continuously and that you know that goes across the web You also don't want You know when Gizmodo gets hacked you don't want your password there to be the same as your password for like Amazon or something like that So just take you know educate yourself and your team on basic One strategy I use personally for that is I have an Ultra-unsecure password that I give to every site that I don't even care about And that way you know if that thing gets hacked great You've just accessed some site that I'll never access again, and it has no information about me You know layer your passwords use different passwords, and it's all about entropy Don't use you know your wife's name or your dog's name or your birthday or password one two three Use something that has enough entropy in it. That's going to protect you and your your environment Entropy So another one is to factor off so not every service supports this but everyone that does that you use you should be using this You might think it's a little bit of a pain and it is a little bit of a pain But it's much less of a pain than you know like having your stuff hacked or your you know Identity taken or something like that and once you get in the swing of it kind of like everything else It's really easy to use the pantheon we use Yuba keys, which are the one on top left You know Google Authenticator are say off the there's like a bunch of great options Two-factor authentication is something you know and something you have If you ask let's say your mother's maiden name and what was the name of your first pet that's not two factor authentication That's one factor twice So just I want to make sure everyone's clear on that and and just to show the importance of the two-factor authentication Everyone probably knows about the target breach That was due to stolen vendor credentials, right? Do you think? It's very plausible that that break breach could have been avoided had they been using two-factor authentication because those stolen Credentials would have just been like okay now. Where's the pen? This all goes to speak to you're not alone and from a Drupal site your Drupal site is no longer alone You know back five ten years ago your site was just your site It was very self-contained now your site is just a cog in the machine that that you are building whether for your client Or within your your own business You're you're accessing Marketing data you're accessing internal data you have your intranets your Your marketing sites you're going out and you're authenticating to external services Authorized net FedEx, you know PayPal notes squirrel YouTube MailChimp Urella all of these are being accessed And they're all trying to do that securely and you need to make sure that not only is Drupal being secure But you're securely walking around and talking to these other services Think of Drupal's role in the future of the web It will even be more so of that you know the hub Integrating with all these different services, so this will be an important security aspect And you don't want and I'm sure a lot of you guys have had this where Just because one service if you look at the cog if one service is Attacked or one service is breached. You don't want to expose everything else in your system And it goes back into your daily life if you have one website login That's breached then all of a sudden you know you have to go change your gizmodo here and get you have to go through all of your Different sites and change them It's because we no longer live in an isolated web. We we live in a very connected web Cool, so we're gonna we're about halfway through We're gonna go into some tips of securing your stack These are gonna be really high level and not super super kind of action-oriented but just kind of want to give you idea of the you know what you should be thinking about maybe a couple little tips and You know this is both for you and your colleagues how you can think about security So I'm happy to go into more depth and kind of references and stuff with anybody afterwards But I think we'll kind of just haul through these so to me it starts like I'm biased I work at a hosting provider a website management platform But I think that's like really one of the one of the earliest points in the in the life cycle where you start Thinking about security so there's a lot of the stack kind of broken out into a couple parts hosting operating system down on the line, but To me when you're evaluating host hosting hosting shut up security should absolutely be a criterion for evaluating hosting You also want to know what the strengths of your team is, you know If you have a lot of like strong DevOps people people who understand security have that security mindset already You know, maybe you can roll your own a little bit more. That's not a strength of your team You know you might want to look at a host that can kind of compliment you there You don't want to waste it as a dev shop You don't want to waste time doing something that isn't in your wheelhouse. I'm not a host I don't do that. So why should I take on that role and waste my time when I can go be doing better things? That's gonna make me happier my client happier in the product better another another thing to note kind of picture of a typical corporate data center and so a lot of people who work in Kind of bigger orgs and like there is a security team and they might be attempted to be like Oh, well the security team deals with that. Well, that's not the case same thing Only you can educate yourself if you're in this environment You might have a you know some new product something little tiny fluffy marketing brochure wear site But that's in the corporate firewall and this has been an exploit that that people have Utilize before to get into the corporate firewall and get into your entire business through the stupidest thing that nobody cared about But was running in your in your corporate environment. So that's another thing to think about What else is running on that server? What else is running in that network? Securing your OS right so fedora Ubuntu, whatever it is the best thing you can do Similar to Drupal install the security updates, right? There are people like this is how you leverage the smart people that are working on security all the time You just apt-get install like yum upgrade whatever it is Really you want to go for a sensible configuration This is kind of about the integration of risk to you don't need to lock everything down But a lot of the defaults on your server really not be help might not be helping you out that much a Couple things I'd really look into our IP tables SSH, please. Please. Don't use passwords use certificates. It's just gonna be way better and and lock down your suitors So, you know if you like even you know We all maybe get a little little buzz when we log in and we get rude But if you're logging in regularly as rude or logging in and regularly using like pseudo-sue or something You are doing this wrong and you are like leaving a vector open Similarly engine X and Apache this these like your web server, you know you should love your web server and So we are web server is going to be one of the quickest places to add or remove headers lockdown stuff like X-Frame options I think I'll show some slides from some of Pantheon's logging and doing another Session tomorrow around elastic search which we use for logging But logs are super important right if you get hacked Or you sense that there's a tack going on you want to understand how they were able to get in or where they were coming from So also take a keen eye on your web server to your access logs error logs that kind of thing that can go a long way to Kind of understanding what happened and help making you feel a lot less kind of vulnerable in the event of a breach Yeah, a couple other a couple other little tips My sequel is awesome Maria DB is awesome. We all love them Drupal You know Drupal loves My sequel Maria DB as well the defaults also don't help you out right change that route password lock down So it's not accessible from other hosts. This is a case My sequel is great because it's really easy to get started playing with right But you know like that kind of comes at the cost of security a little bit So question the defaults educate yourself There's you know a handful of five things you can do that will help really lock it down user route password route is not a good idea So I'll just talk a little bit about data encryption Currently within Drupal. There is no native way to encrypt However, there are a handful of modules and you can see them up there on the screen The thing you need to be really aware and we touched on it earlier is you need to make sure that that key is taken care of But but luckily there are a lot of modules depending whether you need to encrypt your username your password Form encrypt You know you can come talk to us after after the talk here Towns of security sponsors a lot of these so But leading into encryption key management that is the hard part of Security it's often said that encryption is the hard part of security and encryption key management is the hard part encryption although it doesn't really have to be Because hackers don't break encryption. They find your keys So it's just really a bad idea to have the keys to your kingdom Essentially to your front door and that's what you're doing if you're leaving your keys within the database settings file and fairness Maybe that's like under the front doormat. It's like But it's like, you know, if you did get in you know where right where to look But and there was a lot of people here talking about compliance compliance regulations require Key management and there are standards and best practices for key management defined by NIST which is the National Institute of Standard and Technology and From our perspective key management is fundamental to an events in-depth approach to data security So talking about protecting API keys. We talked about this earlier. Don't share your API keys with your developers One of the things that that I regularly say is if you have a disgruntled employee Do you really want them having access to your MailChimp account that they can send out email as you? These are steps that you can take in all of these you can create You know fake accounts developer accounts and give those API keys out But keep the secret keys to yourself. Keep them secret don't just pass them around to every developer that you have and Don't let your developers share keys either make sure you have enough keys that they can all use It's just like sharing a password So Drupal itself obviously keep it updated With with Drupal Ged and all that that came up last year It those security updates and the security team they know what they're doing and when they publish a security update Take it as as a necessity. Don't just you know look at it saying I don't think I'm gonna do that Greg did a great Madison did a great talk yesterday on Drupal's security more in How to secure your site in your code? Definitely go look that up if you weren't there and go check it out The other thing is you've got Drupal 4 Is coming up here So Drupal core is great and you can follow all the security updates that they have but then also No Drupal site is built just around Drupal core you have all these contributed modules and you're basically grabbing somebody else's code and saying I trust you To come and put code onto my server and into my site So you need to look at the look at the module. Is it active are the people working on it? When was the last update? Who is the maintainer? Do you trust their code? Have you seen them work out of other places? If you see a random module from a random developer that does something that you don't really know don't install it on your site and if you do see something like that you can raise an issue queue about security issues and The security team does look through those Yeah, so I think the the last part of like the technical stack isn't very technical It's actually your team and so thinking about like strong passwords Like it doesn't matter if you have the strongest password But like in the world and it's this long and you feel you know You have to type it out all every time and then like your marketing intern has you know Like no password and that's where they come in right so it's about building that security consciousness not just for you But for your team sometimes you can enforce strong passwords through password complexity or enforce you know to factor off or kind of enforce like kind of you know enforce security, but You also want to couple that with kind of just building awareness and building that consciousness and talking about security with So real quick and we do want to leave some time for questions if there are some hopefully you guys are Thinking about this and have some questions But we want to tell you some real-world examples of where all of this is played out in in our example or in our You know professional lives Yeah, so I think like we all kind of have some more stories and like we we have a bunch and I'm sure Like people in the room have some like you know interesting ways They were hacked or thought they might be getting hacked or like you know attacks They were seeing whether it was some you know a DDoS or kind of you know stuff like a base 64 decody Val that they found in their code base or something so love to talk about that that stuff a little bit pretty fun Just gonna dig into a couple things so Drupal get in and also pointing out that Matt's giving a talk in this room At 5 p.m. And it will be an awesome security talk about Drupal get in specifically But this is like some of our pants from our pantheon logging system when we We were able to kind of isolate, you know the signature of the Drupal get in attacks and And log them and prevent them from having an impact So we saw that within you know several hours of the exploit being published We saw just going from a to z across the platform, you know alphabetically down the domains The attacks coming in and the attacks that weren't on our site that were in somebody's alphabetical list We're like hitting, you know go down here. We are anywhere else, right? So pretty like, you know that gave me like complete faith in that like, you know This is like a targeted attack and you need to get control of it We talked about SSH attacks a little bit so like the internet is a crazy base and people are kind of like jiggling doorknobs and a Tool I like is called fail to ban that locks out people who are attempting But you know if you have servers that are accessible to the internet They are just getting just continuously Attacked and you can look in your SSH logs for that and it's generally like this is why you don't want to use a password You know, it's just that you're just leaving Leaving open a vector where somebody doing something really dumb is able to get in we've seen a couple Targeted kind of HTTP denial of service attacks So these are kind of generally for kind of activists sites sites with kind of a political or social message one of them was around a This band called pussy riot that got in some trouble after making fun of Putin and so we were hosting an activism site for that that was getting slammed and These aren't super sophisticated tax kind of script kitty But all the more so you don't want that kind of script kitty attack to take down your website And a good way to prevent those is put a CDN in front That's going to be taking the brunt of that that attack And so your your server your database and and your lamp stack isn't the one that's taking on the full force of the DDoS Yeah, there's a couple CDNs in the expo hall. I think it's tough So definitely if you do find yourself on the more might be a target side of the spectrum Yeah, I probably want to go here and they can fill you in on how they can help For myself I've actually been handed a database not once but multiple times that had Thousands of credit card CCVs expiration dates I had a client call me up and say hey, there's a site that you didn't build but it's in our system And we totally forgot to tell you about it, but it's storing all of these Credit card and we just had a donor call us and say you just emailed my gmail account with my full credit card number Thank you very much. And so you know, I've had to deal with this multiple times another thing that I Really recommend you doing as a developer is stay in touch with your with your clients I gave the clients the ability to create web forms and then I came back when they were having some issues and they called me up and they said hey, we're having some issues with a Homecoming web form we have I go to it and they're taking in credit card numbers CCV expiration date all that because they wanted to process tickets for their homecoming event and I had to kindly explain to them that you are using this completely wrong and you've just really caused a lot of issues so In what you're doing, especially there's a lot of e-commerce everyone wants to do e-commerce everyone wants to take a payment somewhere online now Do it safe do it securely do it right? Drupal commerce is a great great great platform to build off of But it itself is not just the key you have all these other pieces that build on top of it You can be running Drupal commerce and you can be running it insecurely So make sure you protect your credit card data and part of this is just Notifying your clients and in educating them as well and saying look this isn't a best practice You probably shouldn't just store credit card numbers on a database somewhere and please don't ever send those to me again like if We're time out this earlier and even up until walking away, right? If you don't feel like you are empowered enough to You know to communicate that to your client like that might not be a place You want to be when you get that database of credit cards, and I've actually done that I've walked away from a client because it's just like I can't touch this This is way too vulnerable way too much for me, and I just don't want to expose myself and And in my brand I guess there's my company To the level of insecurity that you're willing to accept so Just be being knowledgeable of it and try to educate your clients as much as you can So risk mitigation I mentioned that's one of the other main reasons aside from compliance that I had a lot of conversations with members of the Drupal community and just the security community at large No one wants to see their names in the headlines for a day to reach. I Think Sony had a hard time with that that was not good, but there's gonna be brand damage There's gonna be loss of customers and loss of jobs. It's not just For a company it's not just having to do credit monitoring for for all of their customers There's just you see that there the stocks are gonna tank and it's just not not a good experience as a whole For businesses, so I mean you just gotta do the right thing and I'll use a case study here we recently partnered with somebody that was Working with a international hotel chain at the hotel chain said we need Someone to build this an internet. We're going to have Employee data in there and for anyone to even bid on this project There has to be encryption processes in place and so luckily that The partner came to us and was like is this something that you guys can help us with because we want to win this bid And we also need to make sure that we can you know not have our name in the headlines for when this site get win Or if the site gets hacked We need we need to protect this data and so luckily we were able to protect protect them But I think it's just a great example of someone being proactive with data security So last we want to leave you with okay. You got hacked Everyone this is their greatest fear. They wake up and all of a sudden you realize this has happened to you It's no longer something that you've learned about it's no longer something that you've read about it is actually happening to you Don't panic React first thing roll back roll back your code roll back your database go back to a place where you know is secure and then once you get the Site secure review the the best thing you can do because again We're all going to get hacked at some point the best thing you can do from it is say Why did I get hacked and how can I prevent that from happening again? And and review that with the community and and reach out to other folks if you have no idea And you're like I got hacked my website got defaced and I have no idea what happened Go find somebody because it's going to happen again unless you know what you did. I know what you did wrong Correct so that's that's a good point is that if there are processes in place that if you're in academia that you have to You cannot immediately roll back just unplug it take it offline do whatever you can If you're not in academia, and you have a just a client-facing site or a brochure site Roll back as fast as possible And and basically the the first step is protect yourself and protect the site Do whatever that means if it means you running down to the server room and just yanking the cord out of the wall Then do that because you don't want to allow it to continue to happen. That's a good point Thank you So that's kind of the end of what we've got here We do have some time for questions if anyone has any questions or comments or war stories or anything like that There's a microphone in the middle if you feel comfortable using it because This is recorded so other people on the the webs can can hear about it. Hi. Hi question So we have a website and there's a lot of Interactivity going on people signing in They put in their information As far as we know they're as secure as we can make them Everything you guys have mentioned we do But there's one caveat When the development life cycle proceeds we need to take that database and bring it down to dev QA and stage In doing so we do a my sequel dump and that means it includes that PII correct What's your strategy in allowing for that SDLC to continue? But mitigating that risk where when you refresh dev QA and stage with a dev from production It precludes PII So from a from a Drupal standpoint what I do is and I do this on multiple reasons one is PII The other is I just don't want to have a dev all of a sudden send an email out We've got a site that's got a couple million emails on it I don't want to all of a sudden blast them with a dev email So there are ways that you can script it so that on that dump or immediately after that dump happens before it Goes back into the other environments that you go through and just scrub all of your data Get it all all out Knoll it out do whatever you have to do and automate that process so that it's not up to somebody to click a button and do that Right, so really quick just to follow up on that So we we thought about that and in the my sequel dump we thought well We could write a script that completely or actually during the my sequel dump will take two dumps right The first one is schema only the second one has ignore tables Correct and but as soon as you have ignore tables and they're emptied now those tables have relationships with other tables That PII once it's empty those relationships are gone now your development environment dev QA and stage are useless Yeah, that's tricky and I think one thing I'll point out is right like Dev in stage are two different things right so you might want to just have staging be kind of a more limited access With more more like live data, you know, you could instead of just ignore tables you could pipe that through a script which You know, which kind of just jumbled up email addresses or you know something that you might consider sufficient But I think that it that is a tricky that is a tricky thing. It's kind of It's a one-off that it depends on the data that you're pushing and pulling if the data can be jumbled And it doesn't break any relationships or if you then have to follow the train and make sure that you do the same Changing to everything then that's something you can do as well So one of the things we do really easy was we just change every domain on the you the email list to an internal That just does nothing So then you just got a bunch of random, you know strings of beginning of email, but you don't have the internals But again, that's going to be very dependent on each environment you're in Building off what the other gentlemen said about you know, he's got to yank the court as soon as he's attacked Instead of rollback, I would argue redeploy on new clean hardware someplace else with a new host and everything Correct isolation is key. And so if you're in an environment like a pantheon or an aqua that has a Easy flow for that and you're not going to redeploy right away. You can roll back you can you can copy the database back to your dev place or your dev site and then Roll back, but again part of the review processes. You have to be able to know what happened if you immediately roll back You're gonna lose some of that information. That's a good one That goes back to like like using version control is so important because if someone was able to get code in that, you know Yeah, version control should have all that state All right, so thanks for the talk. It was great Earlier in the talk it sounded as though you may have alluded to the fact you may Be using some form of honeypots By having other accounts that you just keep out there that you don't use or keep any information And do you in fact use them? If you do, do you have any recommendations and how effective do you think they are? Sorry to say that again kind of The question was around like honey honey pot accounts for like Drupal Drupal users or or kind of SSH or something Yeah, yeah, I Haven't had Specific experience with that. I don't think that's like ultimately that effective. I think that might be kind of fun but but not But not probably a strategy actually to mitigate your risk. I wouldn't in most cases I can think of but One kind of thing that's interesting there and you know I think part of the DevOps track I'm big on the cultural aspects of DevOps and like do what you need to do to convince people like that this stuff is real And they should be thinking about it And if that means like somehow like, you know having a little honeypot or like looking at visualizing the logs for the SSH attack or the just the authorization failures from wash dog or whatever else like Not quite a effective mitigation strategy, but like a cool way to be like look guys This is actually happening like we need to you know, we need to talk about this and think about this and here's you know Here's the visualization of that. So something I might be more suitable for like a high profile site So I use honeypots just as a general, you know, keep the crud out But one of the things you do have to realize is that that honeypot takes processing to to run and there is a Very easily targeted DDoS attack where if you know a form has a heavy processing behind it You can just spam that form and even if the honeypots catching those and not doing anything to your data It can still take your site down. So the honeypots a good good Place to go and it gives you some information But I wouldn't solely rely on that Yes Track the IP addresses that they come from you can start doing bands as well Yeah, that's kind of a cool way is like get people to attack and then be like, okay I just know you're a malicious and I'm not gonna with using the keys over passwords Is there any tools that use for managing that for like SSH or SSH or any any keys like that like management? So within within Drupal, we just Are it's in beta right now. We're hopefully going to be going to RC this this week We have a key module that we're asking the Community to start integrating into some of the more e-commerce and all those logos and stuff We've already got patches for a lot of those out there And basically it centralizes all the key management in Drupal into one spot so that you know where all those keys are part of The problem where right now is is in Drupal we treat those keys You know you're using form API or something like that and you can use the password Field to collect that data and it gives you the nice pretty dots But what happens is that password just gets stored in the plane in the in the database and it's getting stored either in your rules Config or in your variable table So we're we're trying that within a Drupal level to centralize that key management and then from there There's several options available to do that Properly off-site and whatnot. I'll just add to that because that's actually where my company lives is encryption key management So basically how with a lot of these modules that we're sponsoring and that you can use to encrypt all the various data Like we said, it's important to get it outside of the Drupal installation And not only just get it separated, but you need to manage those keys So each time that you read and write it changes those keys So how we do that is just with They used to be caller called hardware security modules Now you can go to the AWS or Azure or wherever launch them And it's just an encryption key management solution and we have a Leave the only integration with all of these Drupal modules So that's that's the way if you look at a lot of compliance requirements It's going to say you need to be using this validated encryption or tips 140 dash to Compliant key management based on industry standards, which those are the industry standards And so you by doing that you're able to meet compliance which goes back to the question We asked earlier about how many people needed to meet compliance This is a way to provably be meeting compliance and part of the updates We did to the encrypt module as well as allow for configurations Before it was always one key one one encryption method and when you change the key all of a sudden all the old data was just No, encrypt now has the ability to support multiple keys and cycle the keys and do proper management there And SSH is easy and I can help you after okay I mean like if you have a developer on your team, and you have several different places where they use that key Do you have a central way to manage that key for the central way to manage the key in terms of like a MailChimp key or anything like that? Yes, say your developer has access to ten different Ten different things and they use the same key I would use different keys so you know you can you can easily spin up a MailChimp We've done this a couple times. We've just been up a fake MailChimp account That just sends emails to us and in reality if you do all of your dev process around that in the last minute switch the key You're totally fine You know so that's that's one of the ways that you can manage that And we're we're also working on some solutions that'll manage that as well Thank you. Hey there. It's actually just more of a comment. I was thinking about the the backup Importance of that so just kind of a Experience that we ran into that's worth sharing is that having backups for more than just like the last week can sometimes be important we With one of our clients who wanted to handle their hosting all on their own They weren't making security updates to the Drupal platform and then they were hacked a month earlier Yeah, and then none of the backups on the platform where they're on Had data that was further back than a week or two weeks And so having a rollback to be able to go back a month, you know, this is generally something I would you know Just put out there is yeah rolling back is not necessarily you're not gonna always know about it that day depending on who your client is So before it became very popular to do in the platform level and with options like node scroll and stuff I actually had a bash script that would take Daily backups and then every week it would take one of those and stored off as the weekly backup And they would keep you know five weeks of backups and then do a monthly backup and then do a yearly backup So that you had a directory that's like I could go all the way back one year and not You know, you don't you don't store on a large number of files You're only storing like ten at the most or ten backups at the most but you can go back Historically through and that's a that's a very good point Cool, I think that's all the time that we have so real quick. We'll answer your question and then we'll run out of here So I have some Drupal accounts on shared holds correct and I was looking at a log yesterday the recent log entries and I'm starting to see a lot of page not founds for things like Wp-admin, which is the word press correct. You have any sense of what's going on and so precautions the take in that so shared environments are very They're cheap and they're easy, but they're also very insecure in that matter because a lot of the time Somebody will just go to the server and then just start spamming everything on the server with logins And then it's just them going around shaking door handles waiting for one to open And so I wouldn't I would log it. I would tell your service provider. Hey, I'm seeing these come in I'm seeing an increase of them But unless they're actually getting at your data or if you have SSH that's insecure anything like that then I think you're gonna be okay. Yeah, we see that constantly It's just people walking around the internet grabbing domain names jiggling the doorknob seeing and if it's they're looking for word press and you're Drupal he'll be alright. Yeah, and it happens all the time on on your on your grid service or anything like that I had I had a Was it the there was a caching module and it would cash all the requests that came in and I looked at the cash log And it was just you know all these random Sites and they were all just ones that were on a shared host with it. So very common. I don't think you have to worry about it Thank you. Yeah, thank you guys I'm running in the first three rows for your close participation. I have a quick present for you