 Welcome to the last talk on the last day. Aww. But that's okay, because we've got Tom. Aww. Tom. No. Okay. That's all the heckling you get. Come on. Let's take it down. So Tom Eastman is formerly of Catalyst and of Seistak as a lead security consultant type person. He's really scary and knows a lot of things, so let him scare you in a good way. Hi. Apparently, I am that scary. I don't know why. I'm not a security person. I'm a Python dev. I'm a Django user. I've been using Python for 15 years. Django for close to 10, and going to be talking about ways to try and make things a little bit less scary. But I'm going to do that by leading us down a bit of a thought experiment, which might not be fun for everybody. Who in this room works on web applications, either on the public internet or as a private internet thing or both, or pretty much everybody, because otherwise you'd be in there anyway because you wouldn't care. So the hypothetical question that I'm going to give you is, you get back from KiwiPycon and you are told that while you were here, your website was hacked. Your application was hacked. On a scale of one, two, many, and lots, how panicked are you? Okay. So what if you found out when you got home from KiwiPycon that you were hacked six months ago and only just found out? The central IT department, the assistant admins called you and said, hey, why have you been sending all this traffic to Romania for the last six months? I get a lot of panicked looks when I ask these questions to people. Usually that's because it's usually, usually they're scared because they haven't really thought about it beyond a sort of, oh, please, just don't let it happen to me. There is actually a correct answer to this question. Does anyone want to guess what the correct answer to this question is? I'm not sure if that was totally a guess, but why is this the right answer? It's the right answer because, first of all, it's not panicked. And it shows that you've at least thought about this. You've at least gone down that thought experiment. Imagining how you'll get hacked is how you protect yourself from being hacked because you actually have to be thinking about all the worst case scenarios in order to even work out how you're going to protect yourself from them. So that's what we're going to do today. We're going to take an imaginary web stack and we're going to just sort of start seeing how far down this rabbit hole we can go. We'll start thinking about the horrible things that can happen to you in your day job. The most important thing to realize is that about security is that it's not any one thing. There's no single security mechanism so clever that it stops all attacks from all attackers except maybe unplugging the server, melting it down into a slag, and dropping it into the Marianas trench. And even then the attacker will find the backup of your database that you left in your home directory on your desktop machine. Yeah, there's more than one kind of attack and so you have to think about more than one kind of defense. A secure system relies on lots of defenses in lots of places. You have to start to think about who the bad guys are and what their intentions are and what their motivations might be. When you do this there's usually kind of a structured mechanism to it. You find yourself thinking about classes of attackers in terms of what their motivations are, what their resources might be, and their potential skill level. So let's just play this thought experiment for a little bit. Opportunistic script kiddies. These are the people who are probably trolling the internet for low hanging fruit. They're looking for unpatched systems that have zero days already installed on nice user friendly, click it to hack it, things like Metaspoit. They're very unlikely to use financial resources when they're just casting their net wide, but they're dangerous because they have all the time in the world. And there's effectively an inexhaustible number of them. They're in it for the lows. So you might sit there and think to yourself, they're not going to attack me. Why would they attack me? I'm nobody. And the answer to that question is if they can get in, that's the reward. It's not like they cared about who you were. Organized criminals, these people are usually out for some sort of specific financial gain. They're generally very skilled. They have financial resources and they're organized to at least spend some of that on getting into your system. They often have very specific motivation, which is also for financial gain. So if there's no financial motive to breaking into your blog, then it's possible that, why are you two dancing back there? That's strange. On the other hand, they could just be out to dossier site for blackmail or extortion purposes. Discounted former employees. This is the one that people don't think about enough and they need to think about the most. Nothing at all. But there is a really important point. Oh yeah, I used to work for you. Hi, Nick. Actually, how many people in this room did I use to work for? Couple of you. You need to realize that disgruntled former employees can happen even if you're a lovely place to work. Because you can't actually control the mindset of your employees or your former employees. They have intimate knowledge of the system. They probably still have credentials on your system unless you've got really good policy about the exit plans for people. And their motivation is right there. And then you might have people who might want to make a political statement out of attacking you. And then you might also have nation state actors. This has been in the public consciousness for a while. So I wanted to throw it in here. They basically have infinite resources and they probably don't have a lot of motivation. I mean, if nation state actors really do have motivation to get into your system, you probably have bigger problems and you probably already thought about this stuff, hopefully. What I just went through are a bunch of security personas. And I recommend you do this when you're thinking about your site. You should actually make up little people. If you use agile development methodologies, you're probably used to the idea of personas and user experience design and stuff like that. Do that for bad people too. Make personas for the people who might be trying to attack you and use those when you're sort of gauging the interactions with your site. Just a very brief example of this sort of thing. I spent a couple of years working on a tool for the Ministry of Education and it was a tool for teachers. But the attitude towards the tool was quite polarized. A lot of people were very for it and a lot of people were very against it. There could be reasons to attack the site because embarrassing a government IT project is very in vogue in New Zealand. It's something that you get a lot of free press out of. But the number one threat to this system after all of our sort of analysis really just consisted of a student leaning over the teacher's computer and the teacher walks out of the door. So you have to know sort of what the likely threats are and what the likely things are that can happen. I'm going to recite the definition of attack surface directly. This is straight from Wikipedia. The attack surface of a software environment is the sum of the different points, the attack vectors, where an unauthorized user can try to enter to or extract data from an environment. That's a bit tedious but it's just a point of saying basically every moving part and every exposed bit of your system is a potential point of abuse or attack or throwing random data at, see what happens. What I want to do is dig through a web stack. So your web stack is made out of layers of different technologies. You don't have to think about protecting only the outermost layer it's devastating if you limit yourself to that. There's more places where they can get in. Sometimes they can skip one of these and go straight to another. At almost every layer of the application stack you can perform the same thought experiment. You can say how might this layer of the stack get compromised? What are the potential consequences? In what ways can you reduce the attack surface there? And in the event of a compromise how can the consequences be mitigated if at all? Or at least what would your plan of action be if it had happened? Summarized as hindsight. Find a way of doing your hindsight ahead of time. I don't know what they call that. I think there's a P word for it. Planning. Sorry, I'm rushing really fast because this is a 45 minute talk and we have a 30 minute slot. So moving on. Your web server is probably some combination of Apache or Nginx. Generally speaking. Both have very large communities behind them. Both are mature and well tested pieces of software. But it's also obviously the most exposed part of your web app. Pretty much by definition. It'll be hit by everything. Any bugs discovered in either Apache or Nginx will be automated and exploited quickly. That's just the fact of life. And it'll be anything that happens on the internet will happen to your server at some point. SSL bugs, like the ones that we've seen in the last year, Heartbleed and Poodle and stuff like that. They're the sorts of things that will end up always being tried against your servers. So how do you minimize the attack surface of your web server? You start by disabling everything you're not using. In my experience, that's actually a lot easier with Nginx because it tends to start with a minimal configuration anyway. Apache kind of starts with a whole lot of the modules that it thinks you're going to like to use. And these days, you probably don't use those. You probably don't use ModCGI anymore. You probably don't use... You probably do use... Well, if you're running a Python stack, you probably don't use Mod Rewrite except for a couple of things. Actually, no, you do because SSL redirects. Sorry, I'm making stuff up. One of the best things you can do with configuration is delete big chunks of it. SSL configuration is hard. Jeremy went through this in his talk. And this tool... If you're not a crypto expert, or if you're just someone who actually has a life and needs to get their job done, the myriad of SSL configuration options and things are just going to be overwhelming to you. So get some best practices advice. And this is a really good place to start. Really experience this admins that I know have a few complaints about this thing. They have like, oh, I would change that cipher. I don't really like that one. But they're the experts. And if you're not, lean on experts. So let's say... That's minimizing the attack service. But what if someone got in? This is what we want to do. We want to think about what will you wish you had done? Or if it has happened, what can you do? One of the best things that you should have done is keep the web servers separate from the app servers. If you have any kind of budget at all for this, and you're not just running it on your Raspberry Pi, separating out either in virtual machines or actual hardware or whatever, the web servers from the application servers is a really valuable thing to do. And that's one of the main reasons why I'm moving away from ModWSGI and all my things. ModWSGI for deploying Python apps is really mature and really stable and really good. But it does couple your application server to your web server in a way that probably just ain't that good an idea anymore. But I gave this talk at PyConAU and Graham Dumpleton, the author of ModWSGI, pointed out that he's working on a system that might flip that on its head. ModWSGI Express, I don't think it's ready yet, but have a look at that if you are using ModWSGI and are interested in what options you have to make things safer. Operating system lockdown is important as well. Who here uses Ubuntu a lot? Cool, I'm basically talking to an Ubuntu crowd. Learn more about App Armor. App Armor or SE Linux, if you are using Red Hat, provide a lot of operating system level protections to prevent code exec, for example, on a web server from being devastating. Because they won't be able to shell out and create a bash shell if your App Armor rule says, oh, by the way, Apache doesn't need to run bash. So how about we just don't let it? It's a really, really good layer of internal protection to prevent that initial compromise of potentially bad unpatched Apache code to get to the operating system level. I'm gonna skip some stuff. Oh, I have things, cool. A lot of people skip this step, but why does your web server need to talk to the outside world? Don't let it. Firewall off everything. Your web server shouldn't be able to talk to anything except the server that it gets its security updates from. Why would it be able to talk to anything else? Why would it be able to initiate connections to anywhere else? Okay, so let's talk about your Django server, or your modWGI server, your application server, or whatever. Because application servers are usually written in a high-level programming language like Python, you generally don't have to worry about buffer overrun C-style attacks like you do with Nginx or Apache, but if something does happen, an attacker could achieve arbitrary code execution on your application server by compromising any system commands that your application server runs if you're shelling out to anything as a part of your application server. You're using some magic tools to resize images and you're passing it command-line arguments and those command-line arguments aren't fully sanitized. But the more likely direct attacks against your application server code probably involve compromised login credentials. They probably involve someone getting a level of privilege in your application that they shouldn't have otherwise gotten. Stealing the accounts for the Django admin user, right? Or there could be an indirect attack on your application using cross-site scripting and we'll talk about that if we have a little more time. How do you mitigate this? You need to make sure your application server can't do any more damage than it needs to be able to do if you suddenly lose control of what it's doing. And to be honest, that brings you back to AppArmor profiles again. It's a really good idea to just read up on those and make sure that your green unicorn Django server doesn't need to be able to read files from slash dev by and large. Actually, locking down... I'm talking up AppArmor and I still recommend you do it, but AppArmor for locking down Python things is a little bit of a puzzle game because Python tries to do a lot of things that maybe your application server doesn't need to do, but it's still worth the time. Decent things to do include making sure it's running as a user who doesn't have the right to do anything else. Your application probably needs to talk to the database over a network. It probably needs to talk to the memcached layer or maybe Redis or whatever. Does it need to be able to write to any file on your system? Not for logging. Use a logging mechanism for that. Use this log for that. You can pretty easily write a web app that does not actually need write access to any file on the operating system. And then that's just one step away from having some kind of alert happen if it tries to write one file on your operating system and you go, hey, I didn't expect it to be able to do that. Credential theft attacks, that's a lot more dependent on your application setup, but IP address restriction on admin accounts is a really good start. If it suffers a service or something that you don't need to provide admin people to have access to, two-factor authentication is a wonderful idea. Having some extra restriction on there so that just plain old credential theft won't happen. What are the consequences if your application server does get hacked? It's not pretty. You have to think about these things, but it's not pretty. You probably have to assume that all of your data stores, your database, has probably been fully compromised because generally speaking with one of these web apps, you've got full database create, read, write, access. Customer information may be lost or tampered with. Incident response and forensics will become critical because you'll need to as much as possible determine the timeline and the extent of the breach because at this point it very much is getting in touch with your customers, explaining them what happened, telling them to change their password, but also offering free credit checks, blah, blah, blah, blah, Ashley Madison, topical stuff. Moving on, we have a lot of stack to get through. It gets uglier and uglier and uglier. The database server sits behind the application layer. You'd like to think that they couldn't get to that unless they had already compromised the application layer. Who here is familiar with the concept of SQL injection just to get a rough, excellent. Why are you guys even here? You already know how to be secure. If there's an SQL injection vulnerability in your application code, then it might be possible to compromise the database without requiring a compromise of the application server code. That's the risk, right? Any kind of SQL injection vulnerability, they just bypass all your protections and have full access to the database. The database could be exposed through lateral movement inside your network, or the database backups could be compromised. I won't ask for hands up on this one, but how many people, just, you know, grimace to yourselves if you have a database dump of your system sitting in one of your home directories on one of your staging servers? Cool, I saw a couple grimaces. Oh, yep, it's a thing. I probably, yep, I'll stop talking. I have a microphone. I have to be careful what I say. Database backups could be compromised, but yeah, much more likely is a compromise could happen on a development staging server that's using a copy of production data. If you use production data, you load it into your dev database. Your dev database is sitting on your laptop because you are a remote worker or you are traveling to a conference right now. Grimace to yourselves quietly if you have a copy of production database sitting on the laptop that you brought to KiwiPicon. You don't need to put your hands up. There's not a lot you can do if they get as far as SQL injection, but there's absolutely no excuse for them to be able to get to SQL injection. SQL injection is the number one thing on the OWASP top ten in terms of security vulnerabilities in the world, and it is also one of the most preventable. The techniques to prevent it have been spelled out for people. If you didn't put your hand up and you don't know what I'm talking about with SQL injection, Google it, find the OWASP wiki and read up on it a little bit. It won't take you long to learn what the problem is and you'll find... Yeah, it is very fixable and at this point there's no excuse for it to still be in your web app. Find a way to not use production data on your dev environments. Use test data or whatever. You probably have to use production data on your pre-prod if you have that sort of setup, but don't let it spread and here's the thing that everyone forgets to do. Audit where it has spread to. Just somewhere in your process of these systems are the ones that have production data. If you know it, you cannot protect it if you don't even know where it is. You cannot protect production data if you don't even know which of your employees have it on their laptop. How can you? That laptop gets stolen at a conference. Stay away. Yeah, know where your database dumps live. That's actually really, really, really easy. One of the consequences, the data's been breached. Actually, Madison, blah, blah, topical, topical. It can be a little bit embarrassing is what I think I'm trying to say, but it can be an awful lot more embarrassing because you probably don't even know it's happened. I am going to skip an entire section of this, but I love talking about cross-site scripting and I love talking about content security policy, so bug me afterwards because, and if you don't know what content security policy is, put that on your list of things to Google after this talk. Content security policy, I'll just buzz through that really quickly. Content security policy is a web browser feature that an extra header goes into your response and it will tell you, it tells your browser that only certain JavaScript is allowed to be run on this web browser from this site. So you can say only the JavaScript that is from my CDN is allowed to be run on this web page. And not only that, it can also report if there's JavaScript embedded in a place where it shouldn't be. So your web browser can become a warning mechanism if someone has executed a cross-site scripting attack on your site. It's only a mitigation, not a cure because only the most modern browsers support it. But there's a fantastic feature, which I just mentioned, which is CSP violation reports. I am skipping through, I do apologize about that. Web browsers that support content security policy also have a reporting mechanism where you can tell that browser, your server can tell that browser, hey, if you see JavaScript on my site that shouldn't be there, send a post to this URL and report where it was. So suddenly your user bases up-to-date browsers become your line of defense against cross-site scripting attacks that could affect the older browsers. The older browsers can't protect themselves using content security policy, but using content security policy can benefit everybody nonetheless. So that was the easy stuff, right? All you have to do is freak out about every part of your infrastructure become paranoid and live in a hole. All too easy. What if someone gets your infrastructure as a service keys? I heard that. What happens when someone gets your AWS keys? Who has ever accidentally committed AWS keys to a GitHub repository? You can grimace or put your hands up. So it turns out that this happens all the time. Automated scanners are constantly searching public repositories like GitHub Stack Overflow and Pastein-like sites, looking for strings like Begin Private Key and AWS Key and stuff like that and AWS Secret. There are 100 different things. They'll find them and they'll use them right away. And the good news is that when that happens, usually all that will happen is they'll spin up 100 machines on your AWS credentials and just launch a million Bitcoin miners. Because they don't actually know you or care about you. That's the joy of the automated tools. But if someone who actually doesn't like you, like disgruntled former employee or just anyone who's targeting the hold of those keys, what are the consequences? What was your entire data center in AWS? Is it all in elastic Lambda-e services? Is it all in Route 53 or whatever the AWS product is? Are your backups all in the same place? Are they all backed up to S3? Do you have just a now's the time we wind down the company? Does it work? So, AWS, at least, lets you make lots of sub-keys with limited privileges. And you should be using really, really strong paranoid two-factor plus password-blocked-in envelope for the master keys and then have as little privilege as possible for the keys that are actually managing infrastructure or do your deployments happen from laptops that are here at the conference? I promise you I'm not earmarking laptops to look at afterwards. But, yeah, please, for me, just don't have all your eggs in one basket. Have a backup of all your stuff buried in a vault somewhere or maybe on your laptop, but maybe be careful with that laptop. I don't know. Maybe I'm just teaching you to be really, really careful about laptops. Okay. Final topic. What will I wish I had done when I had the chance? This is the thing that you should always be thinking. If something horrible happened to this part of the system, what is it that I really wish that I had done? What is it that I will have wished that I had done? You'll wish you had better logging. Finding out you've been compromised is a horrible thing to have to deal with, but finding out you were compromised six months ago, how far back does your log rotate let you keep stuff? If you're using log rotate, that, frankly, implies that you're using log rotate the files in viral log on those servers, but if those servers were compromised, you can't trust the logs anymore. Auditing is, auditing and is the key piece of incident response. You have to know as best you can what they took and when. And if you've been watching the Ashley Madison stuff unfold, one particular thing that seems clear is they just don't even know. They didn't even know. They denied that it had been breached. When it turned out it had been breached. They clearly could not even audit what might have been happening. Who does that? I want all of you to assume that your backups have not worked since last time you tested a full restore. Your backups have not worked since last time you tested a full restore. I want you to internalize that. Unless you've been doing full test restores, assume that your backups don't actually even work. Assume that they have not worked since the last time you did that. When was that? Never, probably. Modern cloud orchestration has had at least one benefit where a lot of the machines nowadays are ephemeral. You destroy them and bring them back up. And that's kind of helped with the repeatable infrastructure aspect of that at least helps the backup situation a little bit because you don't have to worry about all the files in certain places. To prevent the worst, you actually have to imagine it and plan for it. Thank you very much. We don't actually have any time for questions. If everyone could make their way over to track one like now because we've got closing and maybe maybe I should check that first. One moment. Sorry, the full version of this talk where I do cover the stuff I had to skip for time constraints is available on YouTube. I gave the talk at PiconAU. So by all means feel free to check that out. They're not quite done over there yet, but I just said what Jen told me to say. We can do questions if we want to be mean. Well, okay, don't be mean. It can be a little mean. Questions? Okay, sorry. The question was if does content security policy prevent you from using inline JavaScript? The answer is the default configuration for content security policy does block all inline JavaScript. And that is its single best feature, but what that means is that you have to plan for it. So back porting content security policy onto existing systems often means you have to turn that off at first and slowly refactor possibly quite a lot of your code to strip out all of the inline in your pages. Every green fields project, every new project you start you should have content security policy enabled with a strict policy from the start and that means that you will put all of your JavaScript and JavaScript files and you'll be far, far safer that way. So yes, that's the biggest sacrifice that you have to make and it can make refactoring it a bit of a pain, but totally worth it. I'm not familiar with that. It's been mentioned to me before. He mentioned Google Tag Manager. All I can tell you is that because content security policy is becoming more popular, it's becoming libraries and JavaScript libraries are catching up and making sure that they're compatible with it by and large. So for example, when I was first using content security policy a lot of stuff in the Django admin section of my site broke because some of those little things are inline JavaScript, but these days it works well. There's one question up the back. When I first tried to use that feature, I pointed it at a page that had like a thousand little bits of inline JavaScript and this was an earlier version of Mozilla where it was still kind of provisional and it tried to send a post for every single violation in the page. That was less than ideal. You tell it which endpoint to go to and so you probably want to make sure that that endpoint at least will be able to handle whatever is coming into it. It's probably safer than that nowadays, that's for sure. I have a question. Yeah. If you've got an infinitely sprawling VALOG and you're not supposed to put it in S3 or Glacier, where are you supposed to store all your server logs? I'm not familiar with the current state of play software as a service log-ketcher type things. I imagine they exist but mainly what I'm thinking is if you most of my deployments have been you build like 10 VMs and you do some stuff rather than fully sort of Amazon type deployments and a separate logging server which uses separate SSH keys to get into that is basically just harder to lateral move into is probably a really good start. What is it called? Logly. That sounds promising. But as long as if you're contemplating the prospect of the application servers or the web servers or the database servers getting hacked or getting compromised their logs should go somewhere else and the log server can be kind of a black hole if you are careful with its SSH credentials, they're not just sitting on your laptop here at Kiwi Picon then it can receive stuff, it can write stuff to disk, that stuff can go to some sort of tertiary storage system, nice thing about logs on tape is that they can't be tampered with. So maybe, yeah, there's bound to be a million in one way of doing it but the main thing is just to make sure that a compromise of this end doesn't give you the logging server for free. I would imagine using something like syslog to live stream logs, yeah. Yeah, as opposed to just SCPing them across, yeah. We're pretty much out of time now, so if everyone can give another round of applause to Tom. Given the time pretty much