 So, hi, after all of that, I promise I'm more technical than that looked. I'm Bill Weiss, at Bill Weiss Most Places, bvatpuppet.com, where in case you can't tell from the slides and everything, I work with principle security architect there. I'm here to talk about really why security people shouldn't have such a bad attitude about the DevOps movement, and how many of you are in security? Like security is your job, boy, okay, rough room. More importantly, for the rest of you, why you should work with your security teams, not just be like, oh, there are the people over in the corner there who tell me no when I want to do things, they're fine. I have had that job, by the way. I've been that guy, I'm reformed, I'm now telling people I do not do that, hopefully a little faster. So I'm here to talk about DevOps and... We have this problem where people talk about DevOps, and it's just like, okay, we've got developers operations, that's it. Somebody says, well, but we need the QA people in there. QA is important, so we're gonna do DevOps. Security, it's gotta be DevSecOps, or SecDevOps, or DevOpsSec. And so you end up with this, this. So it really is DevOps for everyone, right? We named it after Dev andOps, because I don't know why that's what they wanted at first, but it's a whole company thing. I'm gonna start by talking about why it matters. And we use some data from the 2017 State of DevOps report. This is done by some folks at Puppet, and basically the entirety of Doro, which is the DevOps research and assessment group. They do a survey every year, they get thousands of people to talk about what their environment looks like, and what their practices are, and what their performance is like, or indicators of performance, to see if there's correlation in their rates. This is the big delta between 2016 and 2017. Companies are bucketed. They group very nicely in respondents into three buckets, low, medium, and high performers. Those groups map, it turns out, extremely well onto companies that do DevOps, that do the things we're talking about here. Between the low performers and high performers, they are deploying 46 times more frequently. Last year, that number was 200 times. I think this is just there's more data, and we're getting a little richer, because 200 was, that's a lot. Lead time for changes, which is interesting, 440 times faster between the low and high performers. So that is, you have a change you wanna make, my security hat on, let's say it's OpenSSL patch day, it's Tuesday, and we wanna make that change. In the low performing group, that window is between a week and a month maybe to make that change. Last year it was between one and four months, so we're doing better. Whereas the high performers are under an hour. From proposing a change, test it, have it live. How much better could that be? A lot. Meantime to recovery, when things go wrong, how quickly do you get it back running? 96 times. And then change failure rates, which is really interesting to me with my security hat on again, how often does that OpenSSL patch just mess everything up? Five times lower. So that is, in the high performing group, it is between zero and 15% failure rate, low performing is between 31 and 45%. How likely are you to make changes in your production that aren't super critical if it takes you one to three months and you have a 40% chance of it going poorly and you have to mess with it? And then it takes you a bunch of time to fix it. So those companies have like week long outages because that patch didn't go well. It turns out you can do better than that, probably. And the biggest indicator of if you're in the high performing category, we do not sell this. Oh, I forgot to give my standard warning. I work for a company in this space. I'm not selling our software. This is not about Puppet. This is about the class of software. It turns out continuous delivery is the big indicator here. If you can get to continuous delivery, you are probably in the high performing group or you're well on your way there. Last year, I found that the practice of make up continuous delivery, I'll talk about those, significantly predict deployment pain, that predict IT performance and change failure rates, IT performance, because like we all want to do better as IT, but what does it mean to the business? IT performance maps onto productivity, market share and profitability. Who here would not like their company to be more profitable? Okay, go. You back there would, we'll talk, I don't know. So it is continuous delivery, because that's a buzzword, a lot of people sell it. It is five things, it's test automation. Can the computer tell you if your software works? Did that change work? Is it safe? Deployment automation, once you know the software works, can you then go run it? If you're a services company or you're like a SaaS, that means deploying it to production. If you are, let's say you make Chrome, you make a browser, they're doing continuous delivery in there. If you make a change to Chrome and it's successful, there's a pipeline that ends up with your laptop. Integrating security into delivery. This is one of those indicators of, are you doing security in your CD? Why? Because if you're doing continuous delivery and you don't have security integrated, you're just making things worse, right? Like you're rolling out new bugs while you find out about them or after you find out about them, which sucks. Continuous integration and trunk-based development. There's a whole talk and argument about the trunk-based thing, like continuous integration. Is the system, when you're making changes, is it automatically telling you, did those changes work? And then version control. Most people see that and say, it's 2017, dude. Of course we're doing version control. That survey, people are not. There are companies out there in 2017 who are like, yeah, we've got software and software at all. They're in production. If that's you, I don't mean to shame you. I will help you get there because it will change your life. So that holds like a lot of work. Like what would it take to get there? Takes a lot of effort. I don't know, maybe I don't want to do, I mean, you want to do version control. Nobody says that. Maybe I don't care about continuous integration. I just like running tests manually and that's fine. Companies in high-performing groups spend 50% less time remediating security incidents. 50%, half as much, right? You just doubled the efficacy of your instant response team without having to hire a person over there. You should hire more of those too, but they just get better at their jobs because they can do those things faster. Your engineers, not just the security people, engineering as a whole, spends 21% less time on unplanned work. Right, so that's all that firefighting. For every five people you have on that team, like you get one of them back from firefighting forever. And 44% more time on new work, like actually delivering value. That all sounds great, right? So how do we get there and what does it mean for the security aspect of it? Rack to the DevOps. DevOps, I want to talk about what DevOps is kind of fundamentally. I'm not going to give you the whole spiel. It's CAMS or CALMS. CAMS is the acronym come up with by John Willis and Damon Edwards at DevOps Stays Mountain View in 2010, which stands for Culture Automation Metrics and Sharing. These are the fundamental things of DevOps. Jez Humble the next year suggested adding lean to it because it turns out that the practices of lean software development and lean management matter a lot for this stuff. So culture, what does it mean to build a culture, right? In this case, we're talking about a culture of security. You don't want the security team if you even have one. I should have asked this earlier. How many of you have like a security team in your company? Good, okay, I guess this goes to Jeff. How many of you work in massive companies? How many of those teams are different than like your compliance team? Okay, most people like that, fine. They're probably not a huge part of your company, right? If they are 1% of your headcount, your security team is massive. If you're a 10,000 person company and you have 100 security people, please tell me where you work because I want to talk about that. Probably not, right? Companies of 500 people have one or two. They can't be the only ones that care about this. And the Agile Manifesto, DevOps gets a lot from Agile. We're trying, we want to avoid some of the, well this is gonna be contentious. Some of the failings of Agile in terms of getting super prescriptive. But the Agile Manifesto starts individuals and interactions over processes and tools. This is not a software problem, this is a people problem. Security people, get out of your caves. Like I, it's like I came up as like a hacker kid, right, I like wrote AOL scripts and stuff. And so my idea of what my job was gonna be involved like dark rooms, a lot of techno, and like flashing lights while I was doing things. And the rest of the company thinks that too. Like oh the security team's like over there doing their thing. They even have a dark corner to point out. Your security people aren't there. Security people get in this mindset of like I am here to guard the systems from the users. Like I am the thing between the company and utter disaster. Again there's two of you or there's five of you, you can't. You can't like stand in front of the servers. You can't like watch the packets. You have to get the company involved with, you have to interact with everybody so they know what's going on so they can help you do this. Security can't be the only ones thinking about it, right? Security team needs to be talking about what's going well and what's not. There are rare times you can't do this, right? You, the lawyers say you can't talk about it or the SEC came and had a stern talk with you. Okay, you can't talk about this. It's really handy to just be able to tell people like hey, we had a super bad day. Like I'm gonna show you how we lost a bunch of money. Cause it turns out that when you do that a couple of times people get interested and when you're like hey, guess who found out we lost $5 million last week. I didn't actually, sorry. I can, anyway. But when things go well, right? When you say hey, like we had a, this is one I'll tell, this is a real work thing. I started at Puppet. People heard that I was a security person even though it wasn't his security role there. And our finance people kept forwarding me these wire fraud, wire transfer requests, right? Our finance people were getting like a couple a day very legit looking from our CEO like hey, I'm in China right now, I need you to wire these people money for this partnership. He was not in China. Or he was, but he was not doing that, right? And I said wow, how do you, like some of these look really good. How do you know? And they just went oh, we don't, we have policy we don't do wire transfers here, period. Via email because that's dumb and that's how you lose money. I got up in front of the company and showed them like here's how many of those we get a week, right? Here's some that look super legit. Like who can tell me why this is wrong? And that was a nice reminder for people. Oh, that happens. If I see those maybe I shouldn't just trust that Luke like wants a bunch of money and it is like Cayman Island's account. And really, and this is hard, admit things go wrong, right? Getting up there and saying hey, we had a really bad day. The rest of your company, if you're doing some kind of DevOps-y stuff, whatever that looks like, they know when like the developer's having a bad day, right? You know if builds are failing because there's informational radiators. You know you didn't ship on time. You know you didn't meet your financial goals, whatever. Nobody distrusts those teams because of it, right? When your CFO says, you know, we didn't quite make it work out, you're not like this guy. Never talking to him about money again, no. Says he's doing his job, right? Or is she? It says they're doing their job, right? And so showing people that you're doing a thing reminds them that you're there and that you're doing a thing and that they should talk to you about it. And they're more likely to tell you if they see similar things. We talk about blameless postmortems in DevOps, right? You can do that for things that aren't like the server crashed. We ran out of RAM. You can do it for things like, I don't know why I'm on this wire transfer thing, but we wired a guy $100,000. Bummer. We're not getting it back. By the way, I'm just gonna say, as far as I know, PubBit has never failed that one because our finance team is super awesome at that. They just do not trust email for things that I would. I've got emails where I'm like, I'd call them and ask. Nope, we don't do that. That's culture, automation. Automation, I think security people are really down on automation a lot of ways because it sounds like the computers, which I don't trust, are like doing things to the other computers I don't trust and they're making it like worse, right? Like we're on some software and software is terrible. Like it's all buggy and awful and it's like changing stuff. No. Well, automation, automated stuff is repeatable and audible. Those are words security people like, right? Repeatable is great. We can do it a bunch of times. I can watch it happen and it keeps happening the same way and it's auditable so I know it happened. That is great, right? Those are words I love as a security person. Automated is hands off, right? So a lot of security people, one of their concerns about DevOps is like, well, it means that you're just giving everybody a route. All y'all, here's the root password. Deploy software, do whatever. It can be that. I've been in environments where that was. But it also can mean that things that are happening automatically means no human needs to be in there. Instead of deploying software being like three of our devs are at the bar for three hours, like typing stuff, the software just doesn't. And you know what it does because you've seen it. It's reliable. To go back to that repeatable thing, it keeps happening the same way. If I've seen that Chef cookbook and I know it does the right thing, it just keeps happening over there. If I run another machine, it's in the same state. If it's in the same starting state. AppSec people specifically, because it turns out the whole security industry hates the DevOps thing. I understand why, but I think it's really wrong. If you're an AppSec team and you're like, this sounds really great, let's do automation. Static analysis is amazing. That link there is just a directory of great static analysis tools. And I think there's a problem in the industry of static analysis. There are companies that sell software that does this. I'm not gonna name names. Some of them are really great and they're all super expensive and they're really hard to run. The first time you buy a company of these product, you're just gonna lose your dev team for like a month, like doing that. And so everybody thinks I don't have time for that. Static analysis is like you run some regexes on your code. Right, like you search for stuff. That's static analysis. Specific ones that are great. Again, this was a directory of a bunch of them, but how many of you use closure at work or at home because it's fun? Okay, that's not great. Those of you who raise your hand, who knows what readString does in closure? I know, it sounds like a trick question, right? ReadString. It executes code in that string. If you call readString in closure and try to think from the network, like people just run whatever on your machine. That was a puppet bug last year, which was a bugger. And so it's super easy to just have a thing when you check in code to be like, hey, boss, like that readString there, are you sure about that? Calls to exec or a vowel in anything, ever. Scary. Maybe okay, right? There are places that I love a vowing code, but like in general, it's not a thing you do in production, in general. BearSequel, if you see insert or select or delete all in caps in like a string with some stuff being thrown into it, scary. Might be okay, probably not. If you're a Ruby shop and you are not running a Rubikop, you are doing yourself a disservice. It is rad, it is free, it just works, it's gonna find a bunch of dumb stuff in your code. If you're a Rails shop, Breakman is in the same spot. There's a free and paid Breakman, paid one's great, but the free one. How many of my former coworkers in this? I ran Breakman once on a piece of software that I was supporting, and it just kind of churned for like five minutes and then gave me like, here are your 10,000 problems in this software. Ugh, that didn't work. This is clearly broken software. Breakman doesn't work. My developers, not my developers, but like we are smarter collectively than Breakman. And so I took it over to the devs with this dumb piece of software. Would you look at this? And oh, oh, I have a thing or two to do now. Can you email me that? It's great. How many of you write shell scripts, right? Just like .sh files somewhere in your environment? Thank you. That's where just nobody was raising their hands. There's this thing called shell check, which is this pretty amazing, it like understands the bash and csh and everything, grammars, intimately. It finds terrible things in your shell scripts. And you can just hook it up as a ci thing, it's super. It's written in Haskell, so when you try to compile it the first time, your machine's out to launch for like eight hours, but. But once you get it built or you find a compiled one, it's super. Sometimes, right, some of those things are just gonna be false positives, false positive E, and false positives are really annoying. And so instead of being like, nope, build fail, maybe you just have a robot that sends a, leaves a comment. The GitHub folks, I don't know, five years ago when I saw this talk, GitHub is written using GitHub pull requests and stuff, of course it is. And they have a little robot that just comments on a pull request and says, hey, I see BearSequel on there, or I see Val's, or I see whatever. Would you just find a friend to say that's okay? I'm just a dumb robot, like, I don't know, I see a scary word. And it works great, because nobody wants to be the one who like sees that and is like, nah, it's fine. I wrote that code, it's fine. Now you find a friend to look at it and they either say, it's fine, I don't like that. And you have to treat your security failures as bugs, right, they're bugs. Most chops, most, but not all, as we said previously, check for aggressions, right? You find a bug, you like write a test to make sure you don't do that again. Companies never do this for security stuff, right? You have some like, trivial thing you did that it turns out half the internet was able to run code on your servers, like, for free. People are like, fix that. That ain't gonna happen again. It will. You can also look for that thing in other parts of your code, again, the way you would bugs. If you parse YAML off the network, pro tip, if you don't know this, parsing YAML off the network equals bad day, in general. Because you can have code in there. You're not expecting that, but if in Ruby or Python or whatever you load YAML, you just run code in there. If you're not doing it carefully. The first time you find that bug, maybe take a quick walk through your code and look for other YAML loads. And then occasionally try to load YAML as code in it and see what happens. And then this seems silly, but it's one of those things that again seems super hard. There are automated dependency checkers that you give at your software. It will say, here are all the libraries you use, and here are the problems in it. Here's this one's out of date, this one this, this one this. These will save you from things. They also, basically, how many of you know what version of struts, if you're a Java shop, how many of you know what version of struts you're running? Like really, truly know what version. Because if it's not the latest one, like you're gonna have a bad day. And this software can just like that morning be like, oh, fix that. And it's in that piece of software you haven't touched in three years and so you forgot there's struts in there, hey. Image magic was the example I gave before, but struts, struts are better. These will also do license scanning for you, so if you care about which open source licenses you use, this is super helpful, because if you accidentally include some AGPL code in your stuff that you ship, like your company's gonna be very angry at you, because suddenly you just open source all your stuff. Not all of you work in AppSec, not all of you in security, but not all of you are like shipping sassware. Some of you are shipping real software, I shouldn't say real software like that. Some of you are shipping software that gets put on a disk or gets FTP or whatever. It isn't just a thing you run. And in your case, hopefully you've got automation around all of that. You know what is amazing, is having a way to automatically know when things change. Two jobs ago, I maintained this huge machinery, racks and racks of gear, that all it did was just continually scan all of our IPs to see if it could find things that changed, because we didn't know when people brought up new machines. And so we had to know when they happened to do something with it. What if there was a machine that did that for you and you could just say, hey, run this when you provisioned a new machine. You have a new server, like check to make sure it does the basic stuff you require, right? What open ports are there? What versions of open SSL or what version of SSL does it support, right? If you bring up a new server in 2017 and it does SSL V3 or V2, you should have a strong reason for that. And that strong reason isn't like, oh, it looks that configuration, which happens. Even better, if this is all automated, you can do that before machine's live. You build up a new server, before it gets added to the load balancer, it gets an external IP or whatever. You can just enforce stuff on it before you do that, so you never have that weird window of like, oh, it's live for a little bit, oh, it was vulnerable, we shouldn't have done that. Oh, load average is really high, I wonder why. You can do that before you provision it credentials to the database, it's great. Reproducible provisioning, in general, is really great because you know what's running. And so how many of you have tried to configure like SE Linux or App Armor? Okay, keep your hands up if you didn't give up on it after a bit, because it's terrible. By hand, it's terrible. But by this, you already know what's running, you could just put those profiles there, it'd be fine. Reliable provisioning is also great because all of a sudden, you can replace machines. You've got a machine that's in a weird, hinky state, you don't know what its problem is. Kick it to the curb, right? Take that VM, like, put it in the trash or keep it off in a quarantine to work on, put a new one there, service is still up. And you can automatically reason about these machines. You all of a sudden have code that provisions them, which means you can do things like make assertions in there, watch what they do in a way that you can't if your build instructions are like, Hugh takes this CentOS disk and puts it in the new server and checks the right packages, which one's the right ones, and copies this thing from the server. I spend a lot of time on automation there. I love automation. Lean, let's say about Lean. Continuous improvement is the point, right? And what this means for security teams that a lot of people don't do, is you don't have to build this massive security empire and do everything and make it perfect. You just have to get better most days, right? You need to say, here's the thing we're focusing on, better, better, better, done. And I feel like a lot of security teams always look at this as like, we have to just have these big bang releases of like, here's the policy. Okay, all of the scanning is done. Just start slow and roll it out and measure those results. You can actually see if these things are working and change course. And then, if it's not, kill it. Very few people, especially in security, people in general are willing to get rid of process, right? Because like you built it, it's probably there for reasons. No, kill it. If it's not working and you're measuring how it's working. So cyber kill chain is this term that's kind of military derived that's about hackers. There's not like one point of entry, right? It's not like we had a perfect system, a single thing happened, now we're aquifax or whatever, right? There's a whole line of like, the attacker scanned us and did reconnaissance and did this stuff and exploited it and figured out how to get data back and did a lateral movement. You can use your knowledge of that, hopefully with less terrible terms that sell cybersecurity things. You don't have to do all of it at once. You can focus on kind of anywhere in that thing, right? You can say, you know what? I don't care about reconnaissance right now. Our attackers are probably gonna know where our servers are. All I need to do is keep them from that initial point of entry. Or all I need to do is detect that point of entry and move forward, whatever. Metrics, I think a lot of security teams really fall down on this. And again, I know most of you aren't in security teams. You can help build this for your security teams or if you don't have a security team, you can just build this stuff to do better. I hate to say it, but disasters are really interesting, right? Everybody likes seeing like this went super wrong. Winds are also really interesting if you can talk about why they're a win. And if you're gathering metrics, you suddenly have a story you can tell people about what you're doing and how it's making things better. Figure out whatever's important because companies aren't identical in this and so you may not care about how many compromised AWS accounts do you have a month? Some companies just don't care about that because they don't do AWS or they just don't care about who runs what out there. I don't know. You have to figure out what's important to you, right? How many companies track wire instance fraud attempts? How many of them track tax fraud instances? Some companies care, some don't. And you can make like pretty charts, right? Like, yeah, up to the right, it's great. We're like, going up. Or that chart could be like number of infected machines per month. Like how many viruses we have to clean up? That could be time and instant waits for attention because we don't have enough people to work on it. Right, instance are taking longer and longer to get fixed. You know what will break out some budget for you is that graph in the right place. If you say, yeah, we don't have time to even investigate hacked machines for three months because like that queue keeps growing, that's a big deal. And if you don't have those numbers, you're just the people who are like, we need more people. Okay, all of us do, right? Every team wants more staff. But if you can show this is why I want more staff and that thing's important, it'll matter and it'll happen. Sure, I've been talking a lot about sharing but there's specific things you can do, right? As I said, talk about those things that go wrong. At Puppet for a period, we had a periodic talk at all hands that we called computers are the worst. And it was just a detail like this was really bad. My first one I got to give, which is about two weeks after I started was, let me tell you about the time that somebody checked their AWS credentials into GitHub. This chart that ends in $60,000 is how much that cost us. Please don't do that. Didn't name the person, there was no shame involved. They felt a little shame. I didn't name them. And a number of people came up after me, oh, how did that happen? Like tell me more about this so I don't do that. And that made us demonstrably better by just getting up and saying like that was pretty terrible and it cost, I don't know, like a salary. For the record, Amazon is super friendly and we'll refund you on that sort of thing if you seem like you're competent and care about it. If you already have a company like stats, if you have like a weekly or monthly dashboard and you can get some of your security stats into there, even better, right? Here's our cash flow, here's our sales. Here's how many machines and sales like got hacked during that time. That number gets relevant quickly and gets people involved. I'm a big fan of running lunch and learns or something like that where you just say, hey, who wants to learn about this thing? If your budget allows it, buy people lunch, if not, make them bring their own. So these can be literally free for you except for your lunchtime. If you have appsec teams, some of the best bang for your buck you can get in training is run these and run like CTFs or talk about like, let's actually all break in to this terrible PHP app I wrote. Because it turns out once you do that, the developers start going, oh, I wonder if that's a thing we would do. Right, you teach about cross-site scripting and it's like, oh, we don't really do anything with HTML input there. You go to multipliers. Suddenly that whole team cares about security and is doing security stuff for you. It's great. If you're in a sec ops group so you're not teaching CTFs, you still should because they're fun. Instead you could do these about fraud attempts or about times you've actually been hacked or stuff like that. You could run a pretty good 30 minute lunch and learn talking about what wire transfer fraud looks like and how you detect it and how you avoid it. And I guarantee that that will be worth your time because it will save you real money. And this comes down to as a security team you need to seem human. You can't be the trolls in the corner that are like, no, you do not get to get that. I turned your server off. No, because then people don't want to talk to you because you're the jerk that says no. Instead you want to be like a friendly security team that would love to help with things. And then going back to that blameless post-mortems thing, people really want, people are more likely to talk to you if you do not get up and shame them. I did not get up there and say, hey, Steve there made a mistake with his GitHub credentials, his name's not Steve. And he cost us 60 grand, good job buddy. No, we just talked about what happened and said hey, accidents happen, we're fine. And tabletop exercises. These are super important, they're really valuable. You probably can't, especially if you're a small team, so afford doing like huge disaster readiness drills or like actually bringing stuff down or whatever. You can just do like an hour long, what would we do with this? There's this Twitter account, bad things daily that is like pure distilled nightmare fuel for ops people. They just tweet once a day a scenario that you could worry about and they all start with like, we would all cry. Right, like that's just, but then what? It's super interesting, you should follow it, it's great. But it also shows, it doesn't just train those things, it also trains your, it also shows people that you are there and you're thinking about these things and gets them used to dealing with incidents which you will have. Am I like super over time? Nobody's, oh okay, I got them. All right, I'll wrap it up. So if you're an app sec and would you like, what would it take for you to want fewer security bugs in production? Like you would pay good money for that, right? That sounds great. Fewer security regressions. Just we make the same mistakes fewer times. Faster mediation, right? When we make mistakes we get there faster and we fix them and we know they are fixed faster. SecOps, you're ops people, same thing. Would you like faster mediation? Would you like fewer failures, right? Those numbers show that if you do these things and if you're doing the DevOps, you're just gonna fail less often which means your stuff's up and you can do the fun things more, that's great. More confident changes, right? How would you like to just worry less about rolling out those patches which you should be doing but right now are holding off because they're gonna be terrible? Would you like more confidence, what's running where? So you actually have an idea of what should be happening? Well, that's what DevOps is for, so we can learn here. Thank you, my dogs.