 Hello, everyone. So I'm Pete Sheslock, and this is a talk that I wrote when I worked for another company that I won't name, because I don't work there anymore. But I think it's a relevant talk, which is why I like giving it. And it's about some of the specific things that we did and we built at this previous company and that I've done at other companies to really enable this concept of self-service operations, DevOps, for lack of a better term, from scratch. When you start from nothing, where do you start? What do you focus on? We're going to talk a little bit about that. For those of you that don't know me, I am Pete Sheslock. I work for a company called Chaos Search. We're actually sponsoring here. You should swing by. I usually like to answer the question, but I'll do it with a show of hands. So just keep your hand up if you keep answering yesterday's questions. How many people use AWS? Just raise your hand. Cool. Leave your hand up if you have data on S3. Cool, everyone. Leave your hand up if you run Elasticsearch and or the Elk stack. OK, cool. So leave your hand up if you spend too much money for not enough data on all those places. So if you're still raising your hand, you should come find us over there. We have some cool technology that allows you to query your data on S3 using Elasticsearch. It's why I actually left my previous company to join. I was really excited about that. One of the other things I like to talk about is I'm the founder of a meetup group called BossOps. We meet as often as I am not lazy enough to schedule them. There is a meetup group. There is a pretty active Slack channel. So we're probably going to do another meetup here in maybe October or November. So hop on the meetup group joint. It's a networking event only. I'm trying to get people to meet more of their peers where we can get together and commiserate about computers. One of my interesting changes, though, is I made a change. I'm actually now the VP of products for Chaos Search. I used to run technical operations for many years. And you're like, why is there this product guy up here talking about DevOps? Like, what does he know about this stuff? And I'll tell you, I'll be the first one to tell you I don't know a lot about it. But I like to describe a little bit about the change because I had PagerDuty on my phone for eight years. And I got to delete it last month. And that was pretty awesome. It actually feels better than you can even possibly imagine. But I'm not a millennial, but I feel like I'm a millennial by heart. So I like to use Bitmoji to describe what it's like in operations. Is that showing right? Yeah, so that was kind of my life in operations for the past many years. But in product management, it's a little bit different. It's a little bit more like this. So you've probably heard of before, there's this concept. Pick any two good, is that going to animate? Sure does, cool. I spent way too much time on that GIF for it to not work. But you hear that you can't have things that are good. They're fast. They're safe. You can't have all three because the time it would take to do one takes away from the other ones. That in order to move fast, you have to undertake risky behavior. Or to build something that's really good and really secure that you have to move slowly. I just don't think that's the case anymore. I think you can have them all. And for those of you that have seen any of my talks, there's always a unicorn slide. Sometimes it's pooping on things. This one's not. We're going to be a little bit cleaner here. But the unicorn lives in the middle, where you can actually have all of these things. You can run your systems at scale. You can do so safely and quickly. And that's what we're going to talk about. So I like the level set before any talk, which is what is DevOps, right? Is it a title? Is it a whatever? And there's some noted thought leaders in the audience. So I always like to make sure that even we level set with them as well and set this consistent level. So is DevOps a tool that you can use? You can absolutely buy DevOps tools. They're out there. You can spend money on them. You can hire consultants to give you the DevOps. Maybe it's some software that you can buy. But before you buy that software, make sure it is concurrent DevOps. Don't come into my house with that single-threaded DevOps nonsense, no, no. You've got 16 cores on that server, and you're only using one of them. So is it a job title, right? This is a search I did from quite a while ago that's still pretty relevant. How many DevOps engineers are out there? And let's be honest, if you're not called a DevOps engineer, you're crazy. Because we already know that that title elicits another $20,000 to $30,000 a year in salary versus sysadmin. So I got no problem with people calling themselves DevOps because people should get paid. But no matter what, don't be a director of DevOps, not speaking from experience. I definitely never, ever called myself a director of DevOps. But you can't actually get worse than that one. Is Tom McLaughlin in here somewhere? I feel like if there's a serverless joke, I've got to have Tom, but it could be worse. It could be as bad as this one. But is DevOps a team, right? How many people are on a DevOps team? You could just call that the ops team, but that's cool. It's become a qualifier for sysadmins that can code and work with cloud services. A lot of times, the sysadmin world has transitioned very fast, and this term DevOps came along to qualify it. So if it's not a team, software, what is it, right? It's anything your heart desires. DevOps can be whatever you want it to be. That's the beauty of this world. It's like the Zombo.com. That's what DevOps is. You can do anything at Zombo.com. Anything, the only limit is yourself. How many people here have no idea what Zombo.com is? You are all going to have an amazing day today, and here's why. You're going to go to Zombo.com's website from the last bubble back in the 2000s. It was written in Flash, which doesn't work anymore. Someone wrote it into HTML5. You want to turn your audio up. Trust me. Just listen. Take it in. Soak it up. It's great. It's a treasure of the early odds that we all should hold on to. So when you think of that, every time the guy says Zombo, think DevOps. Enjoy. So let's talk about the bad old days. How many people here have been in technical operations, computers for, let's say, more than 10 to 15 years? So probably have walked into a data center, know what a data center is. That's usually the big one. How many people here can remember this world? This is my favorite tweet. Taking Twitter down for a little nap. Message and site delivery back in the four hours. If you look at the date there, it's November of 2007. Sometimes I wish it was now. Just turn Twitter off for four hours, I think it would make us all happier. But obviously, they don't do this in this day and age. Granted, what do they do instead? Most sites, you just go down because you crash. You obviously don't want to do that. But we don't just say, hey, we're turning everything off. That's ridiculous. Now, Twitter's not the only company this happens to. This was literally two months ago. I was on JetBlue's site. And it's like, oh, sad face. You can't book your flight you need. But they were like, yeah, we're just doing some maintenance. OK, cool. Except it was 2 o'clock in the afternoon on a weekday. So thanks. But no matter what, just how bad you think you are or other companies are doing, the US government has just one thing to say. Hold my beer. This is like a Shakespearean drama here. Let's look at a few things. I'm going to burn some slides later to pay for this one, which is look at the end date, obviously, December 31, 1999. But then it's like the time range. But then it's like on Thursday, September 22, 2016. But the planned out is just from 2018. Yeah. So let's talk about the battle days of software. So back in the battle days, devs would rarely have access to machines or production machines, any machines, which means the ops team would be responsible for taking the code and installing it based on devs' instructions. And I'm sorry if there's any devs in the audience, but everyone's bad at documentation, including devs. So let's just say those install instructions were not usually pretty accurate. But how many changes were comprised in that software that was going out? Was it days, weeks, months? Was it a waterfall-type project? Was it Agile, which is just a lot of series of little waterfalls together? When we had to burn data to a CD and shift those CD to customers, how do you get new features out there, right? Service pack one, service pack two. In the infrastructure time, the lead time could be measured in weeks, months to get servers in your infrastructure. And the code could be ready way before any of those servers were ready to go. So you had these huge, long feedback loops between code being written and code being ran on systems. And granted, those systems at the time were lovingly bespoke created, artistically designed systems. And I know there's someone out there that's like, now back in the aughts, I use BFCG2 or CF Engine 1 or whatever. It's like, I know. But no one else did. Everyone else had a hunk of bash that they were using to set up their servers if they were lucky. But the big part of buying servers in advance, which is why Cloud got so impressive, was the servers were purchased, they were racked and stacked. And maybe the code wasn't ready yet. And they're just sitting there unused. And now you're just burning money. So there's been a lot of great conference talks around DevOps and things. I call this kind of the seminal talk for Velocity 2009. And this was where this concept of DevOps, they even called it Agile Infrastructure, came about. But this would really set it off for me personally when I watched this talk. Because here's a couple of people that came on stage. They were at Flickr at the time, poor one out for Flickr, who basically were deploying 10 times a day. And everyone here is like, 10 times a day, whatever, man. But 2009, everyone was like, no, you're not. You're so full of it. There's no way that you deploy 10 times a day. I'm lucky if I deploy 10 times a year. We couldn't believe it, right? And so they had this traditional thinking from their talk, which was it is the dev's job to add new features. And it's ops to keep the site stable and fast. That's the traditional thinking, which we now know is wrong. And you see this slide in a lot of these types of talks where you either want change or you want stability. And people are fighting because the incentives are not aligned between these two teams. One team trying to push change, pushing features. And every time they push those changes, the site goes down. And so when you don't have these shared incentives, there's some miscommunication, we'll say. Sometimes you see pictures online, and you just want a story behind it. This is just too good. So you have these misaligned incentives. So OK, great. So this is post DevOps world, right? We're in a post DevOps state. I got my dev and my ops. They're super happy. But I'll just leave my security team over there, because screw those people. We don't really care about them. They're the people that slow me down. They're the people that tell me I can't deploy. And I feel like I'm saying the same thing we did 10 years ago and more, which is now we have, again, two teams that are conflicting incentives. So how do we get their incentives aligned with the rest of the company's incentives? Now, if you don't, you often find another scenario where this misalignment can represent itself as like a half-baked security solution. What's great, too, is that I have checked a box. I have access control. Check, right? Does that access control actually do anything? Probably not. But I got it. Your auditors. Yeah, your auditors. Like, you show an order to this, they're like, oh, boom. I'm good. Moving on, right? You show your customer that, and they're going to have a much different thing to say. So from that velocity talk, it is not the ops job to keep the site stable and fast. It is for operations and devs to enable the business. We're not just saying, well, it's up to everyone to keep the site stable and fast. It's not. It's to enable the business, whatever your business is. And we do that by lowering the risk of change through both the tools and the culture. And I think that's one of the parts that I always like to stress. If you bring in a new, fancy automation tool to your old, stodgy enterprise that doesn't want to make any human capital changes, then you're going to run into a DevOps transformation that will never work. Conversely, if you bring in people who understand how to deploy software continually at scale and give them a bunch of antiquated tools, you're also going to fail, right? It involves both sides of the coin, tools and culture. So this is a story in three acts. I like to talk about how to optimize and why you should optimize for the ease of software deployment. If you're wondering, where do I start, that's where you start. From there, as you exceed on, you want to turn your metrics into a first class citizen. Metrics are how we understand how our application is doing. And finally, ownership and accountability over all these things that you've built to enable people and giving them the ability to manage their own things themselves. And so each one of these steps build off the last. You should really not jump to the end if you can't get your code deployed easily. So software deployment, right? Modern operations, and it may be a bit of sobering idea for many of you, is how do I install X on Y? I mean, think about it. As operations people, how do I install Cassandra on Elasticsearch or Kubernetes on anything? Or whatever, right? How do I install this thing on that thing? And the tools in which we are using change for all of these, but the act of what we're doing is largely the same. So one of the very first things I did at this previous company and many other companies before was make it extremely easy to deploy software. And I'm talking you push button and your code's deployed. Now for anyone who's been in this space, you know that there's a ton of work around there. It's a big investment. And it was. We optimized for that very early on because we knew it would pay off. So how we deployed code from the first time we hit that push button deploy to four years later as we were deploying didn't really change that much. But what it allowed for is abstracting away the complexity of software deployment, of testing, of automating that code, releasing it out to our customers. Back then, using tools like Jenkins, just hit run, code gets shipped out. Maybe we build a Debian package. We push it to an app to repo. I'm not talking about anything that is so incredibly advanced that it is outside the realm of possibility. It's just tying those tools together to make them work. So here's kind of a simple deployment pipeline. You compile your source, build the package, sign the package, test the package, deploy the package. I would say this is kind of table stakes. You should be doing all of these things. Now you're like, well, my code doesn't compile, whatever, skip over that. But there's this one point I always like to do make because having worked for a security company for a number of years, signing your package is actually kind of important for two reasons. One is verifying the integrity of the code that you deploy is important. But also it's that step right after signing. Do you notice how I signed the package before I tested it? Because how many times do you deploy code, test it? Oh, let me just drop in this one last thing and then it goes out into production constantly. What if you can verify via signature that the code you tested is the code you deployed? I think that's the biggest part. Now let's just ignore the fact that it makes security people happy because you know the code is the same all the way through. But there is a story outside of just security, right? This is a place where multiple teams can actually benefit from this. So we talk about deploying code when it's ready. And for everyone, their concept of ready is different, right? If you are a SaaS business, ready could be when the build passes. Hey, if the build passes, that code's ready, ship it. If you're a big enterprise and you have to follow countless audits, maybe when you have necessary business processes passed, like the build pass, the test succeeded, there's documentation, we've notified marketing because we're releasing a new feature, we've done this, we've done that, right? There may be a lot more things that count as ready. But another question I always like to ask is how many people work for a company that holds customer data, right? Very vague, right? But if you're holding customer data in any way, and it's probably even more than even this, but for those people holding customer data, who has gone through a SOC2 audit, right? So the rest of you that haven't raised your hand, that's probably coming next. It's become the de facto standard, right? Your customers are asking of it more places. I wanna see a copy of your SOC2 report. I wanna know how you do your operations within your business. And so that concept of code being ready is obviously different for everyone. But the scenario is in many places is maybe this is, it's been reviewed by other engineers. It's passed a series of tests. But it's also met maybe some security requirements too. You can loop this all into there. You also hear the term shift left. Kind of a, I don't know, wonky term. I'm not sure how I feel about it. But I like the concept which is move more security testing into the pipeline earlier into the build processes. I think that's something that everyone should be trying to aim for. So I always like to talk about metrics because I think if you can't see it and it's not happening, and it's like the application crash, I don't know, my customer just called and said, you know, I can't get to the website, right? You never want your customers testing your app. But in the early days at a lot of companies, you know, I've always wanted the concept of self-service operations. And metrics is a great way for that, which is I just want to tell my devs, or anyone or ops folks, just if you want metrics for your apps, send your data here, boom. Here's an endpoint, send your data, drop it out there. And then one responsibility on their side is to write the libraries to tie into that. So it's ops to build up the platform, but this devs to instrument their applications. Not for anyone else to instrument the applications, other than them. So we built out something very simple, apps write to statsd, it goes into graphite, move on. You could replace every one of these pieces, but the key thing is I gave them a very sane, clear endpoint that they could shove metrics into. And that allowed them to move a lot faster and without coming to me and bugging me and saying, hey, how do I get this new metric in there? If they just shove that stuff into local host and whatever the statsd port is, they're there. But the key thing is that dev and ops, we both work together to choose the right instance types for their workloads. So any of the cloud users, Amazon, Azure, whatever, you tell a dev like, oh, go spin up whatever you want, right? Suddenly you're spending tons of money on Amazon. And so we work together to figure out, hey, are you really using that much CPU? I've got some graphs that show you're not, I'm gonna downsize you. And we also try to make sure that we're using those right instance types, no matter kind of what the application. And the thing I always like to say is, there's people that are like, nah, I'm just gonna deploy Kubernetes, it's great. Devs will just deploy whatever they want. I gotta tell you, if you think that's gonna solve the, I over provisioned my app, then you're gonna be rudely mistaken. You still have to work with them to understand their use case, monitor and get those metrics in. So how do we actually know this is working? Is engineers went and built the dashboards and the monitors themselves on their own. They didn't ask for help, they didn't come to us other than just say, hey, how do I do this little thing? They all did it themselves. And these dashboards and monitors became a requirement of deployment. You're not deploying a new app unless you're gonna bring some dashboards in. So finally, we talk about ownership and accountability. So if we've made it easy for operators and developers to ship their own code, we've made it easy for them to monitor their own code, it seems like we should give them the ownership and the accountability to go and manage the deployment of this code and manage the overall health of their application. Operators and infrastructure engineers building the tools that enable all those pieces. So the classic line from Amazon, right? You ship it, you own it and I think that model still works. Development owns their applications. They should get paged when it breaks in the middle of the night. If they can't fix it because they need operational help, they should escalate. But it shouldn't be paging to someone who has no concept into what that application does. And giving that ownership is very important because at the end of the day, everyone cares about the health of the business. Full stop, right? If you're at a company and you don't care about the health of the business, you should really find a new job. Whether you're a developer writing code that services some tiny part of the application, it still is enabling the business to move forward. So how do we do DevOps? The kind of generic we, I suppose. On one side, the operations teams, they need to trust that developers will involve them on future and future discussions around their architecture. Going and building things off in a bubble never works out well. And I've seen countless scenarios where, and I'm not to say anything bad about any of these following databases I'm going to say, but I've seen things where an engineer wrote this giant application around console using console for storing of data. But the consoles never ran in the infrastructure. I saw another scenario where they were using Mongo to store some certain type of data, which it is what it is and that's fine. But the fact is that operations team never ran Mongo and the developers had no idea how to run or scale it either and weren't willing to help out. So involving people early gets them involved in the discussion because you might have an engineer say, we don't have Mongo, but we have this completely other thing that would work just for you just fine. And on the flip side, the devs has to trust that ops will discuss infrastructure changes. I've seen scenarios where it's like, hey, we're going to deploy mesos out to everywhere. What about this app that can't run on mesos easily? Well, forget you, man, we're running mesos, right? It works both ways. You got to understand what you have and working with the devs and engineers on how it delivers, how they deliver their applications. So the end of the day, everyone needs to trust that everyone is doing the best for the business. I think that's the ultimate goal is you want to improve the business. And again, I say business generically, you could be a nonprofit, you could be a sole proprietor, you could be an education, it doesn't matter because we're all trying to service other people's needs. We all have customers, right? Customers may pay you, but your customers could be your developers or your sales teams or whoever's consuming the thing that you're building. So I always like to talk about this mythical dev, devop sec. Wait, is it sec dev ops? Or ops dev sec or dev sec ops, ops dev. I'm still working this one out. This joke was funny at security conferences. But the funny thing is is that it's like reading a Dr. Seuss book, right? It's like yet another term that we can continue to have fights in battle about the meaning of it. And I realize that it's been a good 10 years and I don't think we can all agree on even what dev ops is, right? I've talked about it in this talk. Is it a job title, a team, a product, a service, et cetera? I'm really not confident we're gonna have an agreeable definition as to what dev sec ops is either. But I think at the end of the day we're talking about taking what now is kind of competing incentives of security teams that wanna move a little bit slower and understand what's going out. And in operations and developer teams that are moving much faster, how do we integrate security teams into the decision making process? How do we get them to understand the changes we're making? And that's where you hear things like shift left where you say shift the build stuff earlier in the process, which I think is a great idea. And so I've seen companies where security engineers do more ops than most ops teams do. The ops teams enable them with ownership and accountability to do the change they need to and follow the same processes to deploy their features out to systems. But everyone's working together to move towards that goal. So how do you integrate that in? Integrate them just like we did dev and ops teams. Get a shared set of tools and programs and features working together and include them in the conversations. And I like to say teach them how to fish kind of programmatically, right? If you use Chef, teach them how to use Chef. And guess what? They're building things within those constraints. They can help you harden your systems. They'll do it for you if you just show them the tools. And you know, I'm a big fan of Threat Stack for no reason whatsoever. And if you use Threat Stack, which is pretty cool, you can have alerting go into a Slack channel. Who's in that Slack channel can be anyone. And so in this scenario, like I did some changes to some things and Threat Stack was alerting like, hey, here are some changes to DNS, which isn't normal. And I can just say, hey, security team, that was me, I was just doing some stuff, right? And so security teams could take this output and even build bots on top of it. And maybe I don't even have to pop in and say, was that me? I could just hit a button in a Slack and say, that was me. And now the security team knows about it, right? Again, using these tools to improve how we get our work done. So this is one of my favorite quotes from Rich Smith. He maybe is, was, but he was a security, either head of security or someone at Etsy, maybe CISO. But I really love this one because I think it applies to everything more than just security and operations. But abrasive individuals will single-handedly do more to undermine the security brand of your culture at your company than anything else. If you don't have approachable security team, it's not that the bad things won't happen, it's that no one's gonna talk about them when they do. Something I've seen great success is holding open office hours once a week where you can sit down and just come and ask us questions. Whether about asking me questions about operations, how do I monitor my app? I would have people from sales asking me, well, how do you know if the app is up? Let me show you these dashboards. People would go to security office spaces and say, I'm traveling somewhere, how do I stay secure when I'm traveling? It can be about anything, but it creates this approachable culture. At the end of the day, the best security cultures, though, they are collaborative, they're not prescriptive. And it's easy for me to say this, but it does require a lot of hard work from a really dedicated group of people that really want to make things better. You can't just have security decisions made in the bubble for the same reason you can't have developer projects in a bubble or operations projects in a bubble. You really have to include everyone in those discussions, even if they may not have a vested interest, they may have some opinions on some side of that question that they may not know. That's really the key to this whole thing. If you want to improve how your business gets work done and how that technology moves forward, you have to realize that this is long work over long periods of time. And for mostly a green field, which was my previous place, it took us four years to get to a place where we're arguably at today, where we have a lot of teams working together on these shared tools, these shared processes. It was not an overnight thing. So if you're a company that is an enterprise with a lot of technical debt to deal with, it's gonna take years and managing those expectations is the biggest part of it. There is no overnight success to DevOps. You have to change how you think about things and how you get work done. And you have to understand that you can't just flip a switch overnight and say, hey, everyone, we're moving from all these bare metal data centers to cloud. It's gonna be this continual process of change over time. And so the goal is to just make that continual process and just make everything a little bit better every single day than it is today. So that's all I've got. Thank you very much. So you mentioned that it was important, you said if you don't know, if you don't care about the business, find another job, what would be your recommendation for people who are in an organization where they don't actually know what's important to the business to be able to investigate that and figure out what are the things that matter. I mean, it's hard in a large organization to do that sometimes. Yeah, no, that's a great question. So everyone heard that, I don't have to repeat it. So if you don't know really what matters, obviously have these discussions with your teams and reach out and talk to people in your organization, not only in your own team too. If you're at a big company, I worked at a big company where we had a product marketing manager who would just schedule lunches and say, hey, I'll take you out to lunch. I just wanna learn what you do, right? Go and reach out to people in those areas to see what are their concerns and what are they doing and what's important to them. And what you may find is a lot of people actually may start saying the same things. And at that point, you can then start having discussions with management side and say, I've been talking with a bunch of people and everyone seems to think that we should focus on X, but all the public discussion is about this other thing. What do you think about that? Now granted, you might get told to go pound stand or whatever and that's fine. At least you kind of know where you stand in the business and that's where you make the decision to go somewhere else. We were hiring, by the way. But at the same time, I always like to say is don't limit your discussions to your own team. I think that is a big loss. Reach across to other teams. How do people sell your product? If you don't know how your business makes money, then you should go learn and find out. I think we get a lot of emails, personally I do in phone calls from salespeople, but I can tell you, I'd never wanna be on that side of the thing as a biz-deaf person making those cold calls. It's an incredibly hard job. You should just sit with one of them one time because I bet they would want to learn so much about your experience. What would you want to hear if you got one of those calls? So hopefully I answered your question, Matt. And if not, I don't care. Thank you so much. Awesome, thanks everyone. Thanks everyone. Hi.