 You can increase your salary by $70,000 after the end of this talk. So if you're out there getting coffee and you can't use 70K, go ahead and stay out there. I want to talk today about elevating your contributions as an ops engineer. We did the show of hands earlier today and it looks like we had maybe 60% ops folks. I don't want to leave out my dev people. Some of these things will apply to you guys, but I come from an ops background so they say write about what you know, so that's what I did. Sorry. I got a bit of bad news. This is not going to be a talk about technologies. I know people are always wondering like, all right, what is the thing that I need to learn to make myself better? The truth of the matter is technology is very seldom the actual problem, right? If there's one thing we've learned through these DevOps Days talks, it's that it's people, it's process, it's relationships, it's communication. So that's kind of what we're going to be talking about here in this talk. So there's not going to be too much mention of Docker. So first, what are we talking about? I'll introduce myself, then we'll go into four key areas that I think are important in actually elevating your career as an ops engineer. Knowing your business, and a lot of these are going to be like, well, yeah, this sounds pretty stupid. But when we dive into it, you're going to realize like, oh, I had never thought about it from that perspective. Know your business, know your systems that you're working with, know the people that use your systems, and know your priorities. And I think the fourth one is probably my favorite because it gives you a lot of flexibility in terms of how you go about running your operations environment. So first, a brief introduction. My name is Jeff Smith. I work for a company you've never heard of called Centro. We are a media ad services company, but we also are developing a software as a solution, a software as a service platform for media ad agencies to manage their own ad buying life cycle. Is that exciting? Yeah? It's not what I did. It's not what I did. But they sold me on it anyways because it was a great mission. My Twitter handle is dark and nerdy. You can email me Jeff Smith at Centro. I also have a blog that I never update called allthingsdark.com. Feel free if you want to look at a post from like 2015. It's relevant. I think it's about the Batman movie. I don't know why. Before Centro though, I've been at Centro since about November. Before Centro, I worked at a company called Grubhubs, a movie you might have heard of. We'll talk a little bit about that because I think my experiences at Grubhub really help form and shape my career and my viewpoint of how we look at operations. So I wanted to start though before we get into it is what is IT operations? And I think historically we've always kind of talked about IT operations as sort of the stalwarts and the defenders and the stewards of production, right? We guard production. That's what we do. We look at your requests and we say that's stupid. We're not going to do that or we're going to say, well, that's going to make my job easier. So okay, you can go ahead and do it. We chastise you for not having tests and then we don't have any automated tests to ourself. So I wanted to kind of like quickly define how I think IT operations is changing and sort of this old new definition of how I consider IT operations and how that's going to inform what we're talking about today. So the sort of traditional definition of operations management when you look at like business stuff, it's like operations management refers to the administration of business practices to create the highest level of efficiency possible within an organization. It is concerned with converting materials and labor into goods and services as efficiently as possible to maximize the profit of an organization. So there are a few key pieces of that definition that I think are really important and really crucial for how we view ourselves as operations professionals. Think about this line. To create the highest level of efficiency possible within an organization. That says nothing about strictly production. There are tons of things that are happening in your organization that don't actually like are not immediately impacting the production environment. There's an entire value stream that sort of is part of the business process. How do we get involved with those things? Converting materials and labor into goods and services as efficiently as possible. I instantly think of developers. I think of the developers that are writing code. How do I help make their work more efficient and faster? So the first one, know your business. That's an easy one, right? Like we all know we work at companies that sell coffee, that sell operating systems and office suites that are hell bent on creating Skynet that's you Amazon. Yeah, it's happening, right? You ever think about that, right? Like one day someone's going to be like, you guys thought it was a good idea to concentrate 95% of the world's compute power at one company. And we're going to be like, well, you put it that way. Yeah, I guess it was. It probably wasn't the smartest thing to do. But it was so cheap. The APIs are so slick, right? But no, you've got to know your business. So when I was at Grubhub, it was a pretty easy value proposition. Before I went to Grubhub, I was a customer, right? And I was like, I'm hungry. I go to Grubhub and I order food. Food shows up. I can get behind that, right? I understand that. The picture looks easy for me, right? So the thing about being a consumer and actually caring about the product that you're working on is you start to get invested in different parts of it. So if you look at these yellow arrows, right? That's what I call the hangry zone, right? This is the part where I'm starting to get hungry and I'm wondering what the hell is going on with my pizza? Where is it? You instantly get invested into what's going on and learning about the systems that actually make the delivery of your food possible, right? Through that, you end up learning so much more detail about the organization, how it works, how the technology plays a role in that, and how you fit into that, right? Because let me tell you, when you're in the hangry zone, people would just show up at your cube like, latency spikes, huh, Jeff? Not today, buddy. Not today, right? I need my turkey salad ASAP. So then, I went and I moved to Centro. Yeah, research, planning, and then we're, oh, then we're gonna buy the ads, and then, oh, okay, so then the ads get shipped to Facebook, I'm just glazed over, right? It becomes instantly more complicated, mainly because I'm just not that interested, right? I don't actually use the tool, and that's really a huge cop-out and it also is going to impair your ability to actually service your organization as effectively as possible. So you start to just glaze over of all the different counterparts and who you're interacting with, and you're just kinda like, yeah, whatever, I really don't need to know about this stuff because I'm an IT or I'm an engineering. This isn't important to me. It is important, though, because as you go through the organization, you'll realize that members of the organization have different dialects, right? Everyone has started a job and heard all the three-letter acronyms and been like, what the hell are they talking about? They're talking about SSPs and DSPs and PLCs and TPS reports, and you can quickly get lost in it. So you sort of like retreat to your own little corner, but the interesting thing is that IT operations is probably one of the most uncommon dialects in the entire organization. It's probably a handful of people that speak your language, right? When you're talking about servers and sockets and the TCP wait times, right? Accountants aren't worried about that. They don't care. Underneath that, you have developers. There's typically more developers than operations people. So more people are speaking developer languages, classes, objects, things like that. Then you get into the application functions, right? Now people aren't really talking about the specific technology, but they're talking about their app and how it functions and how it works. And then you've got the business function, right? This is generically the C level, how we talk, how we operate, and then we translate that into application function. If you don't understand the dialect of the other people in your organization, you're always gonna need a translator. You're always gonna be just a little bit behind on diagnosing or remedying a problem. A good example is a typical conversation we have in the business. Jeff, RFPs are slow when they're pending insertion orders and I'm like, oh yeah, load average looks okay though, so. Is it really a problem? Right, load is too, life is good. Then we need a developer to come in and sort of translate for us. Oh, the RFPs query third party sites, so checks for latency in those calls. And you're like, oh, okay, that's a language that I understand. But while you're waiting for that translation, while you're sending them to go talk to a dev, right, that's time being wasted on a system that's potentially down. So think about it from a generic perspective, right? Like it's easy making fun of something like, because I work in the media ad agency, right? Most people aren't interested in that. But think about it from a more generic perspective, like say you work at Netflix, right? And someone's like, Jeff, when people search on the Apple TV app, when they search for the kids genre, they're actually getting filmed noir and people are complaining about it and you're like, whoa, I don't know how this app works. I don't know anything about movies. I'm just an IT, I'm an engineer, right? They look at you like, what kind of moron did we just hire, right? He doesn't understand, you know, you never want to be in the meeting like, oh, excuse me, I don't know how we make money. Put it out there, I have no idea how this works. Server load is good. Expecting the stock to kind of follow suit. Right? So you got to know your business. It's really important to get involved and understand what it is that people are doing, what are your peers working on, even if they're not in the technology group, because understanding that language is key to being as effective as a contributor as possible. And we'll have some more examples of this sort of as we go on. Know your system. So that's another easy one, right? So when we talk about systems, you're like, oh yeah, I know, we got web servers and at the web tier, we got a load balancer, we got HAProxy, we're out on traffic, then we got a Redis server, we got RabbitMQ, and it's like, okay, we got MySQL, but how does your app use those things? And you're like, oh well, the app puts messages on the queue and then it takes them off, and then it does some stuff, then it puts a message back on the queue to do some more stuff. That's not really knowing your system, right? Like you know generically that, oh, this application uses this thing like everyone else on the planet uses this thing. But that's not really intrinsic information in terms of troubleshooting and understanding the system. And the other interesting thing I think is like with third-party products, right? So how many people use MySQL or Postgres? I forgot where in Seattle. How many people use SQL Server? I'm sorry, my bad, sorry. The interesting thing I think about is like with these third-party tools, we spent tons of time researching them, right? If you've read the MySQL internals book, you're like, oh man, I've always wanted to understand how the client-connector library functions. Look at all these awesome system calls and function calls, this is amazing, I'm so glad I read this book, right? Put tons of research into it. But the interesting thing is, you're gaining all this expertise into a system with not really a specialized usage, right? Everyone is using MySQL pretty much the same way. It's not revolutionary. Yet for some reason, when it comes to our applications, applications that our organization has written, we're kind of like, well, you gotta talk to the devs. I don't know anything about that. You went out and you spent $40 on a book that you understood 30% of and you're just amazed by it. You've got the developers that built this thing sitting next to you and you don't even go over and talk to them to understand how it works. That's what you're responsible for implementing. Why not? Why not have that conversation? Why not go a little bit deeper into the core piece that actually helps you make money, right? That's the part that makes the stock go up. So we need to spend as much time as we do researching these third-party systems, spending it into researching and developing and understanding our own systems. The other interesting thing in operations, and I'm gonna tell you guys some dark shit today, so. So be ready. This is a word cloud of the top 100 people in my LinkedIn profile that have endorsements, right? So the word cloud represents the number of people. Do you guys have a technology up there? Yeah? Technology, like the thing that our bread and butter as ops people, the thing that we sort of prided ourselves on knowing and understanding is largely becoming commoditized, right? When that key player leaves the organization, your bosses are like, whoa, how are we gonna find someone that supports Redis? I don't know what we're gonna do. There's just no Redis people out there. Like, no, there's tons of Redis people out there. How many people are, let's pick a technology. How many people are Redis experts, right? One person, come on, all right. Pick a technology you feel like you're an expert at. Someone shot one out. Java, okay, wonderful, thank you. Now. I can't hate it, but I had to yell something. Right, you know, I appreciate that, appreciate that. Hang on, I'm gonna get my Marco Rubio on real quick. Your skills are largely getting commoditized. So we pick Java, right? I've never written Java, I've read it a lot, but I don't really write a ton of Java. How many people are Java developers and think they're better at me than Java? Yeah, you're probably right. But you know what I'm better than you at, buddy? Central platform. Exactly, you don't know what it is. It's exactly it, right? I will never hire someone on the street that is competent and skilled in central platform, right? That's where you're adding your value. Because these tools, I mean like, for the Redis expert, you are now replaceable at $1.15 an hour elastically in AWS, right? You are an API call away from unemployment. So you really, really, really want to say like, hmm, well where can I add a little bit more value that's better than this crow command, right? It's gonna suck when you pay your next student loan too. You're gonna be like, God. So it's about domain expertise, right? Because even though someone out here may be better at Java or Redis than me, they don't understand how my platform or technology leverages that. They don't know how it scales. They don't know how it runs into problems. They don't know where the bottlenecks are. They may think, oh, one user we're gonna scale linearly, but they don't know that I know the secret piece of code that is actually gonna amplify that and turn three users into 12 Redis requests, right? And then put pressure on the Redis queue. You have to understand how your applications function because that gives you the domain expertise that makes you valuable and keeps your family fed or just yourself. Domain expertise plus business knowledge equals understanding the business impact of the things that you do. It's always about the business, always, always, always. We love being technologically pretentious, right? Oh, God, you guys are a Ruby shop? Yeah, I'm a Ruby shop. Deal with it, right? It solves our problems. But understanding the business impact is what allows you to safely make that change so that the example that we use is, okay, RabbitMQ has a lot of pressure on it. Okay, we've got 20 worker threads on it. Let's go ahead and reduce the number of consumers down to 10. What does that do? Maybe those workers are processing email notifications. Okay, not a big deal. Email notifications slow down. But what if they're processing real-time bidding requests and those requests are measured on milliseconds and you just have to capacity? What is the impact of that? What does that mean? You don't know if you don't understand your business. So it's important to make sure that you have domain expertise with business knowledge so that you understand the impact of everything that you do because it's always, always, always about the business, always. It's one of my favorite slides. I'm just gonna give you a second, right? It's like, right? You're doing all this wonderful stuff, right? And someone's like, how is this helping us make money? But it's slick technology, right? It is so killer. We've got Docker, we've got Redis, RabbitMQ. We're doing all this magical stuff and it's like, is it making me do something better? And it's easy to stop and say, okay, well, we're making developers faster. Okay, and, right? Like how do we translate that to business value? A great example is like, this is weird coming from the consumer sector, but at our company, a lot of our customers aren't really into rapid feature deployment. They don't like it, it overwhelms them. They're like, well, you guys are pumping out too many features. So we sort of have to slow that down. So the idea of being able to rapidly deploy wasn't a great selling point because they're like, well, we don't want all those deployments anyways, but then when you transform it and you say like, well, we can still deploy it darkly, right? We don't have to turn it on, but we can get this code out there so that when we're ready, we can switch it on and go ta-da, and we're not worried about building up these months and months of features and then dumping it on someone's doorstep and ultimately having it fail. That's a little more palatable. That's more attractive to them. Know your system's users. When we talk about systems, again, a lot of times we're stopping at production. We're like, oh, we know who the users are. They're the ad agencies. Well, your system is sort of this generic concept, right? It's a collection of applications and services and we represent that in production, but it happens there's other implementations of it throughout your organization, right? So who is using those other implementations and where do they come from, right? How many of you have found about an environment that you didn't even know was a thing? You're like, oh, really? How do you deploy code to that thing? I didn't even know it existed. They're like, oh, yeah, we've had it for like five years. Is it still on the five-year-old code that we wrote years ago? Yeah, it's just the logos though. No, it's not just the logos. More stuff has changed since then. So get out and discuss with other people who you're working with. So I want to introduce you to Karen. Sorry about the fuzzy photos, the only one we had. Karen is a customer success manager at Centro. She's responsible for training new clients as they come on. So she has to enter training data for each client in order to prepare them for their training. Now, interesting fun fact. Karen has to train these people so therefore they cannot use the system until Karen is done training them. Therefore we can't book the revenue that we've signed these people until Karen is done training them. Karen's process takes roughly 12 effort hours to complete, effort hours. How many of you have 12 hours of uninterrupted time that you can dedicate to a thing? Show of hands. Anybody? All right, I'll make it easier. Who's getting 12 hours of sleep? Right? So we can't even trade there, right? 12 effort hours. So in a typical day, I probably have two hours of uninterrupted time. Best case scenario. So you can see what this quickly does to the timeline, right? So now you're talking about a week and a half worth of effort or worth of duration for Karen to get this training environment up. And to make it even worse, it's extremely repetitive work. It's extremely repetitive work. It's not like she's splitting the atom. She's just like, oh, then I gotta type this thing in, then I gotta type this thing in. Sorry, rubial moment. So when I met with Karen, I was like, what is it that you do? And she explained it to me and I said, oh, this is a problem we can solve pretty easily. So with two hours of operations engineering time, we reduced the amount of effort that she has to do down to 1.5 hours. And it was really like, I mean, it's not even like we went back and cooked up a plan and we're like, oh, we could do this. I was like, or you could enter your data and I could back that up. And then we could just restore that when you need to do a new client. And I was like, wow, you can do that? I was like, you'd be amazed what I can do. This is the tip of the iceberg. And this is the low hanging fruit that turns you into a magician, right? They hear this and they're like, oh my God, Jeff's a wizard. And I come here and tell you guys like, and guess what? And they gave me a raise. So we were able to reduce the amount of effort down to 1.5 hours. And then we also got her team's needs on the roadmap for our transition to AWS. I forgot to mention that. We're migrating from a data center to AWS. So we put that in the roadmap to say, all right, let's understand what training needs so that we can make sure we're addressing those needs as well. And that's amazing. But the other amazing thing is, as operations, right, we've historically been a cost center, we have now removed the blocker into booking revenue, right? Think about that for a moment. We are now allowed to book revenue faster thanks to two hours of maxes time. And that's an actual impact on the bottom line. That's awesome, right? That's what we've always wanted. We wanted to be more in the revenue stream and prove our value as opposed to constantly trying to defend why we exist. Well, outside of outages. We always exist during outages. Sean, oh, hell, I thought I removed his last name. Sean, we're gonna just, sir. If you ever want to limit your career, don't learn how to pronounce your CEO's last name. We'll call him Sean. Sean has multiple demo environments. He's still out there. He sells to some of the larger clients. And he's managing these multiple demo environments and he has these various data setups for each demo environment, depending on what he's trying to focus on for a particular client. So there's a lot of behind the scenes stuff that happens where it's like, okay, Sean's doing a demo, so make sure the demo number seven is set up appropriately with yada, yada, yada. Okay, that's great. So when I met with his partner in crime and we started to talk about that, I said, well, I wonder if there's something that we can do to sort of assist you with this. So now when our road met, we've got Sean creating new demo environments on demand. He can then select, oh, I want this data set and I want a new environment and I want to call it Sean rules and it provisions it using, I said I wasn't going to talk about technology. Uses a bunch of tools in the background, but it does that. He can even deploy in progress feature branches. So he wants to talk about something that isn't actually released yet in master, but now he has the capability to do that. That's amazing, that's powerful. You're helping the CEO sell to clients in a way that he thought wasn't possible before and the thing that I always dream about, I don't know if this actually happens, but I always dream about it like he's in the meeting and someone's like, oh, well, it'd be nice if we could have a demo environment to try with and he's like, oh, yeah, sure. Let me spin one up for you. And they go, oh, you can just build an environment for us right here. Like think about it, your CEO is deploying code. That's beautiful. I mean, right? You're just like, I'm so proud of you, Sean. I think the incredible thing though is like, these guys didn't even know the ops team. They didn't know who we were. They didn't know what we did. Well, Sean might have known. Karen definitely did. And we were able to introduce ourselves and work with them and collaborate with them to make them more effective. And that's what we talk about when we say, maximizing the efficiency of the organization. We're going out and we're touching people's lives. I'm all dramatic touching people's lives. But no, seriously, 12 hours, like, I mean, come on, she's gotta be a little happy. So now I'm not saying you should go back and kick in your CEO's door and tell them, tell me what you need. But you can make a slow progression through your environment to figure out who you need to be helping. So we start with production, your operations team. But then you move to QA staging. So fun fact, I worked at a company, no names, they're a QA group, ran the QA servers, using the same tool as production, but a different code base. So what do you think the chances are those environments look the same? Yeah, no, no. I mean, you might as well be deploying on another company. So the first thing we did at this company was move and said, all right, we need to take over the QA environments too. But build CI, release engineering, moving into local developer. It's like, you know, if you're using tools like configuration tools like Salt or Puppet and Packer, like you can then use those same tools to create vagrant instances for your local developers. And it's really not any extra work on you because you're already doing it. Now at the end you're just saying, now I'll put that to a vagrant box and make it downloadable for someone. That's powerful, right? And your developers are gonna love you because they spend all types of time trying to figure out how to get, you know, the old version of Postgres to work on their local machine because Brew removed it from the, you know, TAPS library or whatever, right? You know what I'm talking about, right? So they're like, well, we just upgraded everything to nine-five, so maybe you could just do that in production. Oh yeah, sure, no problem. And then ultimately moving into the business units. I think the key though is that like as you move through the pipeline, you are adding quality into the process. And quality isn't something that we can sort of bolt on at the end, right? It has to be baked in as part of the process. And the more alike we can make these environments from an operations perspective, the more successful everyone in the value stream becomes because we're working on environments that look much more like the target environment we're going to. And the common excuse that you often hear with this is like, wow, I just don't have enough people to support that. If you're at the point where your headcount has to scale with the number of servers you have, you've already lost. If you're doing it based on services, that's a different thing because there's a bunch of mental models, you gotta keep track of yada, yada, yada. But if you're scaling based on the number of servers you have, you're already behind the eight ball. So this work should be easy because you're really stamping out the same thing over and over again. Know your priorities. Oh, priorities, they're so wonderful. I think the biggest thing you can do as an ops engineers, when you get that inevitable interrupt of work, you say to someone, well, where does this fit? Especially your boss, because don't think we're immune to it. I'll be like, Ali, I need to do this really important thing. All right, where's that falling towards the other important thing you told me to do? Well, it's less important, but more important than the other important thing I told you about. So make sure we have the three important things in order. Plan work and unplanned work, right? It's a constant battle, right? Unplanned work is all the shoulder taps, the incidents, the requests that are extremely urgent and can't wait. Planned work is any work that you've decided when to actually do it. And these two are in constant opposition. They're always, always vying on your time. Priorities help you take unplanned work and transform it to planned work. And that's something that you have to do all the time. When we look at our priorities, we say, all right, support production. It's always operations top priority, right? So incident handling, things blow up. We need to be involved in dealing with that immediately. But then you've got things like, okay, so we've got the, this is all fake, by the way. Notification feature rollout. That's the second priority. Why is it a priority? Well, because Sean's gonna announce this at some industry event and we kind of want it to exist when he announces it so that he can actually show it. So that makes it kind of important. Third one, AWS migration. Well, why is that important? Well, because we can send the data center contract. So if we're not in the AWS vibe, September, we're just gonna shut it all down. Good run. So when the request for the new database server for the data warehouse team comes in, you can actually defend your choice on when to schedule that. Priorities become your shield for defending attacks against your chosen set of work. You've chosen to do these things. You've done that based on these priorities and that's your shield. So when someone comes up with a request, you say, well, where does this fall in the rank of priorities? If you think that this is more important than Sean doing his demo, then by all means, I'll spin this thing up right now. But if it's not, let me take this unplanned work, convert it into planned work. Now everybody knows and has the expectation set. Everybody knows, okay, I couldn't get this thing in immediately, but they're gonna get it on the roadmap. And more importantly, it also reminds them next time that they just can't come over with a cheeseburger and be like, Jeff, got your favorite cheeseburger, need that data warehouse. Actually, it might work. I love cheeseburgers, man. All right, so what do we talk about? Know your business. Know what it is that makes you money. Understand how that works. Know your systems. Make sure that you're not gonna be unemployed because someone invented a curl command. Know the people that use your system. Get out to the business, talk to those people. Find out who's using it. Talk to your developers. Figure out what their pain points are. You might be able to save someone an entire weekend of work with relatively little effort because they don't fully understand your capabilities, especially with this rapid transformation of DevOps that's happening today because we don't even fully understand our capabilities. So it's nice to be able to talk to them and go, hmm, you know, I never thought about that because we don't have a use case for that in production but I bet we could cook something up for you. And then lastly, know your priorities. Know what you're working on, why it's important, and where that falls in relation to other work that's coming in. Thank you.