 How are you doing? Make DevOps wait again. How many of you are wondering what I'm going to talk about? I share your concerns. So it would be gratuitous and amusing to do a bunch of things about, you know, the parody that is our critical process today. But instead what I want to do is actually, I'm going to say a few things but then I want to have more of a dialogue because I feel like, and I'll go to this slide, I feel like every DevOps talk has already been given. There's like not really new stuff. There's some new tools, there's some kind of new things we could play with, but the actual kind of underpinnings of principles, they're not changing. They're not new. But at the same time, just put in perspective and I know this isn't going to be as amazing as it would be if we weren't having a slicker. But a lot of this starts to sound like pandas vomiting rainbows, which is to say that it's somewhat inactionable or becomes, I don't know, people kind of, there's this pendulum that you see in the talks where on one side people give like a really nice tool talk and everyone complains and they're like, well, there's too many tool talks. There's not enough talk about culture and people. And then on the other side, you've got people who give culture and people talk and then people on the other side are like, wow, there's not enough about the tech. You've got to talk about the tech, right? And so this pendulum, who's ever experienced this? Who's ever said any of those things, either of those things? Like every talk on DevOps was already given basically in 2009 and mostly by me. Who's ever seen any of my other talks before? Who's ever seen my deployment talk from Rossi 2010? Who's seen the, there is no talent shortage. So someone told me once that I give talks where it starts out and it feels like it's going somewhere and then they have no idea where we're going and they're really, really confused and then at the end it all kind of made sense somehow. Has anyone ever had that ride? So this is, this is a, I'm not, I don't have any slides, but I'm going to have, I want to have a dial, I don't have a town hall, but I want to, I want to put a few slides or quotes to frame it. So this is the most important slide, obviously. This is the easiest way to find me, get my attention. Also just to make sure, next time you see me, I might look different. And then I could look really different. So this is all pictures from the last five years, just so you know. And then very, very important, obviously, engage with my brand. So this is Charles Wachowski. Anyone is a Wachowski fan? Yeah. So this is a quote, is that the intelligent people are full of doubts while the stupid ones are full of confidence. And there's a lot of tricks and some of these tricks have to do with how you, how you say things, how you communicate things. I could come up here, I could say some words, I could move my hands, I'll raise the pitch, dramatic pause, sounds like I know what I'm talking about, I project confidence, you'll believe me. That's our democratic process, by the way, too. This is like, this is old school demagoguery back to the Greeks. So who's familiar with two phenomena? So one is sort of well known and somewhat discussed, and that is the Dunning-Cruber syndrome. So the Dunning-Cruber syndrome, for those of you who don't know, is a tendency for human beings to, if you're an expert to underestimate your relative expertise, and if you're not an expert to overestimate your relative expertise. There's also another, another phenomena, and I think this is hardwired in our brains, it kind of goes back to this demagoguery that I was talking about a moment ago. And it is provably true that human beings prefer confidence syndrome, where the people who have the least experience, or the least expertise, but have the most confidence. And then the natural human tendency to prefer confidence to expertise. And you can explain a lot of problems that you guys probably see every day. So this is another one, and I changed the word, I think it, like here's the one of the things I'd like to point out, and that is, it's not that demagoguery sucks, but it's like it twists your body, it twists your mind, like people kill themselves. So this is the guy who basically burned Atlanta. So he's saying, after all the reporters, no one's glorifying war, he said, no one who's ever fired a shot wants this thing, right? I'll read it, dramatically. I am tired and sick of devops. Its glory is all moonshine. It is only those who have either fired a shot nor heard the shrieks and groans of the wounded who cry for blood, for vengeance, for desolation. Devops is hell. So there's tons of work that goes on in organizations, and even organizations that are fairly high performing, that is not so awesome. And I've been inside a lot of organizations, I've helped a lot of organizations, what I feel like is improved, but at the end of the day, even people that I've seen, maybe especially people I've seen, give talks about how they're doing such a great job, but I know for a fact it's basically a shit show. And that's one thing that I think, I've come to realize that everything looks like a shit show from the inside. There's some things we can do better, there's things we can incrementally do better, but let's recognize what's actually happening here, and this is our opportunity. To me what DevOps represents is very inside of this larger theme of software which is optimizing all of human experience, all of human performance, human decision making, that inside of this we have using software to actually improve the work that we do. So that software cycle of improving human performance, of improving human experience is part of this DevOps cycle. It's a small thing, and that would bring me to another point I want to make, which is that it's not one thing. There's all these buzzwords going on where people are talking about continuous delivery. It actually pains my soul when someone wants to talk to me. They're like, we'd like to have a DevOps initiative. You're like, okay, cool. And then after that we're going to have a continuous delivery initiative, and you're like, oh god. Power 1c, you can't really talk about a lot of this stuff in isolation, and I think it's sort of bad for my perspective because my original goal was never to bring developers and operations together. My actual goal as a human, as a person involved in the product that I was involved in, was to optimize systems. So if you think about what it takes to optimize a system to build architecture, there's a lot of people in this room that are probably pretty good about thinking about how this is going to communicate with that and how this is going to fail. We don't put the same kind of impetus into designing our organizations, and my original idea that kind of led to a lot of these conversations were really around how do you build systems that are human systems, and it doesn't just start and stop at dev and ops, right? There's all this other stuff that has to happen to build a healthy system. You need product management, probably the hardest job in the building. You need to have some way to make money, hopefully. I mean, as long as we're going to play capitalism, we might as well play the win, right? Like, what's going on here? Like, we don't get to choose all the rules of the game, but you have this system that you're embedded in, and it's sort of sad, at least to me, that you have people who are like, oh yeah, dev ops, and it's like, yeah, developers and operations are working together, and it's like, fuck the sales guys, right? Like, fuck that. Those guys, man, those guys aren't even humans, right? Marketing, whatever. Like, everything outside of my little tribe, my little spear, they're not humans. Like, there's some other tribe, and if I get them, I'll kill them. This is another, the last quote really, and then I kind of want to have a conversation, but this is the conversation that I want to frame, and that is another Picasso quote, and I actually saw this on the parking lot driving in, so I was like, yeah, Picasso, and I already put it the other way around here. Knowledge without follow-through is worse than no knowledge at all. And I'm actually a little bit sad that no people, or people aren't as excited about the organizational learning stuff as I am. It's really awesome. It's actually, I believe this as authentically as I can believe it, that the difference in the organizations that I've seen be successful trying to transform themselves and the ones that haven't, by definition, is a function of their capacity to learn as an organization. And there's a body of research around organizational learning and a body of research around systems thinking that I think is just trying to lead dormant, and I try to introduce some of the conversations, but I feel like no one's having those conversations back with me yet, so maybe someone can help me out here, because it's all about me having conversations that stimulate the neurons. But this idea of learning, meaning that you didn't actually learn anything until you've changed behaviors, right? If you get information and it doesn't change your behavior, you didn't learn, right? Anyone as an expert in anything in here? Does anyone play a musical instrument? Does anyone play chess? So if I read something about chess or I read something about music, but I can't play the game better or I can't play the music better, did I actually learn anything? And so the rest of the time, what I want to have is a dialogue with people in this room about what you've actually changed in the last five years. Have you actually changed? It's glorious that I can pay my mortgage and talk to people, but it's actually very, very frustrating, too, because you end up having these conversations, and maybe you guys have the same conversations in your organizations, because I also strongly believe whenever anyone comes in from outside a consultant role, there's someone who said the exact same words that they're going to say inside of the building, like the week before. Well, they did actually get paid for it. That's the funny thing about a lot of these organizations is they're like, we need talent, we need intelligence, we're going to innovate, we're going to get the best people we can, and they get in the building and they're like, don't do anything except for what we tell you. Right? It's awesome. So in the last five years in these organizations, I had this conversation over and over where they say, and it's always a little bit of a variation, but we have this intractable problem that either it's like no way to solve, because we made all these bad decisions over here, but we're not really willing to revisit them, so what we're going to do is give you this Gordian knot of stuff that you're supposed to somehow unravel, even though there's only one or two threads you can pull on. Right? Usually what ends up being like the last conversation was what we'd really like to know is how can we get the DevOps without actually changing anything? Right? And so that's why I want to come back to this dialogue about what did you actually change? So I'm going to ask people, if anyone wants to, I saw there's a mic, so I have a mic here. If anyone wants to take the mic, they can. And if no one takes the mic, I'm going to start calling on people because I know some of your names. You can also ask me questions. There's a couple hands that went up, yeah. There had to be a better algorithm for that. Okay. So basically just what have we changed in the past five years? What behaviors have you personally changed, or organizationally changed? Well, basically, being a business owner, a lot of times I have to reevaluate how I do things. And over the past year, I kind of got exposed to DevOps and actually it was actually at the last scale where I got exposed to DevOps. And I recently started using Ansible to automate a lot of the stuff that I was doing manually. And I'm trying to kind of improve my workflow by automating things that I understand. But at the same time, a lot of times, all these technologies can be overwhelming. So I'm really trying to get a grasp of everything. That's good. I have this conversation a few times too where people are saying all this stuff, you go through and you automate stuff and you didn't actually get that much out of it. The thoughtfulness, because the problem actually was, this is the problem most of us have here or a lot of us have here that I've actually probably helped contribute to. And that is that it is often an anti-prime to try to automate what already exists. And that you have an opportunity when you're going to go to automation to think about the things that, when you have something that's really complicated to automate, that's a smell. Just like if you're a developer and there's things that are hard to test, there's always things that, whenever the technology is resisting you, like maybe you want to rethink what you're trying to do. I worked for Small Company X. They got bought by a big company, Y, about five years ago. And at Small Company X, we were devopsy and small and we changed the behaviors we used to deploy code by exposing metrics. We changed the behaviors of the definition of done by getting developers to write metrics and their own deployment codes. And in company Y, when we were purchased, there was a huge silo between development and operations and so far as the buildings weren't even allowed, people weren't allowed to go between the buildings. So I'm here after a couple of years. I'm from Canada originally. And at company Y, we changed the behavior between the development teams and the operation teams. Everybody can free flow. And now we're exposing metrics just slowly, taking a little bit longer. So metrics have changed my life. I'm not angry. Metrics changed my life. Right? I do think that there are some pillars, right? And this isn't something I just came up with. But I think that everyone is sort of in the inner circle of the devops crowd recognizes the cams or calms, acronyms, right? So culture, automation. I like adding lean because then you can say calm instead of cams. And then metrics and then finally sharing. I think sharing is the biggest one for me. And I think that watching... I think sharing happens on two dimensions that are very, very important. And in some places, they only do one. And not the other. And the first one is the sharing that we do inside of the organization. And that is important. Maybe... I wouldn't say it's more important, but it's definitely important to share. If you're trying to optimize the system and this side can't make decisions because it doesn't have information, that seems suboptimal. It doesn't seem like a big debate. But then there's this global community. And this is the thing that I think most exciting about devops as a movement is not that there's tools like whatever, who cares. I can make a computer do something. I could actually... I swear, I could make computers do these things better with bash than half the configuration management stuff we're seeing. Like, okay, fine. I can make computers do stuff. But there's this global community of practice that are all sort of solving these some problems. And they come to DevOps days. They come to velocity conferences. There's a number of other conferences where they kind of gather, I think scale in some senses, kind of like a proto-devops conference with a lot of Linux conferences, have some of this stuff. But that ability to connect with other human beings who have experience, who have solved some problems, or maybe they're just farther along on the problems that you're about to have is a force multiplier that I think is totally underutilized and underestimated. Like, I'm relatively intelligent but the power that I have to solve problems is 10X more than it would be if I didn't have a list of people on my IMs, my Skype, my phone, my email that will answer my questions that I've already done that stuff. How's the mic? Howdy folks. Howdy, my name's from Ticketmaster. So one of the things that we've done in particular the last year is we've actually sped up initiative that we refer to internally as DevOps University. And when you're coming from, as people have pointed out, very heavily siloed, operational versus development versus whatnot, each of those become stumbling blocks for the other. Operations is the stumbling block for things getting in production. Debt is the stumbling block for operations because shit's falling over and there's not a complete feedback loop. So what we've been very aggressively doing is we've been basically taking our operational dev folks and putting together, we do a week-long training course spun up by this gentleman who is probably embarrassed that I'm pointing him out. And we try and teach them all the basics of how to reach across the aisle and how to, the best ways to self-service these, the best way to drill into all the different, whether it's beginning development tricks for the operation folks or beginning operation tricks for the development folks, it's been a significant force multiplier for us over the last year. As a matter of fact, we've had events where if we hadn't had this initiative in the company, we simply wouldn't have managed to pull them off. And from that perspective, it's been fantastic. That's what I'm talking about. That's deliberate organizational learning. They're setting up explicitly to learn, to teach, to learn, to share. Who else does that? How many people solve the DevOps problem of having a silo of operations and a silo of development by putting a new silo called DevOps in the middle? And where the hell are the, like, the least engineers? Where do they go? Does anyone live this story? This guy has a... So, okay, go go. So, I have started treating my infrastructure like code as often as humanly possible and found that it leads to my team using the same tool set and techniques as the developers. And I find that that leads to us understanding their pain points a lot more and getting resolution to their problems faster. Excuse me. So, the blog post that I wrote, what DevOps means to me in 2010, had three main points. And the first one is that developers and fixed admins could and should work together, which is revolutionary, I know, especially in 2010. But in so many organizations, especially when you're coming out of places that have not really traditionally been delivering services, because there's this transformation that has to happen when you go from IT, keeps the mail server going, to IT is actually the value chain that really enables a lot of these conversations. So, having those people work together, we're in some, like, weird ITIL frozen in and they're passed like they didn't. That's a big thing. And then the other thing, which is to the point, now I was just making this, everything is becoming API driven. This is obvious in 2010, and it's even more obvious now, that if I can provision machines with APIs, if I can configure machines with APIs, I can take things in and out of Mario with APIs, I can do all these things that used to be done some other way with an API, writing to APIs looks suspiciously like software development. I hate to break it to you, but it looks suspiciously like software development, which means that you can leverage lots of tools, lots of process, lots of things that software developers have been doing for a while, and that you can start to, as he pointed out, you have now, like, some boundary objects, some things you can share and have conversations about, because you start to have a common tongue. You don't see them as the other. It's like, oh yeah, there's this deep stack that no one of us can actually understand fully, and we need all of us to do it. Hi, my name is Joel. And the main thing that we've done at Holt looked in the last two to three years, I think we've gone through the whole sort of dev-oppost journey from like full isolation to like, oh no, working together, we have that middle silo you were talking about, that sort of expanded the middle silo, like, wrong, wrong, wrong, wrong, wrong, wrong. Like, incrementally better. Incrementally wrong, better, yeah. And so we got to the point today where my only mandate for my team is no craft work. And that's a big, big point to make, because writing software can be craft work. Writing Ansible can be craft work. Automating things can be craft work. And so craft work is loosely defined in those things that you really don't feel like you should be doing, like craft work, like we have to automate this new technology again, you know, we have to do this again for containers, like that's craft work, right? And so really thinking and being thoughtful about being mindful of why we are doing something and what the, you know, alleged business value is, like, that's how we've been able to actually eliminate almost all the craft work. So no one has to deal with Nagios, no one has to deal with a mail server, you know, no one has to stand up, you know, KGM boxes anymore. Nothing bigger, that's your problem. I like it. I'm just going to add this one while I run. I started caring about family and making sure that I was taking vacations and that when it was time to go to dance practice with my daughter, that I went. So that's what I've changed. And I saw. So I look for a school and when I say school and kindergarten through ninth grade. So generally, there is no such thing as DEV in this universe. There is only OPS. The problem is that when you only have OPS, you're at the mercy of the products that you have available to you in the market and they don't always solve your problem. So embracing DEV OPS from us meant creating a community of learning so that we could bring DEV into the house with the staff we already have. Every operations person is now also DEV because we're learning how to do it and how to make our own tools. There's a couple of conversations I've had just recently. Some of them have to do with large governments, free letter, whatever. And they've got these problems that are very similar to you. They actually are trying to run software that they're getting from a third party. So lots of tricks that come when you're trying to coordinate and align people inside of an organization, get that much harder when those people are actually by definition in some other transactional relationship outside of your organization. And the things that were just said I actually kind of want to go back in time. So this is pre is like before Puppet was even called Puppet and I was a developer and Luke was a sysad man and we had a bunch of conversations and there was like I was reading all the agile stuff and there's like this community of agile developers and like you know who some of them are and you're reading the blog post of like Michael Febbers and you know Ragnwalder or whatever and like maybe some of you guys know them and some of you don't. But one of the things that we very explicitly wanted to do was two things. One is that there needs to be this community that elevates operations right like that recognizes like who in the dead world at that time this is going back to like 2004 time. These people that had done these things that were talking about and you can like point to them and they were giving conference talks. But there wasn't really, there was no velocity conference. There was no DevOps conference. So like who are these people that are doing these things? We know they exist because like we see these services. We know they exist because like they have to. So why are they in this community? And I think that that has happened in a way that like it's an existence proof that this room is full basically. Like we have this community and that we can leverage that. And then the other thing that we explicitly had a goal is that you want people to be at the pub by four. Right? Now for Chris that could be the dance recital by four. Right? But that and this kind of goes back to the point I was trying to make earlier and I hope all of you guys understand this. It's not it's about the job that you do. It's about improving human experience. Improving human performance. Right? And not to me if we can continue that and I know we have a long ways to go and I know that lots of people go to work and they have like mud and blood on their hands and like the pages going off till 3 a.m. whatever. But incrementally we're getting better. But there's things that are better. And that allowing you to go spend time with your family, allowing you to go to the pub whatever you want to do with your time. Like that to me is an improvement in human experience. Do you have a question or comment? Here's one in the front. Or back. So one of the things that in our organization that we faced is when we started doing DevOps seriously all of the developers working on the team were quite busy doing let's quote-unquote bigger work and there were some of the smaller things that actually makes the developers life easy. We didn't have time to do that. Like incorporating some chat ops or writing some guides on how to deploy certain things on how to write it creating custom made developer environments etc. So being an educator at heart the solution that we found is actually hiring interns. So we went to local universities and hired interns and here's the thing no seniors or juniors hire green interns who did not start even developing early and you can actually shape them to make those tools for you. That's how we have chat ops on our teams. That's how we have automation driven from code check-ins etc. So I always love when people work in such a way that it helps you get better at what you're doing. Like I feel like if you're really good at whatever the skill acquisition of chats or music or whatever like by playing the song by playing the game you should get better. So we should all be getting better. So bringing green people into me like that warms my heart that's an example of organization learning. I actually want to bring it back to something that he said because I was cringing a little bit when he said crap work because a lot of stuff that you're talking about sometimes is actually super necessary. So like there has to be if you're serving the larger purpose of building the greatest possible cathedral the biggest possible organization you can but sometimes you do things that in the moment you know like this is not fun for me. This is not what I want to do but you do it because that's what organization needs from you at that time or maybe you get a bunch of interns to do it for you. So you mentioned organizational learning a couple times and I heard some of those things about organizational behavior in terms of making devops great again can you speak to that in terms of what has been effective for organizational learning? So there's a talk I gave in 2013 with Rossi who has a paper with the seven dimensions of organization learning from Walk into Mercer and I feel like if you take and do that questionnaire and then reverse the questions and make some statements then that gives you very actionable ways to improve your organizational learning. That's another talk Have you seen that talk? John Kess So basically that framework that is from the 90's so this is the thing that's like crazy so I stumbled upon this paper that someone wrote in a dissertation where he measured the organizational learning and correlated it with job satisfaction in this R&D department in Taiwan it's like the kind of stuff you do for a PhD or whatever and then I unraveled all the references back to this model that these two women professors had made in the 90's and then I was like I felt like I found the holy grail like this is like almost because the only thing I got frustrated with trying to help people along the way is you go into an organization two different organizations might look exactly the same for all intents and purposes they're relatively the same size relatively the same kind of business and you give them the same set of tools, the same set of instructions and one will just basically face plant and another one will like totally blow your way like how good the results are and the thing that I came to realize or recognize, at least what I believe is that the difference is like how they were set up culturally to learn as an organization like what are the impedances inside of your organization right now that actually keep you from learning and there's a long list of them like is everyone able to question things regardless of their rank in the organization is everyone reflecting on their work are you deliberately doing things to learn are you focusing on not just the work you do but actually how well you do the work like so there's this questionnaire each one of those I think has a potential to be inverted as a statement and then be made actionable like make those statements in the organization and you'll be a learning organization John Lewis give him the mic after this guy I had just a few cultural comments more from a management perspective a few things we tried to do at YP one is if you're in leadership we try to sponsor departmental meetings that wish there is no leadership other than lead engineers and we're not there unless we're invited and there's sort of a gap you have to overcome if you do that which is will they go away and come back with things I don't want to hear or will they riot right we get a bunch of angry lead engineers and engineers together and they come to your office with torches and pitchforks they don't although at the beginning they'll ask you to come and come and come back to me with what you think is the right thing to do and then second I encourage people to if they want to do results oriented management we always talk about that like we only care about the delivery of the work not face time not line of sight not kind of all this other bullshit as I say wildflowers not bonsai right so provide environments for great things to grow and happen don't pinch and wire and try to control everything the other part is you have to do the hard work of the asshole in your department you have to and everybody has those people that you've worked with and kind of what I look for is are the people that are different with different people find the bullies in your department that are really polite to some people but a complete dick to these other people so they're just under the radar because if you want to let go of the reins and let things evolve and run the way they should those people will get in your way and you have to have the courage so that people actually go off and make decisions without you being there or you're still managing things that's something we've tried to do to some success so while you bring that over here I'll make a couple comments so there's a quote by Ben Horowitz that says that the level of communication requiring organization is proportional to the level of trust or inversely proportional to the level of trust so if you have high trust you need less and less communication to do work and then all of us are probably at least I hope most people are now familiar with the ideas of the cap theorem and this notion of consistency availability and partition tolerance there's nothing about the cap theorem that actually talks about computers so if you think about the human systems involved when you have a meeting what you're actually doing with that human system is a right lock so when you have a company-wide meeting that's a global right lock of your organization and so when you set up structures where the edges can independently make decisions and take actions you're going to be a much more you're going to have a much higher blueprint and activity inside of the organization as long as you get to the level of trust that everything's going to be right John Willis I was going to save this one for last wouldn't the intercurrent have to fix those? I'm not kidding no I'm not going to say my comment that I wanted to say organization learning is an incredible amount of knowledge on people who have studied Toyota production systems and lean guys like McRafa, Steven Spears Jim Wormack and if you look at what Toyota did that was probably the greatest learning organization in modern IT times in terms of what they accomplished if there's any dispute from 1970 to 2010 they decimated the automation market in IT because they were probably the greatest learning organization I'm going to say there's tons of works studying how they did that so I would definitely look at McRafa, Steven Spears Jim Wormack and Deming so I just thought of a funny part so I had these conversations where at the end of it where they're like oh yeah we really want the DevOps but we don't want to change anything but we'll just use Docker whatever the tool is everyone wants to take the blue pill it's like how do we get the blue pill and we can just go back to whatever we're doing and it doesn't actually matter so I think I'll give you guys a break I'll give you a couple minutes back just to wrap up I wanted to say that I think DevOps is whatever you make it you can put energy into something and you're going to get energy out of it inside of the organization we saw this same kind of adoption with Agile where at some point a transition from people who are focused on actually trying to deliver software which is what the Agile Manifesto says the only measure of progress is code the only measure of progress is this one thing over here and they started focusing on the details of the methodology and the process it's like oh we didn't do this one thing exactly the way the process works so we're not Agile and that's not actually what the Agile Manifesto said it says we're learning how to do software we're learning how to do software by helping others do it and by doing ourselves so that's the spirit that I think DevOps should enter into all of this is that we're learning how to do this by doing it I can do what I do, you can do what you do but we do it and we help each other do it and we're going to get better and better and it's not a thing that sticks we're like oh yeah that one methodology you didn't use Jenkins the right way it's this one new container tool you're not on Unicolonel so get past all the fashion and tribalism to dominate some things, get to the principles that really underpin what we're trying to accomplish and help each other as human beings evolve to optimize the experience and the performance that each of us have together that's what I think we'll make DevOps state again and one last slide even though they look like shit thank you you