 So I want to start off with an interesting little description of what's happened to me here over the last two or three months. So I've been at a number of human factors conferences, user experience, user, whatever the time of the day might want to be, programs, and people ask me what kind of things I've been doing the last couple of years. People I haven't seen for a while, things like that. I like tell them about doing DevOps work. And they look at me like, wait a minute, you're, you know, human factors and user experience. How did you get into, why is it got like human in a place like this? Like maybe a bunch of you might be thinking, why is it got like this in a place like this? And, you know, the bottom line is because historically human factors has been in high cost, high risk, high cost of failure systems. That's the history of human factors. You take it back to aviation industry, military system and things like that. So in a lot of ways, you know, when I get that question, I feel like, no, this is exactly where I belong. Exactly the place I want to be in. The other thing about it is that having done a lot of work historically for phone companies, for Sun, for Hewlett Packard, companies like that on back end systems, this is not only someplace where I have an historic background doing this kind of work, monitoring systems, configuration systems, tooling systems, but it's also a place where I couldn't be any more entertained than I am by the kind of work that's involved. So that's why a guy like me is in a place like this. And let's see whether I can get this thing to advance. We'll just do it manually. First thing I want to do is say thanks to the DevOps community in Kansas City. This is a great meetup group and I really enjoy participating in it. Great leadership. You know, I've named a couple of people there, but I know it's a lot bigger group than that and has been and I really appreciate that community. I really appreciate the conference of organizers and I really appreciate the sponsors. Another thing I want to say is something about Kansas City itself. So I spent a big part of my life living in this town, crowned sooner to me. Historically, I think it was a place for jazz concerts, various things that have been held at this facility over the years. And when I thought that I would end up on the stage at Crown Center, I didn't think it was going to be doing this. So I'm going to say something about four people, Kansas City history, saxophone players, and we go from left to right, Coleman Hawkins, Ben Webster, Coleman Hawkins was from St. Joe, Ben Webster from Kansas City, Missouri. These guys were the cream-the-crop of tenor players during the big band era. These guys started in Kansas City. This is their turf. Charlie Parker from Kansas City, Kansas, completely revised how we think about saxophone being played in jazz. And then, farthest on the right, Kerry Strayer, a good friend, not into Kansas City Native, but a guy who spent the bulk of his professional career in Kansas City, I had the chance, the opportunity, the pleasure of being one of Kerry's students, and he was a great part of Kansas City Jazz scene. So now let's move on to talk about human factors and DevOps. So the first thing to me is when we think about how people operate in complex systems, is a term adapting, okay? And when I started getting involved with this, one of my heroes in this business is Kosinkai Kawaguchi, the founder of Jenkins, Hudson, whatever you want to call it from any one point of time. And what's beautiful about the story about what he did was it's the classic adaptation story. So here's a Java developer inside son, and he's got deadlines, and he's got objectives, and he's got things he's trying to do. And what he runs up against, the thing that's preventing him from moving forward, is as he gets code ready, and he's trying to get it into a test environment, and the operations team comes back and says, yeah, three months, we can get your code on there, and then we'll check it out. And he's gone, I can't wait three months, I gotta find someone around this. So the next thing you know, he's rummaging up and down the halls, and my wife used to work at Sun, I did a lot of contract work at Sun, and I can remember walking through there and seeing the machines stacked in various places all over the place, the people, I'm done with it, it's over here, and he went and grabbed a bunch of them, and he said, okay, I'm setting up this environment. I got, I'm running on VNs on here, and my development group, we're gonna push our code onto the systems that come the closest to what we need to do to be able to make sure that we can move our code forward in a timely fashion. So again, the key concept there is adaptive behavior, and I know this is coming up already, and that is the notion of development walking into operation space, operations walking into development space. And the thing that's so critical there is that as we adapt and as we try and say, okay, how can we continue to move forward at the pace we need to, boundaries need to get taken apart. So all these separations of who does what and what it's about and all those kinds of things, in some ways they have to disappear, developments doing operations business and operations doing development business, and the silos that we exist in don't necessarily help us move forward, but better sharing, better responsiveness to each other as we begin to understand a little bit more about each other's space, those kinds of things, they do help. Picking up practices from another group and saying that would help me and vice versa, that helps and that helps us move forward, and also we end up with a lot broader perspectives about the constraints, the limitations that each one of us is up against. So I threw this in there because I thought it was kind of interesting and I've got a citation for this article, it came from a group in Brazil doing research just on the kind of language that's used to describe DevOps, kind of feeds off what Erin presented earlier today, and the thing is that all of these different things that we're doing down here in the corner, eventually it says, aim at improving process quality and product quality, and so we're doing fairly structured activities and we're shooting for improvement in quality. Certainly that is a key element of human factors, applications, and objectives for all these environments. So when I think about this in a very simple way, I think about Venn diagram, we have these very separate operations and suddenly we're going to say, okay, it's really not very helpful to keep these things separated. But the minute we don't separate them, all of a sudden we've got this really crazy looking world of all this overlapping stuff and overlapping responsibility and who's responsible for what and where did that new set come from? I never even heard about that before. And all those things, and that is very disruptive. It goes back to the things Paul was talking about at the beginning, and that is that disruption is a difficult thing to deal with for any of us because we're trying to move forward on objectives and all of a sudden somebody comes in from the side and goes, oh, you've got to do this completely. And I'm going, oh, I don't know whether I can figure out how to do that. But the thing is we have, we've got to pick up and we've got to act. So we do. So now we've got all the Lego pieces and how the question becomes, okay, how do we get all these Lego pieces back together again? And it doesn't matter whether you're a shop that, I guess maybe I should point down this, you're a shop that's been on this end of the world, and I was joking with somebody the other day, I've worked with shops where they're into the world and they've got one guy whose job definition is different files, you know. The developer's got the file on here and the file on the production system is you run diffs on those. You spend your job all day long just to run diffs. And then you've got other extremes where we've got people who, I mean, the guys I used to work with, the subject matter experts at Sun Classic, you don't ever push anything onto a server. You write code. The code pushes it onto a test server. You know what's going to happen. You execute the code. And so there's a very informal, very formal, but everybody's got to figure out how they're going to move forward. And so now we find ourselves relying on a whole lot of services. And there's, I think, certainly the people that I talk to, there's a big impetus to use open source services. And I've got, I don't care. I like anything that helps me do my job better, so I don't really care about it. But there's some really cool things about open source. But the thing is now, all of a sudden, we've got all this list of all these tools and all this tooling has to come into place. And then everybody has to figure out, okay, how does that work? So I've got big services. You know, like you think about Git and Vagrant and Puppet and Jenkins is just to name some of my favorites the things that I've played with and had to get my hands on on a regular basis. And then you say, yeah, but they don't work on their own. They can't operate on their own. You have to then say, okay, what about, what's it going to take in terms of SSH configuration or SSL configuration or IP tables? What is it going to be? Maven or Composer? What's going to take to run underneath those things to make them all do what they can do? And then, of course, we go a step from now and we say, okay, and how many different code bases are going to have to exist in this environment in order to be able to work. So which ones of these? Which 10 of these do you care about and is going to have to work for you and for your product to be able to deliver to your customers? And then, of course, we can also throw right into that. And then how many different versions of Linux are you going to have to support? And I know people who aren't just supporting Linux. They're supporting Linux. And four still running versions of Solaris and a couple of different versions of Unix and everything else under the sun. If you look at this and you think, well, the world got put back together again. And the thing to me that is so cool about that is that you think about the complexity of that system and I look at the complexity of that system and I think, gosh, that's quite an accomplishment. The thing is, the people in this room, that's what they do for a living. This is what we do. And so we put it back together again and we make it work. Just like we made it work when it wasn't maybe as easy to do. The thing that I like about it from my perspective from having been the sharp end of this a couple of times, what I like about it is I felt a whole lot more comfortable once I got it back together again with a reasonable set of tools that I did beforehand. So it's back to not what middle management and upper management need, but what I need to be able to sleep at night. And I have been in that position and believe me, I really appreciate what these tools could do for me. One page too far because I've got something else down here that I want to, okay. So now I want to talk a little bit about my experience having been involved in actually executing on this. And the things that I would say about it, first off my experience was that the software development team couldn't have been any happier. The software development team went from a position where they were expected to roll stuff and roll stuff in a hurry and they had a minimal amount of confidence in what the outcome was going to be. And then they got hammered when it didn't work. But at the same time they all had a fairly high expectation that it wasn't going to work. And so then they would spend their entire work cycle during their day trying to tool up to resolve discrepancies and issues that they had to deal with because they had no way to test those things appropriately to begin with. So now all of a sudden when they've got an effective DevOps environment, they couldn't be any happier because they are doing all of the same kind of testing work that they needed to do to begin with and they knew it. But now they're doing it on a system that doesn't run production and they're not putting themselves at risk and when they run something in production they got downright cocky about it. They knew the thing was going to work and that's where everybody wants to be. I got really a lot more effective at being able to help them do that because I'm in a position now where I know what the system configuration needs to be. I've gotten rid of all the ministry services that nobody knew what they were doing and then still had to maintain and really drag down what the state of the system was. The other thing was that I was increasingly confident that I had a way to abstract what we needed to run for production in a way that I could run up new services whenever I needed to. That if a system failed or if I needed to roll over the notion that the way to move forward is not to advance the existing system to build a separate set of nodes, get those nodes up and running and then swap them out. Those kinds of things, then I started to feel relatively cocky about the fact that I could keep these things running effectively. As a team, we got much more effective at delivering service, both to customers and to business in terms of doing upgrades, in terms of keeping service running effectively, and that's a pretty fantastic feeling. Then the developers and I were thinking, okay, now is when we get to take any of these ugly, unpleasant things that we know exist in the code base that we know exist in the way we've run operations in the past and we can start tearing these things apart and get things back where they belong, which of course, I'm sure all of you know, this is the point at which management jumps in and says, oh man, now we can really go fast, we're ready to go, we can do all kinds of stuff that we couldn't do before and faster, better, cheaper, just got to be a real thing and we're really feeling that we can start rolling all kinds of new stuff and you jump to light speed and all of a sudden you realize, wow, all those ugly places that I didn't get to, how's this going to bite me? How can I keep going with this? And you'll wonder what's going to happen next, okay? And so now we get into human factors engineering and of course, the whole time I'm doing this, I'm still a human factors engineer and I think about everything as a human factors engineer and that's what I do, that's what my mind is and what my understanding is and what I take into the situation is I'm thinking, if I can understand how we, both as technical people and as people in general make sense out of the world, then I can help design things, design technology that improves people's ability to respond as opposed to technology that becomes an additional burden for them to have to deal with. And so I'm going to start looking at things a little differently about what we're doing in the DevOps world. And so let's talk a little bit about human factors engineering. I'll give you a little background on that. First and foremost, it means taking a solid science-based view of how people process information. This is a very standard graphic in any kind of cognitive science, experimental psychology, human factors program and it comes from a group doing, work on human performance at the University of Michigan. David Curious has been doing this stuff for a long time, spent a lot of time working with DOD projects and they show massive gains in performance on really complex, really high-cost systems. And basically what it amounts to is you're saying that there exists out in the world some source of information, whether it's a display screen or a heads-up display in a cockpit or whatever it might be. And as a person, you've got a certain set of senses that respond to that information. Typically, it's vision and auditory senses but there's other ways that you sense things as well. They don't apply so much in computer systems but certainly they do in aviation. Once that information comes in, then your brain goes to work, your experience, your memory, the things that you've learned go to work in processing that information. Eventually, based on the requirements of whatever task you're doing, you're generating either vocal response or a motor response and that output goes back into the system. So if you think about this and you think about it in a classic aviation sense, I mean this is the standard human-in-the-loop process control, process management kind of environment. And it's really easy to think when you are the operator of a helicopter or an airplane or whatever it might be that you know that the only reason it's staying in the air is because of you doing things. Then you place yourself as a really critical central component of how things work. We sometimes kind of tend to lose that when we think about different operational environments but it's very important to be able to do that. Okay so now on with these kinds of things we can start thinking about how do we operate off of this. So the first thing that we're doing is we're designing for performance, okay. We're focusing on people as the center of the system and when we talk about the system, the system is the breadth and scope of all the things that somebody has to touch to be able to execute a task, all those things that come in. So this is a very broad environment that we're talking about. If we're in a situation where the user can't fly the plane, the user can't drive the car, the user can't effectively monitor a power grid, then that system really isn't operating, it's not functioning. So good design means a number of things but one of the first things it means is it means taking advantage of what we know about human information processing because there's going to be a person in there operating this thing. Follow up on that and this goes beyond, and I think all of us that have been involved, we hear this before but it becomes more and more critical all the time. If we're doing good design, then we need to be public and explicit about what the design's about because in the absence of that you end up with all kinds of participants in the design, everybody's got a different, everybody's looking at this thing from a different perspective and has a different set of expectations and in the lack of that public and explicit definition a couple of things happen. First off, expectations that some people have get left out of the story and the story is what helps drives how we develop and how we get our operation system up and running. The other thing that happens is that all of a sudden the person gets left out, the end user gets left out of this story and we're building things to pass unit tests and God, I love unit tests but in the end of the day if they don't pass, if stuff doesn't pass unit tests it probably won't work at all but the problem is that it's a long way from unit tests to a truly functional usable, useful system. We're set by ISO standards and I know some of the people on the team that wrote these ISO standards for human factors and the descriptions they give are effective, efficient, and satisfactory. You have to be able to demonstrate that this software is effective, efficient, and satisfactory in the hands of the person that's supposed to use it. And I get frustrated for lack of a nicer term when this becomes a checklist because that's not in there because of a checklist. Again, I know the people that wrote these standards and I know it's not a checklist item. It's a reminder that if you don't ask yourself what would it mean for somebody to be efficient in this situation? What would it mean for somebody to be effective in this situation? If you don't ask the question, number one, you don't really know what you're building in terms of a system but then the other thing about it is also you don't know how to verify the design. You don't know what to do in order to determine whether you've got it working or not. As the previous two speakers were talking, what occurred to me was what I think is an interesting little sidelight here. And that is so... In the soundcheck, I spent an awful lot of time around race cars in my life and my hearing is shot. But 34 years ago, I started taking my son to Indianapolis 500, which was a big thrill for him because he's known about it his whole life. We go into the museum at the Speedway and there's this huge line of like 15 cars that were either winning cars of the Indy 500 or cars that changed the way the cars were designed. And this was over a period of about 20 years in this one row. And I started him and went in and I walked in and I said, okay, I want you to point out how come this car won in 1952 and the car sitting next to it was a dominant car on the circuit for the next four years. And he would look at it and it's like he's gone, I don't know, it's a different paint job or something, you know. And I'm going, okay, let's back up here and let's take something very simple. Let's look at the wheels and the tires. Once we've distinguished what's different between the wheels and the tires on these two cars, then I want you to look at the brake system and then I want you to look at the shock absorber system and I want you to imagine as you can see what little you can see in these old front engine cars with big bodies on, I want you to imagine based on what you can see what's the suspension like on this thing. And all of a sudden he's starting to make very astute recognition of dramatic changes in structural components of these cars as he goes. And the thing is that when we build a system, we go in and we say, we think this is going to work. We are making a bet. We are making a bet that we are smart enough to know how to build a system that will work. And the thing I always challenge teams is when you go play cards someplace and you've got a good hand and you make a bet, they don't just let you grab the pot and walk out. You have to stay till the end of the game. You have to see whether you won or not. And you have to go say, okay, what's the criteria? How will I evaluate whether I won on this design? Because the thing is just like playing cards, it's just like driving a race car. It's one hand. It's one race. It's part of a season. You have to decide, did you win on this go round or not? Because if you didn't win, then you got to figure out what you did wrong. What was your estimate wrong? What was your expectation wrong? How come you picked the wrong shock absorbers for that car to make it win on this racetrack? And you have to be able to move forward. So the same research work that we do in figuring out how people process information, those same methods are extremely useful in being able to evaluate the effectiveness of how people perform with a piece of software, sitting in the control seat of the shuttle, whichever one it is. It's the same game. But we have to figure out how to stay in that game long enough to know whether you won or not, and then decide what the team has to do next for the next hand, for the next go round, for the next racetrack, whatever it might be. So this is the point, because as my son looked at that situation and said, you know, I'm beginning to understand the evolution of these race cars over a 20 year period. The thing we have to do is we have to start getting smart about what's the difference between something that's effective for me or not. And when we start to think about tooling systems, and I mean, it can be an IDE that we're using or it can be anything else, whatever it is, we have to look at that system and we have to ask ourselves some fundamental questions. And these are the kinds of questions that we can readily ask and we can readily imagine thinking about that help us determine which one of these cars is the right car, which one of these pieces of software is the right piece of software, what is it that I need to be able to move forward? And so the first point I have on here is that I'll be able to see what this thing does. I'll be able to look at it and go on, okay, I got the basic picture. I can see the difference between a friction shock absorber from the late 40s versus a new style hydraulic shock absorber. I ought to be able to see that they do very different things but I ought to see what they're able to do. I recognize the concepts and certainly the farther you are in as a subject matter expert, the more readily you see those concepts but then the thing to do is to pull in the person who's a junior person, pull in the person who's got a peripheral understanding of the domain. Say, you look at this, what do you think it does? What's here? How does this thing work? Anticipate the relationships. The guys in the DevOps meet up group know I'm a big puppet fan and but the thing is that if I show puppet to somebody and say, okay, when you install puppet, what are you going to have to have run underneath it to make it work right? I don't know. It runs on its own and I'm going, no, it doesn't run on its own. You've got to be able to see what's going to take to make this whole show work and then the possible outcomes I ought to be able to look at this system and say, okay, if I run this, if I run this environment, this broad system, what kind of outcomes can I manage? And again, I'll go back to puppet and the extent to which I've worked with people, when they look at puppet, they see something that will run one node and that's it. Each node is going to have puppet installed in it and that's it. The puppet will run and manage that node and I'm thinking, you've missed it. The whole power of this thing is to be able to run massive sets of nodes, not one node. And so, how do we get to that point? And so the question that I would answer, I just pulled a, certainly not picking on anybody, but I just pulled a set of icons here from the kinds of tools that we use and I'd like people, because there are ones that I'm familiar with and there's ones I'm less familiar with, but I'd like everybody to think about asking these kind of questions when they hit these tools and say to themselves, what do I feel like? Where am I at with this? And the reason I'd like you to think about that is because what I want you to do is I want you to push back on, because this is open source, because it's our development environment, because it's our DevOps community, push back on those companies and say, I wish I could see how this works a lot more readily than I do right now. I want us to start demanding those things. I don't want us to be in a situation similar to some companies that I've spent some time with who basically said, hey, I'm in my dark room with my black screen. I work the magic. I don't need this, because those companies don't exist anymore. Okay. Second topic here and I spent a really fun amount of time with a human factors person, David Woods from Ohio State University. David and I worked together in NASA days so many years ago. And I was doing the launch processing. He was doing failure analysis having come off of doing work on the Three Mile Island. It was really fortuitous to meet with him at that conference and talk with him about this notion of resilience, because the thing is, once we get these systems where we can look at them and I get it, I know what it does, then the next question is, okay, and how comfortable are you right now? In this broad operating space in which degradation could be very gradual and very comfortable and you go in in the morning and you say, gosh, that just isn't working really good, but this degrades at such a soft rate that I don't care if I spend the whole day because the customer will never know the difference. Or alternatively, I'm in this space where I know this is really steep and right now this doesn't look good which means in the next 10 minutes we're down. The whole shooting match is down and I need to be able to know the difference between those two spaces and I need to be able to operate on that. That's really important. The other thing I need to be able to know is I need to be able to know where on that curve I'm operating at any moment because if I'm way over here and I've got a steep hill, I don't care. But I've got to know I'm over there. So these things, these kinds of things are extremely important for us to be able to understand. How do we attack that problem? And again, working with David Woods over the years, one of my favorite things, two things that he would always say, you know, we would talk about touring to prevent failure, managing to prevent failure, and he would just laugh and he would just, it's going to fail. It's just going to fail. So you've got to deal with it. The other thing that he would say is that you need to be at the back of mission control and you need to look sideways. And if you can look out of the corner of your eye and see whether things in good shape or bad shape, then you're in good shape and that's what you need to do. Status check on this. So what's your comfort zone? What's the signs of degradation? How can I see how things are going? What are my known risks, those kinds of things? What do I need to be able to do? You need to ask those same questions. So quick summary here. First of all, remember that DevOps is adaptive behavior and Kosuke adapted and then he moved off the mark and he's doing something different and every time all this new tooling comes we're going to have to adapt again. This world still operates and relies on the expertise of the practitioners. It is not plug and play and it will not be plug and play for a while. I'm okay with that but we have to recognize that that skill level still has to be there. I would like it if these DevOps tools better represent the objects, actions, outcomes, status of these systems so that we were better able to understand what's going on. And then the last thing is don't forget resilience. Don't forget resilience. Don't forget resilience. Keep thinking about resilience. And that's it. Thank you.