 What's up Chicago quick experiment that I like to do especially because I started talking to a few people and Heard some interesting company name. So just out of my own curiosity. How many people by show of hands are in a company with a hundred or more employees? Okay, keep your hands up 500 more employees a thousand or more employees Five thousand or more employees Wow, ten thousand or more employees All right, cool. All right, put your hands down My name is Jeff My talk today is hacking human systems and this is a different talk for me because typically when I give a talk I'm talking about something that I've done or I've implemented or some strategy that I've been working on this is largely going to be like a 400 person stream of consciousness as I kind of Dissect this idea that I've been working on about like how human systems work how they're cornered sort of at the center of everything that we Do I think people in the larger organizations will probably take some of this away big time just because of the whole Let's say bureaucracy of large organizations I'm gonna say a lot of things that you might sit back and be like, you know what I want to fight that dude All right, I'm gonna fight that dude because I don't believe what he's saying That's fine. Like I said, this is a thought experiment. So please, you know catch me outside and We can chat about it either, you know during the open spaces or after the event But please I want to start a dialogue about this stuff So if I see something and you're like, I'm cold and bullshit Please do it because I think starting a conversation about this stuff is really key Oh, I forgot to my $500 stopwatch. All right. Sorry. Just want to make sure we got time. So What are we gonna be doing we're gonna I'll introduce myself We'll talk about systems and what systems thinking is we'll walk through a traditional system that I think everyone should be familiar with Hopefully because you're getting paid We'll talk to examples of human systems in our field and redesigning our approach to solving problems Sorry, it's new clicker. I'm borrowing it from someone. So my name is Jeff Smith I work for a company called Centro that you've never heard of that you both love and hate I'll tell you why I'm the manager of production operations there. I've got a small team of about three or four engineers And we are supporting a Ruby on rails application prior to that I was with Grubhub for a number of years helping them build out their SRE team. You can find me on Twitter. Where we at? All right, you can find me on Twitter at dark and nerdy Feel free to email me. I also have a blog that I don't update all things dark calm I write a lot of medium posts to there so you can get links there if you want to reach out to me But I'm not hiding. So there's plenty of ways to find me Centro is a leading developer of digital advertising software. That's why I say you both love them and hate them You hate them because we help make digital advertisers more effective You love them because you don't pay a monthly subscription to Google search So that's pretty much how I sleep at night advertising is pretty much what powers the free internet. So That's what Centro does So Docker What does Docker have to do with systems thinking absolutely nothing? I just wanted to get your attention I notice everyone lean forward Right, so it's officially a DevOps talk. We can move on but we said it right? I can now officially label this appropriately and slide share So what is the system? Systems are sort of all around us and when you start talking about systems thinking you start to see How the world is made up and connected with all of these different systems and in the technology field We tend to focus on computer systems networking systems But the interesting thing is most of you don't have technology problems Most of your problems stem from the human systems that are throughout your organization, right? No one is like we don't know how to implement a cache layer That doesn't happen. It's not about the technology It's about getting the momentum to actually get the organization to move to do the thing that you wanted to do government systems Prioritization Resource management these are all human systems that are actually impacting your ability to deliver on your technological goals But for some reason we sort of ignore them and we don't pay attention to them And we don't really know how to navigate And manipulate them the people that do are called, you know playing office politics But in reality those people that are playing office politics are getting shit done a lot of times Or the very least gonna raise So what is the definition of a system a system is an interconnected set of elements that is coherently organized in a way that achieves something And we're gonna focus on the something part because that part can be a little Misleading or disingenuous when we think about systems. So there's three components of a system You've got elements interconnections and purpose The elements of a system are all of the parts the resources the agents the people the computer systems They're all of the things in the system in which we measure the stocks Which we'll talk about a little bit that we measure to know how the system is behaving or perform Interconnections are the connections between those elements. How do those relationships impact the way the system performs and then there's purpose What is the point of the system? Now the interesting thing about purpose when we said, you know, we're gonna talk about the something Purpose of a system is defined by its behavior Not the bullshit that you put in the document It's about what the system actually does and we see that a lot, right? Think about since we're in Chicago think about the beautiful soda beverage tax, right? The stated goal is we're going to reduce calorie consumption of our constituents We're gonna make them healthier and live a more active lifestyle. No, we're paying for pensions That's the point of the system and when you look at it from that perspective You'll notice that all the elements and the interconnections of that system are focused and targeted to that goal So it's very important that when we look at these systems we think about what is it that it's actually doing versus what it is That we claim it's doing So at the heart of all systems are stocks and Stocks are the elements of the system that you can feel count or measure at any given time We're gonna put a lot of emphasis on feel Because so much of what happens in some of these human systems centers around emotions Emotions of all types typically emotion is at the center of something We like to pretend that we are Logical decision-makers, but in reality it's emotion. That's at the center of the decision and then we use logic to back that up So here's the basic stocks how a stock basically works in a system, right? We'll take our bank account. I Do have more than a hundred dollars in my bank account. I was just No, I don't We've got the stock our bank account our dollar amount And we have inflows and outflows to that stock inflows deposit increase the stock withdrawals Decrease the stock and the system is largely calibrated Based on the number based on the value of that stock We have this thing called feedback loops and a feedback loop is formed when changes in the stock Effect the flow into or out of that same stock, right? So continuing with that same example, we have a concept of a balancing loop as Your bank account gets lower $25 is probably more accurate for me You work more hours so that you can increase the number of deposits You might also slow down your withdrawals Right once your bank account starts getting low. You're like, all right. Maybe we won't go to Ocevall this week We'll go to Burger King or McDonald's That's called a balancing loop it make sure that that stock stays at a at a reasonable level Balancing loops to work the other way too. Now if you got a million dollars in your bank account, what could happen? You might work less hours You might spend more but you're probably going to spend more right no point in letting that cash sit in there Might as well get a boat Something you can do Then we have this concept of what we call a reinforcing feedback loop and Reinforcing feedback loops are when the stock is able to generate more of itself simply through its existing Common example of that is your interest rate or your the interest rate on a savings account right as the stock grows Your interest rate is a percentage of that and you get more stock Sticking with the Chicago examples corruption Corruption is a reinforcing feedback loop the more corruption you have the more it generates of itself and on and on and on and on So we're going to talk about everyone's favorite system the change control system How many have some sort of change control process in their organization? Okay, not as many as I would have thought Get on that shit guys the interesting thing about a change control system and and I'm being very Broad stroke and generic here, but most change control systems that I've seen Don't actually implement the stated goal of what they're trying to do right they are They are subject to a bunch of other factors that make the system behave in a different way Now the stated goal of a change control system is to introduce Changes to the system in a controlled fashion with mineral minimal disruptions. We talk a lot about risk management, right? That's always a big thing But when we talk about Change management, and we look at the actors and the agents the elements in the system We realize that sometimes the participants in the system aren't the people participants we need to deliver the stated goal or outcome so here's just like a Rough sketch I did of a generic change control system, right? We've got a bunch of business changes business operations that happen. So our two stocks in the system are changes in risk Business operations generate changes When we have enough changes or through some regularly scheduled process We have this change approval process right this cab board the cab board approves or denies those changes and Those change approvals or denials increase risk Once risk gets to a certain point. We do some due diligence. What does that due diligence usually look like? It usually looks like a manager who is way too high to actually know what's going on about a thing Going and talking to the SME that just submitted the change, right? Do you think this is dangerous? Yes That's why I submitted it It's gonna take the whole system down trust me don't do what I just asked you to do it's crazy So we end up wasting a bunch of time so that the person that originally asked to do a thing is asked if they should do a thing So it makes you wonder like What is this piece actually doing? It's risk management theater in a lot of ways And if it's risk management theater if we're not actually reducing the risk Then what is the actual point of the system? in my worldview This system is typically about limiting the amount and frequency of change introduced to the environment That's what we're actually talking about Because what happens when the change goes bad, right when that networking change goes bad, it's like all right guys We're only going to do networking changes on Thursday. Okay, because that's reducing the risk, right like sure We're not taking it down during business hours, right, but they're still an outage Then it moves to all right. We're only doing it on Thursdays at 2 a.m And it's like great now. I'm tired and I'm doing this networking change That's really reducing risk My favorite is the security audit so that the inspiration for this talk started at DevOps day Seattle We were in an open space and we were talking about Security processes and how we manage a security controls and how frustrating that audit processes And we're all sitting there griping about the different types of others We had and these things that they were asking us to do and how we had to document it and I stopped and I thought I said well None of us have actually challenged the validity of these things that they're asking for And we quickly all agreed like well, yes shit. We should be doing So then why are we so up and angst about this ask? Why is it that we wait until the audit comes to be able to like spur into action And do all of these things to make sure our audit assets are covered So and thinking about it. I kind of give it this term audit anxiety, right? There's usually a bunch of things that you have to do in an audit, right? Make sure you got your documented policies. You got to make sure you're enforcing those policies You got to have proof that you're enforcing those policies show me when this thing happened and how you handled it And then also you have to prioritize these fixes But why is it that we wait for the audit to actually start to do these things in earnest? And typically the larger your organization is the slower this rate of change unless you have like a Security czar that is like always down your throat about it I got a guy that a friend of mine. He's got a five-person team and three of them are dedicated to security, right? So that shows you the the sort of Prioritization that if nothing else the boss puts on the organization So I started thinking about it and I tried to come up with like a system flow of what that might look like So our stock is sort of our known risks, right? That's the thing that we're measuring and Through different activities. We create prioritization pressure which flows into those known risks, okay? So internal findings is a big one, right? You're doing a bunch of stuff. You notice that, you know, I don't know Open SSH hasn't been patched recently or something to that effect That creates pending security work that now has a backlog that now is documented somewhere So we're like, okay, we know about all these risks. You also have publicized exploits, right? Something that's out there in the wild people hear about it. Suddenly. It's a big deal But then you also have the time until your next audit, right? And the more the less time there is to your next audit the more prioritization pressure Builds up why because you now have identified these known risks And we decrease the outflow for that stock is the actual security work actually implementing these fixes and changes But the interesting thing is that prioritization pressure isn't necessarily Logically Equivalent to the actual finding And there was no better example of that than Heartbleed, right? When Heartbleed came out It wasn't the worst security vulnerability we've ever seen but you know what it did have a kick-ass logo Right it had a kick-ass logo and that kick-ass logo got it on the news and once it got on the news everybody's boss was like Hey, you hear about this Heartbleed thing. You're like Yeah, but we've got like 15 other vulnerabilities I've been telling you about for six years that we need to patch how many people patched Heartbleed within the first week Is that a normal rate for you? Heartbleed had a had a Mismatched amount of prioritization pressure, but it moved the organization in a way that forced you into action So I started thinking about that and I started thinking like how can we? Manipulate our systems to produce that on a regular basis We can mimic other traits of we can mimic the traits of other elements How do we sort of create that buzz inside of our organization that we created with Heartbleed? Might be hard because we don't have newscasters on our side But maybe we could get Tom skilling or someone to just drop a note in the middle of his weather forecast like oh, and there's a new vulnerability for Apache We can alter the interconnections between elements in the system What if we change the reporting structure in terms of who's notified when a vulnerability is found Or what if we could change who's on the hook for when a vulnerability is found and not patched? What if we change the rules all together? What if we said as An organization we're going to commit to publicly disclosing any vulnerabilities that we have that we haven't addressed within 60 days ourselves That generates a lot of pressure internally right because now you're saying all right We're going to air our dirty laundry if we don't fix our shit now I don't know how you're going to find a CTO that's going to agree to this but It's an interesting thought experiment to figure out like okay What are the levers we can kind of pull to force or push our system into action? Docker just make sure you're with me make sure you're still paying attention right. We're still not talking about it, but Feel free to lean forward Configuration management my favorite how many people are using some sort of configuration management salt puppet chef Ansible okay, not as many as I would hope how many people are working on it right now go ahead raise your hands up Come on okay, and how many people are finding that it's not technologically problematic, but more Resource planning or prioritization to actually get that implementation going Because config management is a pretty easy sell right So the fact that we're struggling to actually get it implemented everywhere is sort of like a tell-tale sign of exactly what I'm talking about So what is awesome about config management all this common stuff that we hear it's easy to recreate environments We get traceability of changes We get source code management that allows us to help say you know who implemented this change to the discussion around it peer review The repeatability aspect is awesome because you know even though there's a command that we're supposed to execute I guarantee you if you've got ten servers Bob is going to screw up server eight and You're going to be wondering what the hell is going wrong with one eighth of all of our requests So the benefits of config management are pretty easy. So what is it that drives us this sort of adoption a lot of times? our stock in that system is the pain of failure and What drives the pain of failure well sort of your rebuild frequency right? Company growth is another one as the company grows you've got to add more servers that becomes more painful than you say man Why are we doing this when there's 15 tools out there that will help us with this and eventually you start working your way towards automation? So again, how can we sort of simulate this thing because like hardware failure is sort of out of your control And unless you're a large organization like Google chances are you're not experiencing hardware failures that much right like the difference between like Google and You know, I don't know the uber of knitting They're gonna have very different like hardware failure rates Company growth kind of outside of your control as well. So how can we manipulate the system in a way to do that? Well, someone has already done it for us. Who's familiar with our buddy chaos monkey? Right chaos monkey is gonna go around and kill servers at random The whole point of it is to make sure that you can rebuild these servers quickly and that your systems can handle the failure With chaos monkey, you have knobs that you control how frequently you're going to do it So you can now manipulate the inflow into the pain of failure And you can say look boss like you know, these are the things that are happening in our environment And this is problematic now I'm not saying it's as easy as you know firing up a VM launching chaos monkey and walking away That'll get you fired, but You know, it's another example of like how we can take the system Manipulate our inflows to try to produce expected behaviors And that's really what we're talking about We're talking about manipulating the system to provide our desired behaviors and a lot of time that is gonna be a play on emotions Whenever we make a business case for something they always say, you know, oh look at the ROI build on the ROI and push that out No, that doesn't always sell it right what sells it is the fire and brinstone, right? What sells it is the boogeyman tails, right? Well, you know, if we don't patch this some 400 pound hacker is gonna be able to manipulate our systems and take everything down suddenly now We're ready to spring to action doesn't matter how true or false that might be We're generating the emotion of fear of panic and then we're backing that up with logical reason So how do we look at these systems and sort of artificially create that behavior? So I wrote a post called I don't understand immutable infrastructure because I don't really understand immutable infrastructure And what I mean by that is I don't necessarily subscribe to all of the reasons why we say we don't want to change a running system I can think of a number of reasons why I want to quickly manipulate a running system so I put it on Twitter like you do and Naturally someone wanted to fight me so we had a nice hundred and forty character brawl and at the end of it he said something that Made everything much more clear to me and sort of is is is sort of the point of this conversation He said to oversimplify it's the difference between you can automate everything to you must automate everything Despite all of the rhetoric or or BS and remember we can fight outside later Regarding immutable infrastructure this stuck to me because it's like Okay, he's really using immutable infrastructure to force a behavior that he couldn't consistently get before He's using it to say hey guys if you don't automate this thing Right, you're not going to be able to get your change out into the world Because you can't log in you can't SSH into this server and just make a change you just can't do it You have to automate it and that really stuck to me because that's it All right, this is exactly what we're doing. We're again changing the system to behave a specific way so that we can elicit behavior Because even when you have config management, it is always especially especially in times of crisis It's always easy to say I'm gonna make this change real quick and then I'll Backport the change in the config management, right? I'll do it hot and then I'll do a fast follow Right, and then someone says hey, man. You want lunch and you're like hell. Yeah All right So now your elbows deep and some barbecue and you're like did I do that? Server blows up and you're wondering why your things broken next time the new server comes up So my challenge to you Is to think about the stocks that are in your human systems What are the things that are sort of driving the behaviors that you're doing fear? Pain the size of a work queue rate of change some people just freak out when they see a backlog number too high They're like oh shit. We got to fix this start closing tickets and there's all right canceled canceled not gonna fix not gonna fix and Somehow that brings about a sense of calm, right? You're like It's a bunch of shit. We didn't do Thank goodness, but that's not helping anybody. It's not helping anybody so think about the things that Are driving your emotions in your organization and think about how you can manipulate them It's not a terrible thing doesn't make you evil might make you an office politic player But it's worth it because you'll be able to deliver on the things that are important I'm not gonna lie. I got ahead of myself I actually said what I was supposed to say in this slide in the previous slide So I'm just gonna spend a minute looking at you and then we're gonna move on We make decisions on emotion tapping the emotion drive the behavior So what do we talk about? Systems are made of elements interconnections and purpose Those interconnections those elements are things that we can manipulate in order to create our expected behaviors Stocks are the basis of any system and it's often the most difficult part to identify in a human system What are these stocks that we're playing around with? feedback loops balancing loops and reinforcing loops balancing loops help keep our stock at a reasonable level we get the deal what reasonable is reinforcing loops are things that Allow the stock to grow on its own corruption All right tech debt Tech debt breeds more tech debt right because you're building poop on top of poop Make poop tackle problems by focusing on what feeds or drains our stocks Understand the inflows and outflows of those emotions Because those are key to managing our human systems. Thank you. Hey, thanks chef That was a great talk. My name is Jesse Martin from Eli Lilly and company in Indianapolis And actually I had more of a comment actually because I thought your comment around how do you take some of this security patching? seriously something that we actually had to do was we have Kind of this culture of you've got to incorporate security in your application development supports problem or quality system We actually had to create a separate kind of infosec organizations with deep experts that were really paying attention to this because exactly to your point you just couldn't get Focus on these things without an organization and kind of people that were attending the you know the Various hacker conferences and really really current on security to actually raise the alarm and actually have very very senior kind of executive Reporting responsibilities that is actually separate from your quality organization to just kind of lend a hand So that's something that we haven't I'd say it's kind of it's working for us so far, right? Yeah That's a common pattern that you're starting to see Specifically, you know doing it in sort of like a DevOps structure where you know these infosec people as opposed to being these outlets that are on top Screaming about what you do their partners and they go in and they help you because in reality the developer is not incentivized through security He's incentivized through features through shipping code and when those things aren't in alignment Which is a common problem in all organizations is incentive alignment when those things aren't aligned That's exactly what you ran into so having a separate work to partner with those guys and to say like hey Let me help make it easy for you Yeah, that's always a winning strategy So I'm just curious how you can I Any tips on identifying what the actual stocks in the system are Yeah, so when I was actually writing this presentation and thinking about like change control and things like that really it was about thinking of the times when Something move the system in a direction that it was uncommon, right? You sort of have the system that sort of stagnant You say yeah, I know change control is always going to take me a week and a half because that's just the process But then look at the items that are the outliers and say okay So what was different about this situation that made the system behave in a particular way? And when you think about that and try to break up all of the parts that were involved in that decision usually The stock sort of like jumps out at you and you says oh it was because there was all of this fear around heart bleed Potentially opening up our world to a world of hurt, and it wasn't necessarily that it was actually More dangerous than any of other of our other vulnerabilities It was packaged better and whenever you talk about packaging a problem and selling it You're talking about some emotion of either fear or joy And then you just tap into that It's not a science by any means great. Thank you so much Jeff. That was awesome