 Eric and all of you guys. And if you're here in the last session, you know we're going to be talking about all of you. And if you don't, you won't tell me. Perfect. You like that? Hi, I'm Aaron. So I'm a community engineer at Elastic. And I met a couple DevOps groups in Connecticut as well. So I'm one of the organizers for DevOps Days Hartford and the DevOps Meetup that happens around that area as well. So I wanted to talk a bit today operating human systems. So if you were here for my last talk, I sometimes take inspiration from words bots. So this is sort of a backwards application of studying how we operate complex computer systems and how it sort of applies to our interpersonal connections that operate those systems as well. So some definitions and level setting will happen at first. And then I'll kind of dive into some more practical things you can actually do. So does people know MTBF and MTTR terminology? Cool, I'll go through it. So this is what they literally mean. When you're dealing with computers, is mean time between failure or mean time to repair. So basically metrics for how your operations are doing, either the time between actual failure cases or how long those failure cases take to resolve. Depending on which version of the abstract I have altered them to say mean time between fallout or mean time to relationship as it applies to people. So if you were fine with your mother for the past year and you had an argument and speak for three days and then you're okay again, that year was the mean time between fallout and those three days were your mean time to relationship. So we also want to wrap some definition around complex systems because words like complex and complicated and simple are everyday vernacular. But there are some specific terminology as we start to think about complex systems theory and how people and machines interact. Has anyone ever seen this graphic before? Awesome, I'm really excited that I get to share new things with people. So this is the Kinefen framework and how it relates to complexity. So these lines between it aren't really quadrants, they're not hard barriers between things, but they're a little bit soft. They're more about categorization of either a situation or a system or whatever you're working with. Simple are really simple. They should be understandable by pretty much anyone. So a Lego set that's disassembled might be a simple problem. You look at it and say, all right, these are Legos that are disassembled. You categorize it, this is the Batman Mobile Lego set and you respond. You follow the instructions to put it together. So even a child with appropriate motor skills can put that together. You don't need any specific knowledge in order to do it. Complicated things require some domain knowledge to be able to figure out. So maybe your parents house, the internet is out and you need to go help them because they can't figure out the problem. You have to apply some domain specific knowledge and figure out, okay, in looking at this, I can see the SSID isn't being broadcast so now I know to go check the router and I can see it's not plugged in. Let me go, we plugged that in and everything's resolved. So it might not be hard to fix but it required that specific domain knowledge of how to, where to look and where to actually resolve the problem. Another example for complicated is like car issues. The front right corner is making a noise. You might not know what to do to resolve it so you bring to an expert to look at it and fix the problem. So complex issues as they're different from complicated. Live in a world that's pretty much unknowable as a whole. There's no way you can look at a complex scenario and actually figure out where all the moving parts are. Some of them might be concealed to you or more often acting inside a complex system causes the system to change. So increasingly this has to deal with the actual computer systems we operate. We're building microservice clusters on hardware that someone else owns in places we're not quite sure where it is that have unreliable internet connections between them or virtual networks. We can't really conceive of the whole thing as they grow more and more. Or if you saw Noah's talk earlier talks about how any simple Ruby on Rails app might involve a ton of different libraries and dependencies that you just can't conceive of everything at once. So usually in a complex scenario you have to do something first to figure out where you're at. You can only really understand the effects of actions in hindsight. So this is a place where you're usually doing new things. You might build a prototype, see how it works, respond to that behavior and iterate on that again. Build a new prototype, see how that responds and so on and so forth. Chaotic are usually incidents, right? Like something went wrong, houses on fire. So the first thing you do is make fire not there, right? Like there's a problem act first, then go back and figure out what caused it and try to resolve the situation. There's sort of this like fold between simple and chaotic because one of the easy ways to end up in a chaotic situation is miscategorizing something as simple. I know what this is, I will do this, now everything's on fire, right? So there's that quick terminology. That's why problems don't sit specifically in each quadrant but can move around. As developers and programmers and technologists we tend to flock towards the complex domain. Usually when we solve a problem even if it requires domain specific knowledge like how do I configure this server? We then get really bored because we already solved that problem. So now we automate that. And now the problem is how do I maintain this automation process? Or how do I control a fleet of servers? Or how do I, it just keeps getting bigger and bigger. We constantly move towards the new and novel. Or new emergent. A novel is I did not expect that to happen and you end up quickly in chaos. So I would argue also humans exist in this complex problem space. Largely even to ourselves we don't always know what's going on. We often go to a doctor who might be able to deal with some problems and not others. There's a lot of maybe this will work, maybe that will work. There's differences in how we act day to day. The same like input doesn't always give the same output in complex scenarios. So, yeah, I think there's a lot there. There's stuff that goes on behind the scenes in our heads that we still don't really fully understand. And I think our relationships are even more complex because now we've got a bunch of complex systems. We're interconnecting. We might not even quite talk on the same level. So now we've got all of these different barriers that we either see or don't see that we have to traverse. Kind of an example of this is I could say to someone one day, hey, that's a great shirt. And they're like, oh, thanks. And the next day I might say, hey, that's a great shirt. And it caused them to be really sad because, and they didn't expect it because that shirt was actually given to them by a really good friend who passed away a year ago and they haven't thought about them for a long time, right? So there's lots of moving parts with people in our relationships. Okay, so does anyone know RCA as it pertains to incidents and computers? Root cause analysis, awesome. So don't do it anymore. So it's not really valuable in complex systems. There is no root cause in complex systems. There are too many things going on and too many things that change. And even now that maybe you've got a problem and you analyze it and you've seen a complex systems phase, but even that analysis and the problem occurring has pretty much changed the system. You can't say, oh, this will definitely cause the problem every time. More importantly, there's so many things happening. You can't prepare for every root cause. So it's not really valuable. Root cause is often a search for closure, not a search for the actual problem. You wanna say, oh, this was the problem, now we can move on. For instance, the root cause to every plane crash is that gravity pulls stuff down. But drawing that conclusion from analyzing the crash isn't really useful. It just lets me say, now I know why the plane fell. Consequently, that means there's no root cause to problems in relationships. We don't always know what's happening, so you can't pinpoint one specific thing that might cause a relationship to fall apart. So the challenge here is when we think of root cause, we think about preventing failure and trying to extend that time between failures. We often build really rigid systems that don't allow for a multitude of problems. We say, oh, we're gonna avoid this failure case, we're gonna avoid this failure case, we're gonna avoid this failure case. There's a lot of, you can't do it this way or you must do it another way. And when we have emergent behavior and we have novel things occur, these rigid structures aren't able to adapt to that because they haven't prepared for those scenarios. It's better to have resilient systems that can kind of adapt to their environment. One of the other things about resiliency is it tends to get better the more often it experiences these failure cases. So you know now what makes it break and you can start to address ways to get around that. One of the problems with complex systems is it usually takes a lot of different things to go wrong to cause catastrophic failure. So when you wait to take care of problems because it's not quite broken all the way, you're not ready to experience that problem case. You build a lot of tech debt and debt like anything else eventually has to get paid. Whether it's tech debt, you could have systems failure. If it's relational debt, you could break off from someone and never talk to them again. If you never pay down that debt from an interaction, it can cause problems because anytime we interact with someone, some sort of transaction takes place and it's not zero sum. You might both walk away enriched from a conversation. You might both walk away upset at the other person or maybe one person has no idea what's happened and just passed by but someone else has heard by that conversation. So something happens and if we just always transact and take away, it's eventually gonna be a problem that the debt comes due. Experience with failure is necessary. That's what helps us understand where the edges of our systems are and what helps us improve those. So with our computer systems, we kinda wanna know how far we can push it before it breaks so we know when to stop and we know how to expand those edges. And for people, it applies too. If I understand when pushing you is a problem because it's gonna damage whatever it is, maybe it's gonna cause some undue stress on your part. I need to know when to step back. But I also need to understand when some discomfort might help us grow together. Whether it's in a relationship or you as an individual, it usually takes some amount of discomfort to grow. So understanding where that edge is and how far you can be uncomfortable before it causes problems is important, which means we're probably gonna cross that threshold accidentally now and again and we need to address it. Okay, so this is sort of the level setting portion. I wanna get into what we can actually do to start addressing relationships when we have these failure cases and we have problems. Sort of required operating conditions is a blameless culture. If you're in a group of people that really doesn't care about fixing it but is really seeking that root clause or that blame to say, oh, this is what the problem was, now we know what it is, we can get rid of that thing, you're gonna have problems ultimately. You need to be seeking the restoration and the learning from that failure rather than seeking what's a blame from that failure. There's a whole other presentations about just culture or a blameless culture but I don't have time to dive deeply into it but if you don't have that, most of the rest of this won't matter. So a few more lessons from computers. So the first thing is sort of about distributed systems and how they interact and one of the things about the company that I work for at Elastic is we're distributed not just in the software we build but our company is fully distributed as well and strangely enough, one imitates the other. Abstractions are really hard to deal with that is to say when I say, oh, I don't know if this solution gives us the best bang for our buck. Most of like half of my colleagues on the community team don't actually know what I'm talking about because they don't have that turn of phrase in Austria or Japan or India. So I have to explain that, right? There's some calendars just in communicating from the fact that our systems operate differently. There's also problems in the abstraction of asynchronous text communication when you're trying to talk to humans who use more than just the actual words to communicate most of the time, inflection of voice and facial expressions play a lot into how we communicate with each other. So one of the things we do is we always make a point to assume good intent. Usually when someone does something or says something that hurts us or offends us in some way, it's most of the time not intended. Most people in the community act together for the betterment of the community with the exception of specific bad actors, but that's not what we're talking about then in this case. So starting from that frame of mind can really help you shift into that blameless mindset of, this person probably meant to do something good but something bad happened. How can we learn from that and improve it? And sometimes abstractions aren't good enough to communicate, so we have the slash zoom command where we can just instantly bring up our zoom meeting room and we can just go face to face with someone. So if you're having a discussion, you recognize like, hey, we're not seeing eye to eye. I've learned to recognize it for me is when I start typing a really long thing, trying to explain a lot, I'm like, you know what? We probably need to see face to face and talk our words directly to another person's ears to figure out where our disconnect is. So that's an important aspect too, to remove abstractions as much as possible. So some lessons in relationship networks from computer networks. Two-way communication is usually better than shouting into the void. I think almost always when dealing with people, which I recognize sort of defeats Twitter's entire business model, but nevertheless. A really good turn of phrase I've heard is what you're saying is X and what I'm hearing is Y. This is sort of like the human TCP handshake. You're acknowledging that someone's delivering a particular message verbatim should be what they said, but what I'm hearing might be all of the other things you're interpreting out of that. So if you're having a really rough time with someone because your pull request got rejected again, you might say, you know, what you're saying is, I need to limit my code better when it through whatever product. But what I'm hearing is that I'm not really good enough to be part of this team and I don't have the skillset and I'm not really wanted, right? So you're acknowledging the other person that you've heard them, but then explaining where you're coming from. This tends to prevent that like everyone talking louder, saying the same thing back and forth to each other, because even though you've technically hear each other, you're not communicating. So this isn't always successful either. And there's a whole framework for problematic conversations and difficult conversations called nonviolent communication. This is generally where one party has been hurt by someone's actions and we're trying to bring that back and rectify that relationship. I sort of think about it like incident response for humans. We've got a problem and we need to fix it. And much like incident response, you usually wrap a framework of behavior around that response of do these steps or check these boxes as you respond to an incident, just like firefighters of specific steps they go through for how they deal with a fire. So the nonviolent communication framework breaks the conversation to four specific pieces. And this is again, like troubleshooting. We wanna break down the problem into specific parts so we can address where we're not getting the behavior we expect. So I will go into each of these a little bit more. So observations first are observable data. They're not evaluations of someone's behavior and they're not judgments on that behavior, just what actually occurred that you'd be able to observe like what a camera or a microphone would pick up, not what you're adding on your end. So for an example, I'll actually go back and give an example at the end because I realize it works better at once. So then there's feelings that come from that because try as we might, we are still squishy things controlled mainly by emotions and hormones. So when something happens, you probably have some sort of visceral reaction to it that caused the relational problem. These aren't thoughts, they aren't things that happen up here as you analyze or think about it over and over again but there's sort of that visceral body emotional reaction that you have right away. It's easiest to talk about feelings if you sort of simplify the spectrum. Language is very colorful but when you're trying to communicate it's hard to have all these different floaty meanings that describe lots of different things. So if we can kind of paint with fewer colors we can make a more valuable image. And generally we wanna avoid qualifiers. You're not necessarily very or a little bit in emotion, you just kind of wanna talk about what it did. I found this sachet acronym really useful for sort of painting a short small spectrum of feelings. And it's just these four. So sad, angry, scared, happy, excited and tender. Sad is usually like a loss of something. Angry is usually either you wanted something and didn't get it or someone violated a boundary that you have where you say, you shouldn't do this and they didn't. Whether or not you're explicit. Scared is usually uncertainty of the future who usually is what makes you scared. Happy is usually contentedness. So it's sort of the opposite of loss you sort of have. You're pretty content. Excited is sort of the opposite of scared. You're looking forward to the future. It's probably more certain about what's gonna happen. And tenderness is a connection with other people. It's kind of like I'm feeling really empathetic. Someone might share a story that hits close to home and you're feeling tender for them. So great. Once feelings are defined needs are the next thing. These are usually basic human needs. Connection, wellbeing, honesty, play, peace, autonomy, meaning or purpose. This isn't an exhaustive list of your needs but they're kind of a good starting point to think of, oh, it's not what did I need in a specific scenario but what is it and I'm after like at a human need level that I'm not getting from this interaction. And then finally a request is made. So what do you want? This isn't a I want you to not something but it's a positive request. Cause it's harder to not do things than it is to do do things. Because when you're not doing things you're always successful until the one moment you're not. You can never achieve not doing something. But if you ask a positive request that's an achievable goal. I can do that thing. You kind of want to think of smart goals in the request form as well so that someone can actually achieve the request you make. But it is not a demand and it's not guaranteed to happen. You can't demand someone do something and they're not obligated to follow through. It is a request or else it kind of shifts the power balance the other way especially in this case we're usually communicating as peers. So NBC framework in a nutshell. An example for instance is an observation. I asked you for help. I have a deadline on Friday. I asked you Tuesday afternoon for help looking over my code. You said you'd get to it in five minutes. It's now Thursday morning and I haven't heard anything. There's no judgment of that behavior. There's no evaluating what happened just simply what occurred. Feelings. So I'm feeling angry cause I requested something from you and you haven't done anything to make it happen. I'm feeling scared cause I've got a deadline coming up and I still need help and I'm not getting it. I'm scared that I'm not gonna do good enough work. Maybe you're sad because I thought we were closer than this and I haven't heard from you so I feel like, you know I'm not as close to you as I thought. Doesn't have to be all those but as an example you can see this sort of a spectrum of what a person's feeling in a situation. And the needs here might be, you know I need, I do need that certainty that like I'm producing good work. I need that certainty. And I also need to feel like my requests are being heard. I haven't heard from you. I need to know that you're hearing it and I'm prioritized in your course of the day. So my request is, you know I ask that you do help me out in the next two hours if you can or we make another time. And I request that I totally understand there might be something that came up that you just communicate with me communicate with me whenever you pass a timeframe if I request help and you say I can give it in an hour if you can't in that hour just communicate with me so I know what's going on. Then it's up to the other person at that point to either accept that request or not and so on and so forth but it breaks down all this complex interaction of my uncertainty of my quality and not feeling heard and not getting what I need into sort of consumable parts that lets another person know that their action whether they intended to or not upset me had all of these consequences that you know happened from that. So the big thing I want to sort of leave off with is the goal for goal setting and all of that is to seek success not avoid failure. If you're avoiding failure you can only do it so long and you're gonna always have problems. There's no like the success condition on mean time between failure is the number infinitely goes up which is not a real possibility, right? You can't have no more failures in the future. So rather than doing that we should make goals to actually fix things when they go wrong seek positive conditions out of it. Fix repairing the relationships not avoiding problems. And then I have a bunch of sources and if anyone has any questions I would love to answer them. Yeah. So one of the problems that I have at my work doing this sort of thing is a lot of people in a profession lack emotional intelligence. So when you have people lacking emotional intelligence communicating it can get yucky. So how do you deal with that? So I found that those really specific frameworks of only use the spectrum for emotions and don't say very or little and this very specific like nonviolent communication process for instance, which if you plan on applying I definitely encourage you to look in too much more than just the crash course I went through. Because they provide very specific like actions and responses to those actions it helps when you don't have the skills to express yourself yourself to have this path to follow and say, okay, this is what I should do next. This is what I should do next. How am I feeling? Okay, well I have to pick from these six so let me see where it fits best. I think it helps provide a tool set to make the gap that you might have in emotional intelligence and eventually you kind of get the hang of it and you can branch out a little bit more from there. Other things as well but that's one of the things I've found. Anybody else? Yeah. So I guess. Just trying to understand. So. Thanks for the feedback. I'll repeat the question afterward if I. No, I'm not. So my question was is this something you've applied to? Like you said you manage teams is this something you've applied successfully to your team? Like how I guess how many success stories from this have you seen so far? Are they specific examples I guess? Yeah, so different different techniques use different places and successful in different environments for sure. So for instance, our team does deal with all the initial stuff that's sort of baseline of assuming good intent and removing abstractions from conversation whenever we have problems works for us really well at Elastic and in my team on community specifically I even recently had an issue with someone on the team we clearly weren't communicating at the same point you could tell both of us were annoyed with each other and it was just like you know what we need to have an open discussion on this and talk face to face so we can figure out where we're missing like have the opportunity to respond and not read in your own emotions to all the words that are coming at you. So that's always been really successful. I've used a version of nonviolent communication in really their high trust like blameless groups. It's pretty much like a group therapy kind of thing but it's really really successful in that case where you might be in close quarters for a long period of time and being able to talk honestly to people that are willing to hear it and not pass judgment right away is huge for resolving those conflicts quickly especially in a group that might be like working together on a really important project in a short period of time it lets you so I've been to, we've used that on we do weekend retreats basically that are kind of about emotional intelligence at base I can talk more about it maybe off the mic and things like that but that is a technique that we use there to deal with inter-conflict while we have a very intense emotional like 48 hours together, so yeah. Do you find having elastic being remote? If you're a partner that sometimes having two people in the same group makes one more intense? Yes, it does make it easier or harder. So yeah, it depends on the problems it is nice in the way that I can step back and sort of process something and recognize if I respond right now it's not gonna be good I need to go take 10 minutes and come back to it without like a weirdness in the conversation you don't have to like explicitly state that you can just walk away from the keyboard but it does make it harder to actually communicate and get through things you have to for instance say you're three hours behind me I really need to talk about this but you're not available so we can't talk till Monday and now I've got to deal with the burden of carrying this conflict until Monday, right when we can actually sit down and have the conversation so some of one some of the other, yeah. So at elastic you guys at least have told everybody else and I believe to be true that you are like truly blameless or like the company believes in this blameless for a company that might believe less than that how might they transition and either from like a top down or from like more grassroots type thing. Yeah, the blameless and just culture which are totally worth looking into the resource here I think is about patient safety like there was a talk last year at DevOps Days Boston that talked about the just culture at Brigham and Women's so that's a really good place to understand this a little bit more in depth. Applying it is tough because it's cultural change like anything else so you run into a lot of the same problems when you say how do we do a DevOps initiative at our group that hasn't never done that before because there's a lot of changing hearts and minds that comes into place. I would recommend starting at something practical so thinking of blameless postmortems for instance and you can apply that specifically to that condition or whatever you wanna apply it to but being able to start at that practical non-personal level might help move into the more personal side of things and say hey, we do this for our incidents can we do this for this argument we're having right now and try and retrospect what's going on. I hope that answered and maybe we can. I got another question. Yeah. So you said that you didn't like complex systems don't have the process. Right. Yeah. All the companies that may not have the same blameless culture you have and we do blameless proponents but in those first part of this area we're going to call them we're often looking for root cause in complex systems so that is like what would you say to somebody that says hey but I think that there is a root cause to your complex system. So what would I say to someone who says there is a root cause in a complex system? First study a lot of John Olsbal who is one of the biggest and most prominent proponents of there is no root cause and furthermore human, what was it? Human error is a myth is the other side of it too as far as how we study reactions of systems. So there's again a lot more where I would probably start having that conversation I think is to address what you're actually after when you look for the root cause. I think like I said earlier when you're looking for root cause you're more often looking for closure on the incident rather than how can I fix the problem? You just want something to say oh this is what caused it now we no longer have to explore or learn from it. So if you can get to the point where that statement is explicit in the group and you sort of understand that collectively I think gives you the best way to move past it from there. I think that's it. Great, thank you.