 Okay, so I am Ben Rockwood. I'm the director of IT and operations over at chef But I am first and foremost a system administrator. That's what I started doing. That's what I always do That's what I will do when I when I die And as a system administrator, I think way too hard about low-level things I always want to go to first principles dig as deep as deep as deep as I possibly can to understand things and then once I've done that I want to extract out tools that can be used to help me solve real-world problems and One of the fun things about the DevOps movement is it has a lot of roots and One of its primary roots is agile right originally before we were DevOps we were agile operations and Agile really has its roots in lean and lean has its roots in Scientific management and so you'll find that I tend to move up and down the stack because I don't believe that you ever really Truly understand anything unless you understand the causal roots beneath it So sometimes they go a little bit deep some people like that some people don't but We're gonna dive through of all the things that I found out of all the causal roots one of the most powerful tools that I found is a3 It's something that I've used in all the operations teams that I've managed over the course of the last 10 years and Every team that I've used it in found became wildly successful using it And so I'm really hoping that you see some some joy in it, too It is in fact the engine of lean it is the engine that keeps lean actually functioning It is the secret behind Kaizen And it is the keys to a learning organization and one of the reasons I want to talk to you guys about this It's a lot of people say hey we should do that We should should be continuously improving and you're like yeah, that sounds great. I want to do that I totally want to do that How do you do that? Or they're like we'll just continuously improve and you're like hey, I'm going to continuously improve How? We'll just try hard But I'm assist admin so I want a tool and I've got a good one So let's first talk about continuous improvement. What is continuous improvement really inspired by the idea Kaizen? Kaizen literally in Japanese means change for the better. The premise is to make many small tiny improvements Day-to-day in practical real-world work not you know just on a whiteboard But in real-world work that hopefully adds up over time to big big gains right and that's not a new concept Right, you know, how do you lose a hundred pounds you lose one and then you lose another one And then you lose another one and then you lose another how do you run a marathon you run? You know a hundred feet and then you run a mile and then you're in five miles right not a new concept Kaizen really is a way of life. It's a way of approaching how you do things all the time Sometimes you will actually hear the the the concept of a kaizen event. Let's get everyone together and improve. Yeah, that works When you think about kaizen though, this is really really key I'm not going to belabor the point but you need to be thinking small tiny So whatever you think is a thing that you could do in order to improve Some individual process or something that you do whatever you think that improvement is it's too big make it smaller and Then make it even smaller than that and then we start getting to things that are actually implementable Okay, so kaizen needs to be tiny tiny tiny but continuous. I Want to make sure that you understand that there's another concept out there which is kai kaku It's the other side and this is radical improvement This is when you realize that making tiny improvements to a process is the wrong thing to do that The thing is fundamentally wrong and you need to blow that shit up Right and sometimes you got to be real honest. You just got to start over. Okay, so blow that shit up But also know the difference right know the difference and if you're not sure debate it But make sure you know both premise with radical improvement and kai kaku is really you know major overhaul when something Is just on the wrong course to begin with and start a new course as soon as possible So you can get back to kaizen we always want to be doing kaizen But that doesn't mean we want to improve something that's fundamentally flawed. Okay, use it very sparingly, but know that it's there Okay, now I told you that that dev of has its roots in agile which has its roots in lean Which has its roots in scientific management? These are the science the the pioneers that really led up to lean guy on the left is Shoeheart then we have deming then we have ono Right, I'm sure y'all have heard the history of lean lean is the American term for the Toyota production system The Toyota production system was created at Toyota from ono back after World War two when they believe that they had three years To catch up to American industry or the country was entirely screwed in order to learn from American industry After we bombed the daylights out of Japan We sent all our best managers over there to teach them how to rebuild their industry because that's what we do as Americans were fucking awesome More over beers later, right? One of the people that went over was Deming Deming taught him a lot of things, but the funny thing is is Deming never claimed to be an independent genius actually He was a disciple of shoe heart shoe heart is awesome. Why because he worked at bell car, right? Come on, we're assistance people you got to love bell core, right? Okay. Anyway shoe heart was a master of statistical process control and the Japanese really wanted to learn it so Deming came over and taught them all of shoe hearts tricks on how to do statistical process control And so a lot of these ideas that informed What ono created his Toyota production system and then became lean all actually came out of Deming which came out of shoe heart Okay, so you got to know the history one of things that Deming kind of at the end of his life He sort of brought together as like the great framework of what he was trying to propose was the system of profound Knowledge because it sounds fucking awesome, and it has four components appreciation of a system We always think in systems system system systems, right an understanding of variation incredibly important There are two types of variation. There's special Common cause variation and then there's special cause variation, right when you're looking at those graphs You see that graph go up and down up and down up and down all day long That's common cause right just normal sort of stuff that happens that big old spike that makes you go shit What was that that is your special cause? That's the one you want to focus on not those little tiny blips This is one of these things in seasonality and data analysis. We want to think about then there's the theory of knowledge What the fuck does that mean right? That's a little ambiguous in Psychology because ultimately all these systems are composed of people the theory of knowledge comes from something called pragmatism right All you philosophy people are like Pragmatism America's homegrown philosophy right we created one. It's awesome. It's pragmatism Came from Pierce on the left there really built on by a Henry and then Lewis CI Lewis is less well known but the most influential of them and also do we made it very practical Now why am I telling you about pragmatism? What the hell does that have to do anything? Being a pragmatist does not mean that you're simply a pragmatic person Sometimes when we say pragmatic we mean somebody who follows their gut somebody who just does something that's very practical and sort of works We talked about pragmatism. We're talking about something very specific Which is an approach that assesses the truth of meaning of theories or beliefs in terms of the success of their practical Application we're talking about is epistemology How we know what we know and It's really important that we consider this idea even though it's a real base one We need to think are we focusing on on on a? Are we focusing on audiology and are we focusing on kind of the wholeness of truth? Or are we focusing on what actually works in our environment to determine what is or is not true? Okay, just because Etsy does it doesn't mean that it's necessarily a thing that you should do in your environment because your environment Is not Etsy's environment same with Amazon same with Netflix same with anyone else How do you determine if something is true or not you determine if it's true or not as to whether or not it works in your environment, right So always thinking from standpoint of pragmatism that I am going to solve problems in my unique context for my unique Context not as universal constants is a really important place to start and that's where we always want to start, okay? That theory of knowledge that Deming talked about actually was a reference back to this book. This is the C.I. Lewis's book a Mind in the World Order it is a philosophical proof, which means it's hard to read It's incredibly interesting, but it's one of those books like you read it You're like wow that was amazing, and then five minutes later. You're like, I don't really know what it said So you read it again, and you're just like I love that book Okay So these ideas of a focusing on what is is something that was a seed at the beginning of the 20th century That really was important to shoe heart Three C.I. Lewis's work then was passed on to Deming then came up through Oh, no And now is one of the fundamental roots of everything that we're doing today that originated out of that movement So real important thing to know that's out there. It's a big kernel of that truth And it comes all the way through to today, okay? all right, so One of the ways that they would needed a tool to wrap around this idea of pragmatism and to To bring it into day-to-day use and so they created this thing called the PDSA Cycle plan do study act some of you will have heard this as the Deming cycle Which was actually PDCA check act The original shoe heart cycle was PDSA and the funny thing is is even though they call it the Deming cycle Deming never actually called it. He always called it the shoe art cycle. So I don't know the messenger gets more more love But this is something you guys have probably seen before right and you probably think I know that every time somebody is really boring and bureaucratic They totally bring this up now. There's this weird dude in a kilt on stage. She's telling me that I should do it I'm very confused Why is it important because it's the scientific process? It's what it is. It's a scientific process brought into business, right in the scientific process. We ask a question We construct a hypothesis Then we test it with experiments. We analyze those results and then based on those results not what we want the results to be But based on the results. We actually find We draw conclusions and then we decide what we're going to do next That's precisely what we're doing in this PDCA cycle. We plan what we need to do. We have an idea Well, I think I need to do this. I think this should be done. Then I'm going to go and perform some experiments Then I'm going to look at those experiments and say do they actually match? Is this what I thought I was good to see? What does this tell me and then act? I'm gonna finally get to decide what to do with it and this is a little bit confused I remember the first time I saw it so I'm like you do and you act At the beginning in the end. What's all that about? But it's really performing experiments and then deciding what you do with those experiments And the thing is this PDCA is something that kind of goes off the rail So Toyota came up with this concept called the a3 framework the a3 framework is a way of taking this PDCA cycle and encoding it into a document one of the things that created this document is that I'm sure you've never seen this before People were coming up to ono with like 36 page project proposals and going hey something's wrong Here's this thing tell us what you think and he was like, no, no, no, thanks. You can keep that I don't want it. He's like, you know what? If it doesn't fit on a sheet of paper I Don't even want to read it make it fit on one sheet of paper. So being engineers. What do they do? They found a really big piece of paper This is an a3 this is where the term actually comes from so this is an international standard a3 sheet of paper and to give you an idea of the size Fold it in half and you have us letter paper. Okay, so this is bigger than legal is a big piece of paper So they said, okay, we'll put on one sheet of paper and over time they refined this and What you have here in this this document is that PDSA cycle Drawn out here in the first we started the top left-hand side if you guys are familiar with the business canvas This is very similar. In fact, I think business canvases actually ultimately came from this We have a problem statement at the top. What's our problem? We have a background out there We have current condition Future condition. We have an analysis. We have our experiments. Then we have our results confirmation and finally we have our future plans, so Break these down a little bit more And in a3 this is a thinking process. This is not a form. This is the first big thing You got to understand about about an a3. It's not about the piece of paper. It's absolutely not about the paper In particular because when you fill out a form, what do you do? It's a point in time. You sit down you felt for This is not a form that you fill out at one time This is a thinking process and it's a physical artifact that constrains you to work through that process to make sure that you're following it well So on this piece of paper, we're gonna have a problem statement What is this is gonna be a very brief one-liner sort of what's the problem right monitoring blows? Right, but you're gonna set a context. What are we talking about? The next thing you're gonna do is describe the problem background. Why should anyone care, right? This is great if I if somebody just goes and reads your a3 why should they why is this a problem? Why should anyone care about this? This is what some of us might call business case It should be very short, but it should be packed with data Then we're gonna describe two things that are the key here. You're gonna describe your current state What is actually going on right now? Guess what? This means you actually have to understand What's going on right now and a lot of times on our systems are so complex. We're like it's a crappy system Why is it a crappy system because I have no idea how it works That's why it's a crappy system like no we need to go and figure out how it works gonna do the back work Figure out. What is your current state? What is actually going on right now? And then you're going to have to figure out what it is you want not a goal Not a simple goal of I want I want to go this far Describe the future. This is a vision. This is what I actually want. I want to not ever wake up at night ever Right? That's a good future state. You have to describe that Once you know where you're at and you know clearly where you want to go now. We can do a gap analysis This is called a root cause analysis Different people call it different things, but essentially what we're doing is a gap analysis Why am I where I'm at and how do I get to where I want to go and the real focus is going to be for a lot of us Why aren't you already there? Right? If it's so easy, you'd already be there There's a reason that you're not there and these have to do with constraints. Our environments are full of constraints And one of the important things to do with the desired future state is to really think outside the box I mean think think big if you want to about what kind of future state you want Because if you don't know what you want to do without constraints, there's no way in hell You're going to know what you want to do inside constraints Constraints are the thing that limit everything we do anyway So once you've done this analysis, and you know where you're at and where you want to go and You've done that analysis about what what's what's going on there Now you can go through and figure out what you need to do to try and close that gap And these are going to be the experiments you're going to go off and you're going to you know lay them all out This is what I want to do. These are the things we're going to do and you can do a lot of these in parallel I mean this does not have to be one plan. These are experiments, right? Try as many different variations as you possibly can because the purpose here is not to solve the problem the problem here Or the thing you're trying to do here is to get towards what's going to solve the problem and knowing it rather than just thinking it Once you've done those experiments now. We have to go back and Review them what worked what didn't what were the trade-offs? What do we think was going to happen what actually happened? And this is where you really got to be looking at what is that future that we want? Are we getting there and did it match what we saw in our analysis? Once you've done that now you've learned something Now you're a learning organization now you've learned something, but what do you have to do once you've learned something? decide what to do with that new knowledge Are you going to make it part of your standard process? Are you going to adopt it or you're going to throw it away? Does it turn out one thing that will commonly happen when you start with a threes is you'll find out that they that you'll get through This process you realize you have a bunch of other things to do so you'll spawn off a bunch of other a threes for That are smaller because you realize you bit off more than you could chew right This is the flow that starts to push kaizen This is how kaizen really works and so you're thinking like I'm a computer dude. I don't use paper paper. What the hell is paper? Obviously, I don't use paper. I use confluence You can work in any system. This is a confluence template. We have in the chef wiki That we use for our a threes so I have three teams that report to me. They all use a three and Anytime we have a situation where we're not quite sure what we want to do. There's still a fuzzy sort of There's a fuzziness and whatever the problems we're trying to solve or how we're doing it We always it's like a three. It's time for a three get an a three going right And we'll sit there in a meeting and start filling it out and start thinking it through right where where we good Where are we weak? Where do we know? Where do we not know and it gives us a framework to push through and Like I said, you're going to go through this in over time. This is not something you just fill out at one time, right? There's obviously you can't know what the results of your experiments are going to be before you've done the experiments You can't know what experiments you're going to do until you've done your gap analysis. So it's a process This is an example of a real a three from chef. This was for the Ubuntu 10. Oh for EOL anybody deal with that one Yeah, yeah one brave soul admitted it We had a bunch of 10 of the four systems that were still out there and we needed to get them upgraded And so we wanted to think through what was the best way to get this done, right? And so you can see some of the the problem Description up there in the top the background The current condition the bottom kind of got cut off here And then on the right you can see each of the experiments that we did In order to get us going in that direction There's a number of different times that you can use a three. It's a pretty flexible tool Problem-solving is definitely the most most common Use Project planning it's really great when you want to do project planning You're not sure what's involved project proposals and initiatives when you want to show out an idea again You're going to go through that same process of what's my current state? What's my future state? What's the analysis and what do we need to do to get there? Really good in retrospectives and postmortems about you know, hey, we see keep seeing the systemic problem How do we actually get towards something that's sustainable and just about any other? Case you can think of very very powerful and flexible tool Some best practices when using a three is as a process is always just keep it simple don't get clever clever doesn't help anything Keep it simple as possible If it won't fit on the form so in the case of a paper you're physically constrained by size in a wiki Of course, you can just keep going on as long as you want to but there's a point at which It's just like come on dude. This is too much It becomes obvious and that that's what you should be used to its strength when you when you're running out of space It's a good sign that your problem is just too big Reach for them as often as you can this is why it's a wiki template that we have we can pull them out Whatever we need to and again never ever ever forget. It's not about the form It's a logical thinking process Right in a very powerful one now the the way that you'll see this if you actually ever go into a manufacturing plant When you walk on to the on to the factory floor You'll normally see a wall covered in a three sheets of paper And what you'll see is is all their current improvement initiatives that are moving out with their a threes So it's kind of fun to look through and everyone knows the state of everything all the time In in our teams we just keep them in our wiki on you know linked off the home page So everyone can see that we're doing and it's a real nice way for people to come in and get a sense of What we're doing it's it's even more useful than just having a Kanban board around because Frankly, it's nice that everyone has X year Kanban board It's a lot of times they look at it like no idea what any of that stuff means I have access to it hooray But what does it mean? These are a lot more clear and these are something we can build on I mean this is a this is a basic tool That can always be built upon so one of the things I wanted to do was to incorporate the ideas of design thinking Into this process because if you look at the process of doing design thinking These steps you empathize you develop a deep understanding of a user's challenge like really put yourself in the end user's shoes Like what is what is a consumer or or a user really? Grappling with and then you define based on that you clearly articulate a problem that you want to solve You ideate you brainstorm potential solutions that will solve that that problem select and develop a solution Then you prototype what that solution is going to go to look like and Then you test it iterate put it in front of your user see if they like it if they don't and then iterate iterate iterate Well that design thinking process sure sounds a lot like what we are doing in in the dev-up side of things And it looks really similar to the agile flow. It looks really similar to what we're doing in the lean continues improvement side So what I decided to do was to merge these things together and I created what's called the a 3.2 Template and you'll notice just a little bit of change here. We've got the problem description So I call it investigation exploration reflection and implementation Problem description stays but then I add in a section on stakeholders and the stakeholders is a really loaded term That generally turns a lot of us off so what we're going for was stakeholders in the sense of of design thinking in that I want to go around to all the Different people who have some stake in whatever we're doing. They're gonna see it. They're gonna touch it They're gonna they're gonna judge us on it. Whatever they're gonna be somehow involved in this process I'm gonna go to them and I'm gonna start asking them drilling penetrating questions to figure out What is it you really want? What do you really want? Well? I want a thing. I got it Okay, you want a thing. What do you really want in the thing? Well? I want it to do this. Okay. Yeah, yeah, yeah Why why why right and eventually they'll be like Because dudes a jerk and I just want to get them off my back great Now I know how to make you happy is to get that dude off your back got it go around to each of your stakeholders and really drill down What is the thing they actually actually really want so that you can? Judge your solution based on that then we got the current condition. We've got the desired future state We still have our analysis But then I change The experiments really into prototypes right which I think captures the idea a little bit better too We want to do lots of things and we just try them to see what works Right reflection and implementation. What do we do once it's done? Do we create a policy? Do we onboard a new tool? You know, what do we do some fun quotes from the Hiano who I showed you earlier? Why not make the world and easier and more interesting so that people don't have to sweat the toyota style is not To create results by working hard underline it Don't work hard It's a system that says there's no limit to people's creativity people don't go to toyota to work They go to think and this is a manufacturing company, right? This is what they're saying to line workers on an assembly line. You're not here to work. You're here to think And then shingo Who really was kind of the sys admin of of the TPS world? Shingo was the one who came up with a lot of the technical innovations that actually made the toyota production system work a Relentless barrage of whys is the best way to prepare your mind to pierce the clouded veil of Thinking caused by the status quo use it often There's a real belief that what we should ignore lean because it has its roots in manufacturing And what does manufacturing know about knowledge work? And when you go back to Henry Ford where this all really started when you look at Oh, no They don't really see a difference the application in implementation looks very different But the fundamental root ideas are the same and I can prove that because of the fact that we're all sitting here Right now There's a bunch of books out there that can help a three understanding a three thinking managing to learn It's a famous book a three problem-solving but always if you want to understand lean the best roll-up book of all the fundamental Concepts that is the best use of your time if you're going to invest your time anywhere I always recommend the fifth discipline. It's the most bang for your buck. It'll get you where you need to go It's definitely required reading. So thank you very much Thank you