 The session we will be having now is called DevOps Culture Simulation, and I'm very happy to welcome Dana Pailaeva on stage, the speaker for the session. Dana, you're all set? Excellent. Yes, I am. All right, so I'm going to start the presentation, and I'm excited to be here today. Welcome, everyone. To get us started, you will need a couple of things. And one of them is access to your phone. If you don't have it, we can make do with the browser, but having the phone is better, because we are going to start with a little Mentimeter questions. And in order to access the Mentimeter, you will need to go to this site, www.menti.com, and enter the code 2260513. Or if you have a phone, and that's when I'm saying that it's better with the phone, because you can actually scan this QR code, and that will get you right into Mentimeter. All right, awesome. So we have 10 people that have been on both sides. We have people from development, from operations, and one person who has no idea, and that's okay. That's okay, too. All right. Looks like most people coming from development. Some people have been on different sides. I'm going to give a few more seconds for people to respond. And awesome. So we have mostly people from development. We have people who have been to both sides. A couple of operations walked in. Welcome. And one person who has no idea where he's coming from. That's so cool. What if someone is not apps or from development? So I guess that's a great question. And I'm missing one of the options. Thank you for pointing this out. So I'll fix it for next time. So yeah, then I guess, I don't know. I don't know what to tell you. I'm missing one option. So I guess no idea is going to be the catch all. So that's if you're not from apps or from development, or if you don't remember where you're from. So that's going to be on that option. Okay. So then the second question for you. So welcome everybody. We're happy to see this diverse representation from different sides and free bird. Yes. I like the free bird option for someone who is not from apps or from development. So that's fixed for next time. Okay. And the second question for you is this one. What's your experience with DevOps? And just go ahead and type in anything that comes to your mind when you think about your experience with DevOps. It could be anything from I have no clue to I've been, I've been working in DevOps organization for a number of years. So anything that is right for you, that's the right response. All right. Very nice. And we're getting nice mix of experiences, nice mix of backgrounds. Some people just starting up, some people have experience and that's really nice. Always has always have that additional benefit from having a mix group of people mix experience in the group. Very nice. Just been reading about it, hearing about it, theoretical knowledge. Yes. So that's all wonderful. Because this session is going to be very introductory session. So if you've been doing it for a number of years, it may be too basic for you. But you know, I'm inviting you to figure it out for yourself. If it's too basic, feel free to drop off. Or if you're finding it valuable, by all means taste. But this is going to be a foundational level session. And typically when I run it in person, it would be an experiential workshop. Unfortunately due to COVID, I'm not going to be able to run the leg and chocolate session with you because I can't pass leg and chocolate. But I am going to share the same information, the same learning. And we're going to also talk a little bit about what are some of the experiential learning moments that people share when they run this session. Okay. So with that, yeah, yeah, new to DevOps, middle journey, reviewed a lot of project, no depth hands on conceptual level. So welcome. Welcome everyone to this session. I'm happy that you're all here. Okay, a little bit about me. My name is Dana Puliva. I'm an Agile and DevOps coach from New York. And in the past 20 years of experience, I've worked with lots of organizations being on both sides of this spectrum. And I started as a developer. And as a typical developer, I was very shielded from what's happening on the operation side. And so then at some point in my career, an opportunity came up to become a DBA manager. I just grabbed the opportunity with my both hands because it's managerial position. And I'm going to be managing DBAs. And what is that to know about that DBAs? I knew how to create tables. I knew how to write store procedures. So I figured that's enough. So little did I know that life on the operation side is very different from the development side. And so what I walked into was the life of emergencies, you know, store procedures that are failing, not just in the ones that they're running normally, lots of emergency, lots of, you know, 24 by 7 type of availability, pagers, and an emergency that needs to be solved in a very short period of time. So that's what I was not aware of when I was in the development side. And so in a way, by finding myself on development and operation side, I experienced life on both sides of the wall of confusion. And if you... Actually, I'm not going to make any assumptions. Just let me know in chat if you heard about, give me plus one if you heard about wall of confusion and give me minus one if you haven't. So plus one if you heard about wall of confusion, minus one if you haven't. Okay, okay. So we have a few people who heard about wall of confusion but majority haven't. And that's cool because I love talking about wall of confusion. So wall of confusion is a term that was coined by Andrew Sheffard. And he specifically referred to the wall of confusion when describing the disconnect between operations and development in terms of goals, in terms of procedures, in terms of way of working. And that goal misalignment is the one that's driving that disconnect. And in specific way. So if you think about development team, development team is focused on speed and delivery. Most of the time in current situation, development team will be working in Scrum. And they will be working in those sprints. And as you probably know, at the end of every spring, the goal is to deliver potentially shippable incremental for product. So what does development team do with that in traditional organization? They just throw it over the wall to development for them to deploy it in production at some point. And development team moves on to the next sprint. They're taking the next stories from the backlog and iteration continues. So that's cycle of taking stories, finishing them up, throwing the code over the wall to operation and then moving on to the next set of stories. What happens with those finished code, finished stories development team doesn't necessarily have any concerns over. They're done with development and they move on. So what happens on the operation side? Operation side is a life of emergencies, escalation procedures and continuously being focused on stability and reliability. So as far as operations are concerned, keeping the lights on is the main goal, which means that anytime when a change is being introduced, it's a danger because with any new feature that development is throwing over the wall, they're potentially throwing a lot of bugs. And so the best way that operations know to keep production stable is prevent any deployments. So as you can see, there is that disconnect where development wants to get more code in production and operations want to reduce the number of time that we're changing production. So in traditional organization, the best way to do it is to reduce the number of deployments. And that's where a lot of times when we're working with the traditional organizations, we hear that we have monthly deployments, we have quarterly deployments, we even have in the past, we used to have one deployment a year. And so you can imagine that when you have such a massive deployment and massive amount of code being deployed in production, it's going to cause a lot of problems because it's just so massive. And so in this situation like that, when we have this disconnect of the goals, what happens is that it leads to this low trust culture. And I'm going to share this formula because it really speaks to the point that when we have a low trust culture, we actually take longer to deliver anything into production. So if you think about the time it takes for us to go from plan all the way to having things in production, anytime when we don't trust all the steps in the process, we don't trust all the different teams that are involved, what we're going to do, we're going to introduce extra steps. So we're going to plan and we're going to re-plan and then we're going to design and we're going to have architecture review, we're going to have sign-offs, we're going to have all these different people involved in going through all these different gates because we want to make sure that we are covering up for any possibilities of failure. So that's again, traditional organization approach. This is not where we want to be with startups, but this is what's happening in a lot of organizations because there is a low visibility into what's happening across the entire value stream. There is low trust between the different players in that value stream. And so as a prevention measure, as a covering up measure, we are introducing all these extra steps which makes our process of getting scenes to the customers even longer than it needs to be. And of course, I'm going to tell you that DevOps is one of the ways to solve this problem. And based on the information that's coming from state of DevOps report, those organizations that introduce DevOps practices and embrace DevOps culture, they are seeing amazing results. And so these are some of the numbers from what the elite performers are seeing. They're able to do 208 times more frequent code deployments. So they're moving faster and they're achieving the goal that development team has, more deployments, and faster time from commit to deploy. And they're also achieving the goals that operation has, which is faster recovery if incidents happen and much lower number of incidents overall. And so in a way, they increase in speed and they're able to increase stability. So if you're wondering who are elite performers. So the way how state of DevOps defines elite performers are those who are able to perform multiple times per day or on demand. If you compare that to traditional organization where deployments are happening from every half a year to every month, you see this huge disconnect between how many features the elite performers are able to put in production and how a few features and changes on the traditional organization are able to put in. And naturally if you're hearing your company saying that we want to compete with Amazon, okay, and how many times per month are you deploying? One time per month. Excellent. So guess what? Amazon is doing 130,000 deployments per day, which is 130,000. That's a huge number. And if you're doing it once per month and then doing it so many more times per day, then good luck to you being able to try to catch up with them and compete because they're going to outperform you because they're able to do that deployment more frequently. They're able to address customer's needs better because they re-architected the entire infrastructure and they changed the architecture to be able to do that. Okay. And the good thing is that it's not only about doing things faster. It's also about releasing time that used to be spent on unplanned work, on rework, on fixing things, releasing that time and making it available for innovation. Because any developer, and we've seen quite a few of them here today in this session, every developer wants to work on something innovative. We don't want to be working on something that's boring and we don't want to be stuck in maintenance. We want to be working on the cool things. And with DevOps, you get that opportunity. You have more time to spend on working on cool things, working on innovation because you don't need to spend as much time on rework and fixes. Another cool thing is that DevOps driving cultural change. And I'm sharing this information from the interrupts DevOps survey. And as you can see on the top, mostly the changes are happening around development, becoming part of application deployments, also operations becoming part of the new products and feature development. So they're becoming a part of the other group's processes and they're sharing information, which is awesome. Another interesting thing that's happening, and it's on the very bottom, which means it's not as widespread yet, but it's a good trend that's happening. So salary and bonus plans for development, QA and operations are aligned. So remember at the beginning, there is a big disconnect in the goals, in performance objectives for these groups. So this is what's addressing that. So now that bonus and salaries are aligning, what's happening is that developers are not only responsible for quoting their part and forgetting about it, they're also responsible for making sure that these things work well in production. And why shouldn't they? Because they actually know what they coded, they know how it works, and if something breaks in production in traditional organization, operation is the one who has to respond to that. But they don't know how developers implemented it. So typically the only thing that operation can do in that situation, they can restart the server, but it doesn't help to fix the actual problem. So in those organizations that merge the goals of ops and dev, and they include developers into production support, what happens is that as a developer, we don't want to wake up at 3 a.m. in the morning, right? So if that happens and our code breaks, you better know that we will fix it so next time we don't need to wake up at 3 a.m. And we know where the problem is, we know what's causing it, and so we are definitely in a better position than operations to address those issues from production. Okay, it all sounds like magic, right? And it's not surprisingly that, yeah, the unicorn, everyone recognized the unicorn. So many companies that were using operation, I'm sorry, using DevOps in the beginning or they were paving the path towards DevOps practices, they were called unicorn companies because they were doing something that no one else was able to do. And the very first company that started sharing some of the amazing things that they were doing was Flickr. So in 2009, they presented at one of the conferences, Velocity Conference, they presented the progress in this area and they presented this presentation at code 10 deploys per day. And at that time 10 deploys per day was something absolutely out of this world. It was magical, it was fairy tale-like because everyone else was doing deployments every couple of years, every 18 months because we were all back then in the waterfall type of world. And so Flickr was the first one to say that, hey, it's actually possible to be fast and to be able to deploy things in production multiple times a day. Since then companies like Amazon, Spotify, Etsy, they started changing their architecture and reorganizing the way how they work. And in 2011, Amazon shared another number which was 11.5 seconds, I think. So that was the time that it took for them to deploy, which is another amazing number. And since then in 2020, it's no longer for unicorns. It's for large organizations also, it's for organizations who are able to not only play around in small areas, but the organizations that are governed by all sorts of regulations, financial institutions, all these companies are able to embrace DevOps practices and introduce DevOps culture. And I mentioned this number a little earlier. Now, actually in 2018, so I'm sure now the number is even higher, Amazon shared another number which was 130,000 deploys per day, which is something absolutely out of the normal people's world. But again, this is something to know that it's possible. Doesn't mean that everyone has to be as frequent in terms of deployments, but knowing that you have a way to deploy on demand or at very high frequency if you need to, it's a good thing to know because that makes it possible for you to experiment with the practices in your own organization. All right. And what's interesting is that even now some companies are shying away from DevOps because of many reasons. And these are some of the reasons that came out from the survey. The one that puzzles me is this one. There is still a confusion and lack of definition around the overall concept. And before I share my favorite definition of what DevOps is, I want to hear from you. So in the chat, just go ahead and share your favorite definition of DevOps. And it doesn't have to be anything. Let's say it could be your own way how you understand what DevOps is. Because I know a couple of people read about it. A couple of people know it from the practice. Some people are new to it. So just how you understand DevOps right now. Okay. Thinking DevOps and operations, DevOps automation and opposite automation coming together. It can do magic, yes. Building deploy continuously. Okay. Develop and operation with first DevOps. Provide value to end user. Let's do it together. Practices of DevOps and IoT operations. Development, testing, operation, security all coming together. I love that because, yes, DevOps it's not just development and operation. It's entire value stream. And in some places you start hearing DevSecOps, base DevSecOps, base DevSecQAOps. It's becoming unpronounceable. But the initial meaning of DevOps was bringing the entire value stream together and being able to deliver end to end to the end user. And I'm going to share my favorite definition. Electric and stable. Yeah. That's pretty cool. Azure software delivery and operations. Okay. Set of operations for iterative development, integration and delivery. Yeah. Many different, as you can see, even from this group, we're getting many different definitions and continuously integrate and deploy bringing value to customers. Yes. Yes. That sounds wonderful. Okay. I'm going to share my definition, my favorite definition. So it's my definition, but it resonates really well with me. So it's about being mixed with patterns intended to improve collaboration between development and operation. DevOps addresses shared goals, shared incentives, as well as shared practices and tools. And it's important because, again, if we have multiple goals that are driving us into different directions, we're not going to collaborate to achieve something that is solving a problem for the customer. So here it's about goals being shared, shared way to incentivize people for good work and processes and tools that are either shared or understandable by both sides because it's hard to learn all the intricacies on both roles. But what's possible is to extend your skill set a little bit to understand what's happening in the operation side. How, as a developer, building your code, how to make it more testable, how to make it such that it's easy to monitor, how to make it more secure, so knowing that information is helpful in your own work. And same way from the operation side, if you know a little bit more about what development is doing, you can know better how to support the code in production. So having that ability to extend your skill set outside of your narrow role definition is something that is very useful in not only DevOps, but even in our own little scrum team. We talk about the T-shaped skills, cross-functional teams, so very similar taking that and extending it to the entire value stream. Okay, and normally when we are on site, this is where we transition to actually playing the game. So we can't play it today, but what I can do, I can walk you through what is happening in the game and what are people discovering as they're playing the game. We do it in three sprints, and typically the first sprint is when we are simulating the way how traditional organization works. And I call it feel the pain, because we start from every table being a separate group and separate group representing a functional group in a large organization. And so as you would imagine, lots of silos, lots of miscommunication, lots of information not being shared. And second sprint is about introducing first steps towards DevOps and the third one, we're simulating continuous value delivery. And we have nine roles in the game. So every role has very similar type of responsibilities as you would imagine in a typical organization. And in addition, they have lots of dependencies. And so in the first sprint, they actually struggle a lot because they don't know who those people are that they depend on. They know there is a role they depend on, and they have no idea in the large room where to find these people. So just like in the large organization, a lot of times you don't know who is responsible for what, that's what's coming out in the first sprint of simulation. And so this is the overall flow of the game. We have the business person who is in charge of accepting and placing different orders. We have Patricia Product, who is the product owner, who works as the liaison between the business person and the development team. We have the developer who is building the packages based on what Patricia tells him to do. We have tester who is testing the packages, and then once they're ready to be brought over or thrown through the wall to the operations, that's when things come into an operations role. And the roles that we have in operations, we have system administrator Adam Admin. He is in charge of building environments. He's in charge of monitoring production. We have Sarah Security, who is a security engineer. She is looking out for vulnerability security issues. And we have Robert Release, who is in charge of building deployment packages and deploying things in production. So as you can hear from the terms, very similar to what happens in the actual world, except when we're running the simulation, we are playing with Lego animals. We're building Lego animals as their products and we're using chocolate as our user documentation. So what happens in addition to that? In addition to that, some players get secret instructions because just like in the real world, we receive information and we not always share it in traditional organizations. We work with the marketplace board, and so this is where the business person is in charge of not only placing orders, but also changing the price of those Lego animals through the game. So what that does, it actually shows to the other people in the room is that if they're not fast enough in delivering to the market, the market demand may change. And so in a way, it's a simulation for your competitors can beat you if you're too slow. And so through the use of this market board, that's what we're able to show. Okay, and this is how deployment package, development package look like. So this is little Lego animal, small label in the package. The label is the one that SARA security will be using when she's looking for security issues. And there is little chocolate in the package she's called. So that's what people are building. So what's interesting is that when we start working on this game, a lot of times people are very confused in terms of what to build. And specifically, I don't give any instructions in terms of how a little dog needs to look like. And they expect instructions, they expect that people will give them specific instruction as to how it needs to look like. And the point of the game is that they need to actually talk to the business and figure that out. And that's another learning that comes from the game that without that communication with the business, without daily interaction with the business, it's hard to build something that is addressing their needs. Deployment package is something that's put together by Robert Release and SARA security is going to run the scan on it and what's interesting here is that as you can see, there is a whole bunch of smaller packages in this large one. And in the game, when SARA finds an issue in this large package, the entire package is going to go back to development team, which means that it's an analogy for when we are building large deployment packages in our actual world, in our real life. And we are waiting for these infrequent deployments. What's happening is that we're introducing a lot of potential challenges because when we have the large batches of code that we're deploying, that's an opportunity for having more issues because as any developer knows, when we start merging large feature branches, we will run into merge conflicts. And by the time that the large deployment package comes to almost being deployed to production when we run security scans, if the issue is identified, it's much harder to find it. It's much harder to troubleshoot it because now the time has passed from the time when it was implemented and until the time it was integrated and brought for deployment. And so that's where through the game, again, we're learning that deploying things in the large batches is less productive than looking for the small type of deployments for more frequent but smaller deployments. Okay, I'm going to take a few questions from the chat. So there is one ample number of benefits of DevOps but still organizations shy away from it. What could be the potential risk on not a promising thing about DevOps that keeps organization away? Yeah, and so that's what I mentioned that a lot of people, a lot of organizations are either confused in terms of what DevOps is or they might be shy away from it because of the role security. So think about how this introducing new way of working, some of the roles are going to change and people who have been doing those roles for a number of years, they may not be happy about that. So sometimes it's a resistance that you're seeing from those individuals or from those departments against DevOps because DevOps is changing the way of working. Just like any change, even though it might be a positive change for organization on the ground, individuals might be resisting the change. Okay, so when we start the first sprint, it's a technical value delivery with Scrum, which means we're simulating sprint planning, sprint execution, demo and retrospective. And in the first sprint, we have development and operations working as absolutely separate teams and system administrator control the schedule, security test run at the end, so just like in the real world. And this is how a room looks like when we run the game. So at the beginning of the game, as you can see, every table is separate and people are working at their tables. So even from the room dynamic, you see that they're all separate silos and they're not interacting between the different tables. Another thing is that this red table, this is where operations sit. And so these people at the yellow tables, they're all development teams. And as a part of the setup, every developer needs to get their environment configured. And in the game, we're just using the masking tape to create that environment, but the only person who has that masking tape is an operation person from this table. So when these development teams are about to start building their LEGO animals, what they discover is that actually they can't do anything because they do not have environments. Does that sound familiar? In many organizations, when developers start, they have to wait for someone to build them an environment. Yes, I'm seeing lots of thumbs up, you're recognizing the problem. Yes, and that's what in the game, it's easier to experience it, right? Because it's much faster, much more visible in the simulation. But when we're working in the real world, these are the things that are hidden. So we're waiting for things to happen, we're waiting for the environment to be created. And so as developers, we're losing our time, right? And we are not satisfying any customers if we're waiting for the environment to be created. So that's another challenge that DevOps is addressing because with DevOps, we're moving away from manual environment configuration that can only be done by a specific individual in operations and we're moving away from that. We're moving more towards automated environment configuration that can be built on demand. So any developer starting up in a specific group can build that environment on demand. Okay, I have no clue about DevOps. So I would like to know when do you know you have set up DevOps? Is there a blog book that would make me understand high-level end-to-end processing steps? So yes, lots of books about that. And I'm going to share a few of them in this session as well. And how is DevOps different from Agile methodology? That's an awesome question. So if you think about Agile methodology as it was intended to be, there would be no difference because one of the things in the manifesto is that we need to satisfy the customer with early and continuous delivery of a valuable product. So it's early and continuous delivery. It's not about building and putting it on the shelf. It's about delivering things to production, to the customer, satisfying the customer. So over time, unfortunately, what we're observing is that Agile teams are focused on building more than on delivering. And so DevOps as a movement came to solve that problem because they're now taking what development we're doing in terms of building things in the iterative manner and extending it all the way to putting things in production. So changing the processes that are there on the operation side, enabling that automated deployments, enabling continuous delivery. So automating that second part of the process that stayed between developers building the features and the customers actually being able to use it. Okay. So lots of great questions coming in. So I'm going to move on and I'll take more questions a little later. So typical findings and debriefing of the first sprint. Organizational silo, local optimization, missing market opportunities. And so what I mean by that is that in the first part of simulation, we actually get developers to build something. But the way how the game is designed is that we introduce code phrase, which only system administrator knows about and developers have no idea about it. So they keep building, they keep building. And then when we did the debrief, the first thing I start from is asking them so how do you feel about the first sprint? And every developer saying, oh, it was so good, we delivered so much and we build it and it was great. And then I turned to the business table and asked them, so guys, what do you think? Are you happy? I heard lots of them delivered and business table goes like, what do you mean? We got nothing. And you can imagine the faces of people in the room because there is that big disconnect that becomes visible. And now developers locally, they were very productive and happy in their own team. But as an organization, we delivered nothing because of all the old processes that we were simulating in the first step. And so this is where we start talking about DevOps because the first part is just about simulating how things run in traditional organization. And the things that we introduced about DevOps is specifically the three ways of DevOps, basic principles of DevOps culture. The first one being system thinking. And I'm going to go into all three of them in more details, but right now it's just I'm going to talk about all three together. So first wave system thinking. And as you can see, there's a line that's going from business to customer. So it's all the way the entire value stream looking at the entire process and optimizing it end to end. So it's not focusing on specific areas like let's make work and development better or let's make work and operations better. It's looking at it end to end. How can we eliminate bottlenecks in the process? How can we speed up the entire process in such a way that we're able to respond to customer demands faster? So the second part of it is the amplify feedback loop. And so you see that error goes back from operations to development, which means giving information to the developers so they know how the things operate in production. They know what are some of the issues that the code is running into, maybe some scalability issues, performance issues. So giving the real time data back to developers so they can act on it and they can actually create something that's more usable. And the third way is the culture of continual experimentation and learning. And we'll go in more detail on this one, but as you can see, there are lots of little loops happening here, which means there is lots of learning, lots of iteration, lots of continuous improvement over everything that's happening. So over different processes, over way of working, over different ways of solving problem. And that's where the culture of learning comes into picture because it's also culture of safety. It's a culture of psychological safety where it's okay to experiment. No one is going to punish you for that. Even if you fail, that's part of the experiment. And so that's okay to do. Okay. So let's look closer at this first way of DevOps system thinking. So I'm going to show you this picture because to me, it represents how in many traditional organizations, processes and bottlenecks are happening between the different departments. So when you look at this and you think about the flow of value from all the way from when business is asking for something and delivering it to actual customers who can touch it and use it, there are lots of places where we wait. We wait for things to happen. We wait for approval. We wait for environment to be built. We wait for things to be deployed. So there are lots of places where there is an opportunity for streamlining the processes. And that's where when you start introducing DevOps practices in your organization, a lot of organizations fall into a trap that let's start with automation. And if you automate your convoluted process, if you have lots of bottlenecks in the process and you automate that, it's going to give you some benefit, but it's not going to give you as much benefit as it can. If you first look at how can I change the process to reduce the bottlenecks. And only after that, how can I automate it? But first, how can I remove those, you know, silo mentality? How can I remove those extra steps that they have in my process or extra approvals, extra bottlenecks, extra weights? How can I reduce that? And I love this quote from Phoenix Project by Ergin Kim. Any improvements made anywhere besides the bottleneck are in illusion. And so that's what happens in organization. That's also something we discover in the game because when we try to make changes to development team who was already very productive and happy about delivering, about building, our bottleneck is elsewhere. Our bottleneck is in deployment. And so making developers more productive or asking them to work three times harder, it's not going to change anything if the bottleneck is in deployment area. Your customers are not going to get anything. You will make developers working harder, but it's not the problem here. The problem is in deployment area. So looking at how can we solve the bottleneck there, that's a better way to, you know, approach that problem in the value stream for that organization. Any questions so far? I know I've taken a few. What is better for DevOps team Scrum or Kanban? Oh, that's a really cool one. So this scrum, you have this cyclical value delivery, right? So you have, even if you have a small sprint, it's still probably a week long. So if you're truly working in a scrum fashion, then you're only going to be delivering every week. This Kanban you have that continuous flow. So you're able to work on the small number of items at the time. So you have that work in progress limit and it helps you focus better. It also helps you get things into production if you have everything set up, right? So you can put things in production without waiting for a specific end of sprint. So for some organizations, there is a mix of things where they use Scrum for development practice and for planning for having that cadence of specific events. And so they're able to retrospect so at a specific cadence, they are able to plan for a specific sprint goal, which is helpful. But another thing that they do, they can separate the planning part from the deployment part. So even though they're working in the sprint fashion, they're able to use the continuous delivery, continuous deployment in parallel with that. So in a way, it's become a scrum. All right, what else is here? Which practices is best practice for DevOps team to work when we have multiple development teams assigned DevOps member to different teams? DevOps manager managers assign work to the DevOps member which come from different development team. So great question. And option number one, assign DevOps members to different teams. So that's solving one problem, right? So it's solving the problem that you're bringing the expertise to different teams. And so development that operations team member is available to help when needed. Another option, so the option is the manager assigning work. So that's something that I wouldn't recommend because in Agile world, in DevOps world, we are encouraging self-organizing teams, which means that people are volunteering for work rather than development manager is assigning work. So that's already an anti-pattern. And another thing to consider is something that's called T-shaped skills. So that's helping people in the development team to learn those additional skills that they're lacking at the moment. So in this scheme of things that member of DevOps team may become this temporary mentor or person who will be able to help get up to speed cross-train people in the development team to help them build those additional skills. Automation is another thing that's helpful here. So the DevOps team member may help with creating, for example, Chef recipes or specific scripts that enable that environment creation on demand and then teach members of development team how to apply those recipes, how to use them to create environments. Same thing with building the deployment pipeline. So there might be a member of the DevOps team or the operations team who is going to pave the path, but then the idea is that the more self-sufficient the teams, the development team become, the better it is in terms of their organization overall. Okay. All right. So yeah, fix your forest bottleneck first. So what do we do in the game next? So we have the conversation around the default bottlenecks and naturally in the game, we have lots of them by design because we want people to experience the bottlenecks so then we can start fixing them. And the very first bottleneck that we fix, I'm going to talk about that, but we also bring things together in terms of this is the game, but what happens in the real world and some of the bottlenecks that you see in the real world, environment creation. And I remember mentioning it and everyone then comes up because that happens a lot. And so this is one of the things that, when you're moving towards DevOps, one of the things that you need to look first, how to automate environment creation. And that's where you see lots of organization moving into clouds because it's easier to create a new environment in the cloud than to provision a new physical server. Because it used to be that you have to order a server months in advance and then by the time you get it for your project, there is a lot of delays, lots of vendor getting it into the data center and someone has to provision it, someone has to configure it. And so there are lots of, not only weights and delays, but also lots of dysfunction happening around that where people in operations or people who had contacts with operations would reserve space in the servers in advance hoping that whenever they needed the space will be available for them, but that was causing us that now people who needed couldn't get the environment. So lots of dysfunctions were happening from that. So this move to cloud, that all goes away because now you can automate environment configuration, you can provision it on demand, and so this is something that makes this bottleneck go away. Large batch size code mergers. I mentioned it a little bit in passing that every time you do that, you run into a problem that or fixing any issues that are found and that large batch size code is going to be more difficult. Merge conflicts, something that every developer knows when we're working on the feature branches, and this is something that a lot of organizations are actually moving away from. And this Amazon and Google, they actually have master, everyone is working off on the master branch because they're doing a continuous integration and so every feature is not going into the long-lived feature branch. They're working in the way that the code check-ins are very frequent and so that way the entire code base can be in the ready-to-shift state at any time. Code deployment. Again, it's a bottleneck in many traditional organizations and this is where creating that continuous delivery pipeline, CICD pipeline is something that's addressing the bottleneck here in code deployments. Test set up and run. Overly tight architecture and this is the one that makes it difficult for many traditional organizations to move towards DevOps because if you have monolithic application that you have to work with, it's very difficult to think how can I make new changes in the small size batches because if you look at the monolith, there are so many dependencies that it's hard to change it in a small way. There are many areas of the code that are going to be affected and so what many organizations do to overcome these bottlenecks, there are at least two ways to go about it. One is to stop everything and re-architect and that's what Amazon has done when they achieved their first amazing number that they shared prior to that they actually had to stop and re-architect everything. The other way to go about it is something that's called triangular pattern. With that, the new architecture or new code is being built around the old one and just like if you imagine the large tree which is the monolithic application and this little new plant that goes around the tree and then eventually it strungulates the tree because it becomes more powerful, the new plant, so the same analogy is here. You start building out the new application, the new architecture in a way that it's eventually replacing piece by pieces replacing the older architecture and so at some point that new architecture becomes your way forward and then the one that's been the initial monolithic application is the one that you're going to decommission. So that's kind of step by step moving away from this bottleneck number five. And my favorite bottleneck, people unwilling to change. So just like I mentioned a few minutes ago, sometimes that's what's driving the biggest resistance to DevOps transformation because people think that with DevOps, like if I'm in operation and I know how to do things manually, now you come to me and you tell me, Dana, if you're going to automate everything you're doing, how do you think I'll feel? I'll be resisting, right? Because I don't want my job to be automated out. So what's important to notice in this area is that this automating some of the manual tasks that operations is doing today, we're actually helping them to not only create space for more innovation because they don't need to do that manual work anymore. They're also getting an opportunity to learn a lot of really cool new technology that comes with DevOps automation. And they are having this opportunity to upskill themselves and become marketable in the job market because DevOps skills are very in a high demand. And so with the knowledge that operations have in terms of how all this hardware works and how to tune in, how to optimize it for production performance, all that knowledge is not going to go away. They can reapply it in a different way using new skills and new tools and they're the position the best in making it happen. So it's not that the job is going to go away, it's going to be redefined and if you're on board with it, you're going to be in the best position to upskill and to become really marketable in the job market. So that's that. And what we do in the second sprint, we change a few things. So we move security engineer into the development team. And what that does is that now instead of running security scans at the end, we start sharing information about security issues from the very beginning. And so that makes a huge difference in terms of how much better developers are able to build things in the game, but also in the real world. And that's when you hear people talk about DevSecOps. This is one of the elements of it is that we're making security information available to developers from the beginning. So instead of testing at the end for security, we are building security in or we're doing something that's called shift left on security. So that's one of the practices that helps to deliver faster because now we are not stopping and rebuilding, fixing issues that we find at the end. Instead, we are building them in such a way that they're secure from the beginning. And another thing we do here is that we do this exercise that simulates the building T-shaped skills. And so I'm going to ask you another question. Give me a plus one if you heard about T-shaped skills and minus one if you have no clue what T-shaped skills are. Okay, okay. All right, most of the people heard about T-shaped skills. All right, all right. And basically T-shaped skills and a few minus one. So I'll quickly explain what it is. So basically think about if you are highly specialized in one area, you have I-shaped skills. So just like letter I in English. So just like that, it's I-shaped. So it's like very narrow. If you have skills that are in addition to your main skill, you have complementary skills in other areas that may be not as deep, but they're helping you to expand your knowledge but also helping you to be more effective member in the cross-functional team. So then you have T-shaped skills. So it looks like letter T in English. So you have the deep skill set in your area and this shallow but very broad skill set in other areas that are complementary. And so what that helps is helping with removing bottlenecks because if someone is, let's say we need specific operation skills in the area, but operation person is busy and others know a little bit about that, then operations person at least can guide others how to do it. Or maybe there's something small that others can help with addressing while operation person is focused on the more difficult task. So in the game, this is what's happening. These are people that are doing the cross-training, they're doing the T-shaped skills. And you see the energy that generates because people are now connecting with other people in the room. And just like what's happening in the game, same thing when you are allowing people in your organization to build additional skills that generates energy and that opens opportunities because a lot of times when you have developers of operations or any knowledge workers who are stuck in the same role for years and years and years, they get bored. And if they don't see new opportunities for them within the current organization, they may decide to leave that organization and go elsewhere to find the new opportunities for career development, for career development, for progression, for achieving mastery. Because the knowledge workers, we know they're all more driven by intrinsic motivation, so autonomy, mastery, and purpose. So those are three things that we look for. And having the opportunity to cross-train, to expand the skill set, and maybe even move into different areas of the technology organization, that's an opportunity that you're given to your people in your company to achieve mastery without leaving your organization. So that's another reason why you're building the two-shape skills and encouraging that role-blending, role-switching, or expanding the knowledge is helpful for the organization because you're keeping all the talents in your pool versus seeing them leave and going elsewhere. Okay. A couple of things that we do throughout the game. I'm using a lot of techniques that are called liberal instructors that are amazing for debriefing. And I'm going to share a couple of them with you. Again, because we don't have breakout rooms in the way how I could use them for liberal instructors, you'll have to imagine how they work. But I encourage you to give it a try because they are very useful in bringing everyone's opinion and everyone's ideas into shared conversation. So one of the simplest one is something that's called one, two, four, all. And this is where we start with one minute. Everyone's just writing down the response to a question. When we're doing the debriefing, we're asking the question around the simulation. But you can use it for any other purposes. It could be a problem you're trying to solve in your current sprint. It could be something around questions for the next release. It could be anything. You start with one minute, every person individual in silence just writing down as many answers as they have or visit that one minute. Then next, you're going to come together in Paris. In Paris, you're going to talk to another person sharing what you wrote on your little post-it or piece of paper, sharing with another person your ideas. This is where this structure is awesome for when you're working with a mixed group, when you have introverts, extroverts, different level of seniority. It's awesome because it gives you an opportunity to collect everyone's ideas, share them in a very safe way because you're just sharing it to another person. You're not sharing it out for the entire audience to start from. You're starting from this two-people conversation. In that conversation, you create more ideas because that's a lot of times when we feel safe, we're able to build on each other's ideas. Next, you're going to go with your partner into another pair. Now it's a group of four. Again, you're talking to a small group and you're now able to bring ideas from every group and think about what is the best learning, what is the best idea that we want to share. This is an opportunity for extroverts now to represent this four-people group because the next step would be coming together as all and sharing out what came out from these small group conversations. As you can see, we're building on starting from an individual idea, then sharing them out in a small group and then coming to what's the best idea from the group. This way, we're getting a lot of different ideas from the entire group and the group could be anywhere from, it could be as small as four people, but it also works amazingly when you're working with the large groups. I use it a lot when I do live conference presentations for just collecting questions from the audience because that way everyone got to ask the question. Not all of them were heard by me, but all of them were heard by at least another person in the room. It makes that so much more engaging for people in the audience, but also it makes it safe. That's something to try when you're running it in the organization. Some of the aha moments are coming from this spring, too. I took a few screenshots from the actual posters when I was running it, but as you can see, lots of people are commenting on broadening the T-shaped skills as a developer, more engaged with the product, more collaboration, more gets done. Again, through this simulation, they're able to see that positive effect of working in a different way. Instead of being a siloed, now working together knowing more about what's happening in the other group, expanding that system-level perspective. That brings us to the second way of DevOps, the Amplify Feedback Loop. This is where we talk about small batches because just like in the congested highway, if you're in the large truck or in the large car, you're stuck. As you all seen, I've seen it with my own eyes when I was in India, if you are in a small bike, you can move so much faster. You can just drive around all these different traffic jams and you can get to your destination faster. Same way with the code. When you're deploying in the small batches, this is how you can get through your pipeline faster. If you have large code batches that you're trying to deploy, you run into many issues because there's just so many more opportunities for failure. The faster you get it to production, the faster you're going to get that feedback loop because you're able to get it to the hands of the customer. Another thing about the feedback loop, you might have seen this diagram. I'm actually going to talk about the diagram. CICD, that's another thing that introduces the feedback loops. The third type of feedback loop is giving information back to the teams from the live production monitoring. I covered that. Someone was asking about the books. Continuous delivery by Jess Humble and David Harley is an amazing book if you want to get into details of what is continuous delivery and how to implement it. I highly recommend this book to get started. This is a diagram by Jess Humble that I wanted to share because it gives a very good insight into different levels of feedback loops that you get with continuous delivery. Just like you see here, when the code gets checked into version control, it triggers the automatic testing. Then building unit tests are going to either fail or pass. If they fail, then feedback is going to get immediately sent to the developer. The developer is not going to wait for weeks or months to know that there is a bug immediately as they're checking in. The unit test runs. This is where they get the first feedback. Your feedback loop is closing really fast. If unit testing failed and the build was successful, then it gets promoted to the next level, which is we're now running automated acceptance tests. The same thing here. If it fails, immediate feedback to the developer. The code that you just checked in passed on the first step, but it failed on the other. The issue that you introduced was that last piece of code that you checked in. That's where, again, the larger the code that you're checking in, the more difficult it is to find the problem. The smaller it is, the faster you're able to find the problem and fix it and move on. Same here. If it doesn't fail here, it gets promoted to the next step and the next step and the next step. This is where you see there is little approval. These two steps could be done automatically. It could also be done manually. Some organizations, they don't need the actual production deployment to be done all the time. They want to time it with specific marketing events. They want to time it with maybe some other business-related, it may require manual approval. But having an infrastructure that is in place and you can by the push of a button send that code into production knowing that everything prior to that was well-tested and there are no issues and no security issues. It's ready to be given to customer hands. That's what CI CD enables you to do. Live production monitoring by the teams. We're talking about telemetry. We're talking about having different ways to monitor how your code operates in production and not only monitored by operations but also sharing this information with the development teams. Sharing how well the new feature that they deployed, how well is it being used? How well is it performing in production? Is it scaling well enough given the number of users that are accessing that feature? It closes that feedback loop faster and it's useful information to developers to know how it's performing. It's also very rewarding for the developers to see that, hey, the new feature we were working on is in production and look how many users are accessing it and using it. So seeing it in action that whatever they've developed is being used, it's very positive feedback loop that helps people to stay energized about their work. So that's also a positive aspect of that. On-call rotation. Having developers initially self-managed the instances of production servers. So again, different types of feedback loops. With on-call rotation, you get developers who wrote the code to be part of their on-call, which means that they have a page, and if something breaks, they get paged and they will need to address the problem. Sounds very different from what we typically do in traditional environments, where there is that separation, segregation of duties, separation of controls. Here we actually get that information into developers' hands because they are the ones who know how to fix it. And they also take responsibility about fixing it because, again, they don't want to wake up at 3 a.m. in the morning, right? No one does. And then having developers initially self-managed the instance of production server, that's again, we're talking about with the cloud. We are able to do that because that way developers can run the code on the production-like environment. So it's not production, but it's production-like. So all the configuration is the same and potentially any issues that could become a problem, they can be identified faster in this pre-production, production-like environment where developers have access to and they can tune things to make sure that things are working as needed. And so then that configuration can roll into production as well. And again, some people don't want to do continuous delivery for many reasons. Job security, people are reluctant to automate. Antiquated processes, we've always done it this way, right? Have you heard that phrase before? We've always done it this way? Yeah. Outdated understanding of segregation of duty. So that's something that's very relevant to many financial institutions where they have lots of, you know, processes, lots of audits, lots of governance in place where they need to respond to the auditors to make sure that they are limiting the risk. And segregation of duty is one way to limit the risk. And that's where, when we move into DevOps, having conversations with the auditors and helping them understand how DevOps practices are changing the way how we minimize the risk is very useful because there are other ways to manage risk. There are other ways to make sure that no one is accessing, that unauthorized person is not accessing production, that no one is going to bring harm into, you know, the financial products and then everything is secure. We have lots of automations, we have lots of logs with the continuous delivery and continuous integration. And so through having access to that logs and then monitoring them and making sure that we're storing them properly, that can address many of the concerns that segregation of duty is addressing in the traditional organization. And involving auditors in redefining the process is something that's going to overcome that. External dependencies and fear of change, that's a big factor here as well. So it's unknown how that's going to work. And so it's scary. And sometimes if organization is coming from the place of pathological or bureaucratic culture they are looking at any change as something that potentially fail, can fail, and failure is not an option. So in those organizations, introducing DevOps is going to be hard. So it's changing the culture as well. But if you're still not convinced that DevOps is a way to go, I'm going to share another insight, which is that's based on the Gardner, IBM and Sonotype research. It used to take 45 days in 2006 from the time that security vulnerability is discovered and to the time that it's exploited by hackers. So guess what? From 45 days, it went down to three days in 2017. I'm sure now it's even less, which means that if your organization is not set up to roll out new features or new patches, new security patches fast enough, you're going to be vulnerable to these type of exploits. So again, you don't have to be an organization that deploys new features frequently because of many reasons. But if you have an infrastructure in place, you have practices and culture in place to be able to do it in the emergency situations, that's still a big step forward. Okay, yeah, I love the other comment in the chat. If something is not broken by fixed methodology, yes. That's kind of along the line with, you know, we've always done it this way. Yeah, but just like what I'm saying here with the security issues, may not be broken, you may not know it's broken. And once you find out it's broken, then you may have very little time to fix it. So it's better to be prepared ahead of time knowing that, you know, if something like that comes up, you're able to roll out new features, new fixes in a short period of time and protect yourself and your organization from exploit. All right. And then the third screen is what we do. We simulate reduced batch sizes. We simulate containerization and environment creation on demand. Again, it's all using, you know, props and logos, but the learning that's coming from the game is very real. And this is what's happening. So as you can see, remember that picture in the first spring when everyone is sitting at their own tables. So spring three, everyone is working together and people are standing because they're all involved. I'm sure you heard the term swarming. So they're swarming on this work. They're working together. They engage, they're involved in bringing business, bringing value to the business. And so these green packages that you see on the right corner, that's all the packages that getting delivered to the business. If the first print, nothing got delivered, then, you know, next print is actually they're getting the value delivered. Okay. And the learning that comes out of it, we do it another debrief. And that debrief, if we're using another technique from the Bureau of Instructure, that's called fishbowl. And again, I'm going to talk about it, but I encourage you to try it when you have a chance because it's another way to bring out lots of ideas starting from that one minute writing things down on a piece of paper to doing the second step, which is that fishbowl debrief. And this is where you have five chairs, one chair is empty and then the other four people just, whoever wants to talk, they can come out and sit in those chairs. And they start having conversation via debriefing on the game, but it could be anything, right? This is another liberating structure you can use. So they have a conversation with each other in the fishbowl. Everybody else in the room, they're observers and they're listening into this conversation. They may have any questions that they will ask when they're in a fishbowl because that empty chair is an invitation to join the fishbowl. And so the way how it works, anyone can jump in on that empty chair. What's going to happen is that the other four people that were in the fishbowl, one of them has to give up their chair because the rule is there has to be an empty chair at any point as an invitation. And so it creates this interesting dynamic that for one thing, people who are in the fishbowl, it gives them a sense that they're just talking to each other. It's like they have this conversation, the debriefing, they're sharing what they found out through the game. And then others are almost like watching talk show, except you can actually join the talk show because that chair is available for you. And because he had written some of the things on the post-its before going into a fishbowl, it makes it easy for even the introverts to jump in because they don't need to sink on the fly. They can just read what they had on their post-its. And that's already a way for them to contribute into this discussion. And again, give it a try because you'll see how valuable it is in bringing out different people's idea into shared space. Okay, and these are some of the things that come out during Spring Street. So people comment on improving transparency, more products getting delivered. I love the harnessing energy and enthusiasm. You can actually feel that you can see it in the room and it gets people excited about DevOps, gets people excited about seeing that they expanded the roles. There is a little bit of confusion that gets created. That's normal because as we're learning something new, there will be an opportunity to be confused. But if you don't accept that that's part of the learning, we're never going to try anything new. And with DevOps, there is a lot of new things that you need to learn. And that's where organizations that enable and that learning culture are going to be the one who are going to win. Which brings us to the third wave of DevOps, culture of continual experimentation and learning. And what this is about is about organizational learning, safety culture, and even doing practices like injecting failure to test resilience. One of the best examples in that space is what Netflix created as something that's called Chaos Monkey. So give me plus one if you heard about Chaos Monkey. All right, it seems thumbs up. Okay, very cool. So Chaos Monkey is a special application that is designed to kill production servers at random. So just like something is working in production and things may fail, instead of trying to solve the problem when it happens unexpectedly, they run this Chaos Monkey application in the controlled environment, which means that people are watching as Monkey is doing its job and killing servers at random to make sure that if that happens, the application that has the business functionality is continuous to work and continuous to serve the customers. And if it doesn't, then they're looking for a ways to address it because everyone is now focused on that, right? Because it's done in controlled space. But that's exactly what this injection of failure means, is that injecting failure when you're watching, that way you can build your system in the way that it's going to be resilient when things fail, when you're not watching. And that speaks a lot about culture of learning, right? Because it's expecting failure and being resilient to that failure and knowing that it's okay if we fail as long as we learn from it. And another thing, the safety culture, so if you've seen the 2019 report, State of Debups, and if you read the latest book by Jin Kim called Unicorn Project, psychological safety comes in front and center because this is something that's helping organizations to be more effective because this is something that enables us as individuals to contribute at the best possible way. Because we're not afraid, because we're not trying to filter what we say. We're not trying to think about, I have an idea but what if someone is going to think it's stupid? I'm not sure. So we're not going to go through all of that. We're going to share freely what we're saying and contribute in the best possible way and that will generate innovation and new ideas, new opportunities because of that culture that enables that to happen. Very quickly, some of you attended my session two days ago and you've seen this slide. This is my version of Rand Westrom's organizational cultures. So three cultures types, pathological culture. If things go wrong, then organization is trying to hide it and then punish the person who found it or exposed that. Bureaucratic culture, we're fixing things locally, we're not sharing, we're not learning as an organization. And generative culture is where we're actually looking for opportunity to learn from the failure and share that information with others. So that's where practice is like blame the sportsmortem happening, the leaders are inquiring about what was drawn in the system that caused someone to make that decision that was a wrong decision. So what do we need to fix at the system level versus who do we need to fire? So that's a big difference between the cultures. So DevOps is thriving in the generative culture and that's what enables these practices and these cultures to be adopted, the culture of learning, culture of experimentation. That's another interesting chart that I'm going to quickly share. So this is information about DevOps practices penetration across the different practices and what's interesting here is the question was asked to different people in organization, people at the team level and management and the C level execs. And what surprised me and actually speaks a lot to that culture of pathological bureaucratic culture is that there was a huge disconnect between how people on the ground knew how things are happening and what people on the top of organization thought that is happening with DevOps. My favorite one is this one. Incidents response are automated. So people in the C suite think like, oh my God, everything is under control. We have all the incidents responsible that are automated and people on the ground know that that's not exactly the case. So when that happens, when people are not feeling safe to share even information about how DevOps practices are being adopted in organization, it's a problem, right? Because it means they're not going to share anything that's even more serious or potentially dangerous that C level needs to know. So you have to raise more fear, pathological culture or more learning generative culture. And so DevOps is paving the way towards generative culture and more learning. That's what enables DevOps practices to become effective because of this culture that is creating. And so in the chat, my last question for you, if you were 10 times bolder, if you had all the bravery that you can gather in the world, what big idea would you experiment with in your organization? Just think about if nothing was stopping you, if you were brave enough to do it and the culture was supporting you. So just give me a few ideas in chat. What would you experiment with in your organization? For days of work, Rick. Sounds good. Chaos monkey, yeah. OKR is OK. Self-stores the team, love it. Very cool, very cool. Come on, I need more brave people in this chat. Come on, people. You're not saving this chat, right? So we're not going to share it with you employers, so just feel free to. Come up with more ideas to experiment with. Knockdown functional departments, make feature teams. Yeah, love it. Innovation. Yep, yep. Both in banking and no-springs disruptive behavior. All right, all right. Yeah, very cool. OK, thank you. Thank you for those brave souls who responded in chat. Compete with Amazon, daily delivery numbers. Yeah, that's awesome. Thank you. Thank you, everyone. Fantastic, and there are some other books that I highly recommend looking into. DevOps Handbook. This is one that if you want to read and get to know how to do it, this is the book. If you want to understand why Phoenix Project is amazing for getting your head around some of the typical challenges in IT and how to solve them. Continuous delivery. You can already accelerate another great one. Fearless Organization. That's the one that Amy Edmondson wrote about specifically leadership practices that enable psychological safety. Goal is a fantastic book. Liberating Structures. This is a collection of all the different facilitation techniques. And then by introduction to DevOps with Chocolate Legons Crumb Game. Once we get back into the physical world, you can read up how to facilitate the game. And with that, thank you very much. It's been a pleasure to give the presentation to you. I appreciate everyone's involvement and our brave ideas that are coming in. Thank you.