 Thanks for coming. I'm Henrik. I'm from Sweden, and this is the first time I'm in India, so it's very fun to be here. I'm only, this is a case study I'm going to be talking about. It's a 60 person project that I was involved in. Is that a big project? I'm only here to think 60 people, that's a big project in your experience. That's about one third of the room. How many of you think that's a really tiny period of the project? A few are like ass people. What about the rest of you? Medium. Medium, medium. You can almost be Swedish. Swedish is always about medium. We actually have milk, which is like medium fat milk. And that's the most popular one. Everything has to be medium. The reason why I made a case study out of this was because I learned a lot. And it was actually the first time I saw a successful project at that scale personally. I hear other people talking about it. Personally, I've just never seen projects succeed once you get around that site for some reason. And that of course begs the question, what is a successful project? I'm going to use Alistair Coburn's very simple rule of thumb. It's actually not about if the project was successful, but if it was the way it worked successful. It was a method successful. He spent a lot of time traveling around the world meeting teams and deciding is the way they work. Working, right? And he concluded that, and the way he measured that was basically by checking two questions. Number one, did they deliver? Because it didn't, it probably wasn't successful. But did they actually deliver? The second was, the people who were in the project and stakeholders, would they consider using the same way of working next project? Right? And if yes and yes, then successful. Because sometimes you have projects like, yeah, we delivered, but we never ever want to work that way again. And it probably wasn't a very good approach. So this was one of those projects that, yes, we delivered. And yes, we made an impact. The product pretty much delivered the impact that we hoped it would. And the people in the project were like, we want to work like this every project in the future. And that's the first time I've heard people say that, like such a big project. So that's why I wanted to capture what did we do and what did we learn, right? So that's what this is. It's a story about where we learn. And it's by no means a perfect project. We had our share of problems. But relatively speaking, it was one of the more successful projects that I've seen. So I've tried to capture some of the things that I think helped us. So a bit of context. This was Swedish, it was a government project. So not only was it a fairly successful project, it was a fairly successful government project. What? How often do you hear these two words in the same sentence? Government, successful. This match here. It seems harder for government. I don't know why. But what happened here is we had this interesting situation. This was a Swedish police organization, right? And the background was that the Swedish police wanted to get more effective. In fact, the Swedish Minister of Justice had pretty much placed a demand on the Swedish police organization saying, you guys better shape up, because you're not effective. And pretty much made it clear to the whole Swedish population that we are going to shape up the Swedish police and become more effective. And basically one of the things they wanted to do was reduce the amount of manual work around handling of crime investigations. Specifically in this case, what we want to address was the issue of suppose you get caught drunk driving, right? Don't you hate when that happens, right? You get caught drunk driving, right? And so what happens in Sweden, I don't know how it works here. What used to happen in Sweden was that they would capture your information, they would find out or interrogate you, get all your personal information, and then probably take your driver's license. And then the police officer would drive back to the station and do some paperwork, right? And inevitably they forgot to ask some questions, they had to get back to me and ask something else that they forgot, or they wrote something wrong, misspelled or something. So there was a bit of defect handling. And then they hand off to an investigator who had to follow up exactly how am I going to get punished, and when, and there's always bureaucracy. And the information moves through different systems, different people, and maybe three months later I get my fine or whatever, right? It's quite a long time for what is actually a quite petty, simple crime. So we wanted to automate this and basically say that, well, the police officer should have a computer in the field. They should have access to all the systems they need. And when they're handling this crime investigation, they do it in the field while talking to the criminal, right? And they put all the information right into the right systems. They get validation, right? And they don't have to drive to the station to hand off to somebody else. So now the police officer is the investigator. That's a huge cultural change, right? So this is really, the big thing is really the cultural change and not the technology. Well, I'm not going to talk so much about that, the cultural stuff. I'm going to talk more about the practical aspects of building and delivering this system, right? But another piece of context was that the police organization, and this is off the record, so don't tell anybody. I told you this. They had tried this twice before. I hope the camera's not on. Live streaming, right? So they tried this twice before to build the system. It's quite complicated because this, the software inside this computer is talking to all kinds of different systems. Things like, you know, confiscation of weapons, all these different things. It has to integrate with this. So it's pretty complicated. They tried this twice before and totally failed. Spent a lot of money and done pretty much what Craig talked about this morning, these big project setups, and failed, like, like, miserably failed. And this was the third time. And now it was high-profile because the Swedish Minister of Justice had told the public that we are not going to fix this, and we're going to fix it by spring. So, oh, shit. This is going to be a problem. And what they realized was that we can't use a normal way of working. Because it's a government organization, they have kind of waterfall-ish methods from the past, and those are still kind of, tend to be used. And the experience was we can't use the normal way of working. So we have to try something different. And that's why we got to try something different. So basically, this was like a bubble of agile inside like a whole ocean of waterfall kind of. So the reason I say that is because what I'm describing to you now is not the typical way Swedish government works. However, it is changing fast. So I noticed that since we now have one case where the Swedish government project using agile methods kind of properly and actually making a big impact, this is spreading rings on the water. So now we're seeing a lot of other government organizations catching on. It's interesting to watch this effect. Anyway, so yeah. So this was all about reducing the processing time for petty crimes, right? And the result was quite successful in the sense that it used to take like months to get a crime closed after the system was put in production. It was a matter of days, sometimes even hours. So we can see that this section actually worked. But now let's go back in time and look at what about the project itself? How do we do the work? What did we learn? So this was a background. Around 2009, mid-2009, that's when the project was launched. I wasn't there at the time, so I don't know the details. But roughly 10 people started trying to scope out what is this about. And they started building it. It was quite traditional. They had some traditional use case requirements and all these kind of things. Because there are some things that are fundamentally different. And I'll get to that. But anyway, and so they're starting up this project and they're building this thing. And then about a year later, we had about 30 people. And it was growing. Here was the first release, the first pilot release. Okay, one year. That might seem like a long time, but I don't know. Does it be working government? We'll probably see that this is actually quite fast from being government projects. At least for the Swedish police. We had done some previous project which took seven years before anything was delivered at all. So that was one main difference. We actually delivered something before it's all done. We delivered something before it's all done. But since it's a very complicated, big system of building, by that time we had 30 people and then it was growing. So I came in somewhere around here as coach because now we're starting to see kind of like symptoms of chaos and scaling pain. There's confusion. Before we could use kind of like vanilla scrum. And now it's like, where does it go? And then there's a little bit of waterfall stuff still left from the past and how are we going to manage this? So first pilot release there and then a bunch of follow-up releases. So I'm going to talk about kind of what we did from here to here instead of making some major changes in how we work. Now one really important success fact in any kind of big project is dividing it up into a series of smaller things. And this is interesting. Right now I'm working with a Swedish product company named Spotify, Music Streaming Service. And we have about 400 developers and testers. And so in a sense it's bigger, but it's also in a sense smaller because we avoid big projects. We really, really avoid big projects. So most of the teams are pretty much independent during their own little thing. Sometimes we need to have four or five teams collaborate for a certain period, but the basic idea is don't get good at big projects. Avoid big projects. So almost none of the stuff I learned in this project is applicable on Spotify. So each scaling case is different. But here we had a big project, but important is slicing it into small pieces, right? It's just basic iterative development or incremental development. So how do we slice this elephant? In this case, two dimensions. One was by region. It's like we don't have to deliver the system to all of Sweden at once. We can deliver it to just one region, right? And that saves some effort. But anyways, we don't have to support all different types of crime. We just start with just one. So our first crime that we supported was, it was two actually, it was drunk driving and petty weapons crimes, like, you know, the police says, hey, you're drunk driving and they pulled out a knife. Okay, it's also illegal weapon possession. So that's all we need to solve. That one single crime type in one region, right? All of the crime types for all other regions are handled the old way. So that's one way of taking something big and turning it into something small. Next release, add more crime types and some more crime types. By now it's like, okay, we'll have something that works in this region. So now let's try two regions, right? Like that. And, well, the step from one to two is very big. The step from two to many is not as big. So then we can spread it out to the rest of Sweden. And this was the kind of the main release, the one that the press was writing about and et cetera, the one that made the impact. These were kind of not really visible to the world. So this was considered to be the main release. After that, we just added more crime types, more integrations, you know, relating with different types of systems, et cetera. And it's, even today, it's just still evolving. So the nature of agile development is, of course, that you're never done, right? We don't distinguish between development and maintenance phases. It's all maintenance from the day you write your first line of code. Okay, so that's one really key thing. Without this, I think it just would not have succeeded. That is user involvement, right? That sounds familiar. But almost all research I see around success factors for projects, the failure factors, put somewhere at the top of the list, usually number one, something about involve the users. It's very interesting. They did a good job with that here. In fact, I've rarely seen a project of this scale with such good user involvement. Here was, okay, this wasn't really a user. It was more like an internal customer. This was an in-house development. Okay, the police have their own development organization, although we actually brought in like 30 consultants. If you want to look at it that way. But it basically was in-house development. The police organization's development department built these stuff for the police force, right? So we had an internal kind of like a main customer or chief product owner who had this high-level list of these other things that we need to support, right? Nothing strange about that. But he was an interesting thing. We also had an on-site user. This is a real police officer or investigator who actually is in there mingling with the teams, having lunch with them, hanging around. Every, you know, one day per week. To the end of the project, when we were doing releases frequently, then we had two people full-time, based on a rotating schedule, hanging out with the developers. This was very, very, very useful to get early feedback. Like, what? That's crap. You can't build that way. I need it like this. Another thing, we had an acceptance test user group. So our basic cycle was two months. Once we had the first release out, we were doing pretty much every second month doing a new release. At the end of that period, we brought in about 10 or so real users, police officers and investigators. They came in and they sat down and they played with the system for like one or two days and gave feedback. So that helped a lot. And then finally, the pilot users, right? Like, I told you about the pilot release to one region. So these are people who don't come into the building, but they're sitting in their cars and they're installing our beta version, our alpha version, and they're actually using it. And of course, they're hating it. They don't want to make them hate it less. So the basic assumption is users will hate it, but give it to them anyway so we can learn why they hate it so we can make it better, right? It's very, very useful. So I'm very thankful to them for, you know, contributing their time to do this. So a lot of user involvement. Very important. Right, so now let's jump in and look a little bit at the structure. And a bit of history, based on past habits, they had these different departments, requirements, analysts, developers, tests. Does that sound familiar? Departments, right? And they had that, which wasn't very good. What happens when you, what typically happens when you have this constellation? Handoffs. Handoffs. You get this, right? You get a bit of this, right? Confusion, like, but we don't get what you mean or how are we going to test this, right? I don't understand this requirement. So even though it was kind of agile, some of our old problems still, we still had handoffs and we had problems with that. So gradually we morphed this to a hybrid. So we didn't go all out. We still had these kind of departments, but we started morphing. So what does this mean? Well, we put requirements people inside the development team physically, right? And we put tester inside the team physically. So now this is a kind of typical cross-functional scrub team, which is a good first step. But we also had this, we had a virtual team. So basically three teams that are sitting here working together physically and just, you know, 10 meters away, we have these four people. And what are they doing? Well, they're not in here. They're looking at the big picture. So they're looking right into the future. They're designing the high-level epics, right? Well, this person here kind of follows a feature into the team all around the production. And by having one person and one analyst follow the feature all the way through, we don't need as many big documents, right? I mean, the document is now a person, a domain expert who follows in. So this turned out to be quite nice, like a hybrid model. We get the big picture and we get the feature focus. And same thing on this end, exactly the same thing. Testers in the feature teams, the development teams, they're trying to help this team get features done, right? And working. But these people are looking at the high-level things. They're doing system-level testing. They're looking at performance. They're making plans for how we're going to make the whole system fit together, right? So they don't care so much about features. They're looking at the whole system. And this was a quite lightweight model. In the sense that this person could go here one day and this person could jump in there for one day. There's a little bit of floating in between here. But basically, this was our kind of hybrid model. And it turned out afterwards that it actually worked quite well. So I wouldn't have recommended this based on previous experience. But since we started from here and I didn't want to just change everything at once, we moved to here, and it turned out that we were quite happy being there, gave us a compromise. Yeah? The guys were outside. What is the reporting? Is there any seniority or reserve from a competency standpoint? Is there a difference between the guys who are inside and outside? So the question was, is there a difference between that person and that person from a kind of line perspective reporting structure? No. In fact, we didn't change anything about the line management structure. And that hybrid experience from previous companies, you can make changes without changing the... There's a structure on paper. And then there's a real structure, which is how people sit and how they talk to each other. And I found that sometimes it's easier to just let the paper be the paper. And then people sit in whatever way they need to work. And then over time, the line structure sometimes catches up. But maybe that's cultural. Maybe that's... In Sweden, we tend to be quite informal. I haven't worked here. But I imagine maybe there are situations where you'd have to be more specific about looking at the line structure. I'm not sure. Did you think about any criteria about who should be in and who should be out? Did you think about any criteria of who should be in and who should be out? And not really. It's really up to them. Basically, everybody agreed that this was no good. And so I proposed that idea. How about if someone here just joins? Like, yeah, let's do that. And I don't know exactly how the decision process happened. But basically, somehow they agreed that, yeah, I'll join those guys. I'll focus on the big picture. So I don't know the details of how that was decided. So who are the roles in those team requirements? So what are the roles inside here? In the first one. In the past, there was one manager of this group. And then there wasn't really any roles, but there was lightweight specialization. Some people focused on certain feature areas. Some people talked more to the real users. Some didn't talk as much to the real users. But I don't know the details of that. Is it like business and product managers? Yeah, business and product managers, requirements, design, that kind of stuff was here. But not a very formal role structure inside there. So the confusion was due to the size of the team? So you saw it getting more confusing when the size of the teams increased or it's even applicable for a smaller team? That's a good question. Was it confusion because of the size? Partly yes, because when there were just 10 people in total, I mean, people can just talk to each other anyway. But once it got bigger and bigger, these confusion points were amplified. And people started communicating more and more through the documents. And there was less tacit knowledge. If you have 10 people in the room, we can all have the same picture in our head. With 60, it will start diverging. So now I'm starting to get questions that are stuff that I'm getting to. So I'll just continue the presentation. Did you need to reinforce the team with additional people when you changed the course? Did I need to reinforce with additional people? No, I didn't in any way influence how... We didn't actually... We didn't change the number of people as a result of this at all. Excuse me, once again. We're not really moving in out of scrumpties because these were fairly stable. How do we know it's happening? Yes, I'll continue the presentation. So this is just one tiny part of the whole picture. So, yeah. Assuming that looking at this structure, some people are not co-located. How do you manage it? If they're not co-located, all these books... If they're not co-located, yeah. That's one of the success factors of the project. We were all co-located. But that's cheating. Which brings me to a funny story. Which brings me to a funny story. Kent Schraber, one of the scrum guys who created scrum, he has a very, very funny story. He's worked with a big company and they're like, we want to try scrum, but we don't want to go all in. We want to try it just in one place. First, okay. So they broke out a cross-functional team from the static structure. They broke out a separate team and they gave that team direct contact with the customer, put them together physically, and released off and they did scrum by the book. And it turned out that this team was wildly successful compared to other projects. Surprised, right? But then the management said, that wasn't because of scrum. They cheated. They were sitting together. That's why they succeeded. You know, all the teams are sitting together. So let's cheat more often than how about that. But yeah, it is an interesting question. I have no idea how this would have worked if we were distributed. Distributed is different. The same principles apply though, right? Like, I would rather see a distributed group working like this than a distributed group having these hand-documented handoffs. But I'm going to leave that question because the purpose of this presentation is to give you an example for real life and then we'll save that philosophical question about what might have or what would have happened. All right. So you have a lot of great questions and I hope some of them will be answered if I get a chance to move on. So, we had this thing that I called the Daily Cocktail Party, which is basically a series of daily meetings, right? You walk in any time between these times and you get surprised. Everybody is standing up and it really feels like a cocktail party. So what was this about? Well, it was really a very disciplined structure which we evolved into. All the feature teams had their daily scrum. Just a daily scrum, right? What did we do yesterday? What am I doing today? What's my impediment? But then at 9.45, the different disciplines, like test requirements and development, they had like sync meetings at that level. So all the testers would meet up in front of a board with stuff that is related to tests. So that means this guy over here would go here. This tester here would be here. This tester over there would go here. Plus all those testers that are outside, right? So all these people would meet up over here, which means they bring fresh knowledge, fresh insights, and dilemmas from here into here. We talked about that. At the same time, all the analysts are here. So they might realize that, well, yeah, our team is having a problem. They don't understand how this part of the system is supposed to work. So they have to help clarify that, right? And here is a dev sync. Not all the developers in this case. It was one developer per team. In fact, it was a team lead, which actually corresponds to Scrum Master in that case, but I won't get into detail. One developer per team plus a development manager met here. And they synchronized all the things like who built the build, and things like, we have an architectural issue to figure out. Techies sync here. And they did this in front of a big project board, and I'll show you that in a moment. And finally, the project sync, which was basically one person from each feature team and one person from each discipline. This was a semi-static group of people, which we called them the cross team, which is maybe a silly word, but it's basically a cross-section of the whole project. So Atesta are a requirements person, a developer, a dev manager, at least someone from every team here. And we had some support roles, such as a project manager, considered to be a support role, me, which was a coach, we had a configuration manager, some kind of outside roles were here. And together we did what we call the project sync meeting. And we looked at the big picture. Are there any blockers for the whole project? Where's the bottleneck right now? Things like that. This happened every day. And for a while I was like worried, like, do they want to do this really? There's a lot of meetings. There are so many problems in this way. Just by talking to each other. Because there's a nation leaking between these meetings. So if there's something important that is a problem, it will get solved locally, or get escalated here, or get escalated here, and then get dealt with quickly. It turned out that a lot of things that we otherwise would have had to create process for didn't need process. It said it was just talk to somebody. So let's take a closer look at this thing, which was also an important success factor for us. I have this little trick when I don't know what to do, come into a new client, and I don't know how to help them. There's this default move, which normally helps me get one step closer, which is visualize what the heck is going on. Because that seems to always be useful. Just visualize what's going on. In this case, we did that by setting up a board. And over time, that evolved into a structure. And basically our structure is that this is one single board for the whole project. We have epics flowing in. High-level epics flowing in here. They get broken down. This is like an analysis phase. Look, there are phases here. An analysis, development, test, and production. What the heck? Is that waterfall? Is it? What's the difference between this and a waterfall model? Why are we at shorter cycles? We're doing all the phases at the same time. If it was waterfall, it's here first. And then all the cars would go to here. These are all happening at the same time. Phases. A feature goes through different stages, but the project doesn't. That's the key difference. An epic gets broken down into features. These are basically what we call the deliverables. They correspond roughly to user stories. In fact, we phrase them in the user story format. As an investigator on blah, blah, so that blah, blah, blah. This is kind of like the correspondence of the backlog, but it's not prioritized in detail. Instead, we put in the next 10 here. So this is like a subset of the backlog which is prioritized. Next 10 features. In real time, we populate this top 10 list. And they get sucked into development. So here's like a backlog, and here's what's in progress. It gets done. It gets combined into a system test. We take a bunch of different features, put them together to release system tested. And then once in a while, the users are coming in, right? They accept this test, and they do that. One of the things that I didn't write in my book, I have a book which is called The Trenches. One thing I didn't mention there was these envelopes. In case you're wondering, this is where we put all the cards inside a release. So we put the release number on the envelope. Just for practical reasons. It got less crowded on the board. So a bunch of cards got put in an envelope, and that release is what the users are testing. And that's what goes into production. This is our basic flow. And like I mentioned before, the first thing we did was just visualize what's going on. And of course, what typically comes up is a mess. Right? And a lot of confusion. So our board showed us that, and instead of saying that's a problem with the board, we realized that's a problem with our process. And the board is just revealing our process to us. So we timely cleaned up our process and cleaned up the board, and instead of stabilizing to the extent that, hey, we're actually finding we're now stabilized our process roughly, and it's working kind of. So now let's get brave and put tape on the board. Right? It's not evolving as quickly, we can put tape on it. We can still move tape, of course. It took about a month before we started putting tape on it. Okay. So I'm going to talk more about this, but first I want to mention cadences. We had some cadences. In Scrum, you put together everything in one rhythm. Right? So one sprint is really like four things. You're doing planning, you're doing prioritization, you're doing a release, or at least a demo, and you're doing retrospective, always on the same cycle. Right? Well, that's one way of doing things, but we didn't do it like that. And in fact, we stopped doing sprints when we kept the rest of Scrum. Instead, we had these being different. So retrospectives, well, some did it every week, some did it every other week, but typically on a project level, you might say that retrospectives were every second week. That's where our main points are in looking at how we work and improving it. Right? Planning was kind of on a bi-weekly schedule as well, and I'll talk more about that later. Demo-assistant test was not in a cadence. People demoed stuff as they got done, and the system tested kind of in real time. So we removed that part from the cycle, and I'll talk a lot more about that in a moment. So demo-assistant test on demand, and then the releases were every second month, which to me was, I thought that... I considered that to be a problem. It turned out later on that it wasn't as bad of a problem as I thought. What I normally see here is when you have releases every second month like this, you have a lot of crunching going on over here. But for various reasons, we didn't have that. So, okay, we could keep releasing every second month. The reason why we did this two-month release or a release like this was historical reasons, how the whole project was organized, how it was set up. There was some overhead in bringing in 10 to 15 police officers for a couple of days, taking them off the field to bring them in. So maybe that was the reason why we didn't do it every month. But anyway, this was our basic set of cadences. And look at this board. It turned out that there's a bit of education going on. People need to understand what are we doing here, what's the purpose of having a flow visible in this way. What I realized was this metaphor is very useful, the notion that, hey, we've all been stuck in traffic. We know what that feels like. So we all know that having an intuitive idea for that, it's a bad idea to stuff the road with cars. This is 100% research utilization. Research utilization, then you get terrible flow-through time. Takes forever to get stuff done. If you have a bit of slack, if you don't fully utilize this road, even though it was expensive to build, then you get fast flow-through. So we call it as a traffic system, and each car is like a car, and we don't want to fill up the system. We don't want traffic jams. That helped to communicate this. So in Kanban, there's an notion of limiting whip. How many of you use Kanban or have seen Kanban? It's spreading quite quickly. So there's the whip limits, right? And how many of you who say you use Kanban don't have whip limits? Nobody will admit it. So it's kind of core. Visualizing stuff without whip limits is great, but it's not Kanban. Just point of confusion. In this case, we just put on the board, we put limits. This is Swedish, right? Maximum of five stories per team, in progress. We had limits across the whole board. We added them gradually. We didn't start with limits. Our first step was just visualize. That's useful. Another time as we learned where the bottlenecks was the reasonable amount of work to do at the same time, we added whip limits across the whole board. And we also found other ways of using this board. The purpose of a board like this is first of all to create clarity. If everybody sees the same big picture, it's easier to collaborate, right? Get less sub-optimization. But the purpose is anyway to raise the issues, right? Make our problems visible, because then people can solve them quickly. So we use pink sticky for that, right? This feature is blocked because I need a barcode reader. It says Jim on March 12th. And that's great because every day, we have these project sync meetings. We look at the board and it focuses these. What can we do to get around this? Can we go buy a barcode reader at the retail store? Or can you mock the barcode reader? What can we do, right? So these become high priority. And sub-blockers are at a general level, such as our build system is too slow, right? Or the air is too bad in the office, right? Or whatever it may be. So impediments put them right up there. We also realized after a while that especially once we were in production and had these releases going out to pilot users, sometimes we had issues coming in, urgent dog fixes and stuff, right? So if some of these were more urgent than others, we would put a police car on it. They got that metaphor, right? Ah, you put up police cars. They don't get to crash into everybody. But they kind of get priority, right? They kind of sneak past. So that was the idea. Don't mess up the flow. But if I'm a developer and I have capacity today, I have spare capacity, what am I going to do? I'll just finish the feature. What am I going to do next? Oh, this one has a police car on it. I guess I'll focus on that one. So this was helpful. And kind of fun. So scaling then. How do I do a scale this? I mean, 60 people in one big board. I mean, that can get pretty ugly quite quickly, right? So we had a two-tier system. What this means is a feature flows into here. So far, there's nothing to do with which team is going to do what. We don't care, right? It's all about how do we take this vague requirements area using Craig's terms, such as weapons confiscation would be a whole area of requirements. What doesn't the concrete pieces and only in a just entire fashion do we decide which team is going to do what. In fact, they decided themselves at these daily meetings, right? So who's going to build this feature? We don't know, but we'll find out, right? Some team will pull it in when they find that they have the time. These teams were symmetrical. They were balanced. So it's not like they were specialized in any specific area. The idea was that any team can do anything. It wasn't the entire truth. I mean, sometimes it's like, well, we built something similar last week, right? We were in that code last week. So maybe we should build that feature. It's a lightweight specialization. But generally speaking, these teams were pretty much interchangeable in the sense that they could build whatever needs to get built. These had huge implications on our version control. So I might have time to get to that. In fact, I should check the time here. All right. So they get pulled into teams and each team finishes the feature to the point where they think it's done. Right? And then it gets put in system test and then it goes into production, right? But how do we manage this in practice? Well, it turns out that what worked for us was each team has their own board also. Right? So here we have three types of animals in the system. We have the epics, which turns into features, right? Or stories. And these features, once they end up here, they are physically copied. Not only by hand. We like to take a new card, copy what it was written there and put that on the team board over here. And then break it into tasks. So it's like we have three different types of animals. Epics, big features, business-valuable tasks, which are set up to database tables. And we have a lot of tasks here because that will just go too much stuff. So this turned out to be very, very useful because A, we get different levels of abstraction. The team members don't have to be here all day long. They're focusing on this. But we also had a connection from the task all the way back up to the feature. So that's the kind of fundamental how we scale the boards. Now, I mentioned the handoff problem. That problem didn't just go away. But we found ways of managing it in a good way. And that was having a very clear definition of ready for development. I'm sorry, Indika. I just missed the point. Do you say that you're going to use a different board for the tasks alone and then do a mapping? We have one single board for the whole project and then every development team has their own board as well. Features for tasks also. So did we stop at only developer level? Did we stop as a developer or as a model level? The developers were in the builder or in the building or the one in the team which I mentioned. The comments team, the team together coming to them and it's forming a one team. The dev team. The smaller teams. So this board only talks about the build activities or we talk about... when it's assigned to a developer like developing, designing and stopping at the board level. Or is it that model level or requirement owner will be there or there will be the test will be there? I can't answer in that detailed level. I don't quite remember. It was a couple of years ago. Yeah. Sorry, I don't remember at that detailed level. All right. So definition of ready for development was really important. What we did here, because what happened before this was, okay, we have a nice board. It's always shared. It's all nice and pretty. But all the requirements people are kind of focusing here. Right? And when they finish, then it sits here and they don't care anymore. All the developers are kind of here hand off the test and they don't care anymore. And the testers are waiting over here and they don't care what's going on over there. So what the board did was reveal that problem but then solve it. It turned out that what really helped was setting a clear definition of ready which included things like there has to be a test. We had this test written on the back of the card. Oh, now we need to involve testers. Right? And there needs to be an estimate. It's a small, medium, or large. That's all. Right? A small, medium, large. It's a rough estimate. And the rule was large stuff doesn't get built. We don't build stuff if it's large. We've got to break it down because it'll just eat up. It'll swallow a little choke on it. But small or medium. And the developers need to be involved in setting an estimate. Right? So by setting all these constraints, an acceptance test, now we have to collaborate because nothing can go from analyzing to ready until we've had these different roles involved. I see a definition of ready for system test. Before we had all these stupid bugs being discovered in system tests, mining layout issues, things like that, they could have discovered that earlier. They raised the bar. So it's not ready for system test unless you've demonstrated it to somebody, that feature. You've checked it in on a trunk. You've merged, right? You've done all the regression tests, automated, whatever test was written on the back of the card. All these things, it's raised the bar. It raised the quality of stuff. So by the time we got the system test, the feature was squeaky clean. The problems found in system tests were truly system problems instead of these feature problems. And of course, in order to fulfill this high bar, they needed to test around the team, active on the team. So these things really helped drive collaboration. And it took maybe a month or two, I'll say maybe six weeks, to get away from the culture of here's my part of the board, to a culture of this is our board, this is our project, right? To the bottom, like here, or here, that's everybody's problem. So as an example, we realized after a while that it's like, okay, so there's always activities that have to happen. When are we going to do this break time? When do we do the estimation? And when do we write this test? So there has to be a collaboration point. That's when we introduced the notion of, we were standing in front of the board and this question came up. So I was like, I have an idea. What's your idea? How about we just meet up once every second week? A few people from every team here and a few people from every discipline, basically the cross team. We go into a room and we work together. We split into smaller groups and we get things. And then we do this work of grooming and estimating and so on. And they're like, yeah, let's try that. What should we call the meeting? How about sprint planning? Like, yeah, so we call the sprint planning, although we didn't have sprints. It was more of a grooming meeting than sprint planning, but we had to have a name. So as a result, that caused collaboration. So here you see that specific meeting. You see planning, poker going on, there's prioritization happening, because we've got to decide out of everything in the whole world what should be inside this top 10 list. We have four slots available. What should we put in those slots? So that was a very useful thing. All right. So what about tech stories then? I mean, there's some stuff we build that aren't customer-driven, such as giving an example. Well, maybe we need to update a framework. We need to get a better build server. We need to whatever, right? There's techy things to do. If they're just small, then just do them. But sometimes they're bigger efforts and we need to kind of manage those the same way we manage our feature priorities. So we decided to make those visible, but distinguish them by putting a green sticker on them. Because this is stuff the customer doesn't care about. They're not tracking it. They're not interested. But it has to be done. So we put those here. So top 10 features, or the next 10 features, and the next five tech stories. And don't ask me why it's 10, there are five there. I don't remember why we did that. But it doesn't really matter. The point is we have limited queues, unlimited buffer here, customer-driven features, and a limited buffer of kind of investments, technical investments. That's a star of one another. They both have to happen, right? So teams would pull stuff from there, and we would pull stuff from there. And we tried to set up rules for how much effort we put on what, and we stopped after a while. We realized we don't need static rules for that. We talk every day in front of the board, right? So during those conversations, during the daily cocktail party, there would be discussions and trade-offs around where should we be putting our effort today on something there or something here, or our balance. So here's an example of when a tech story was discovered. This was a meeting I forgot which meeting it was. Somebody brought a printout of a class. He printed out the code. It was seven meters long. It didn't even fit on our biggest conference table. And he's like, we need to refactor this. Everyone was like, you can see, right? Yeah. So it was a very nice way of making a case. And that became a green card. Refactor this big mess of a class. All right. What do you see here? How would you, if you turn on your EQ sensor or your emotional sensor, what do you see here? There's a lot of pink stuff. That's true. Of all the impediments, we actually had a top three prior filter there. So a lot of this stuff is not top three. But what do you see in the people? Don't look at the board, look at the people. What? Concerned? So the reason why the word was because of this. Right? All the, every piling up the system test. Big pile. And this was happening every day for several weeks. Right? Why was that happening, do you think? Right? All these answers. Yeah, basically the developers are being very efficient producing lots of code. But not high quality code. So it's just piling up here and these guys are running behind, right? So what this, and we also measured velocity, I'll get more to that, but we counted, it's a bit silly, but we just counted how many cards are in done by Friday. Every Friday until we left, just all the six cards in here, and we wrote that down, right? The whole project velocity was just how many cards are done on the right side of the board, by the end of the week, and that's all. And, oh look, what do you see? Zero, zero, zero. We got 60 people working very hard, but nothing getting done, right? Just stuff moving around on the board and nothing getting done, nothing reaching customers. So this was a way of making it clear that this is everybody's problem. It's not a test issue. Everybody's problem. And the board helped us frustration become explicit, right? So these mechanisms I showed you helped us deal with this problem. First of all, somebody brought up the great idea of let's add a question to the daily meeting. Every team, every day, asks this question. How can we contribute to system tests today? Everybody asks this question. I write some more unit tests, go out and help them, do the testing, do better test automation, whatever, right? Everybody asks this question every day. And they came up with like, okay, we should stop building new features. So they did, they just stopped, right? This is empty. They asked them, what do you need to not get stuck there? And they said, we need better test automation. We're finding all these dumb bugs that we can just automate, right? Or we're doing these tests. We were running tests manual that could be automated. So the system test people became the customer in a sense. They wrote down the top list of automation stuff and that became green, you know, text stories here. So now the system test people are driving the whole development process or the priorities. So if you look at the board here, it's all green stuff. Some of the developers are doing test automation because they can see that there's no point building a new feature because it's only going to get stuck in this pile of stuff waiting, right? So a very, very useful way of driving a cultural change. Although it took a while. And they improved test coverage quite quickly and became more quality aware and we can gradually get back to building more features. All right. Let me just check on where am I on time. Excuse me. Right, so how do we have the bugs? Right? Ideally, we don't want to put bugs in the code but we're humans. At least I am. So here's how it used to work. I told you about the two-month cycle, right? Code, code, code, code, code, code, code, code, code. Test, test, test. Fix, fix, fix, release. That sounds familiar. So this was not good. Instead, we changed to this model which is don't wait for the release to be ready before you test it. Instead, just take whatever is ready and test it as if that was the whole release. So just take a few features, pretend that this is the whole release and run the full system test thing on it and fix whatever bugs come up. And then once you've done that, well, by that time, there's new features built so we just do it again. So system test is like continuous. But of course, when time counts through the actual release, because we're scared, we still run the full set of tests on everything. So this work of testing everything at the end is still there. How do you think the testers reacted to this idea of working like this instead of like this? What was their spontaneous response? No, we got this so many times. Ineffective, right? Here we test everything once at the end. Here we're redoing the test over and over again and then having to test everything at the end. Anyways, right? So that took a little bit of convincing but what helped a lot was this picture saying that, well, yes, it's less efficient for you, testers. You'll be doing more testing for release. But if you count the time for bug fixing, it's a lot more effective for the whole project. To fix things, to test things early, find bugs early, fix bugs early, gets a lot cheaper. And even though you have to run all the tests at the end anyway, you'll find quite few bugs at that point. So it'll be less risky, right? So after a while, they bought into this because everybody had this and we were standing in front of the board every day suffering because of the bottleneck problem. Everybody agreed that the current model is no good. So let's try this model. And it worked really well. In fact, everybody was like, I'm never going to go back to this ever again. But it was a bit of a hard sell initially. So in practical terms, what does that mean? Well, okay, somebody finds a bug. A developer finds a bug. But let's say you're running system tests and you find a bug. So what do you do? In the past, they put that in the bug database. There's this big process with CCDs, right? Change control boards and long lists of hundreds of bugs and there's allocation of bugs to developers. It just wasn't very effective. So instead, we just change to this process. Find a bug, okay. The person who finds a bug makes a decision. Is this a blocker or not? Does this bug mean this feature cannot be released? Make a personal judgment. And if I'm not sure, we'll talk to somebody, right? But make a judgment. Don't put it in the bug database. Instead, write it on a pink sticky and put it up as an impediment on the whole project, right? So write it on a pink, put it here. And that automatically makes it high priority. And not only that, walk around the office find a developer to fix it now. So we have this little mechanism for, you know, how am I going to find the right developer to fix this bug right now, right? So basically walk around, walk to the first team and say, is this the kind of stuff you guys are working on? Yes, okay. So who do you think I should talk to? Bob, right? Bob, is this a bug that you know anything about? Oh, I know this. I was working on this, you know, a few days ago together with Jenny, right? So this is a blocker. Okay, let's fix it now. That was the basic rule. Blockers always go before anything else. So we don't schedule it. We just stop whatever other work we're doing. We fix it now. And check it in and retest. Yes? Do you deploy it immediately? No, fix it and declare it right now. Right. Yes. So I'll talk more about version control in a moment. But this was very useful because now we did not put a bunch of stuff in the bug database and we shortened the cycle time, right? Blockers are fixed now. So they basically become invisible. Fix it now, get rid of it, pretend it never existed, right? Okay, suppose it's not a blocker. It's a bug. We should fix it, but it's not blocking the feature from being delivered, right? So what do we do with those? Well, we look at the top three lists. And don't... The number 30 come from, I just said any limit, how about 30, they're like Ls to 30 and then it's stuck, right? But sublimit. And the rule is, okay, this bug is not a blocker. Is it more important than any of these top 30? Right? If no, then close it right now. And that was a hard sell, right? I found a bug and we're not going to fix it. Let's just be honest. What's the point putting in a list of 700 bugs? Are we going to be fixed? So just say, we're probably never going to fix this, right? Strictly speaking, we actually did put it in the bug database, but we gave it the tag deferred, which basically meant garbage can, right? Everybody kind of knew that. And if it is more important than any of the top 30, then replace that one, right? So we'll put this one in, we'll take that one out and close it or defer it, right? This helped a lot, because now we didn't have to manage long lists of bugs. And we also shortened the time from finding a bug to fixing it. So I'll show you what we did with these 30 in a moment. In fact, I'll show it right now. This is what happened. So the blockers came up here, fixed it right now, done. Gone, right? The top 30 bugs, we had a section of the board called the next five low priority bugs. So this may be getting complicated, but out of the top 30 inside the bug tracker, we selected the top five and put on the board. Does that make sense? And they came here. So now we have three input queues to the teams. There is the customer-driven features, there is the tech stories, and there is pesky little bugs, right? And we've got to balance our time between these three things on a daily basis. Don't they take priority? Do these take priority? That's on a case-to-case basis, right? Because if it was a blocker, yes, we wouldn't even put it here. We put it here and fixed it now. But these are the things that we kind of want to fix, but it's not a blocker. So we've got to make a trade-off. Today, should we focus on this feature, or that tech item, or that bug? We can make a decision. So if you had to put it there next to the features, so you have a team which gets together to groom the backlog, so will you involve the team's decision on that, whether it's a blocker or not? Or you let the person who raised the bug to go ahead and... The question is who classifies? Basically, the person who finds it, the bug, whoever it is, does a classification themselves, and if they feel uncertain, they talk to somebody, and if it's really an important edge case, then we actually did have a weekly meeting. It was kind of the last three days of the CCB. It was a short meeting every week, where the cross team got together to look at the edge cases. Like, how important is this really? And that would cause prioritization. Are you referring only to the continuous testing... What? Are you referring only to the continuous testing bugs here, or is it also production defects? Production defects, and it's actually both. Regardless of where a bug is found, whether it's in production or in system test, treated in the same way. Usually there are political impacts and production defects as well. We would push for it even if it's not in the system. Political impacts is one of the variables that influence priorities. Yeah. All right, so another important thing to realize after a while. Finally, after the deferred bugs. What? The deferred bugs... The deferred bugs were basically forgotten about. No. Excuse me? The deferred bugs sometimes get fixed, not because somebody picks them out of the deferred bug list, but because they come in again. Like... It's like, what the hell? So maybe this is important because people keep reporting it. So it ends up in the top 30. And if it doesn't come up again, it probably wasn't important. Yeah, we've got to get away from this notion that it's valuable time spent to manage long lists of stuff that we're never going to fix. It's really an illusion. Find every single queue anywhere in your system. Limit it. It's really hard to do politically, but the value is immense, and once you decide to do it, it's very simple. You just make up a number, and then you decide not to let the queue get longer. This triggers just-in-time prioritization. This is really a lean trick. So we have sometimes recurring bugs. Testers are like, why is it that we always get these bugs like the layout is wrong? The text is longer than what can fit, so it gets cut off, right? Little things like that. It keeps coming back. So I asked them, can you guys, the testers, identify the top three, always limit these things, top three recurring bugs, and they put this up on a board. Top three recurring bugs. So these are examples. This type of bug, here's an example of a bug that gives us deja vu, right? So what happened with that was we had teams meet up sometimes and take one of these and do root cause analysis. So the question was, why do we put these bugs into the code? What causes it to put the bugs into the code in the first place, right? Because it's not enough to fix the code, we want to fix the process as well. This turned out to be a very useful way to drive on process improvement. We put the bug in the code because we were stressed. Why were we stressed? Because we had an unrealistic release plan. And it turned out we found these very concrete ways of improving quality indirectly by improving on the way we work. All right, blah, blah, blah, blah. Now you need a break. Two minutes, turn up, talk to each other, just shake loose, right? Shake it a little bit. You'll fall asleep if you just sit for an hour. Yeah, it doesn't work. Bingo, talk to each other. I don't know. Yeah. Thank you. Version control. There's a little bit of how I did version control. I wrote this article called Version Control for Multiple Agile Teams. If this topic interests you, you might look at the article. I did not invent any model, I just tried to describe it. It's called the mainline model for the stable trunk model, right? But I wrote an article about it because I wanted to clarify it. And I found that it works quite well in many contexts. And it turned out that it worked quite well in this context too. With Version Control Systems, like Git, you can use the mainline model. It's not a higher-20-specific paradigm or a specific tool. High-level, high-level, high-level summary. But what we needed was a stable trunk. And this means that trunk is always ready for system test. That allows our definition. For a long time we debated should the trunk be ready for production or ready for system test. We decided for various reasons to say that. Trunk means ready for system test. And what that means is features from the perspective of the trunk some of that pilot code pops in and now the system has a feature it didn't have yesterday, right? That's what happened. And like that. So we don't see half-baked features coming in there, right? No junk in the trunk because it's a natural. So what does that mean? Well, here is Jeff, a developer, and I'm going to check in codes. I don't want to leave my stuff unchecked in. But I'm not finished with the feature. It hasn't been unit tested. It hasn't been demonstrated. I had this column ready for system test. All this stuff. I'm not even nearly there yet. So I'm not allowed to check in on a trunk. So that means I have a perfectly good excuse to use a branch. I want to check in my code. I'm not allowed to put it on the trunk. So team branches is what we had. I check in stuff kind of all the time. The error is more lenient. Unit test must pass. But that's all. So a lot of checking in going on here. Every day, or typically multiple times per day, my team members are pulling from the team branch. So we're in sync all the time, right? Any merge conflicts, we handle that quite quickly. But we want to get stuff to here. So what happens is the team is focusing on getting a feature to stable state. So we can put it here. Ideally, several times per week, they'll push stuff up to the trunk. So fill up when the feature is done. But that means you need to do some fairly significant verification first. That it really is stable, right? Before we put it here. Now what happens then? Well, the daily routine for every team was we start the day by pulling from the trunk. So that means that whatever other teams put on the trunk yesterday gets through every other team by the next day at the latest. And there's an interesting dynamic here. Whoever checks in first wins a case of conflict, right? Because, you know, if team 2 pulls in some code here and that conflicts with the refactoring I'm doing, well, that's your problem, right? So sort that out. And if you need help, go and find the person who wrote that other code, right? Talk to each other. But basically you own the problem. Which means that if you can get your stuff checked in first and you kind of win, that creates a good incentive to get your stuff on the trunk as quickly as possible. So this was a way of getting a little bit of isolation and having a fairly up-to-date system. So branches are fine, but if you think of the lengths of time that a branch is diverging from the trunk, that's technical debt, right? So having a branch is okay, but if you keep that time short, then you're fine. But if the time is long, then you're basically fooling yourself. You have a big technical debt in the sense of a big, ugly merge coming up in the future. You were talking about the incentive for checking in early, right? It also leads to the anti-pattern of quality and this, right? Yeah, there's a downside to that incentive. I might check in stuff that's not good enough, right? So that's why we need to be very clear about no junk in the trunk. Make that policy clear. And if people start checking in junk, then we've got to police that, right? So these daily needs are very useful to find the right balance between checking in too often and checking in to seldom. Continuous tweaking for the balance. And flow every couple of days became possible over time. Initially it wasn't. Initially it was like everybody checks in at the end of the two month period and we have chaos, right? But gradually we migrated to this more continuous flow. We never got to continuous delivery. But we got to a point where we had pretty much continuous integration. Until you reach that point, you still have this trade-off because teams are feature teams with span across components and there's going to be a lot of affinity between two features with respect to one component which gets... Yeah, as soon as there are multiple feature teams and we work on the same code base, you have merge conflicts, right? And that's a fact of life. And the way to alleviate that is by checking in more often and having good tool support and having a good branching strategy or a good version control strategy. I can strongly recommend reading Jezz Humble and another guy about his name. The book continues to move. It's a pretty dense book but it's very powerful. There's a question. Let's say team one is working on something which is not yet going in production. Okay, and now you continuously merge it back to trunk and trunk is someday going to go in production but you're still not ready with all items or is it backward compatibility that you do? Good question. In fact, the rule was not only no junk in a trunk. The rule was only put stuff in a trunk if there's something you want to release as soon as possible. If there's something that for various reasons is not supposed to be released soon let's put it somewhere else. Because if you put stuff that you want to release as soon as possible and mix it with stuff you want to release later you've poisoned the trunk, right? Pretty much killed the process. So, one related question. So, as you said, right, you moved from version 1.1, 1.2, 1.3 then once you released production it's related to what he asked. Now, not only I have a trunk I also have a released version. Yep. So, and you have a issue in production now to fix you to margin both the That's actually this picture. So, thank you for helping me get to the next picture. You have a lot of great questions. I know that I'm going to get this done in time so I might have to de-scope to talk a bit. But anyway, so here's a trunk, feature A, feature B, right? And the system test happens. So, what system test group does is they just take whatever was here and they branch out and they set up a stable environment to test the system. Some people adding new features while they're testing this because this is now a release candidate in a sense, right? But the trunk is actually always a release candidate. It's always stable. But we want to pull it out and run the system test on it, right? So, we'll have, and the policy there is only bug fixes. We don't want to add any functionality there. What happens then is while they're doing this the features, more features are being added but they're not visible here because we need them to have some chance of getting done. So, if a bug is found here then fix it now and fix it here. And then immediately merge it down to there that way it propagates the whole organization. So, stuff flows down but it never flows up, right? And then when they go to get to a point where they feel that this combination of features which is a release candidate would be good enough to put into production. We might not put it into production because it's too feature incomplete but it's technically ready for production. We don't need to run any more tests on it. Then we're done, right? This is done. So, what happens then? Do it again. So, it's continuous system test. And so, what does this actually do? Well, we'll zoom out a bit, right? Branch out, system test, done, right? During that time, all these new features have been built. Branch out a new one which is all the old features plus this new one is now a new release candidate where all the system tests get done. Why is this line getting shorter and shorter? What? It's not because it's close to the end. It's basically because in the beginning, system test was testing everything that's been done over two months' time. A lot of stuff, right? But every time we do it, there's less and less stuff that has been done since last time. So, by the time we get, we got to the point where before several weeks after one or just a couple of days, two or three days, and by that time, there's not so much new that happens, right? So, the next test cycle, we don't need to run all the tests every time. We can say, well, we have a stable release, right? And now there's just two more features added. So, which test do we need to run to feel confident about this release? We don't need to run all the tests, right? Of course, at the end of the release cycle, we're about to actually print into production. We do all the tests also. But in all the intermediary system test parts, we don't actually need to run all the tests all the time. We can make a risk trade-off, right? So, this was very, very helpful. I got to the point where it kind of felt like continuous system tests. But we could not do this without changing the version control system. In fact, that was what caused us to change. We didn't have the idea of short-lived team branches. We didn't have the idea of this before. We just had a completely messed up trunk and a whole bunch of branches that nobody really knew what they were for and when they were going to be merged. So, just being explicit. Every branch has a purpose. It has some kind of life cycle, right? And it has a clear policy. And we only create new branches when there's no place I can check in this code. Right? So, in this case, can we say that, you know, as the customers had an option of, you know, giving or changing the requirements only till the time the system testing was, you know, did not start? Once it started, you did not welcome them from changing their requirements. The promise can change anytime because it really doesn't impact this at all because whatever the teams are pulling from the board, right? And we have this pile of cards on the left side. And as long as we haven't started development, it's quite easy to continuously discuss what's the next in the top 10 list, right? So, yeah, I don't know if that answers the question. Every release you had a system test cycle, right? That's what it does evident from... We did system test all the time but we also had kind of an extra at the end, like an extra just-in-case system test. So, in addition to the continuous testing, you had one final... Yeah, yeah. We called it acceptance test. But it pretty much was. So, only during that final part you will not incorporate any new requirements, right? No. In fact, same here. These guys are testing a certain set of features. And they don't care what other features are being built, what new features are coming in the next release. They don't care. They say, this is the system well right now. Let's get it to work, right? And then they just redo that all the time with whatever has come in since last time. All right, I'm going to move on. Because version control is an interesting topic but it's also an in-depth topic and it could take up all the time. So, I wrote more about it in the book. You can also find the article. Now, a really key thing was retrospectives. Really key. Each team had their own retrospectives. Then we had a cross-team retrospective at a higher level. That's when the cross-team got together. Remember the cross-team I mentioned? One from each feature team or one from each discipline got together as a project manager. We met every second week and talked about high-level improvements. So, what are the recurring problems we've seen? Things like branching, shit. We can't do continuous system tests because our version control system doesn't support it right now. So, we need to change the way we do version control. So, this meeting was where we made all the important decisions, such as changing the structure of all our teams or changing the structure of the board, changing our policies, etc. The setup of the meeting was quite typical agile retrospective. Once working well, what are our problems? Prioritize the problems, find the top two or three deciding actions and experiments. To address them. Of course, after all, we realize that we're moving too fast. We're changing things too fast. It's a 60% project. We had always great ideas and we introduced change, but then it maybe took a couple of weeks for that change to really propagate, to actually be implemented to that change. Especially if it's about changing the behavior of people or the structure of the organization. So, we realized that we had to slow down and really change. So, the way we did that was we introduced a little bit of bureaucracy. A little bit. We introduced a bastardized version of the Lean A3 process. I called it the A4 process. We basically said, we're not going to do any change that impacts multiple teams, unless you make a case for it. You've got to take an A4 paper or document and write down what problem are you trying to solve with this change? If you can't phrase that, we're not going to even discuss this change. Second is, who's going to be affected by it? And then, what are the actual implementation steps that are making this change? So, it's a little bit of analysis. A little bit of upfront analysis. This is a real-life example of one of those A4s. That's when we decided to create a tracking mechanism to see how these features tie back to the high-level epic. Simple tracking mechanism. And a little bit of detail. In our process improvement meeting every second week, the retrospective was, we had these A4s coming up. Anybody could write one, right? And if we had more than two or three, we would prioritize. So, yeah, all these five ideas are great, but we're only going to implement one. Which one is it? We choose one. Because that way, we also get a chance to see did it work before we started doing additional changes. So, slowing down, I guess, managing the rate of change turned out to be very important to us. Most of our clients have the opposite problem. We're changing too slowly. But this was an interesting case that we're changing too fast. We've got to manage that. Another thing we supported our process improvement was some simple metrics. And this was actually a bit silly. I wanted to add metrics, so I'm worried about complicating things. So, I added very simple metrics, and I assumed that they would be too simple and they wouldn't be very useful. But I figured it would be a starting point. But to my big surprise, it turned out they were very useful and they didn't need to be more advanced. So, the simplistic version was perfectly enough, even for this pretty big project. All we did was this velocity and throughput time. This is sometimes called cycle time. So, velocity is features per week. How many features does this project deliver, production-ready features per week, on average? And this is, how long time does each such feature take? So, features per week, weeks per feature. There's a difference. It's kind of like bandwidth and pain. Different. So, how do we measure it? Well, I already told you about velocity. You just count. How many cards are there? Since acceptance test happens every second month, there was a natural kind of buffering happening here. So, we didn't include that in the cycle time. What we did was just look at how much stuff have we gotten to ready for acceptance test. Because we learned after a while that once it got to there, most of the problems are ironed out. We didn't have a lot of surprises coming up at acceptance test. You had different sizes, right? Yeah. If you measure the number of cards, will you give you a red picture? Yeah, that was one of my feelings was, if we just count the features, some are big, some are small, maybe we should take that into account. Well, hold that thought. I'll tell you something funny. So, we just counted about that in the memory. This turned out to be very useful because, first of all, if the number is zero for a long time, that creates a sense of urgency, which causes collaboration to happen, right? And then we noticed patterns. Like, okay, it varies a lot in the short term, which this project spits out about five features per week. Great. Now we can plan. Some are small, some are big, but the average is out, right? So, now we can create a release burn-up chart. How many features have we gotten done? A cumulative, all the time. So, here's some magical date. This is when the press release is going to come out or something, right? How much stuff will get done by then? We can just draw, you know, the optimistic and the pessimistic trend line based on our experience that the velocity is on average five. So, very, very lightweight planning mechanism, which turns out to be superior to all the more advanced planning mechanisms we were using before, which included things like calculating man-hours and doing estimates and all this stuff, right? Well, never mind. We don't care about hours. All I care about is here's a bunch of people. And standing outside, I see that there are five features dropping out on average every week. That's all I need to know. I can plan based on that. Of course, you can't get there unless you have, you know, a kind of continuous or at least regular pace of delivery. If you have big bag globally at the end of a cycle, this doesn't really work. So, it started out like that because of the system test bottleneck I told you about. We were stuck, but once we had shaken a ketchup bottle, right, and got stable flow, we started seeing that, hey, look, you know, we can actually, we can use this for lightweight forecasting. Very useful. Of course, it also caused some frustration sometimes, and it turned out that the customers were hoping we would be there. And now it's like, we won't. This is data. We won't be there. Sorry, it doesn't matter what the contract says or what the release plan says, it's wrong, right? This is data. So now all we can talk about is what is the most valuable stuff we can put in by that date? So it caused friction sometimes, but a good type of friction in the sense that instead of having a big surprise at the end, we had little surprises happening all the time. Kind of thing. So this allowed us to talk to stakeholders, talk to customers and say, well, by this day, all of these features will be done. Some of those will be done by not all, and none of those will be done, right? So prioritization within this group is extra important. To be honest, we're making variability visibly, right? And, yeah, this is useful in other contexts. This is my son. I have four small kids, and it turned out that, yeah, sometimes he likes to use this to get focused on homework. Okay, wait a second. What is this? This is the number of pages of homework I have to do. These are five-minute intervals. Five-minute intervals. And what this helped them see was that when I'm not being focused, this is like, which page am I on? At which time, right? He thought it was quite fun. Which page am I on? If I'm not being focused, time keeps moving anyway. Right? This is called bedtime. That's our fixed time, right? And there's a line here called I'm done with homework, which is kind of like our scope, right? And if I can get this line across that line before bedtime, I get playtime, right? It's a very simple note. I've never seen a better way to get kids focused. It's simple visualization. Whoa! Gotta get done, right? And I think that's why this technique is so simple it even works on project managers. Right? You just can't misunderstand this basic idea of a release production, right? Okay, so that makes the data visible. Otherwise, it's useless. It can't be numbers in a spreadsheet. Make a visual, put it up on the wall, bring it to meetings, talk about it, right? Quick question. You had 60 people and you had a milestone to meet. So did you increase the team size? Maybe yes and no. But how did you make the decision that I have to either reduce or increase the team size? Because you find a velocity, but you don't know what are the features left out, how long they will come. Or did you make calculations? Well, a bunch of questions, I guess. Sorry, I don't... So you have a milestone, right? Yeah, we had this buy yearly release because it was buy monthly and there were some expectations about what's going to get done by then. The minister just said on this date how we're going to go live, right? Yes. So now you have to backtrack and check that if this is my velocity, am I going to hit or not? So did you guys... Okay, here's the important thing. To get certain that we release on time, we don't do that by planning a velocity. We do that by, instead of waiting and then releasing on the final date, we release every second month. So we know we're going to make the date. We're already in production, right? The question is just which features will be in there, right? And this is just a simple way to trigger discussions around expectations management and which features are going to be in there. But we did not do this at people at the end of the project because it just slows things down. Okay? So you had one acceptance test just before the release, right? Yeah. So was it a well-taken decision because your history was smooth? Because what if during the one acceptance test at the end, you had a lot of surprises which, you know, halted your release? So what happens if they blow up an acceptance test? That actually happened in the beginning, the first few cycles. So that's what we need to do is continue a system test, have users in the building looking at the product as we build it. Which means that once we get the acceptance test, there's no real big surprises. We kind of know that this is pretty good. The stuff that acceptance test people find are kind of minor things. We can fix them right now or defer them to later. We chose to not include acceptance test because it would have made it look like a staircase. That's why. Alright, so here's another thing. Using this, we can see improvement. We can say, look, our velocity used to be on average three features per week and now it's six and a half features per week. And we can celebrate that. This was a very nice way of seeing that, okay, velocity is not everything, but it's something. And it shows that we now have a higher capacity than before without having added a lot of people. Melometric was through the time. So all we did was when a card gets to here, which is kind of the decision point that now we're actually going to build this feature. All the way until here, it's just potential, hypothetical stuff we might build. But now we're starting to commit real resources to it. So we put a date on it. We just literally write a date on the card, like that, using pencil. And then when it gets to here, which is our kind of, you know, as done as we can get right now, after that it's just waiting for a while. That's what we measured, and we just put a date there, too. And now we could just count how many days was this card on the wall. Very, very simple. And we wrote this interest spreadsheet. So each feature now had the throughput type logged. And we could graph that and see trends. On average, things that take 50 days or 40 days to cross the board. What do you think happens when people look at a picture like that? You get this, oh, shit, moment. What? What? It takes 40 days? That's silly. This feature is only a week of work. Yeah, but it still took 40 days to cross the board. Because of handoffs and queues and all these things, right? So this is a great way to trigger a sense of what the heck we have to improve. So through techniques like WIP limits, cross-functional teams, collaboration, shortening the cycle time, doing continuous system tests, all these techniques together cause cycle time to get reduced quite quickly. This was cut in half. Before it was on average six to 14 weeks, how long each feature took. And over here, by the end of the project, it was quick, three to six features per week. Which brings us to the question of size. So yes, we estimated the features. Large, medium, small. Very rough, right? And we assumed that there would be some correlation between cycle time and size. Well, it turned out that no, there wasn't. First of all, we avoided implementing large features. If they were large, we tended to break it down instead. I don't build it. Some exceptions. We have three exceptions here. But generally speaking, we said large doesn't get built. Well, if you look at the colors here, these are features that are built. And the color is, what was the estimate? No correlation, pretty much. Look at that. That's a large. So why is that? Why was there no correlation, do you think? Maybe the people who work were different. That's one of the reasons. Yeah, a small feature can take a long time if you don't focus on it. A large feature can be very fast if you focus on it. So it depends on lots of things like handoffs, process effectiveness, what's our priorities. Those things influence how long things take. Much more than I have actual amount of hours of work to build it in this case. I have a question here. Also that in the initial phase, there are a lot of unknowns. And as you progress, you have pretty much matured set of features. Yeah, we get better at estimating over time. Yes? Yeah. So things get stuck here for a while, right? We start building it, and it gets stuck here for a while. And so this feature took maybe two months, but it actually was just five days of work. And the reason why was because we said it was prioritized and then it would never change the mind. And if you have a situation where you keep changing your mind about stuff you already started working on, that makes everything slower. And cycle time helps to reveal that. In fact, that's one of those cheap ways. It's a way that has a big impact, but it's very easy to fix by just getting more focused. Sir, do you take in, if it gets deferred or put into onboard, to also take that time into consideration? Yeah, in this case, there were some exceptions. They got up here and then we took it off again. Okay, you raised the date. It wasn't a perfect exact mechanism. But the basic idea was, when something gets important enough to start working on, and only when it got to here it actually did get worked on, then we measured the cycle time. And by doing that here, that gives us an incentive to keep the queue short. Right? Pretty much. So can I just check if I get stuck answering questions here? Because I'm going to need to de-scope a little bit. So, you know what I'm going to do? I'm going to be a jerk. I'm going to get through the material. And then those we have questions, stay after, and we can talk about those questions. Because I think some of this stuff may be answering some of your questions. But okay? Because we just have five minutes. So just in summary, what came out of this was, yes, we can measure that we got faster, a lot of good impact for various reasons. And we learned that estimates are useful for conversation. That the numbers themselves don't really matter. Because we don't know how long things are going to take. So, but we also added subjective data, which turned out to be critically important, to not just measure velocity and cycle time, but to add a level of subjective field data. And all we did was we looked at the goal, I'm not going to translate this, it's in Swedish, but we had a pretty clear goal of what do we want to achieve by a specific date. It's not an exact scope, but a high level goal. And we phrased that goal and also a partial goal. And the question was, are we going to make it? Are we going to have a system that reaches this level of usefulness by this date? And it turned out that the best way to get that data was just ask people, what do you think? So we asked people, you know, do you believe this current goal is achievable? Certainly, probably, barely, probably not forget it. When I was thinking about, oops, okay, it doesn't matter what other data we have, this data shows us we are going to fail as it is right now. Therefore, we have to do something, right? Change where we work, change the scope, fix an impediment, something. So this was so useful, so we made it a habit. It started like this, but then we changed it to standing in front of the board. We asked this question every week or so. Whoever happens to be at the board, the cross team usually, we just asked them, so here's the goal, right? And the release is in a month. Are we going to make it? What do you think? Five fingers, four fingers, one finger, and now it's this column here, right? And you can see that, oh, looking bleak, we have problems. Next week, oh, looking more positive, we fixed some of the problems. This was one week before the release, we felt completely confident. So just a nice way to add a bit of subjective data to complement the objective data. Right, so wrap up. There's some things that really helped this project succeed. These are things that were put in place way before I came in. I noticed that they were there, and I think the project would have, it would not have succeeded if they didn't do this. Or it would have been harder to succeed, right? One is collocation. And I know distributed teams, how do you do that? Well, you have to find ways to compensate the distance, right? Fly, meet each other, use tools such as Skype, et cetera. There's all kinds of tips and tricks you can do to get closer to the feeling of collocation, even if you're not collocating. But in this case, we were actually in for the same building, which was critically important. It really, really helped us to improve quickly. Included delivery, like I mentioned, right? Don't build a whole elephant. Once find ways to deliver small elephant pieces that will add up to an elephant, right? And user involvement, all the way through the project. In different ways. Real users talking to real developers. We even had situations where a developer would go out and sit with the police inside their car for a whole day and just be with them as they drove around using the system. And they came back with like, holy shit, guys, we have to fix this, you know? I saw all these stupid things that he never really thought the police would do with the system. Right? Because they're in the field, right? I mean, they're standing in a store interrogating a shoplifter and using a computer at the same time, right? It has to be very simple. It can't be like, oh, wait a second. How the hell do I... You can't have that, right? Okay. And the second was creating a culture of continuous improvement. And the way that we did this, which I think was helpful, was clarity. Having a single picture, it's a model. A model is never the entire truth. But it gives a model, it gives some kind of sense of what are we doing, where's the bottleneck, who's doing what, what's our focus. Very important, a single shared view. Complimented by a number of local views, right? And communication. A board like this is useless if it just sits there and not used. So meeting in front of the board every day, making it part of our, you know, and then basic data. Just a little bit of data. This is not the most important thing, but it really helps give a little bit of support to what we're doing. Seeing the consequences of our improvement efforts. So, yeah, I think that's it. Thanks for coming. And I'll be around for questions. Come up first in that case, right? Thanks a lot.