 Welcome to Agile Roots 2010, sponsored by version 1 Rally Software, Virio, Amirisys, Agile Alliance, and Xmission Internet. A Tale of Two Cities, Early Adoption Stories by Scott Duncan. Hi Scott, how can I do, basically, these days, nothing but Agile approaching the training. For the past three years, I've been doing it for organizations that are big. They are places that are scattered all around the world, all around the country. They have locations all over the place. And most of the side, they want to do Agile. And most of them get brought in after they've already decided to do that, and they've always decided to start it in some way, and they either realize that things are not working the way the book said or the class said, or they run into something that the book in the class didn't teach them about, or they just are having a terrible time wanting somebody to help come down now. Now it turns out, these experiences I'm going to talk about when I was working with a very large consulting firm. So I was kind of indebted in these organizations for months, like it or not. Now, this of course comes from the Dickens story, and it's actually a play from the original Dickens Tale of Two Cities. Does anybody know what the first words in the book are? It's the best of times, it's the worst of times. Yes, so you may see what that means in the context of this, if we get into it. So, there's Company F and Company N. One was a financial services company, one was a network hardware kind of development company. The business territory for Company F and City F was basically North America, Puerto Rico, the United States, Canada. They had 2,500 to 3,000 offices where customers could come and get services from them. They were not credit card, they were not banking, it wasn't that sort of thing. For City N, they were all over the world basically. I mean there were some parts of the world where they really didn't have the business going, basically they were all over the world. Why did they decide to do Agile and by the way they were doing Scrum? Well, City F had some projects that were already running using Agile and Scrum, and someone had convinced this particular project that this would be a good thing to do. Turns out about 6 months later, after they engaged me to be involved with them, they had an Agile coach who basically had read some books, and they finally decided to get someone who had done it for a while. And so they brought me in, actually they brought me in for a week while this guy was on vacation, and I left and they kind of said the next week where'd he go? We went back. But they had had some projects and they went to see one of these projects finally. They heard the reports of some senior managers and said, oh great, they went to see one of the projects and they said, gee this is nothing like we even read in the books about what the heck is this Agile about? But they were going to do Agile. And the other one, City N development manager really thought it was a good thing to do and other development managers in the company were doing it, so we basically said we're going to convince the development people to do Agile. And so those are the two reasons they got into it. And I could never find any deeper reason behind it. Well, eventually they learned, oh we want to reduce cycle time, we want to do this, we want to do that, but the real reason they got into it were those reasons. So what were the projects? What were they working on? Well, City Act was basically rehosing and integrating with vendor middleware a system that basically ran their 2,500 to 3,000 locations. This is the software that ran their business day after day after day after day. They were basically doing a bet to company rewrite of this system. Because it was on mainframes with kind of custom developed middleware from a vendor who like it disappeared years ago. A lot of the screens looked suspiciously like they were basically copied from 3270, but now they had colors and buttons on them basically. But it was running their business. City and basically they were integrated with a vendor product and it was used by maybe 300 sales people around the world. So City and basically sold to resellers. They didn't deal directly with consumers. City F dealt with consumers directly. Both systems though, regardless of what the companies were, their business was, both systems were internal symbols and account support systems. Except that because of the fact that City F actually dealt with end users and customers there was an aspect of the system they were enhancing to allow customers to remotely be able to check up on their services. I will also say that City F system was such that if you went into any one of those 2,500 to 3,000 locations and started doing business with them, if you went to any one of the other locations those locations would not have access to all of your information. So you'd have to carry it with you in a sense on paper. And then when you went in there someplace else in the country to try to do business with them you had to know information that that office wouldn't know. Everything was on local servers in offices and they would upload kind of a summary file. But basically you couldn't have access. So that was another impetus for doing this was they wanted to integrate all of this so that if you went into one of the offices you could actually be treated as if they knew who you were no matter where you went in North America. And again the other one was basically being used by a few sales people comparatively speaking. Each one of those offices by the way had 5 or 6 people in it. So basically the software was touching anywhere from 15 to 18,000 people. The sales process. This company had one sales process. All those offices used the sales process. The other company since it was spread all around the world had 5 different locations. 5 different world regions. Each one of those regions had their own way of doing things and within the regions people did things in different ways themselves. So they also had the challenge of trying to make a system that would either work that way or would move the sales process to something that was a little bit more similar around the world. Which is really what the company wanted. But the sales business was so very independently run that that was a struggle way above the level of the project I was working with. So who was involved in this? Well when they decided to go through some training basically neither side trained any contractors or other employees except for the few who were going to be directly involved as either product owners or scrum masters or basically those kind of folks. So nobody else got any training outside. So they went to basically a scrum master training class. That's how they got their training. And they picked business and IT managers and some business staff to go there and presumably become product owners and scrum masters. CityN basically sent business and ITVAs and some development leads to be scrum masters. And a few others. A few managers from CityN went. All the managers actually on the anticipated project and that was about 15 went to the scrum master training class so they were doing exactly what everybody else was experiencing. They had to understand what their approach was being taught. Those scrum masters in CityF are basically business leaders initially. That's where they came from. On the CityN side there were a combination of business and ITVAs and a couple tech leads. Product owners for the one company were also business VA's. There were no product owners in the other team. They basically committed to this person called a release manager. We'll get more into that later on. These were what would be called fully distributed teams. In other words, they didn't just have a team here, a team here, a team here. The members of the single teams were distributed. CityF, the first iteration, they were doing four week iterations, CityF. They had one team of about 20 people. And they were distributed basically in one city on the east coast of the United States and a couple locations in India. The next iteration, they had three teams with about 15 people on each one of the teams. And that lasted for about five iterations. And then finally they ended up with five teams of about the same number of people and that's what they kept going up until about the time I left, 10 months later. At that point, they dropped back to four development teams and that's what they called solutions team because of some issues they were running until I can explain later. CityN, I had three week iterations. They started with five teams of about eight iterations. And then about halfway through the time I was with them, they went from the other three teams with about the same number of people. And that's what they went forward with after I left. They had folks in besides India and besides the United States, they had them in Denmark and in Singapore. We'll talk a little bit more about that later on also. Those are physical environment. CityF basically had committed up front to build basically a co-located environment. They had dedicated team rooms that were within 30 feet of one another. They had a corporate-wide video and they made sure that their offshore vendor had it as well. And so they did their daily stand-ups with video. And so they were, they were, onshore offshore basically did their stand-ups together. Now that required some interesting overlap and working from the onshore perspective. But basically that's how they did it. Now like I said, they had five teams, ultimately all within not many feet of one another. CityN on the other hand had Wi-Fi around their whole building and they had conference rooms with audio. And that was kind of their environment for the teams who wanted to work that way. They had one team that was kind of dedicated but had to share a conference room. In other words, by sharing, I mean they had to share it with anybody else who wanted a conference room space. So every single day they met in a different room. And one of the jobs of the Scrum Master was to find out what conference room was available big enough because they could also only reserve rooms about a week and a half in advance because the rooms were so heavily used by everything else. So you could imagine these people. Now they could get away with it because one of the main restrictions was we don't have to plug in anywhere so we can go anywhere and do this. The others worked and called in from their desks or their home or wherever they were in the world. One of the big battles I had with CityN was getting people who were co-located to actually be willing to walk 30 or 40 or 50 feet and meet in a room together to have even the 15 minute stand-up. Everybody was used to dialing in to everything from everywhere. And again, people were used to working from home and everybody felt like it because nobody really expected them to be in the office. I was on the floor with that company and I don't think I ever saw more than 10 people or 15 people on that floor and that floor was as big as this whole building, one floor of this building with cubes in it basically. And I hardly ever saw more than 15 people at a time physically in the building and there were more than 15 people working on just my project or the project they were working on. So they had four-week sprints, they had 7 to 12 for those sprints per release and they were going to be doing some releases over the years and why were they releasing codes every single month? Well, you can imagine with 2,500 to 3,000 offices and all those people had to use this offer every single day they just couldn't start throwing things into production every week or every month. And basically before they started there had been a lot of upfront commitments so they had planned that way. Also, part of the CityX issue was they were going to change out the hardware in every single one of those offices. They had PCs at that time that were 7 or 8 years old and they determined in order to be able to do all this wonderful sharing and centralize all this data that the PCs they had would never do it, would never make it, they just couldn't handle it. So part of the reason they were releasing things on a schedule they were was because they had massive amounts of other stuff that had nothing to do with the software changes that had to go into place which was basically to re-host the whole thing, take out most of the servers and replace it with stronger communication connections to a central location. So they had all kinds of other things going on. CityAnna was basically doing three plus one sprints of three weeks each. The plus one was to catch up on all the stuff we didn't get done in the first three sprints. It was just overflow just in case we didn't think of something we'll stick it in there. As far as daily stand-ups these folks have every single day worked really well almost from the very beginning they followed the stand-up process really nicely, worked very effectively for them. For the first month they didn't have the video though, it was just over the phone. So basically the offshore lead talked most of the time to the frustration of the Scrummasters. The second month they had two more teams and still didn't have the video. By the third month everybody had the video and everybody was talking about how wonderful the change it was. It also allowed the Scrummasters to actually see the people offshore and be able to encourage them more to speak for themselves which was a cultural issue they had to overcome. CityAnna had one team that was co-located, had a couple of teams that were scattered in a couple locations with phone in and then they had one team that was the most interestingly distributed I ever worked with which doesn't mean other people have it. Their BA product owner type person was in Denmark. They had two developers on the east coast of the United States. They had a developer and two testers on the west coast of the United States. They had two more developers in India, three or four, and their Scrummaster was all by his lonesome in Singapore. That was their team. He ran two stand-ups a day just to get people on the phone in some kind of reasonable hour. And I used to be on the phone at 6.30 in the morning and at 10.30 at night so I could sort of listen in and see how the stand-ups were going. And that's what he ran. Later on I think it would come up again, basically from what you've heard so far, CityN's answer to most issues and I would raise it as well, do you think it would help you if you did this or that, was we're used to working this way. And that was their reason for not making any changes when they just didn't feel like they wanted to bother us. We're used to working this way. What was their technical environment? Well actually CityN has a better end to that one. Their teams were in charge of everything. They didn't have any dependencies on anybody else for any of the code. They didn't have to hook up to anybody else's systems. They ran their own builds. They did whatever they needed to. It was all kind of wonderful. CityF on the RAN had builds done by one group in one part of the country. They had architecture, data and object model groups who kind of were in control of that and weren't sort of part of the team. They had inter-team dependencies which they managed to work out pretty good actually. And then they had other development efforts. One of them in a completely different state that basically they had to interface with. The first phase of their work, the interface requirements were about two or three other systems. Phase two of the work which I wasn't there for but had prepared them for, they had 30 to 40 dependencies on other groups to do work for them that had to work so that their delivery of their software would work. So how did they communicate? Well there was basically a couple of things they did. They would always stand-ups. CityF actually would meet with me during the day and they would also meet with their management. Usually the management meeting was within 30 minutes of the stand-up and so they would get together in one of the manager's offices and there would be another 10 people in there. Every single manager on the project, development manager, test manager, the build manager, this manager, that manager. All of them were there to hear the scrum managers report on the status of all the stories and any impediments that they thought management needed to work on. Tremendous amount of management commitment to participate and the management did follow up on this stuff and support the teams wonderfully. The other, and then usually just before or just after lunch the scrum managers might have been meeting at cafeteria so it's kind of like how's it going kind of thing. And I'd mentioned some things that I'd observed going on and they'd talk about whether they likely would address that or not. So basically it was kind of a mentoring training session for them. I would do similar things with some of the management also. I didn't do much coaching of the teams themselves. I worked through the scrum masters. I didn't feel that after a couple of weeks of just observing I didn't feel that they really needed anybody in the room kind of making it look like the scrum masters needed coaching. So I basically let the teams focus on the scrum masters as their kind of authorities for the process and stuff. City and basically had no scrum master daily coordination and we didn't meet with their management at all almost ever. I found out later on that I was brought into the city and technically to be a development manager. Coach was something that the consulting company I worked for said was what I was coming in for because that's what I did but the company thought the consulting company was delivering a development manager to them that had agile experience. Basically because the development manager got too busy to pay attention to what the teams were doing and so I wanted to hire a consultant to kind of be the intermediary between the teams and the actual management structure of the company. So how did you plan it? Well, City F actually decided to bring in field people to be involved with this project. It was going to last for three full years and those folks would do things like work on user stories, work on use cases, work on things like managing the backlog and all They also had co-located POs. Every single team had a product owner person on the team and then there was a senior product owner that they would report to just to coordinate things from the business perspective. Like I said, they had these user teams. They had real users, real people from these 2,500 offices and about three of these teams who came in and were there to be a resource for the teams to ask questions about how is this work in a real office? Both organizations had product requirements, budget deadlines, basically pre-committed before they started to do this. From a product requirements perspective, City N basically had its regional hierarchy. They met once a year, they had this two month planning cycle and at the end of that two month planning cycle this is what we're going to do for the next year and this is what we expect to have delivered. So basically it was a big bunch of stuff that everybody knew that he had to get done. As far as business commitment, like I said, there were these co-located POs and so there was like instant turnaround and instant feedback on things. The City N had VA and media areas. They really couldn't make any decisions. There were just people to ask questions of and they'd go back and sometimes maybe they got an answer in five to ten days on a requirements question. Hence the one extra sprint at the end to do all the stuff that they couldn't get actual real feedback on. By the way, if you had a guest, it was the best of times, it was the worst of times, this made a presentation on how wonderful agile work in our organization. This is how it really worked in a few very large organizations and by the way this is how it really works in most of the large organizations I've been with. They had some issues that they, at the onset, are not willing to deal with and they sort of make accommodations in many ways. So, some specific things about observations that was all by the way of sort of telling you what was going on. That was level setting into what the things were. City N, their scrum masters were generally not very keen on their role. I did some personal interviews with them after a few sprints to ask them what they were going on. They were picked. They really didn't ask to do it. They were told this was a way to gain management experience which is what they really wanted in the first place. They didn't want to be scrum masters. They were VA leads or development leads who wanted an opportunity to gain management experience and the company decided to do it in an agile manner. So they said, well, scrum masters are the closest into a manager we have, so we'll make you scrum masters and I'll give you management experience. Most of them felt they were giving out their VA and development lead roles and for most of them they didn't like that. They sort of liked being the person that people came to and kind of could give direction and be the lead and said here's the way to do it and stuff. So they didn't like the step back role of scrum master really. Not that they weren't necessarily bad at it. This really wasn't what they wanted to do. And they felt they were more prepared. They went to a two day scrum master class. Yes? You know, we're working on implementing parts of Agile where I think we're scrum left in the presentation yesterday much like you just described. But you know, I think we're struggling to see the difficulties. I think it's great as a project manager. Actually project managers and business BAs feel less like they're giving up a lot of what they do by becoming scrum masters. More technical ITBAs and development leads really feel like they're attached to the product a lot and so they sort of don't like, most of them don't like the idea that they have to give up some of that. I won't call it authority and control but give up that. I'm the go-to person now for decisions. You know, turning it back over to the team was kind of hard for them. Not that they were necessarily totally bad at it but ultimately I think all of them decided that they would eventually, as this project was over, not want to do this anymore. One non-technical BA did like the role. This is the person I had no training whatsoever. This was a guy who was all by himself in Singapore. He wanted training but never did get it so I would like be on calls with him sometimes when we talk about things. I spent the most time talking to him, frankly, and then the one co-located grum master who actually got his team together every single day and kind of insist on working that way. So I don't know what this means. I don't know whether the person who had the worst situation to be his grum master liked it the most. Maybe that was because he had to do it the least. Basically he was just coordinating calls and sort of saying is everything on track? Because he couldn't do this grum master role, frankly. To the end, there was an expectation by the development manager that I would see a wreck that was supposed to be filling in for that one would do all the agile stuff like burn down but nobody really used it. The teams who were never trained had to use it be it in tasks or in story burn down. It wasn't tracked and paid attention to by anyone anyhow. And overtime was a normal week. So basically when I started talking about the agile idea of a sustainable pace, their sustainable pace was a minimum of 50 hours a week. So overtime was built into their schedule to start with and that's how we do things here. That's how we work here. The release manager really didn't care about it. They didn't miss it. His attitude was I know these folks, I trust these folks to get the work done. We've always just gotten it done. That was their project approach to tracking. So I bother with this agile stuff. And then, as I said before, all these requirements were kind of a given ahead of time. It was just a matter of how do we break it up into three months of four or three week sprints. That was a plan. So why bother planning? We've got to do it anyhow. It all has to get done. So why bother being fancy about estimation planning? We just kind of get it done the best we can in these chunks of four or three weeks' risk. So there's no value in seeing the better estimation. But we're used to doing things this way. So the app on the other hand had some, they were people with their biggest issues was defect reporting. Now they were using two different offshore consulting firms. The development consulting firm evaluated their folks based on defect reporting against them and the testing consulting firm evaluated their folks on defects that they didn't let get by them. Basically, defects found compared to those that got passed them. Well, you can imagine what that did. These scrum masters, we didn't know this was going on for like four months, five months into the whole process. And CDF is working with these offshore vendors for years. It's the way they always work. So nobody, like, even said anything about it. So what was happening was the offshore developers didn't really want to check in code too frequently. They wanted to bang the dickens out before they exposed it to any of the onshore testing. So we sort of overcame that. I actually was back to visit them and I asked them about it and they sort of still do that. But they made an agreement. They made one agreement that no defects found during the spring would go into any kind of tracking system whatsoever. The only defects that would get tracked were those that still existed by the end of the spring because they were willing to carry some defects over and that didn't take more than two days to eliminate. Because of the planning cycle and the offshore cycle, they basically said that the first couple of days, one of the things the offshore people would do is they'd fix some of these trivial bugs. So if they could fix them in two days they didn't put those into a tracking system. So the only thing that actually got tracked against anybody was something that was so serious that it would take more than a couple of days to knock out by the end of the spring. They had huge amounts of coordination problems. The second sprint when they had three teams, three teams of 15 people apiece managed to deliver two points of functionality. Because they had a dependency on a group that threw some stuff over the wall that didn't work and all the teams dependent on one of the teams working and that team dependent on this stuff working they learned their lesson about how to anticipate these problems. They also had unplanned data and object model architecture changes in the middle of their sprints. The builds were controlled by a remote group and they couldn't get two builds done a week initially. Within three or four months we had the manager working very hard down to a build a day and two builds a day in the last week of the sprint. They had extensive retrospective change feedback. We established kind of a formal action item system. It was really for the managers and they really did get a decrease out of that so they made some changes based on that. Basically their biggest problem with the manager was that somebody looked at the schedule and said oh you have to go this fast to get all the stories done. That was a constant battle to this day. However it's done very pleasantly among them. Everybody realizes that that really is the truth if we're going to hit the schedules that the larger organization wants. We've got to do this. Some of the product owners would pull back on some of the functionality in order to do that so everybody cooperated to do it. It didn't mean people didn't work overtime occasionally but it wasn't like you team are the only ones that have to do this. Initially the scrum masters just went frantic over this. This is not the right way to do agile stuff but they came to certain accommodations and everybody worked together to pull this off. They had this solutions group that actually was a team that worked a sprint ahead and the major goals was to shield the teams from dependencies that weren't really ready so before the team would have to pick stories this group would already pre-qualify their dependencies and say that's not pick stories that you're dependent on this stuff because it isn't ready yet and this helped them avoid a lot of stubbing and things like that. Initially it was still a problem though. So they were doing what everybody called scrum fall fundamentally at some levels. They fell into many waterfall patterns with this and that took us about four or five sprints for both of these groups to try to get out of it. I tried to teach them about what I call iteration in the head not by the calendar and not by the clock. They were basically saying well within this iteration we'll do waterfall type stuff if we have to. I finally got them to understand what it meant to develop stuff in two hour increments and try to check things in from that point of view. They were very co-located, very distributed and everybody who was co-located loved it and this is a phrase that one of the people said to me and I'll remember forever and I'll say it over and over again every place I am. I'd love to hear that from every team that I have. We had to overcome a few culture differences entering technology not a big deal from that point of view. The training was a big serious problem. The main problem with the training was that we really were never introduced to the agile values and principles and so they learned practices and techniques and when they didn't work they weren't sure where to go which is why they eventually brought me in to try to show them some of these things. And they got into all kinds of battles about what was pure and what was not pure which was wasting a lot of their time because all they knew was the techniques says to do it this way if you don't do it that way we're not doing pure agile. So there were battles over that. So my argument is I made it yesterday understanding the values and principles would have helped avoid a lot of these problems I believe. But my work as an agile coach I think kind of fits in with what's the last thing it said in the book. The guy's about to go to death. It's a far far better thing than I've done than I've ever done before being an agile coach and it's a far far better place that I go to than I've ever been before. So that's my experience with coaching and especially in large organizations. Thank you very much.