 Hello, everybody. My name is Alexei and I work as an agent coach in a company called ScrawlTrack. Previously I worked for a startup called Quick that was then acquired by Skype. Today I'm going to tell you about how things worked at Quick and what happened when Skype bought us. And how we resolved the problems that we had with the Skype process. So, first let me ask you a few questions. How many of you worked in ScrawlTrack? Okay, great. Then probably... I'm okay with this talk. It doesn't hang on me. I mean, how do I do it? Can you see me? Okay, fine. So Quick. Quick was a startup which built a mobile application that allowed to stream video. So the idea was that you can share your life moments by streaming video from your mobile phone to the mobile phones of your friends and family and to the web. And Quick was a startup, meaning it was a relatively small team, about 50 developers at the end. So, you know, it's hard to talk about the process when you really don't have some meaningful process, I mean. As it usually happens in startups, you just do the work, right? So, we had two locations. One was Redwood and another one was Moscow. Our product owner was in Redwood and the development team was in Moscow. A few times a month, no, once every month or month and a half, a product owner released to come to Moscow to meet the team, to discuss the strategy and so on. At other times, we had team leads that were keeping in touch with product owner releasing Skype. Well, you know, sometimes it's hard to keep in touch using Skype when you have 12 hours time difference. But, well, we came to work earlier sometimes, so it was okay. And what happened when product owner wanted a new feature? He spoke to team leads and we formed a workgroup. A workgroup is a group of people that works on that feature, right? Questions like architecture design and some, you know, implementation details were decided by this workgroup. So, sometimes product owner gave us only the high level vision and we could figure out the details on our own. This, of course, required our engineers to understand the business. But in a startup, this is usually the case, right? So, that's actually all about the process in quick. By the way, if you have any questions, feel free to raise your hand and ask them as I go because I want this talk to be valuable for you, right? How many size of the workgroup? Depends on the feature. It's two people, two, five or seven people. Not to be. Is it a team? Yeah, it's a feature team, but the thing is that unlike in the feature team, one developer could belong to several different workgroups. So, they intersect. Okay, so what are the advantages of this? The first advantage is obvious really short time to market. We were doing rapidly, right? So, small features took us one day, two days, maybe big features took us a month, month and a half. We were first to release video calls between iPhone and Android, by the way, faster than Skype and anybody else. So, we had almost no bureaucracy. Any issues were resolved informally. We had a few mandatory rules related to operating with production environment, which is kind of understandable. You have several million users. You have to do something to make sure your uptime is fine. We felt responsible for the product. Because of this time difference, we couldn't rely on our product owner to make all the decisions. So, we had to make product decisions on our own and it was okay. And we were very close to the user. We had support engineers that received feedback from our users and they send us weekly report that contained the top issues that our users had. And we decided basically what to do with it, whether we need to fix it or not. And then in 2011, Skype came in and said, we want to buy you, right? And, I don't know, partly they have chosen us among several startups that did the same thing. Probably because of the, I don't know, the same positioning, I think. Skype's concept was, we're doing service for you to be closer to your family. So, that was also that we did, unlike Justin TV or Bambuser and other similar services. And we were excited to become a part of a large business. Because Skype was, at the time, about 900 developers. And it was estimated to be $2 billion for the company. So, we were excited to become part of a large business. We were waiting for the Skype to come in and to give us some secrets how you do business and so on. But at first, Skype didn't touch us in the sense they just told us, you dig a business as usual and that's okay. But we wanted to do some features for the Skype also. And they asked us to allow Skype to send video messages. So, they acquired us in January and our forecast was that in March we would be able to release video messaging in Skype. Guess what? In March we couldn't release anything. And the reason was, well, there were several reasons, but to generalize it was because Skype had certain constraints that all of the Skype products should follow these constraints. And we wasn't able to do that under these constraints. So, after we failed to release that in March, Skype came in and said, well, okay, we have to fix your processes. Now, we have this structure, we have these processes and you have to figure out how to work within our structure. So, what was the Skype process? We had this concept, yeah, a little disclaimer. Since I was an agile coach for the Moscow side, my view might be not complete. So, I will speak from this point. So, the Skype process. So, we had this concept of release vehicle. A release vehicle is something that can be released independently of other parts. Examples are configuration management, for example, or authorization, things like that. All of the systems that could be released independently that had a team associated with it that had a product manager and a product engineer manager. These are two people that were responsible for the dependency management and for interacting with other release vehicles. Release vehicles had backlogs and they were doing scrum. The concept was that release vehicle is a service, like, you know, service-oriented architecture. So, there is a team that provides value to the other teams by doing some meaningful software inside them, right? And providing through APIs or something like that. So, for example, if you work for a Mac client and you need something from configuration team, you have to make sure your task is included into the backlog so they can do it. And either product manager or product engineer manager have the power to come in and say, we want this feature. When will it be done? So, now, imagine you have many services. We had about 50 of these and they all are dependent on each other. So, there will be certain problems, right? How do you manage that kind of organization? How do you know if there are problems or not? If everything is going fine? How do you forecast when you have something ready? How do you know what's going on with your organization? To answer these questions, we had a system of RV reports. Every product engineer manager had to fill in this report at the end of each sprint. So, was the ration successful or not? Means whether everything planned for the sprint was completed or not? Was the release successful? Release was not successful, considered failed if more than 25% of the sprints were failed. Usually, each release included two or four sprints. So, in that case, if you had, for two sprints, if you had just one sprint failed, the whole release was considered failed. When you have tasks or features that were not completed, that you included into the sprint backlog initially. So, out of ten tasks, one is to vary, that still means that it is failed? Yeah. If you had release failed, then you had to explain why it happened. So, what were the root causes? You had to do a root cause analysis and report the results to the management. And we had regular RV review meetings. It's when product engineering managers and executives together in one room go through these reports and talk about it. So, what do you think of this? Okay, I can't hear you. Well, I said that using these principles, that would pressurize the team. Pressurize the team. Yeah, probably. So, the question is, if you are an executive in a company that has 1,000 developers, how do you get transparency? How do you know what's going on? Right? And from this point of view, I think these reports probably look like, I don't know, good solution, maybe, I don't know. So, the problem was that in 2012, in May, we started to work on a new product. And we thought we could do it in several months. But the thing is that we initially planned to release it in September. But in September, it so happened that we had zero features ready. So, how did that happen? What was the problem? Okay, imagine that you are on a front-end team. And you need something from a back-end team. What do you do? You are in an interaction planning and you figure out that you need something from the back-end team. You go to these guys and say, we have a dependency, please do this for us, right? What do they tell you? They say, oh, no, we have our old sprint running, we can't take this thing right now. Because in this case, we will have to drop something else. And our reports will be read. And we would have to explain to the management why we have scoped drops in our sprint. But we can take this thing in the next iteration. Which starts in a week. So, there's a four-week delay, right? And in six weeks, it's probably you will have something ready. Now, you start to work on this thing and figure out there's an integration defect. And you need another iteration. So, the most simple thing would take ten weeks. In case you have a dependency. There were a lot of dependencies. So now, take a look at this diagram. What happens if you need to get something through the chain of four RVs? Cycle time increases dramatically. So, for our new product, we have six RVs, six RVs dedicated to this new product. So, there was two back-ends, a library in the middle. And three front-ends. And most of the features require at least three RVs to do something. There's a metric in agile called cycle efficiency. What is cycle efficiency? It's basically, it's a measure of how much potential you have for process improvement in terms of shortening your cycle time. You have a value at a time, which is the time when you actually do the work that adds value. And you have a wait time. Cycle efficiency is the ratio of value at a time to the total time. So, in this case, it will be 30%. What cycle efficiency is good? I would say, for Scrum, it's about 20%, right? Because normally you have two week duration, you probably have about five stories in this duration, you release it at the end. So, you work on a feature for two days of ten, so two by ten, it's 20%, right? Now, take a look at this picture. This is our way to measure cycle efficiency for one of the features that we were working on. So, what we did, we asked our product engineering managers to name us a few features that are typical. Typical means it has an average amount of dependencies on other teams. And this graph shows how this feature has been worked on. So, rows are the teams and columns are months. As you can see, it could have been done quickly if you do this more parallel, right? So, why did that happen? Sorry, I have something missing. So, the thing is that the process was meant to make teams work more efficiently. Why did Skype adopt this process in the beginning? Because probably management thought that some teams might be inefficient, right? And to help them grow, we will make them use Scrum and we will send agile coaches to them to make them more efficient in terms of Scrum. But the reality is that that's a local optimization. And because of this local optimization, the whole chain probably gets worse. Why does it happen? Because of the silence that we've heard about in the previous talk, right? So, this is usually the case, by the way. When you optimize for local optimization, your cycle time increases. Questions? Real problem. Because even if for one particular feature, for one particular RV, it seems to be delaying the RV by a lot of time. Yeah. But at the same time, simultaneously, a lot more work would be done by the different teams. So, is it really a problem, is my question. Is it really a problem that the work is delayed? No. So, while some one particular RV is getting delayed, there must be some other work that is paralleling going on. Because the teams are not setting idle. So, when they are not working on this particular RV, they are working on something else. So, is this really a problem, is my question. That's a good question. Because in terms of efficiency, probably the manager's view of this is that they are doing valuable things. They are working on something else at this time. But the reality is nothing is completed so far. So, several months past, nothing is completed. Is that a problem? So, are we doing the right thing? Are we working on the right thing? My question to you. So, if you have a lot of things in process and nothing is completed, how do you make sure you are doing the right stuff? Maybe most of the code that you have written so far is not valuable, right? So, maybe it's not what users need. Maybe it's not going to work when you try to put it together. So, that's an inventory waste. Now, working products. So, what Agile tries to do is Agile tries to make cycle times shorter for you to get feedback faster. But when you work on many things simultaneously and you don't finish any of them, you don't get the feedback. So, why would that be? I can't hear you. If you have a legal architect, not based on the architect, you have no security. So, what I'm saying is that you could have been cross-functional before that particular review. I'm sorry, I can't hear you. I'm saying it. Later you had two RVs. Two RVs? So, why would that be non-structured based on that particular architect? So, if you have a three cross-functional team, you could have pulled up each team dedicated for that particular feature to the counter-unit. So, why don't they work on a feature together? Your original approach. The thing is that that's what we did at the end. So, the problem is that we have these reports and the problem is there are dependencies and when you start working on this thing, something changes, you have these reports, you have to stick to the scope that you planned for in the beginning of your screen and you can't just drop things out because other team has dependencies on you. But, yeah, that's the idea. So, let me continue to what we have done to overcome this, right? So, we have a series of meetings with the product engineer managers and we agreed that regardless of the process that we have in Skype, we want to create value in the first place and sticking to the rules is not a priority. And then we started we agreed to track our work in progress by putting together a product board. This is our first version of our product board. So, before we had 23 user stories on it. 23 is a lot and probably who knows what's little's law is? One person, two two, three. Okay, a little's law is about is a relationship between the time work in progress and average in this way. Suppose you have a queue and you have a certain amount of people in the queue. So, how long will one person have to wait? You divide a number of people in the queue by the condition rate and you get this delay. The same thing is applied to averages when you have not a fee for a queue but any kind of fee. It says that regardless of the time of the queue, your average cycle time is your average amount of things in process divided by the average condition rate. What does it mean to us? It means that we probably want to make work in progress less to focus on less things but to focus on completing these stuff. So, how to do that when you have six teams which have independent plans? You can't just throw away some work and say okay, we're not going to do this, we want to focus on this because in that case your developers will sit around doing nothing, right? You don't know this situation. What we agree to do, we agree to take the most important things and to make them high priority means when you have to choose whether I should work on this team from the most important list and something else, you always choose the choosing features, right? So, we made the second version of our product and we would aid the most important user stories only. The agreement was that if we discover any dependency on any of these aid user stories then your team have to drop anything that it does currently in the current screen and jump to resolving that dependency. That was the rule. By the way, by that time we learned how to hide score drops from the reports. That was safe in that sense. Shortly after that we discovered several things about our process. First one, guess what? We could have figured that by just looking at this diagram, right? But the thing was that there was a bottleneck. Library team was in the middle of the process. Most of the features were dependent on their speed. So, what do you do with bottlenecks? Theory of constraints says we should identify it, subordinate, remove some work that can be removed and then make it wider, right? What we did, we come through the library team backlog and we found a few things that we handled often in other teams. So, we did it. So, another thing that we discovered was that there was a lot of rework between android and library team. The reason was that library team consisted of engineers that were previously IOS developers. So, IOS didn't have problems with that. But android team did have this problem. So, when I said rework, I mean there were features that were handed off many times there and then back again. And code was rewritten several times. So, to fix that problem, we created an integration team. We took a few people from both teams and make them sit together and have unified backlog for that. And another thing that we discovered was we had scope increasing continuously in these eight features. So, what we decided to do we decided to fix the release date and not take in any more scope. So, it was, I think March 2013 and we decided that in May we want to release something even if it's not ideal even if it worked slower than we wanted it to work but we decided that on a certain date we will release it to internal dog food in release to get feedback. So, getting feedback was more important for us than doing it ideally from the beginning. So, that's what we did and actually after this May release we were able to release at monthly cadence regularly. It was a good thing because the delivery problem was actually resolved by the time I should tell you that the product itself pivoted several times after that but at least we could deliver on regular cadence and we could finally prove to the management that we're not just wasting their money show them a working product. That's basically it So, to summarize a little bit in sky process local efficiency actually sky process was optimizing local efficiency that was increasing cycle time dramatically and what we did we visualized feature delivery and that helped us control work in progress and we were able to reduce rework by creating integration teams and after we fixed the date we were finally able to get the desired feedback. So, that's it basically questions. Yeah, so actually in the beginning we as geologists tried to speak to the executive team about adopting Canva and we were told that the word Canva is prohibited in this organization. Probably the reason was that management thought that if we decided to do strong we should stick to that. If you just follow some rule it's regardless of what this rule is from set of rules, right? If you just follow the rules and you don't think what it means to the value creation you are in trouble. So we were told not to create any feature teams or integration teams or anything like that. The thing is that we had to cheat the rules actually to do that. Is this a good thing or bad? I don't know. We saw it as the only way to deliver a thing. Executives didn't know about it. Yes. Can you please speak louder? Scrum mandates like you should have cross functional teams. When it talks about local efficiency and talks about using Scrum if you have a front end team then you have back end teams. Ideally it should be together. So why did they have springs not synchronized? You cannot synchronize 50 teams. They have to be so there were too many of these teams. So you could have synchronized the springs but what you can do in this situation you can say that all the company works in the same cadence right? But what would be the value of it? So as I said typically what do you prefer? Let's say you have to set up a team and you have a team which you know structure the team which will be cross functional and in order to deliver your front end team I am sorry. As a thumb rule when you talk about Scrum you have to be cross functional at least from delivering a feature yes. The theory says that since you are a coach what is your recommendation when you see that we need to do this for example we need to deliver this feature what is the best practice? I am sorry I don't get this I don't get this No it is not really perfect But as Scrum mandates we want to form cross functional teams right So why not take all the teams that are out there and create a cross functional team that's a good question and we consider this opportunity the problem is when you have a technology stack with too many things too many specialists in it by putting them in one team you will get a lot of other problems like the standard won't be meaningful they will have bottlenecks inside the team you won't be able to balance the load so there will be different problems so actually integration team is a kind of middle solution it's not official team on one side and it's not a company team it includes specialists from several company teams but it's not complete end to end right so in that sense integration team is the best we could do so if we try to put specialists from all the company teams together that team just wouldn't be able to execute work assigned for the particular team is not complete you brand that sprint as paid yes now would that give rise to the tendency of getting less work to be done in the sprint do you think that will happen that's a good question actually this is how we hide scorn props so we can do and include dependencies later but the thing is that management position was that you should not do that you should plan in advance what are you going to do the reason management wanted us to plan for two weeks in advance was that they wanted forecastability they wanted to to understand when something is going to be done and when you do things like like that you just don't get that foresight plan for less then yeah that's what the team would do given to them they would plan for less amount of work than what they could do and some teams actually did that and our advice to these teams as I mentioned was that guys you have to have two kinds of velocity the one is optimistic and one is pessimistic and you have to include the amount of tasks to cover pessimistic okay yeah to share a bit more unfolding work in progress especially different workflows limiting work in progress unfolding work in progress so did you have like situation where you had one stage doing a lot of stuff where there is little less work less than what you have for work in progress and how would you manage those typically and that question is how much time do you need to optimize this kind of this is what works well for us and let's have these as a limit for the work in progress so let me get back to that slide with this one this was what we discovered that was actually going on so we put this work together by simply putting all the work that was in progress the thing is that when you have this network of RVs you don't have a direct mechanism for controlling work in progress so what you can do is you can prioritize the features at the feature level I mean and even if you don't do anything else just giving each feature a priority first, second and so on you already tend to decrease your work in progress because what happens if some team is blocked by another team what is the usual behavior the usual behavior is taking in another thing but if you focus on finishing something they start thinking differently they start thinking what can I do to unblock this and whose help do I need to unblock this the thing is that by just visualizing work in progress you already make this problem less okay that's a sensory question did I hear you say there are 50 different RVs 50 RVs the whole company so the question is what does planning mean so far you must have had either a ton of them between different groups or 50 people in a room all talking I can't the thing is that in the beginning all these RVs operated independently planning was ad hoc product management had to make sure that the product they are responsible for is okay in terms of dependencies and this was based on personal relationships later we had another guy his name was Chris Matthew probably could have heard about him he was working on creating a process for product planning he ran two or three days workshops when a lot of product managers were gathering in one room 50 of them and they had their plans for the next quarter on the wall okay in your example that you mentioned the QLIP teams and the android teams you said that the QLIP and the android team had some problems yes now my question is how do you spot that I don't know do you think that's a challenge is it as easy as people raising their hands up and saying we have a problem on this board we had stickies of different colors and each color belonged to one team so it visualizes what does all the team have to do to complete a vision and we had stand-ups near this board twice a week so during the stand-ups these things just came up so why do we do this again so the problem is that when you have separate teams that have their own plans they have no communication that can uncover these kind of problems when you make a board like this and ask them to communicate around this board these problems come up so we didn't use any special tools for that it just came up so sorry there is another question related to that now like you said this all happened near the board for companies who are spread a little bit geographically they are distributed in nature is there a mechanism that you recommend suggest this might be a way to spot that and not only once but consistently throughout the process in our case we were located on one side right now that I work with different companies I ask this question again and again and there are tools for visualizing these things across many sites I can name a few target process if you tune it correctly link it and now there is a startup in Russia that also creates some great visualization tools but it's enough for testing right now so it's called kaiden.io but I bet it's not open for public yet so there are tools the idea is that you should make things visible to visualize product delivery and if you do that things come up any more questions going back to the last slide which one last slide the very last this one summary and the last point next day next day reduce scope I would just like to hear your opinion on this whether you ever had the challenge where the date is fixed let's say we are going for quarterly releases but the product management says that I won't be able to reduce the scope for this particular release and I want if at the end of the quarter you want to release then I want this feature it is must have and the team is running out of capacity and did you ever had any challenge to negotiate with product managers in order to reduce the scope yeah of course we had this challenge I mean product managers don't ever give up scope but the reality is that you cannot deliver everything that they want right so you have to convince them of course so I mean we had these long debates when product manager insisting that we have to do all of this to release something but the thing is that we just had to explain to them that all of this scope would take much longer right you cannot do anything about it if you want all of this to be done it can be done much later but we need feedback as soon as we can get it right so let's do this now and postpone other things when you mention restructuring teams when you mention restructuring teams third no summary slide last slide so last slide summary okay we work by restructuring teams I can't hear sorry we work by restructuring teams we just yeah so exactly what do you mean I think you would do this restructuring or you could do it more on an adult basis can you please come a little bit closer so do you do restructuring any time or more on an adult basis when could you do the restructuring of the teams no we only did it once but the pattern is that you you better watch for rework if you see rework it means that these guys need to communicate more and it's a good idea to put them on one team so that's the pattern so the idea is that what we do now as contract is we visualize the value delivery network and track where rework occurs and we create integration team at this part thank you any more questions is it time thank you