 So I think we can start now. So before we start, let me check. How many of you have attended retrospective? Maybe within your team or you have conducted? So I don't have to go and explain the basic principles. But what do we do in retrospectives? Anyone from our audience, what do we do generally? Yes, we inspect and we add up. That's the key ceremony where we go and adapt and see what is the changes we have made, whether it's working or not. Well, we have one team. We had some problems. We looked back. We improved it. And we found some solutions. That's the agile cycle inspect and adapt it works. It's great. But imagine we have a big group, let's say, or a big product of 10, 20 scum teams working on it. Or maybe we have to do a program-level retrospectives. How does this fit into it? Because we will have people from different backgrounds. Or maybe different functional roles. We will have managers from sales principles which are single-think. My experience at work, but large retrospectives which I've done in my US office plus in my India offices. And you can ask me questions anytime which you feel you need some explanation. Basic principles which you have seen with a single team holds true here also. This we already know what is the agile retrospective. Now, this is the case study which I am doing here is the first large retrospective which we did it in year 2012. It was in our US office. We had this big new program, new initiative. The top management was there. And the customer was always at our back because they had to spend around maybe close to more than 20 millions. And they wanted result faster, but we are not able to scale it up. So to improve it, we added a few more scum teams. And it was a lot of UI changes also happening there. So we had a UI team also. They had the proper scum or scrums, the retrospectives. Everything was good. But we are not able to get the productivity or there were a lot of problems within the team or the cross communications, the technical adaptability. A lot of issues were there. And many of the times, these teams had to stay late to work. Well, they had retrospectives. The teams had retrospectives. But when you look at the large program level, we found some common patterns across the teams which were hidden, which was not coming up to this retrospectives because the visibility of each scrum team was up to a certain level. So then we decided, after our first phase, let's do a combined retrospective of all the stakeholders, all the future teams with all the business stakeholders. And let's find out what is the problem which they are not able to identify. What is the issue? Before you do a retrospective, what is the first thing you do in our meetings? We send an invitation to the people. And we'll say that, OK, this is a sprint first, or maybe a sprint five. We are going to do a retrospective, or maybe this is a goal, and all those things. So before we started, or before we did it, and the audience was so huge, we had our scrum teams running on agile. But we had our salespeople, our business management, the top management, the VPs. They were completely new to this. So what could we do to bring them into it? So we went with all these problems. In fact, I remember drafting all those emails, creating graphics, and the problems, and the cost metric analysis. A lot of data we do analysis. We send it back and try to convince the people, to the VPs, the salespeople, the business development people, and talk to them about they have to be here. And I remember there was one of our top delivery managers. He started getting so enthusiastic. He started pulling all the book counts and all the emails. And he was sort of in a firing mode. So we had to go back, talk to this managers that this is to be a retrospective where all people are equal. So they had to remove all their manager powers. And they had to come as a normal team member. And it was a very difficult challenge at that point of time. Remember, talking to a VP of such a big corporation and how they reacted. But then when we presented our data and say what we want to achieve it, they agreed. We had set the goals about several of the issues like the late-night working of the teams, plus the delivery, which the customers wanted, and the scaling up of the product in the next releases. And then we went to make hype about this. We advertised it. Notably, we created the emails with the invitations, all the goals. In all the pantries and the scrum teams or the scrum areas, we just put it all these big posters, advertising such as a big event. And then we blocked around two days, two entire days for this part. Then we asked the team to gather data, because we are doing this around mid, or maybe around August time in 2012. So it was like around six months after the team has the project has started. So a lot of events or maybe multiple sprints has happened. So they gathered the data. And we had the meeting with architecture team. They brought their own initial analysis from the teams. The teams started with their meetings, their retro notes, the individual team retro notes. The project managers, all those people, they started getting the data for us. And we had multiple reminders for them. And then since we had multiple scrum teams and people from different departments, we created a group of scrum masters, or I would say a team to work with all these people to agree a common working agreements. Because the scrum teams had a different set plus, and maybe to an extent our direct managers would understand that. But when you talk the same thing to sales managers, or maybe the business development people, it was like Greek and Latin to them. So we had to work with them, what is their priorities and how we could adapt them. So it was itself in a small project for us to get this working agreements. Then once we finalized the working agreements, we had invitations sent. Formally we went and talked to the people, invited to this. We sent email invitations with the agenda, all the goals, the working agreements and what we are trying to get achieved by this. Which was a big venue. In fact, we had to move couple of, we had like a series of seminar rooms. We removed couple of portable walls, made a big room, removed all the artifacts from the wall. And I remember our sales facility person looking at his some precious paintings. And I thought he would kill me. But then it was fun, tough working with those people, getting as a work layer. The biggest challenge I faced when I did something similar for India team in our India office. That office was more trouble. But then ultimately they also gave us the permission. Then we set up the venue. And their venue we created across the wall. We put white chart papers. And then we had the working agreement space, the goals across where every people could see it, almost all the audience could see it. And we arranged some light refreshments for the people. This was our meeting preparation before the day. And every day, every scrum team, we used to go and talk about this, review the things because they were the people who were supposed to talk. And we had to go back again to a coaching to our managers so that they don't come as a managers, but be only as a participant. Our opening day, well, the fun begins. Here we had our, all the people assemble early morning. And we had people which we are seeing for the first time. I saw my counterpart in my sales department for the first time and all other people. And the team where we had initially planned to start with a coffee so that they could mingle each other. But then we asked everyone to introduce themselves, their name and what is the play department they are working. So that they could at least get familiarized. At least they could speak out in all, in front of all the people. Then we started reviewing the agenda, what we planned to do the two days. Then we reviewed all the working agreements which has been agreed by all the people. And we called our senior VPs and other top management people to reiterate the project vision. When they started this project, they had a certain vision to this. What is the plan? What is the, what is their intention? So we asked all those people about to reiterate their feelings and how, what is their current feeling at that point of time? Then we went back to the audience or maybe all our colleagues and asked how do they relate to this vision? According to, was it a success? And well, we had a very surprising, we asked the people to raise their hands and it was a lot of mix. It was not as a major success which we thought or originally. And it was a really eye-opener for a lot of people when they looked around and saw that the perception about the success differs a lot. And we talked about the emotions part about what they felt so that the people get talk to each other without any hindrance. Before even this, by the time when we were starting preparing for this retrospective, we had asked the people to gather the data about emails, the defect numbers, stories delivered, all those things. And even some of the meeting minutes or even some of the team retrospective notes. They brought this and we asked them to paste across as a timeline. We had volunteers in the group who went there and added it. And we asked the people to move around and look around and see what they could add to it. What is the things they're missing in the picture? Once everything has finalized, we got some volunteers to start explaining the story from the day the project was started and to that day. We started this because there are a lot of people who joined us in between and there are people who left to other groups and there are managers or maybe other department people who didn't have a complete picture of this. So after the storytelling, they were able to visualize the project as a whole. They were able to understand the emotions when the people talked about it. And I remember a couple of deep interactions when you're talking about story. Some of the people got very emotional about a couple of the stages and this was the first time they were able to talk about their feelings and about how they planned to do it and which they were not able to do. So it became a sort of platform about emotional outbursts plus some other facts. And up to this, once we have the story, we talked about generating insights. So we had a lot of points to discuss. So what is the top points? What is the points we should be taking forward? So we formed subgroups within the team who could formalize or maybe go through all the stories and find certain discussion points. And then we had grouped this discussion points into certain themes. We had a lot of issues with our surprisingly, our test-driven development was big problem. Our development manager was supposed to push it across the teams. He was pushing his own version and the team was not so happy with it. So we had a lot of discussions around that and a lot of points. So we created, we grouped all this into different themes and we asked the team or the subgroups to work all this individual themes and brainstorm it or find the ideas. What is the problems which you see there? Or what do you want to talk about this specific theme? We rotated people across themes and then they brainstorm each other and find the discussion points to generate the insights. And once we finished this exercise, in fact it was the end of the day one, we asked the people to go back and sleep over the points which we discussed. And the next day, because that was a purpose we had around one hour left to, we could have followed it up but then we thought it would be a nice stage where could we stop it and go back. So the people could mull over all the points, they have seen it or heard about it. The next day when we came back again we reiterated our vision, our agreement so that people don't forget about it. After that, we asked the people, the subgroups, to go and discuss it, the themes. What is the problem? Present the top things or not the present all the issues. And then we asked the group as a whole to go and prioritize the points in each theme. We gave colored dot sticky notes for, it was red and green which we usually took. Red means it was neat for improvement, green was best, keep it up. So based on this privatization by the entire group, we were able to identify the top three points which needed our attention for improvement and another three which was to be kept on. And once we did it, we again had our analysis came back or the problem areas. We asked the teams to go with a couple of analysis metrics which we explained like Fishbone or Five Eyes, couple of other, but they were free to take any of their analysis methods. They analyzed it and they found what is the problem? What is the issues which they were trying to tackle it? And they formalized the action plan. But let me ask you a question. You have done a lot of retrospectives within the team. What is the problem? We actually have seen it. Anyone? We should feel the retrospective didn't work. Any? As I will add to the point because this was a very big discussion point at that point of time. Why one of the person said it to me that all the things you are doing, it's great, but it's not going to work. And I was surprised. We put so much effort, energy into this and this person openly saying that it's not right. So why doesn't it work? Why doesn't it work? Because all the retrospectives, his master was doing it, just put it in notes, put it in a SharePoint, finished, nothing. Or maybe a few things he will work on it, but there are a lot of things which they and they were not able to view the progress. You have the task board and all this sprint backlog which you see where you progress each and every day. You can visibly see the progress, but the teams, they were not able to see the progress on the retrospectives. They had retrospectives sprint after sprint, but what happened to the discussion points? How many of things were solved? How many of things were blocked? What is the status? Yeah. I think perspective point is for the team to improve. Exactly. There's no master to change it up. Yes, exactly it's for the team to, in fact, that particular team was in a specific, where the scrum master did every retrospective, the team will collaborate it. Couple of action item which was planned to the team, it will be given to the team, they will have visible to this. But there are retrospective points which has given to the management or other people outside the team which they don't influence, which we expect the scrum master to go and work with those people and facilitate it. But that area as a team didn't have any visibility because that was outside their. In fact, it starts with, it starts with your daily planning meeting. There's something wrong with the equipment. The scrum master should have been able to bring it up to the management and say something wrong, right? Exactly. Exactly. The second thing is, you mentioned something is put in the share point and it's just lost. Are you not going against the basic and the process of people and interactions? The process? I specifically want that to happen. So what is the thing that was happening was all those people who put in, they go through it, all those things. But the visibility part was not, was missing completely because you put it there. Yes, you might go and review it once in a while, but you are not seeing it every day. Because you see every day, you find actions, right? So one of the things which we did was, based on the discussion, we said, we are not going to put anything. We will put it in retrospectives because there are people in offshore locations who want to see it and maybe in future reference. But what is the thing which we came out of? Yep. The retrospective focus, because you are saying that this Chrome master is not reflecting the team's retrospective point, right? So is anybody even being uttered and asking something? It is like... Retrospective for me, it's a lot of, it's a data of the team. Let's say, even if you solve everything, you look at a picture of one year, you will know how the team has evolved. Not to reflect the team's point. Nobody should really worry about what is going on with the system, right? It is the team is playing something and then they are going to change that in subsequent sprints, if possible. Yes, yes, I agree with that point. There is a little more thing we can even discuss and let me finish this, wrap it up and I will just come back to your question. I'll just point it. Okay, so one of the points when we plan to track action items was, all the points we should track, which we discussed, we assigned to different people and we just put it like our task board. We created some quick task board for this. We kept it so that the people could go and see in the physical task board, what was in progress, what was it blocked and it was assigned to whom, whose person was walking on it. It was a completely visible backlog for those people and we had even created people like some volunteers who could go and track this because it was such a big initiative. We had so many people around. We can't expect one or two people to do it. So these people individually, we have one developer who was very much good in TDD. He said, okay, it will be his responsible to go and teach a program with all the other teams to improve their TDD practices and there are, we had some issues with facility problem. So one of the scrummers, he will go and work with the facility with respect of all the group. So we had this volunteers for all the points as you get it and it was visible for the team. So we were able to track it and even though we later on we published it in SharePoint for the people to see it but for them, it was in the world in the scrum team world where they could go it. Then once we were able to identify the big problems and couple of the good things, we were able to keep it up and we had a good plan on who is going to take what actions and how we are going to track and how they are going to get the feedback about this and everything was happy. We said the right time to close the retrospectives and we summarized the entire event for the two days. We learned, we read out the key learnings not only from the retrospective which a lot of things which I explain which I got to know the summarize but we asked the individual people what they learned from it. They were able to share their experiences and then we took a retrospective on the retrospective and see what could we do, improve the next time we did it and then couple of the volunteers who did it that they are going to track this. They shared their contact details even the cell number and the email address for the people, the group so that they could contact and ask them even if they go back to their different buildings they could go and contact this person and see what is their action item and then at last we appreciated the everyone's help and effort in coming up. That was like this is a general format which we are used starting from the first large retrospective. After that we did it multiple times in different locations. Couple of the activities which I have mentioned here I've changed but the format would have remained almost same and I would say it improved us a lot because the second phase which we read it was a grand success for that particular initiative. Customers were happy and couple of, they have like repeated contracts for us or and for the groups and it was a wonderful experience for them also after seeing this. So that's all from the format all the things if you have any questions I can take it up. It's almost on the time. Well, first we did it in US. It was not distributed. We had all the local teams here. Came back to India also. They're also, we did it for the local Indian teams. When we did the first thing they were not working together. We did it once we put the VDAC conference. We had to and we, at the time we brought some people there. We went to the US. We played out few people down there and we did it. Happening at the same time, different locations. We brought people together in the same location from different locations it was a good experience. But once we tried, we did it with a small scale where we had distributed teams in US and India. Two different scum teams. We did it through video conferencing it. Well, it was, it didn't give that effect but it was okay better than what they were doing it individually. But given a preference I would still fly what people to the same location to do at least a retrospective. Yes, yes. Because they were also a key stakeholder for the success of this project. Exactly, because when they saw this they know that their contracts was killing us. How they planned because they give us a big bunch. Exactly, because they were expecting to give us a big specifications and other things and even though they will work with the product owners to get the story points but they were not fully agile. They were still in that age-old model. So when they saw how their actions are affecting the team in the final delivery they were more open to change their ways at least work with the team in a better way so that they could interact it. And how could they communicate that to the end customer? That was a key learning for them.