 Hello, can you hear me? I'm not going to sing, I promise. This always starts bad. Thank you for coming here this afternoon and staying with us. There were seven other exciting talks to choose from. I really appreciate to see all of you. I'll provide some disclaimers before we start, so you know what are we starting from and where are we standing. Is this too close to my mouth? That's the first check. I just want to provide the best user experience. All right, so the disclaimer number one is that what we are going to talk about today is a linear process. That means it started 16 months ago and it's ongoing. We are not going to provide you final solution or the ending. So if you are hoping for that, there is still time to go. The second disclaimer is that there were no animals involved in preparing these presentations, nor there will be appearing on slides. So if you are hoping for an acute picture, there is still time to go to Christian Heim's presentation because I know that he prepared them for today. Unacceptable? All right. And finally, what you are seeing on this stage is obviously me and two other guys. But what we are, we are just a sample group and what we'll be talking about actually involve more than 30 people on the various stages. So that was real teamwork. And we'll be trying to tell our story. But before we do that, let me start with introductions. So hello, I'm Petro Bornik. I'm the engineering manager in the VIP group at Red Hat. Hello, my name is Petr Czerch. I am a physicist and I am also a quality guy in our free IPA. And I am also one of the product owner. So that's how it works. Yes, and I would like to summarize that Petr is self-proclaimed product owner because he volunteered for that role from the QE organization. So that was great. And my name is Dominika Bula. I'm agile practitioner from P&T, agile practice team. And I'm helping these guys and also supporting them as a scrum master. So there are actually three different teams in free IPA right now. And the total is, as I mentioned, 30-plus people. This is who we are going to talk about. So the most important thing, and this actually happened before I was even hired for Red Hat, was change. These guys, on their own, as a team, bottom up, stopped possibly sitting and watching things unfold and they decided to take action. So the change happened on their end. And how did that go? And what was the reason for that change to be introduced by Petr? So we have several reasons to do a change, but I think that the most driving one comes from the fact of how we worked before. So just an introduction, like we were first two teams working on one project that was free IPA. The team were a development team and a quality engineering team. And the way how we worked before was that it was the traditional waterfall model. So basically the development team created something, so-called threw it over the wall. The quality engineering team tested it, reported bugs. But there was an issue with that because usually the testing happened quite late after the development itself. Like the features were dull for some time when the bugs were reported, people might have forgotten when it was fixed or what was the reason for that. So it takes more time, it is more difficult than in the end, it costs more time and doesn't work very well in the long term. So we knew that we want to change that, or we want to do something. So basically what we want to do is, basically was to throw down this waterfall wall and start with something different. So the thing which we wanted was basically to have a way how we can bring the testing much closer to the development or ideally do it at the same time. So our actually big change was to actually set up the environment so that we can communicate quite well. Basically one of the biggest problems with waterfall is that the communication doesn't happen very well. So the idea here was let's merge the team together. So they can communicate at the same time, they can self-organize, do stuff as needed for creating the stuff. So we knew that to accomplish that, we basically need some geometrologies because they all live with this mindset. We are thinking about which one to choose and the one which we choose was Scrum. And it was not because it would be better than the others, but mostly because we have engineers who are experienced with that from previous engagements or also because the cost engineering team also experienced with it before. And that's why we just think, okay, let's try it with more people. So I will let Dominika talk more about that. All right, so these guys had the idea and what did they do? They went to the Scrum guide and they started with basics because you need the pointers when you are starting. The Scrum guide can help you with that. It talks about pillars of what it's needed. So first, you need to assure that you have a transparency. Hint, one tool helps with that. The second is ongoing inspection. So you commit to inspecting your own work. This is very important and it's often missed. And at the end, you adapt. So you introduce the changes as you go. And how to look for value in all of that. People sometimes are so focused on doing their daily work that they lose the bigger picture. And Scrum is actually helping you with this providing specific values. So what it's asking you for is asking you to focus on now and here. Are you with me? Are you focused on now and here? So by focusing on now and here, you can actually learn what to improve next and we'll be asking you for feedback at the end of this presentation. So focus is one of the things. But then you have the courage. You have the courage to admit that no one is perfect, including you. You make sure that your work stays open. Commitment is something very interesting because people might be thinking, oh my God, they committed to agile practitioners. She'll be staying with them forever and they will be just reporting their status all the time. This is not where we talk about the commitment. The commitment is dedication. What we are dedicating to our specific things. Like we are dedicating as a team to specific definition of done. We commit to delivering quality. We commit to improving day by day, every day. And finally, respect. We have a respect for each other. We understand that we are all different and have a different opinions. And this is what making, it's making all of this process great. This is theory. How does this look? You might be thinking, how all of this scrum looks like on the diagram. So I have the diagram for you too. No animals as promised. And before I would describe it, I wanna give kudos to Alex C. Monville, a red-hatter who actually created this diagram who are just borrowing it. So for some of you in the audience, you might recognize that this is a scrum workflow. But for others who are new to the process and came to the stock as a beginners, I wanted to briefly describe of how all of that works. So it always starts with division. Of how the team will be working on and what is actually their backlog. What is actually on the roadmap. And we are bucketing that to the product backlog. And there are many parties involved in that and many rounds of feedback. But once the product backlog is created, we as a team with help of the product owner create a specific sprint backlog. And this is when the time boxing starts. And this is very important because this is the time when you learn. You learn in the small cycles. So with the free IPA, what we do, we have a three-week sprint. And this is this time box time when team is working on what they selected for the sprint. And they meet almost every day. The stand-ups should be daily, but the team decided that what would work best for them is actually Monday, Wednesday, and the first day meeting. They wanna have off on Tuesday, but they promised to communicate on IRC. And Fridays, we try to have minimum meetings. But this works for them, right? They are doing this for a while right now. So we have a product owner working with the team. Each of the teams have a specific product owner. And I'm supporting all three teams as a scrum master, helping them facilitate the meetings. What happens at the end of this cycle is a sprint review where we get together, all of us, all three teams, and we review what was done during the sprint. And then we separate and each team is discussing what went well and how can we prove for the next time. And then the process starts again. This is still the theory, but what we wanted to share with you here today is actual lessons learned. What did we learn during this time? What did we learn during last 16 months? I'll be obviously not be able to tell you from the beginning what team learned, because I wasn't there. But I would hand it off to Pat to share with you the first lesson. But before we actually start with the first lesson, we also need to think about how we actually do the change, because we have the theory, we know how that should work, right? But we have two teams which work a certain way and how to get started. So what we did, we had a gathering in October 2017 and we just put it into some structure. But basically, we just jump into the water. We change our meeting structure to that. And then we just write, we created three separate teams so that we have smaller teams, not two, but three. And we distributed development developers and quality engineers equally into each team. And we just started doing that. So let's go through the journey. Thank you. So lesson number one. Welcome to November 2017, because this is the time when we really started. And I want to show you how we started. On the beginning, there are two roles. One was the people from the team. I mean the developers or just QE as me. And the second role was managers, many of them. And because we should somehow start, we just try that product owners, stakeholders and also scrum-hasters were our managers. And after some cycles, we see some very interesting things. So please, next slide. One of the things which we see is that daily scrum or stand-up, maybe you know this word. Oh, yeah, maybe we can go, no, we are here correctly. Our daily scrum is tool for our team to share the blockers, the code review and so on and make decisions of how we achieve this. But if our managers were scrum-hasters, some people just do status reports. So this tool, the daily scrum doesn't work properly. So we very soon see that we need something else. Please, next slide. So we created demand for a scrum-haster. It was created bottom up, which was really nice. And because our managers were connected to us and tried to do it too. So we got our scrum-haster, which is Dominicum. Six months before we really started. The reason is because we need to create this demand and yeah, it just take time, you know. What is also very important is that we use the scrum pillars how to achieve it. I mean, firstly, we do some in-spec inspection. So we see that something is wrong. Then we do adoption and on the end we have a commitment. So Dominicum is here and that's great. Next slide please. Yeah, and after another six months, we see that our managers are really great. But actually we have, I think three of them and all those managers are also stakeholders. So each manager has little different direction, little different goals, and we didn't know what we really should do. So we also create demand to have product owner and because we are three teams, so we have three of them, I am one of them. And it is because now we have one per person in which team who just collect the feedback from all our managers and then somehow create the correct backlog. Yeah, so it looks also another six months and yeah, it just works. So I think that about another lesson, Petra will say more. So when we look at this Dilbert strip, this was not our expectation. We knew from the beginning that doing it in a jail way will not us make more effective. But when we are working in the environment, we also need to look for not doing the exact opposite because as I said, we split the team into three. But one of the biggest challenge which we had at the beginning was how do we actually distribute the people between those teams? How do we split those teams? And at the beginning, we did not have good answer for that because we were thinking like, we cannot just split it by components because every component touches some aspect of the project so there is no clear division. So we did not split it by that, but it brings problems because free API is already established project. Like it is 10 years old, it has a big feature set. And this means that there is some maintenance, bugs come and bugs need to be groomed. And when you are doing grooming, like how do you decide which team handles the block? Because if there is no clear division. So the way how we did it originally was that we had one grooming meeting but all members from the other teams joined that grooming meeting, which means that the meeting can have a lot of people. It usually had around 12 to 18 people on one meeting, like doing triage, discussing bugs. And as you can know, like usually how it goes with those big meetings, it is not very efficient. Usually only a few people talk. The rest is either idle or just listening or just say one thing to there and there. But usually stay silent. So we are thinking, and in the end this becomes quite expensive, right? Like you don't want people on meeting and do nothing or just listening. So we are thinking how we can actually change that, how we can do it more efficient. And that's why we came up with some stable team split. And the biggest change was that we gave each team four things. First, the team received a long-term mission and the mission was derived from the whole group long-term mission. But that is not enough because you might have a mission but we also need to measure that we are going according to the mission. So for that mission, there are also a definition of success, basically measurements to check whether we are doing stuff according to the mission. So that people when they are designing stuff or being on the grooming meetings, they can cross check that. But this solves only half of the problem. It solves only the new stuff, like creating new features or working on some new problems but it doesn't solve the maintenance part. So for the maintenance part, we had basically two buckets of things. One was long-term maintenance responsibilities. For example, building packages, maintaining CI and similar. So basically every team received a bunch of these long-term maintenance responsibilities and also each team received a specific feature set because when we are thinking that we cannot choose, we cannot split the responsibilities by components. We did not consider the time that we can actually split it by features. So what we said that each team is responsible for certain features and when a bug comes, it is now immediately clear which team should handle that because the bug touches some features. Yes, you can have like an ability. Like you don't know which, like it may touch two features, right? But in these cases, the product owner can just talk together and decide which team will handle that and that's settled but in most cases, it is clear and it's so far where it's for us. So let's go to the next lesson. So lesson number three. When we started with all those things, we just have really, really long backlog. Also, we saw that everyone is just focusing on their tasks and yeah, if you want to find some documentation, it was very hard because we have many places where it was stored. So we saw that we need more transparency to our work. From a backlog point of view, what we did is that we created the definition of done and we provide definition of done to each task which we have. So on the end, if somebody should review it, there were a note what is really what we want to have. So it was very nice. I think that Petra can say what else can help us with these tasks. So basically when you listen to all or various of agile talks or read some books, you will learn that tools doesn't matter. The important part is the mindset, the way how we think and how we work. And this is true but if you have a distributed team on four continents, you really need some tools and these are just an example of the tools which we use. I won't go into details because I think that you mostly know that. So we use Jira for tickets, Confluence for storing documents and until we will have some holograms, we will need to replace face-to-face communication with the video conferencing. That's why we have Lujins and for IRC. And we use IRC for everything else. But you might actually wonder how could you manage tickets with three teams and this is my, isn't more interesting question. So what we did is, because what we want to achieve is that each team has own backlog which is ordered by priorities. But if you have a one project, like the tickets just came into that project, like it doesn't magically divide between those teams. So we basically work with one Jira project which has one backlog. But what we do is that we label each ticket with a team name. So basically then you can go to some Jira views, for example, a backlog view or some, let's say Scrum Board. And you can easily click on a filter and it will show you only the backlog for one team. And this works quite okay. So everybody can look what each team does or you can look on the global picture. So next lesson. Thank you. So lesson number four. On the beginning we saw that there are many communications one way or one to one. Also people have some time, PTO or something like that. For example, right now we are on the conference and not on only three of us but I see many of my colleagues here. So it means we should somehow manage those things and also if somebody is missing in the beginning, the knowledge was missing or two was really different. So we need to change mine mindset and Dominika can say something more. Right, so for some of you here in the room when you look at the slide here, this might look like scaled Scrum framework with the specific community involvement flavor. This is what we do in Red Hat. We cooperate with the community on the bottom and the teams communicate backwards, right? What I'm calling this diagram basically is getting things done diagram. You can see here with the green lines where the communication is happening. On the first level, we communicate with community. It's important to remember that we communicate with the teams and this is sometimes mistake that is done, it starts with the team. People go to DevCon and actually don't update their tickets. Nobody knows what is happening. After DevCon people get sick, still not updating their tickets and we don't know what is going on. So we promise to each other during their respective that we'll make sure that at the baseline, we'll make sure that we make the again for the third time, sorry, we'll make sure to communicate on the team level. We use tools for that, right? The Scrum Master, I'm actually the Scrum Master for all three teams. So we're doing this in the parallel. I help them with facilitation of the meeting but if that communication of the level of single person wouldn't happen, this is when we fail. What happens next? Team communicates, right? But the product owners also should be communicating to stay in sync and how do we do that? The guys actually meet every week to discuss not only priorities but also how they can potentially help each other and unblock each other and make sure that the work that the teams are doing actually completes. On the top of that, we have communication with the stakeholders. And you can see there are many stakeholders. People manager fell into that group but we also have their documentation team and Red Hat support teams. We collect feedback from all of these people to make sure that we are being successful. And the stakeholder meetings happens every first day. This is done for a reason to make sure that the product owners then again on Mondays can discuss and make sure that everybody is in line and be ready for the grooming meetings with the teams to re-iterate this message. And this is how we work. There were some lessons learned but we are still learning every day. And there are things that we'll be not able to solve 100%. And that following lesson is something that many of the scrum teams are struggling with. And I'll hand the mic to the guys to explain more details. Thank you Dominika. So the lesson number five is unplanet work. Actually, it is really very crucial because if you don't want to address it, you are in a risk of no-go burnout, which is a very, very bad thing. So what we are doing? First thing, we don't plan to full capacity of our team. That's the first thing. The second thing, if we really see something which is so important that we should put it to our current sprint, we just remove one-to-one, which means that we remove something with the same size and then we can put it into the sprint. Also, what's necessary is reach out our stakeholders and just say, hey, we did this change in our sprint because there were some reasons because they should be aware of it. Yeah, another thing which is also very good is plugging the deadlines to our stakeholders in our meetings. It is because then you shouldn't be scary about some deadlines which just appear tomorrow. Yeah, tomorrow, not yesterday. Yeah, so that's the thing. And also if you follow those rules, you do risk mitigation and it is also about communication and how we can see transparency. Let's better to talk about last-land lesson. The last lesson is basically what we was talking about, the whole thing, because it actually takes time. Because as you heard, we started in November 2017 and we did not start with the green field. It was a project existing for multiple years. People were used to work in a certain way. But we just jumped into it and tried. And we needed to adjust during the way, like as we said, in March. We got our scrum master because we needed it, Dominika. She helped us with recognizing some issues and we together created a plan, how to improve staffed. This plan was executed in November, so we split the teams into more stable ones with vision. We created product owners from the group. And we still have a lot of things in front of us. You might be thinking that scrum cannot work for you because I don't know, you are distributed, you are big or for other reasons. And the truth is that it actually can, but I think that for everybody, the journey might be different. You might learn different lessons. But in the end, let's just try. Thank you. And now we have time for questions. I know that all of you want to ask questions after that. That was intense. Some of them might be in that journey already on the other end. Some of you finish that journey for some reason and move to Kanban. So are there any questions? What we are doing is weekly groomings and repeat the question. So how the work from upstream gets executed if that takes priority of all of other work planned for the sprint? Did I get that right? Yeah. So we were also asked if we are prioritizing that over NFI fingers and basically replacing the items that were selected as the current goal for the sprint. So it depends what it is, right? What is the priority for that? Because if this is something very important and aligns with the sprint goal or overall strategy, sure, we will be working on this. But in general, it takes discussion during the grooming meetings. What is the approach for that? And this is not the thing, what is the best approach for that? And this is not a single decision of the product owner. We actually have a round of discussions. So this will be not happening overnight. If you're a contributor waiting for something to happen from the IDM site, we'll be not doing this on Monday after DevCon. We need to discuss it first. Did you want to add more? Yeah, definitely. Also, as Petra mentioned, like we plan for unplanned work by limiting the capacity and doing upstream work is part of the job because we don't only handle like some maybe pull requests but there is also a users list. And part of the job is also help people on the users list and this is nothing which can be planned. But what can be done is that it doesn't usually create big surges of incoming requests. It's usually each week or each month. It is more or less the same amount as the last month. So we actually can predict what will come. So the question is that with three teams, how do we do retrospections? How do we proceed? They don't know. No, what we do is after the sprint review, we meet separately every team, one after another for various reasons because we have like one team with Trivino who sits there. Thank you for coming, Trivino is one of our product owners. It makes sense for them to go as last because they're all based in Europe and we prioritize other guys like in India. We need to make sure that it timely, it makes sense for everyone to join this meeting so they're not stressed that they have to go home. We do that every three weeks. And then what happens? We share this feedback. We share this feedback in confluence. Everybody can go and read what did we agree on. So if somebody from the Parar team, Raven or Falcon is interested what Eagle team for this example is doing, they can go and read and then they can discuss with each other if they feel like they wanna discuss. What we did here right now after almost six months after introducing product owner role, we had a joint retrospective with the cake all together to discuss how strong do we feel like as a HR team. And the guys were basically answering yes or no to the basic questions. What we did is we created a barometer of the things that we wanna discuss. And then over the cake, we're basically discussing what do we wanna improve as a team to make us stronger. Thank you, Pavel. Anybody else? Yes? No, the QE organization is separate from the engineering but we're trying to make this work together. So what is separated there is a people manager, right? This is a good question, why? This is the specific setup that we are going to the world with the soldiers that we have, right? So in this case, we are just trying to put everybody together in one team and setup. But this could work, this could vary, right? In your company, this could be different. You could have everybody in one organization. This can possibly change, right? Nothing is set in stone. So do you want me to answer why does it matter who is your boss? So we were showing you this, right? Bosses might be having a different priority. But once you are adopting a specific workflow and everybody is on board and you should start with your management, is that their first and the biggest job is to support you. So if you are implementing basically any of the agile methodologies, including a scrum workflow, it's important that everybody is on board because if only one side will be trying, this is not going to work. So the initiatives can happen bottom up, like it happened on the example of this team, but it's important to have management support and it's even more important to have a buying if the organizational split is similar to what we have in Red Hat right now when the engineering and QE is separate. But we reached the consensus that everybody is agreeing and supporting us right now. So that was the first thing that we wanted to agree on. When the team had maybe recall on that linear slide, when we look into the improvements, what can be done in the team, everybody was actually put into the room and to confirm that they are agreeing with each other, that they are going to support this structure. Any other questions? Yes? So the question was that if there is any shared code or library, how we are handling the dependencies. And this is difficult and at the same time easy because most of the code is to certain degree shared. So we split by the feature, so people can just work on that, but in the end, it all goes as a pull request for example to GitHub and it can be, it is usually reviewed by the members of the same team, but it can be also reviewed by members of the other team as situation allows. So there are actually no clear boundaries like who does that, who does what, it's more about like we need to decide who does it, but it can be done by anybody almost. But the reason why we actually split the features like assign each team a feature also to share knowledge because what you want to avoid is silence and if everything is responsible for everything, then it might then be more difficult to share the knowledge because knowledge is easier to be shared within smaller group than within larger. Did it answer the question? Yes, we have all product owners attending their specifics. The question was if the product owner is attending the sprint planning, if he's having an impact on the sprint planning or if the only team is doing this, correct? Does it happen after the sprint or before the sprint? So what we do is like every Monday, the guys are meeting and they pre-groom the items with the PO who is getting feedback from the stakeholders and then every three weeks the sprint is planned. So they're not going on the discovery of what to actually work for next three weeks. They have a groom list of items and they just clarify things on the, items that were already created. So do we all understand this the same way? What are the acceptance criteria? Is everybody clear on the same page? And then this gets selected for the sprint. So this is a gradual planning process. I mean grooming process, sorry. Any other questions? This is possible but what we are trying to do then when we have a more complicated task with the more dependencies that might be affecting other parts of the team would split this task and actually use this feature in JIRA to mark the dependency so you can actually see that this item is dependent on the other one. What item is that? In which sprint and that the guys have to complete this and they also have this visibility. Hey, there are other folks that are being blocked by this. So we'll be most likely prioritizing this. Sorry, the question was that it is possible with that complex product that had basically 10 years of the development in the back that some tasks will be shared between teams. And then, right, how do we deal with that? So we will try to split this dependency, make sure that the task is actually smaller assigned to the team and map the dependency in JIRA that this part requires to be done first and then the partner team can act on it. And if I go to the right part of the question also was how to deal, like, thinking of the dependency between teams, right? And for this purpose, what we use and maybe Patrick can talk more about it is that the product owners has a sync meeting on Monday as I mentioned earlier and they can talk about these interdependencies so they can synchronize on timing of those changes. Okay, we have two meetings. One is with our stakeholder and this is one of them. And they just provide us what we should do and they also provide us the vision if it is important or not. Then we go to our meeting with product owners and we can discuss it. And we really discuss it. We really see if we have everything that we should have because we have still time. You know, sprint has three weeks so we have two another weeks to discuss it if everything is clear. And when we have consensus, then we go to our teams and we just provide information to our team. And our team also can say, oh, wait, there is something that is missing or we don't understand this dependency or something like that. And then I just collect this feedback, go back to my colleagues as product owners and I say, hey, there is an issue. And if it is bigger, then we collect our feedback and we go to our stakeholders and say, no. Or we just say, help us please. This is more common, yeah. Other question? So the question is, who is accountable in that structure for delivering work? The answer to this is that whole team, it's accountable. So it's a very important first thing, not measure the productivity independently as a productivity of one person. But every time we start a new sprint, we specify what is our goal as a team. And it's fine if somebody is finding something very, very exciting to work on. We get this. But we have a discussion between each other if that person is doing this. And this is on, as I mentioned, the stand-ups, Monday, Tuesday, and first day. We discuss with each other, what are we working on? Is there something blocking us? Is there something blocking the sprint goal? If yes, then we have a discussion. It's possible that during the sprint, something else will take a higher priority than what we planned because we have a customer escalation, for example, right? So then we try to focus on this and work together. But the accountability lays on the team. There might be cases of the low performance of people not fully participating in this model. But then what happens, the discussion between the person and the people manager has to happen. And we need to clarify, why is that not working for a specific person? Is there anything we can do to help you? Maybe more training is required. Maybe you are interested in working on the different things, not necessarily on this product. So it depends on the situation, but this is the general flow. I think with that we have like two more minutes. And to the end, we have a time for last question. No last question. So thank you very much for coming this afternoon and being with us and hearing our story. And I wanted to thank you guys, session chairs, for doing amazing job on hosting this room and making sure that everything goes smooth. Thank you.