 And so I'd like to weigh it. We have two fantastic slow birdies here from red hat three bean and might be They we'd like to welcome you I'm obviously between two beards and They have two fabulous faces of hair like I mean it is wonderful and before we get into Talking about continuous improvement. Let's find out a bit about each of them. So First, thanks Ralph. I know that you normally shave your beard off at around this time of year And I'm so thankful that you've kept it for this event So, but why do you shave it off in the middle of winter instead of summer? That seems crazy That's that's kind of personal, you know, you know, I actually stopped I just had to stop doing that Yeah, it's because you kept insulting me about it Well, you are too beautiful without your beard, I don't know if anyone's seen him He is very beautiful and might be you have any great beard trip tips for people. I do Beard balm never leave home without it. I take it wherever I go a little on there gives it a nice sheen highly recommended Excellent and you You know, you might want some some miracle grow and some Rogan and the top of my head too Yes, and so both of you have a hardcore rap names Mike B three bean Who's your favorite hip-hop? group and Give us an interesting backstory about why you have this cool name I don't know a tribe called quest very very very good. Yeah, or maybe you I don't know do you rap? My nickname so three being it actually comes from my my name people things because I had a kid There'll be three people in my family, but I'm Ralph being the third And your name is boring I'm just not gonna really ask you about it And so Ralph I heard you have many cats How many cats are in your beard right now? And do you hate dogs? Oh? Well, that's what's going around Mike be I heard you are an interesting scuba you have scuba stuff. Yeah, and do you keep your snorkel in your beard? Okay, okay, and what's your favorite fish? I guess who would probably have to be the humu humu nuku nuku up. Wow because it's fun to say Wow, I didn't expect that Okay, down to business down to business. Okay, let's you guys are both part of the amazing factory 2.0 team which despite your involvement has become highly functional and they work with consistent goals of self-improvement and Mike can you give us a rundown of the history of factory 2.0 and the products you've created? Yeah, the factory Factory 2.0 initiative started way back in the day from Conversation with some executives and they said we don't like the way the pipeline works So we want you to fix it. It was literally the only The only instructions that I got so we started trying to figure out what the problems actually were Actually did a survey of stakeholders Ralph was involved in that we Took the survey results had them vote on the highest priority ones. We got a list of about Seven priorities out of that and then just decided to figure out how to start addressing them and some people are from red hat So maybe tell them what does factory 2.0 trying to do so the basic concept is improving the product pipeline We have heard a lot of a lot this weekend probably about like monolithic monolithic applications that people are dealing with so We were no different Applications that have developed over the course of a decade or more. So we're breaking those down into microservices applying some modern development and deployment processes to the services in the pipeline and using that to accelerate delivery of new features and Enabling us to innovate more in response to the business And What are actually what are all the products that we now have I have a t-shirt with a bunch of them, but So results DB is very much as it sounds it's a database to store test results and centralize them provide an API to query them Freshmaker is a service which allows you which responds to events from other services and triggers other events So we use that for automating things like rebuilds Greenwave is a policy engine. In fact, it was originally named policy engine until we got told that wasn't interesting enough And it tells you when a set of criteria for releasing something has been met And then on-demand compose service also somewhat self-explanatory, but it runs composes on demand Okay, I'll have to go back to my screen soon, okay So cool and Sophie's choice Ralph if you can only save one product of all the products that the team has made which one are you going to change I? Can't I can't answer that I Can I have to it's green wave, I guess okay. It's green wave and you I'll say results to be because it's like the beginning of the start of it Okay, I might have to log back in Okay, and surprisingly some people actually use your products And so here's a slide of people saying wonderful things about them And I just want I don't know if Josh is here, but Josh doesn't look like that at all I think that's his tinder picture. I mean, where do these people get these pictures? Okay, what we're gonna talk what we're gonna talk about next is actually about continuous improvement and what we did with our team So First of all to get your team to have a continuous improvement mindset. They have to be a team and So Mike how did teams start though? so the team really just started with me and Ralph and we we came with the plan and Proposed that to stakeholders and they agreed and then we said okay. We're gonna need some people so we Just started borrowing people from other parts of the organization and we got some people from DevOps who were involved and So, you know explain the the concept and the approach that we were taking to them and Launch some some new projects. Actually, I should step back a little bit The first thing we said was well What can we repurpose that already exists to help us achieve our goals? And so one of the first steps was bringing in some of the pieces that already existed in fedora like results TV That was a component that had already been used in fedora as part of the task patrol on project And we redeployed it internally and said this is gonna be How we're aggregating test content So how do we like move on from there and build on that and then figured out some some new services that we needed to develop started a design process and Continue from there cool and and When I entered the team you guys were developing using scrum and then we quickly moved to doing Kanban and I said some stuff all the time and do you guys repeat after me? same sized work if you're doing Kanban everything has to be the same sized or else you'll fail and so but in order for our team to Be thinking about continuous improvement. We took some steps to make them feel Like they were a team so we worked on and we had goals of having them feel connected We had a goal of having it being fun to be on our team And we had a goal of them being technically challenged on our team Okay, so Mike do you want to talk about some strategies we use to help people connect? Yeah, we were always a distributed team. It was sort of the nature of you finding resources where they were available and so we would We had video conferences as most distributed teams do we encourage people to you know, actually be present on their cameras We would do various things to make sure that people understood What we were trying to accomplish and what value we were bringing as part of our efforts I think it's really important so that people feel the value of the work that they're doing and then Yeah, we made sure that they the team understood who the stakeholders were What they were trying to achieve with you know with every sprint and then we had longer-term quarterly milestones about you know Actually delivering Yes, we would we would go back and forth between different time zones that was another challenge with distributed teams is that we had people in In Australia and Bernou and Beijing east coast west coast of the US So time zones were a challenge and so we try to spread the pain out and Switch when we did retros so that Everybody could be inconvenienced equally Yeah, we did face-to-face is whenever budget allowed we did a big one Associated with flock when it was in high-end sms juice and so you may have been there And that was really great. It's it's really valuable even though you might see people on a stand-up or Retro every you know every few days every week getting people together in a room and being able to talk and have really high bandwidth conversations was very valuable Thanks, and And then Ralph although you are fun challenged You wanted the team to have fun at work So what are some of those things that we tried out to make that happen? Yes, I'm not not fun challenged in general It's just the the mandatory organized fun that a little resistant to but but we did we did do some some cool things One of them was we had a we had like a funny hats challenge week So like stand-up was getting a little dry We were doing the same things over again So we said we're a funny hat to stand up But on a surprise day and so people would start stand up but they turn their cameras on and we have a good Chuckle at the beginning of stand-up. I think you won that Neil. Yeah Yeah, it's tremendous hat covered in owls. It was like a dr. Seuss hat We did another thing where we did a team team trivia, you know We we work a lot with each other and we understand each other's technical preferences and what we're focused on But we would we had a people submit questions and answers about themselves and then assembled it in big Maybe 20 question survey that we put around a team You know, who's the member of the team that has this one attribute or another and and then we reviewed them in our Retro, we just basically scrapped retro for the the whole the whole sprint that week and then just like one by one Systematically unveiled the answers of people guessing and betting on who was who was who in the team and it was cool We got there, you know, learn things about each other and get that background So yeah, I think Mike you had shoes that melted in a volcano. Is that right? And Mike you Like to be challenged yourself technically by waking up and turning a computer on But in terms of people that actually do work on the team, how do you keep them challenged? So we have a lot of different areas of Work that we were involved in so we would we would probably just ask people like what is What which one of these services are you do you know the most about which do you know the least about? What are you most interested in learning more about? So just being engaged with the team and understanding what their interests are and where they feel like they could grow we also created very recently actually created a apprenticeship program so that people who were more familiar with a code base couldn't mentor people on a Who are less familiar with it and bring them up to speed? Which is great for not only reducing the bus factor, but also getting people involved in in code bases They weren't familiar with and spreading the expertise across across the entire team So not getting not having people be siloed in one code base or one particular area of the pipeline Yeah, that's a pretty cool program We had and we started it with a survey so that people volunteered for getting into products that they weren't able to get into and we had and so we had people Volunteer and it seems again. It challenges them technically We also do surveys quarterly to see if people are satisfied with the work They're doing to make sure that they still want to be part of the team right because we don't want them to be leaving Okay so So we worked on becoming a team that was that's really important And the next thing we worked on was what is our vision? so so you can't get to somewhere if you don't know where you want to go and Mike Pete you were responsible for coming up with some of our vision our secret vision of having three cats live in your beard has worked out but What is the factory vision? I? Touched on it a little bit earlier, but the overall vision is really to To move what had been a pretty legacy style pipeline into into more modern best practices of of software engineering deployment so you know using microservices moving toward continuous deployment Continuous integration across all of our services and using that as an enabler to be more more reactive and responsive to the constantly changing needs of the stakeholders and the business and So we have this overall vision of how factory 2.0 is going to transform our pipeline And then we also have something we call ok ours or on our team We call them original crazy raps and so Ralph you What were some good or bad raps that we've used you might really want to tell what people what they are They weren't raps that ok ours the objectives and key results and so this was a way that we ended up trying to keep the team Focused and have a direction and that have something that was also measurable We would know whether or not we were making progress towards it and so in the beginning it didn't didn't really work out Well, we took what were our epics in Jira the major chunks of work that we were working on And so one of the tools we were Is on the shirt is fresh maker and we had a big fresh maker epic in the beginning But the thing is is as work continued and as we got new information and new feature requests coming in that epic Just continued to grow and grow and grow and there was no point when it would ever be done So that was about the point in time when we stopped and said we need something that is measurable And that's gonna have gonna have an end and so we we established some ok ours Like we need this feature set to be delivered by to be delivered by this point in time But it ended up being like like abstract and disconnected from what the team was doing and things really transform We took the ok ours from just like a kind of project management level and brought it down into the team and sort of assessing on a week to week basis and stand up like How are we at moving towards this particular goal at times less than just focusing on all the new cards that are coming in and all the new cards that were That we're working on we ended up doing some some process improvement ok ours there in the mix to select summer feature driven We want to be able to we want people who use the pipeline to be able to do this new thing And then other ok ours would be like we want to change the amount of time that it takes us to get through code reviews to reduce that by We'll say 10% in this quarter over the the previous quarter and those two when we would just keep them at a tracking level And we weren't talking about them in the team proved to be of no value But it's only when you bring them in and work with the team to talk about them on a regular basis that they they tend to have that that feedback loop Yeah, and I think this is a bit of the secret sauce for how the team has helped keep vision keep focused and work on things was how They tie every quarter the work that they're doing to something and change it and work together on what that is and when you when you pull work you Know that it comes from like it's attached to The epic that you're supposed to be working on sort of thing, right? and Okay, we only have five minutes left so Mike You're gonna talk about how we use how ok ours although help prioritize the backlog though Sure so we when we started thinking about ok ours we really wanted to Make sure that we were Tracking things that were visible to customers so that we're actually making a measurable impact to the people that were using the Pipeline so you know internal engineering QE People that engage with the pipeline so we That would help us you know focus on what was what were the important things that we had to work on right? We like every team or we have a ridiculous amount of requests that come in from from all sources Even things that we want to fix you know technical debt Bugs with the services request for new features And you know that all gets into if that's all just piled up in a backlog It can be pretty daunting to figure out what you should actually be working on next so When in the context of ok ours we were able to think about what are the Those things in backlog that are actually the most The most painful or like the most pressing for our Customers and our stakeholders and then organize those into larger pieces of work that we could then address Over the course of you know a month or a quarter and deliver and show real value to the to the stakeholders That was helpful Yeah, again, I think one of the big secret sauces and so we have a team feel like a team we have You know set up some crazy wraps and now we are doing some experiments and seeing if they're working and especially with process changes so we were doing Ken Bam and We analyze our flow and we monitor our Q sizes our whip and Our throughput and the cycle times if you want to find out what those things are and what those mean talk to me afterwards but basically it allows us to see our work and We really watch where our inventory builds up and look for where we're having problems where that's happening and We had a great Thing where we were watching this so Ralph if you want to talk about the deployment whip breakage And then what we did about that Yeah, yeah So so as a development team primarily a development team We have some tickets and when you work on a ticket it moves through various phases You go to in progress you do some code review and as a development team after code review you're done Right, you close the ticket because it's done But for us as it as a DevOps team, that's not true It actually has to make it into production and for our team in particular It's even a little bit more complicated because we were developing services that we wanted to deploy to the Fedora production environment for Fedora's use cases and Also to the internal production environment to produce to produce red hat products And so it ended up being this sizable chunk of work for us deployment has always worked But for us it was double because we had to do it twice and it was this hidden work We were thinking we're working on these cards. Why aren't we making any progress on our development cards? Oh, it's because everybody is drowning in deployment times So we changed the way we did can ban and added new columns on the side for that a card isn't actually done until it's deployed In production in both circumstances and what we knew to be true already kind of intuitively that we have a whole bunch of deployment stuff going on We could see visibly on our board So when we get to stand up and there's some cards in progress and there's like 25 cards depending on pending deployment We're like why spend any more time adding new cards through code review and getting them into the queue when we just need To get this stuff deployed to get it out and making that work visible as a part of the tool that the team used to talk about their work Every day was was was key. You know, if just you or I was monitoring it. It wouldn't make a difference Yeah, and we definitely saw an improvement from I think 20 days to get stuff to deploy to about 10 days or Less now, right? Oh five days now. Wow Amazing. So that's it. I don't know if we have we might have time for a question, but we have one minute left so Thanks everybody for coming and thanks for the beards for putting up with my jokes and I hope you had a good time Any questions? Okay. Thank you