 It doesn't work, it doesn't matter. All right, so, whoop, hello, welcome everyone. My name is Fernando, I'm a senior software engineer at Red Hat, I work on the networking services team, and today we are going to talk about pay programming, a technique that we have been trying in our team, some context. So I work on Red, and especially network management tools, tools like network manager, NM state, network role, NISBOR, and all that. We're a group of six engineers, one PO and one manager. As I say, we work in four different projects, with different backgrounds, different environments, different CI, packaging process, everything is different on each project. Then we are a full remote team. So we have like four or five time zones in total and the time difference is too much because we have engineers in US, Europe and China and it's really, really hard to get them together. And then we have been starting to do, of practice, agile methodology. So we started with a scrum. We have two weeks of spring. It's quite great because we have been able to idealize everything. And well, that's some context. So we noticed several problems and we decided to address them. The first one is that we needed to extend collaboration. Well, we noticed that most of our engineers were used to work alone. That is good, but at the same time, it could be a problem. For example, when they get stuck, usually they don't ask for help or when they need something from another project or someone else, they are not used to say, hey, will you help me out? Hey, could you let me know how this works? And that's just simply not efficient. So we wanted to solve that. Then we wanted to avoid both of the next situations. Right, we have six engineers, four projects. You can imagine that there are one engineer that knows about one of the project, two engineers that know about one of the other projects. Et cetera, et cetera. So when we have this kind of simulation and then suddenly one of the engineers comes to a meeting and say, hey, I'm going to be a father. Right, so I'm going to be on leave for four months. Great, how is it going to do your work? No one knows about your project. If some issue came around, what are we going to do? So we started to lend all the projects really quickly and that, believe me, doesn't go well. And we wanted to avoid that. Also not because of lonely, lonely, it could be also because of PTO, holiday or whatever. And believe me, when someone takes a PTO, that's the day when the customer report the most important bag ever. Then we wanted to extend the sense of ownership. So, as you say, we are doing a sprint, we create a sprint, we assign the task and we start working on that. So we know there's something. The engineers usually solve their own tasks. And after that, instead of looking at the ball and say, oh, something is not complete, I'm going to help or I'm going to take it or whatever. They usually say, all right, let's take something else. So we wanted to extend the sense of ownership. So all the engineers should feel like they are owner of all the projects. So when they finish everything on the sprint, they should be able to go to all the tasks that are on the sprint and say, hey, I can't help you with this. I noticed that you didn't start with this, so let me take it. And we can complete the whole sprint instead of adding more stuff to the sprint and then some other engineers not be able to finish everything. And therefore, the scramp is a little bit messed because you are not completing any sprint at all. And then the other formula was the bonds between beliefs. So since COVID, we have not met. And that is around three years ago. And that's a big problem because in the end, when you are working with people, you need to trust them. You need to be confident about them. So you need to have a bond with your teammates. This bond is really important, especially when you are asking questions, especially when you are asking for help, when you need to discuss a specific technical topic. Imagine that you have an opinion about technical topic, the other person have an opinion about technical topic. If you don't have a bond with this person, probably there can be a lot of misunderstanding and you can taste things personal. It's hard to know each other well if you never see each other faces. And especially if you are working all remote and then you just have one call a week and in which you say, hey, I'm doing this, this, this, and this, okay, see you. Right, it's quite hard. And also it's good to feel support at work. If your gut dies, it's great to have someone that say, hey, it's okay, I can take your tax for today. No need to worry, I heard of all your cards. Try to feel good. So how we decided to improve this with paper whammy. So one day I was talking with my manager and he said, hey, have you heard about paper whammy? I was like, no, take a look. So I volunteered to start implementing paper whammy in our team. So I decided to look over a lot of documents, talks, et cetera, about paper whammy and try to understand how it works, what are the advantages, disadvantages, how we could implement it, how to implement it, what are the different styles, et cetera, et cetera. And after doing that, we are here doing this talk. So I hope it's going to be, my research is going to be useful for you. So paper whammy, the basics. Basically this is always a driver and a navigator. So it's just that simple as two person, two programmers usually get together and say, let's work together on this specific issue. They see it and they stop working on it. There's one condition, just one keyboard or one computer. So now it's not two people programming at the same time. That's something that we used to do even remote. But it's two people on a call or in person looking at the same screen saying, oh, okay, the driver is the one that manage the keyboard, do the coding, et cetera, et cetera. The navigator needs to see on the bigger feature. So for example, if the driver is designing a feature, the navigator should say, oh, but if we do this, we are going to break another customer case because we know that there is a customer using this other option, which is a conflict with this one. This kind of thinking is the responsibility of the navigator and also is reviewing the code all the time. So if you are programming in C and you do a malware, then you need to do a free. So that is part of the navigator responsibility to stay looking at the code. And if the driver doesn't do it, you need to say, hey, you forgot to do that. So right, this is the basics. Then the switch position constantly or frequently. The idea is that they can, after five minutes, 10 minutes, 15 minutes, it depends a lot on your style and your preference, the switch position. And the driver becomes navigator and the navigator becomes the driver. I must say that this can be done with experts between experts, between someone new or more junior and an expert between junior engineers. It's beneficial in all the situations. So there are different styles. There are more of them, but I think this is the more important one. So the first one is unstructured failing. Besides the usual driver navigator, which is the one I described, there is unstructured failing, which is very similar to what people doing in an office. You have a problem, you are working solving it. You go just, you ping your teammates and say, hey, I want to fix this with you. Sure, let me help you. And you start working together. In this case, there is no structure, there are no rules. Maybe you decide to do computers. Okay, that's fine. The idea is that you work together. This is the simplest way to start doing it and this is the simplest way to start implementing it in your team. So we started doing this, unstructured failing. We just get an issue. I think a teammate, hey, let's work on this. Perfect, let's get into it. So like I say, I'm working on this problem. Would you like to work with me? They can't see through it and they work together. This is quite simple. Then we have ping-pong-pailing. So if you are a fan of testing and you're a fan of test-driven development, if you don't know about this test-driven development, it consists on first you develop the test that should pass and then you develop the backfix or the feature that needed so that test passes. So basically one person go to the other and decide to do pay programming and they decide to do test-driven development. So if you don't like test-driven development or in your projects, you don't have test, you should, but if you don't have test, you cannot do it. Then the first person create a test that is failing after that as a driver and then the other person as the navigator. And after that, the second person, the navigator, change position and create a code or the implementation that makes the test to pass. So then they switch position again and this time in this case for the second person creates now the test failing and the first person create the implementation so the test passes and they repeat. This is quite good because at the end, you have the feature for the backfix and tests for them. So you are doing two work in one. I must say that this, for my point of view, it's more adequate when you are working between two experts because it could be quite hard for a new bike to create a test for a project that they don't know and the test must fail. So it's going to be quite confusing for them to create something that won't work having on mind the idea that in the future it will work. But between two experts, this is a really, really good scenario. And then we have the backseat navigator which is perfect for a junior or a new bike and an expert. The idea of this is quite similar to a driver navigator but in this case, navigator is not thinking only on the bigger picture. He's thinking on what is the driver going to do? So it's like when you are driving and you have someone by your side saying, oh, now to the right, to the left. Now, go straight. So it's basically the same but programming. So the first person go to the other, sit together and do a paid programming session. But this time, one of the persons, for example, to stay bought, and go to the driver and highlight the navigator. And the navigator is going to give direct instructions to the driver. So this is really, really good for new bikes because if the new bike is the driver, it's going to get used to the workflow, it's going to get used to the files, to the structure of the architecture of the project, to, I don't know, the CI, to the tools that you are using, et cetera, et cetera. And they also found that this could be quite good for an interview. For example, imagine that you are interviewing someone and you say, okay, so you are the navigator now. Please, let me what I need to do. And that could be great. These have a lot of new bikes on the team. So these are the three main ways of doing paid programming. But now we have some very, very important considerations. The first one is that both programmers must be active. I mean, they must participate actively because if not, you are basically in a mentorship session telling them or explaining them something and the other person is just in silence. That's not paid programming. Or if the other person, you are programming you're the driver and the navigator is looking at the email, believe me, that is not paid programming. This is like you are programming alone and there is another person in the Google called listening you programming. That's all. Both must keep up a running commentary. So it's similar when you say paid programming, it's also programming out loud. So when you are the driver, it's quite hard to, for the navigator, it's quite hard to understand what are you thinking about if you don't talk. So the driver must always describe what are they doing. So yeah, I want to implement this. I want to create a function that is going to be added to this class and this class is going to be in this place and we are going to add this event to the loop and this way the navigator is able to think about your idea and say, oh, but then if we add this event to the loop it's going to fail because blah, blah, blah, blah. Another thing is that relationship between the programmers are important and managers, leaders, or whatever person is implementing paid programming on the team must not force them because if you have to engineer that they don't get along, believe me, it's not a good idea to force them to work together in a paid programming session because they are going to have conflicts and they are not going to be good at resolving them. So it's really important to first solve them. All right, so the result of our experiment. We have been paying programming for a couple of months, three months or something like that, or even more. We have been doing regularizations and initially it was quite hard to get people used to it. Like, we proposed it and it was like, yeah, we could try it out. The people were not very convinced but then we kind of forced it, like, yes, let's do it. And I volunteered to that so I started to propose sessions to several engineers and I assumed that they are kind and polite, they didn't reject them. I hope, I think that they were considering but they didn't do it. But after a couple of sessions we got positive feedback and say, hey, I like this. I really enjoy it, it was good and I would like to continue doing it. So now the engineers are starting to propose them to all the engineers without my help or without any other people's help. So that's a great thing. And we make sure that everyone in the team had a meaningful pay-per-money experience. So that means that almost everyone were able to have at least a session or a couple of sessions to understand that. We are also flexible so that means that we are not going to force anyone. So if one engineer say, oh, that's not for me, I don't like it, that's great, you don't need to do it. This is like using BIM, Emacs or whatever, it's just a tool and you suggest it's good for you. And also, right, so some advantages. So first of them, now the sharing, it was incredible. I mean, doing this I get a lot of experience in all the projects we already managed, which is quite great. And all the people, all the engineers from the team doing this are getting that knowledge. It's really interesting to see the different scripts that people use, how they build the system, how they build the package, the CI. It's really, really good and it's really nice to have someone explaining to you why they are working on it because, yeah, it's really easy to say yes, it is really good. And the docs, you see, last time modified, 27. Right. Okay, so then we have focus mind. Other than that, it happens to you, but at least it happens to me. When I work alone at home, I'm on the computer and say, hmm, what am I going to eat today? Oh, in the afternoon, I need to go to the grocery. Yeah, oh, look at the doc. Hey, doc. So it's quite hard to keep focus for a long time. But when you are on a meeting and you are working actively with one person, that change because you have a person and so you're not going to say, hmm, what am I going to eat? No, you are focused because you are waiting also all the people's time. And you think, oh, but then I need to work more because I'm going to be more focused on the job so I'm going to get more tired. Yes, but you are going to do the work in less time. I mean, if you really focus on the task, maybe you can do it in one hour, but if you are thinking in all the stuff, changing context all the time, maybe it takes three hours from you or two hours. Then we have resilience to interruptions. I know that your email probably looks like this. Mine one looks like this at least. So when I work in something, I get a Slack message. I immediately go to it. I read it and I start to engage a conversation. Then I go back to my work and then I get another Slack message. It's terrible. And when you are on a meeting, working with someone you're talking all the time, you get this Slack message and as you need to keep talking with this person and focus on the job, you are not going to leave it. And then you learn to ignore them. And this is one of the best things that I learned during the program. That you just learn to ignore all these messages. And one extra thing, which is the code quality because at the end there are two people looking at the code and four eyeballs, three more than two eyeballs. And we even use the coordination efforts. So when working together on a program session, everyone were aware of the context. So all right, so this issue is blocked because the last time we were working we noticed this is a problem. And then if someone has, oh, what is the status? You don't need the only person working on the issue to reply, you can because you already know the context of the issue. So we reduce the coordination effort. But we noticed some disadvantages. So the long time, right. It's good to work with peers and it's really great, but you also need a long time. You need time to reply to emails, phone calls, whatever need you have on your position or even work on a simple coding issue by your own. That's great. So I don't recommend you to do 24 hour pay programming session. So translate all your work to pay programming. I think that's a terrible idea. That's not going to go well. You're going to get banners immediately. So try to do not do that. Try to set amount of time that works for you and make you feel comfortable. Then Fabi, so it's easier to take last time when you are working alone because you don't feel like pressure to continue working. So you say, okay, I'm going to drink some water. You go to the kitchen, you relax a little bit, drink some water, think about the problem. That's great. But when you are with someone that you might feel uncomfortable saying, oh, wait for me. I'm going out for some water over there. So in order to avoid this, I recommend you to create like if you are going to say, we are going to have two hour session or one hour session. Okay, so let's schedule one five minutes break after 30 minutes or 10 minutes break after one hour. So this way you have time to relax and then come back with more energy. And skill levels. Skill levels could be opponent. If you put one junior engineer with a really, really experienced engineer and for someone that is new on the team, that new person is not going to be able to speak up. So they have a great idea and they don't feel confident enough to say, hey, I'm the navigator. I'm thinking what you are doing is wrong because the other person have been working on the program for 15 years. How cool I know more than this person. Sometimes fresh ideas are good and you could have a different mindset that this person and you can bring a lot of value to the project. So in order to avoid this, it's really important for the expert on the session to promote the behavior of speaking and make sure that the newbie or the junior feel comfortable enough to ask questions and also challenge the idea. Like if you say, for example, all right, so we are going to do this function and add events to the queue and the junior is the navigator saying, mm-hmm, then you say, why do you think this is correct? Or what do you think about the end? Force them to reply, force them to provide their opinion. Maybe it could be, I think it's correct because blah, blah, blah, blah, blah, blah, and then it's accurate and good, okay? But maybe they say, no, I don't think it's correct because this is going to have a conflict with all the part of the code. And do you think about it and say, oh, that's right. And the first thing that happens, the junior feels more and more confident and the next time you are not going to force them to reply, they are going to reply by their own and that's really great. Conclusion, so we were able to extend collaboration. People feel now more comfortable collaborating with each other, which is quite good. And we avoid bottleneck situations. We were able to understand the context of the whole project, so people now can contribute to different projects and if someone goes on PTO, leave or whatever, someone else can take their position. And one thing that went well is extend the sense of ownership. It wasn't fixed at all. We improved a little bit, but we are just working with this, so we are trying to fix that. Maybe next year at that point, there's another talk of how to fix that, I don't know. And then one thing that improved a lot is the bonds between the code. So thanks to pay programming, I learned a lot of things about my teammate. And I mean personal things like, oh, what, do you like to play an instrument? Oh, I do this, do you like video games? Do you like working? And then you notice that you share hobbies and you share information about these hobbies and I found myself talking with them about random stuff, hobbies or whatever, after work. Like, hey, I saw this movie and I really like it and you like the other one, so maybe you like this one. So that's great because now you have a better connection and it's easier to, when there's a problem, it's easier to challenge the idea and to share different opinions and get an agreement because you understand the other people's context and also because you have a personal bond with that person and usually you care more for other people's feelings and other people's ideas when you have a bond with them. Maybe the solution for a lot of learning is to have a bond with customer, I don't know. So all right, what I recommend is to try it out. If you want to try it out, we have a workshop of Peiko Rami today. So please feel free to join. We are going to have some hands-on experience and all of them that if you are on a team and you are a manager, a software engineer, dimly, clothes owner, whatever position and you want to try this out, feel free to reach me out or follow these advice. Don't force people if they say they are not willing to do this. Try out first unstructured painting because it's going to be much easier to implement. If they like it, then you can change to ping-pong, you can change to whatever other method that you would like more. And yeah, the last one is the make, if you want to go a manager and you want to implement this, talk with one engineer that's have some interest and say, okay, you are going to promote this on the team. And this person needs to see some talks, read documents, et cetera, et cetera, gets a full form of opinion and try it out on the team. All right, so I think that's all. Thank you very much. So any questions, suggestions or comments? How does this thing work for you? With what, sorry? With code reviews, learning questions, and so on. Right, you counted it's pretty accurate, but I repeat it once as you reported for the hour. Oh, okay, perfect. Yeah, so the question was how it works with code reviews. So with code reviews or managed requests, for request reviews, it works well. I mean, you can, in the end, you can apply for programming to almost everything. You can apply for programming for a feature, backfix, or even debugging, which is quite useful as well. But also for code reviews. I mean, imagine that someone, there are three persons working on a project, and two of them decide to review the other person code. It's quite good because both of them at the same time, so it's parallel review, so that's great. And at the same time, you can do it with the person that leaves the code and ask questions. So you get immediate feedback. When you write down a comment in GitHub, like, why did you do this? Then the next day, the person replies, I do this because blah, blah, blah. Oh, I don't understand exactly what you mean here. So it's a really slow process. But if you go to this person and say, hey, let's review your code. Hmm, you really, okay, so why did you do this? Oh, I did this because whatever. Oh, really, and why? Then you get immediate feedback. So that's completely... Yeah, but I meant a slightly different question. Oh, sorry, right. Do you regard the code which is produced through programming as being already reviewed? No. No, when you review the code, someone else out of the pay need to review it. So, because also when you are working on the issue, you can get by us, like, yeah, working on that. So it's always great to have many reviews as possible. You can get 10 reviews. If that's possible, please do it because usually someone is going to spot a bug or a problem in the code or whatever, or something that could be improved. Thank you. More questions? Yeah, please. Is there a problem if you can't finish your task within the appropriate session? All right, I mentioned this is only one hour. You do it, and the next time you're going to task in the middle of the task, and then probably going to be computing. Yeah, so the question is that if it is a problem, if the people cannot finish the task in the paper-running time box. So, yes, I know. I mean, for example, you're going to start doing paper-running for one hour. You didn't finish it. You can continue by your own. That's completely fine. Then you get more reviews. That's that. Or maybe you can stop, take some notes. Like, if you were coding alone, when you're coding, at least on my case, there are sometimes in which I sit down, start working, and I don't finish the task. So then I need to start again in one day. That's also happened when you work alone. So it's the same situation. It's both solutions are fine. You can just continue with the paper in another day or in another time, and just think what we did. Oh, we did this, we implemented this. And sometimes that is also useful because you read the code that you wrote together and say, oh, but this is wrong. So then you get just all that problem because you have a flesh mind and you are not tired of coding. And also you can just continue on your own. Mike, please. So you mentioned that the navigator and the driver would switch positions every 15 minutes. How do you handle like, like different maybe navigator and the driver have different IDEs? How do you update the stuff that you have been working on, it's local on the driver's computer. So now we have to like sync. You have like a draft branch or something where then the other person pulls and you know, continue for your house or stuff. All right, so the question is, as I mentioned that driver and navigator switch, how do you do it if they have different workflows or they call it local or different tools? So the thing here is that it's, you must be flexible. So depending on the situation or what you are working on, you can switch every 15 minutes, maybe 30 minutes. Maybe you switch each task. So you have three tasks scheduled and you say, okay. So the first one, you're going to be the driver and I'm going to be the navigator. Then we are going to switch for the next one. That's also an option. The thing is that you need to adapt the method to your project, to your team, to your situation. It depends a lot. As some considerations, I didn't say and it's really important to have a stable internet connection because it's not that if you're remote, that won't work. If you're in person in the office, that's fine. So as I say, it's just try to adapt it to yourself and to your team. And as I say, maybe I described three styles. Maybe you think, oh, but I like to do it different. That's perfect. I mean, that's completely fine. Great, that's beautiful. I'm open just for this question. All right, so I think we are out of time. So thank you very much and I hope you enjoy it. Thank you. We're good to go. Good morning. Everybody's still kind of tired. Okay, we'll try not to be too boring. Let's see. Next generation of data stores, hot queries, cold storage. Let's see if we all had the same thing in mind what happens in this talk, but we see. So for my background, I work at Elastic, the company behind Elastic Search. We are investing heavily into this world as well. So that's kind of like my background to that. My general question is like, who uses data stores? And I assume everybody will raise their hand now. And then the next question is like, who likes managing data stores? And that's normally when all the hands goes down or you make this gesture, right? And then everybody's like, hmm, I don't know. I'm not sure why I want to manage data stores. And the question is kind of like, why is it such a pain to manage data stores? Or where is the problem coming from? Or yes, we all need them, but nobody wants to run them, right? And maybe it's the old thing where you say, oh, it's an ops problem now and good luck. And we just pick the data stores and you have to run it. But why are data stores such a painful thing to run? Or where is that problem coming from? And I think in part it's because of the, I call it the classic architecture that we have all used when building something. And a lot of my examples will be kind of pipe elastic search because that's the system that I know really well, but this will apply to pretty much any data store system that you have. So let's assume we have a distributed data store, we have three nodes here, and we have some data that we have split in shards. So we have one, let's keep it simple. We say that one table or index, that we split into multiple shards. So I have three primary shards, shards zero, one, two. And then I have replication of those shards like this replication of shards zero, one, and two. And these are distributed and it's a distributed system that works as expected, right? So you have the data, one of the nodes goes down, you have all the data replicated on another node which could be automatically promoted. And in theory, this is all nice and works great and it should make operation easy, but it kind of does not, right? And why is it often such a problem to manage the data? I think in part, because depending on how the system works, in the case of elastic search, for example, there is one node that we, for historic reasons, call the master node, that basically manages the cluster state and it knows like which shards are located where and what is the structure of the shard and what is kind of like the metadata in state of this node. And like this is a component that is often, I don't want to say brittle, but it's like if something happens to that, it doesn't matter that you have all the data nicely sitting somewhere, you kind of like it's the map to the data. So if something with that happens, your cluster will have a very bad day. And yes, this is also like a thing that is like shared across the nodes and if that one dies, another node will promote itself to be the master node, but it's still like there are a lot of moving parts in that. And if communication breaks somewhere like, I assume everybody knows the cap theorem, consistency, availability and partition tolerance, if the network breaks in theory, it all works, but in practice, sometimes you get stuck in a weird state and all of this makes operating data storage kind of part of extending storage and just managing that storage and managing all the nodes in conjunction. So you don't really want to have that or it's like we are very used to that. And I always call it the Stockholm syndrome that you get so used to something that you say this is the only way and the right way to do something, but maybe it's not actually what you want to do. And then there is the great promise, I want to say of our industry, which is called serverless. And yeah, there is the old joke, there are servers in serverless. Yes, the point is more that the servers are managed by somebody else. And I'll build up to what that means for data storage over the next few slides. I assume pretty much everybody is familiar with the term serverless or if you want to explain that to somebody, I always like this example where you say we have pizza as a service because here everybody likes pizza and then you have these multiple options so you could have blue is for self-managed and green is managed by somebody else. And then you have this thing where you have like the homemade pizza where you buy everything from the flour to the toppings and whatever and you make the entire pizza at home yourself from scratch. And that is pretty much like you have your infrastructure where you have your own data center where you manage the hardware, the software and everything around it. And then you might have like the taken bake and I know that the Italians will not be very happy about that but the rest of the world often eats frozen pizza and then warms it up at home. That's kind of like the next step. Then you would have the delivery that you get the pizza delivered to your home or you go to the restaurant that's kind of like the serverless model almost. You have outsourced all the work you just show up, you read and you go. And for computing this would be, this is kind of like your own data center that you manage. This might be an easy two instance or a virtual server that where somebody has managed the hardware but you do the software and the operating system and everything yourself. This might be maybe Kubernetes where you have abstracted everything away behind an API but you still need to manage a lot of things yourself and that would then be serverless where you basically throw your code and something in it will scale it out and run it and do everything for you and you don't care about the infrastructure anymore. So that's kind of like the on-prem all the way up to the serverless setup based on pizza but I think the comparison holds quite well. And then it's serverless and then the other thing that often comes up is stateless and just as with servers, serverless, there are servers with stateless and data stores. There is state because some people are very amused like how can you have a stateless database? Are you writing your data to Dev Null and it will just disappear because maybe that's a stateless database but that's not really the point of that. You want to have a data store where again somebody else manages the state for you. And then the question is what is the storage standard of our industry today and has been for a couple of years? Any guesses? Sorry. So I would say it's S3 that kind of like that is where everybody puts their data and everybody has, even though it's a proprietary standard, everybody has implemented an API that is kind of similar to that. So I don't want to be Amazon specific here because you have plenty of choices like Google has something similar, Azure, DigitalOcean. You can run your own with Minio and there are tons of compatible APIs. So I think S3 is kind of like the storage standard of our industry, even though we might not want to admit that, especially when you come from an open source world or it's kind of like painful that S3, this proprietary Amazon thing has kind of won but it seems to be what as an industry we have more or less standardized on. And this is also where lots of players who want to say like we are stateless, this is where your state often goes. There's one small side note because we've had a lot of pain on S3 compatible is that what everybody does might be or people might have very different opinions what S3 compatible might mean. So we've in an inter elastic search we have built a free repo analysis tool that basically checks if your implementation of S3 is compatible enough for us to work because we use that for backups for a long time. But even then we want to make sure that you support all the API calls that the network is fast enough that you can store large enough files like what you would expect because depending on the vendor specific implementation S3 compatible is one of the bigger lies in our industry that everybody says yeah we support some APIs but the finer details are then very different. But looking at, I'll pick some GCP examples but again this will translate to pretty much any cloud provider or any environment. In most cloud providers you have the choice between different data stores and things. So the object store that is the three compatible thing where you can have a lot of block data and store those. You could instead have the block storage which where you basically attach some storage over the network and use that. There's also the file storage which is normally something NFS compatible or so. For most cases this is the classic approach how many people have been running data stores in the cloud that you have some network attached or maybe even block the storage to the instance. And the local data is more ephemeral and then you have persistent disk if you can mount over the network but it also has some trade-offs that you for the most part can only attach one for writing maybe you can attach that image to multiple instances for reading but only one can write to that. And NFS at least we have had pretty bad experiences around performance when we tried to even just run 90 benchmarks if it is compatible or works and we kind of gave up on that. So for most data stores I think NFS is not a great choice. Local attached storage is kind of like the classic choice but where everybody wants to go is block storage just because you have this outsourcing of state. This is kind of like what most people understand if they say like we have a stateless data store it doesn't mean there is no state but you outsource the state to a block store for the most part just because management knows the cost is normally very good. But what are the trade-offs there? So your ability is normally very good that you put the data there and then depending on your cloud provider you will have I don't know four or five whatever the lines in terms of your ability that your data will still be there which might also mean that you don't need to do replication anymore because the data is guaranteed and Amazon for example replicates the data internally so you don't need to care about replication itself or anything more. Latency is often a bit worse. That's a trade-off we can then discuss how much time to make. Cost I think depends if you have a good access pattern and you write large enough chunks of data and you don't constantly transfer them it's potentially very cheap to run. If you have a bad access pattern especially around network traffic that can get quite expensive across zones then it might also be more expensive but if you do it well it can be very cheap in comparison. On latency there's for example Datadog had a very nice blog post a while ago where they describe one of their home-grown data stores that's not publicly available but you can split into what Datadog is doing and it's called Husky and they basically said like the trade-off with S3 was I think both the cost but also the management and then the tame latency was good and better even though the average latency was a bit higher but for what they wanted to do it was more than good enough to have that slightly higher average latency and this made a lot more sense to run their data store and I feel like a lot of people trying to run some system in the cloud come to a similar conclusion that while there is a trade-off and the average latency is a bit higher if you have the right access patterns then a blog store can perform very well and for depending on the task be a good solution. The other thing that you might want to do then is from the classical perspective that was normally another thing that you split up storage and compute that you have some instances that can scale up if you have more ingestion happening so you need more processing of data coming in but you just write it out to the blog store and then when you don't have any incoming traffic anymore you can basically shut down that index or ingestion layer and because you don't need to run it anymore you just put all the data to S3 that's where it sits and then you can scale the ingestion independently but you can also scale the retrieving, reading, searching whatever you call it, layer, independently so for example you have almost no or no reads on the weekend you can mostly reduce your reading engines whereas if Monday morning hits you can automatically or based on requests scale it up automatically and just save costs again because your data, the state is put to S3 where it is living but you can very easily scale up and down both the writing of the data and the reading afterwards as well to scale independently and go faster but also reduce the cost after that The other important thing here is normally that you want to have for a truly serverless or state-assisted you want to have scale to zero so you can even say over the weekend nothing happens so we just shut all the computing services down we potentially save a lot of money and energy you can still do local caching so the different disk formats and access patterns you can still exploit the locality of data that if you have some hot reading path and you always read the same data that you cached at locally so you get the advantage of having local disks for that but a lot of the data can just be on the top store and you don't have to move it around all the time and then you have much better scale and potentially also much better cost in there The other thing that is, I don't want to say the holy grail but it's kind of like the thing that everybody wants to have is the paper execution that you would say that you know what one query of a user basically costs you I also feel like with JetGBT recently that has gotten a lot more attention because there you can see like this request cost you X cents so this paper execution is an interesting model that I think a lot of people want it has historically been very hard with any other data store let's just say you have a Postgres instance you have no idea what one query really cost you in the past that somebody was running or with pretty much any other data store it was very hard to figure out the cost per requested you could either attribute that to a specific client or a department and so the cost was always like very much a ballpark but with serverless and like building on these primitives it might be much easier to actually figure out how much cost you're contributing or one specific query or the ingestion volume how much cost that is adding to your system One thing that is also important to note here is that object stores just like the compatibility of the API that the performance characteristics vary why it can depend very much on the data store and again in a key we always love abstraction but abstraction is not magic you still need to know what happens under the hood and otherwise you will find out the hard way at some point and object stores are no different so for every cloud provider for example there are different like limits that you have for example in GCP if you have less than a thousand writes or 5,000 reads per second you will be fine but for example if you need more you will need to ramp that up over time so that the service adjusts for you this is very cloud provider specific I don't think S3 from Amazon for example has something like that and I haven't seen anything on Azure either but GCP is pretty explicit about that that you basically start around these numbers and you can double them approximately every 20 minutes but if you know that you will have a lot of load coming in you will need to load a warm up your system over the right amount of time to actually scale to approximately that load so that you will be in a good state and not just get server errors back the other thing and that is more evil or equal around cloud providers this in small variations basically applies to S3 or Amazon and Azure as well just as to GCP is that you want to avoid hotspot in because well like we said there are still servers behind the scenes so somehow they need to distribute the data and handle all the requests and if you have a hotspot in your data so for example if you have a naming convention and then you have like you create a bucket and then you just increment the bucket name by one or it just has the date in it then you will always have a lot of locality for example if you have a bucket where you write all the data you can write to a single bucket today but you read over the last year the right bucket there will always be one bucket that will get 99% or more of all the right requests so you always have this hotspot there and that's a pattern that you should generally avoid because there is all the hardware down there and it will not magically solve the problem there is some distribution mechanism but normally it does not well if you have a naming convention that goes always to the same place for example other systems have to apply or use a workaround for that anybody knows how other systems work around that where you have the sequential naming so for example in Elastic we would have the ID that is the distribution key we would hash that so even if you just increment the value by one through the hash function it would even be distributed over the entire key space and you can avoid problems from what I've read that is not how most of the block stores work but if you have like a prefix and you always have the same prefix and then add some consistent pattern in increments you will have a hotspotting problem because the data will always end up in the same place so those are things you just need to figure out and understand if you have enough load and data because otherwise you will suffer from the shortcomings of the implementation even though you don't know it or haven't seen it but you will feel it at some point that there is hotspotting on the foot the other thing of course that is still a thing is round trips are still expensive so if you put your block storage in the US and you will try to retrieve the data in Europe the physics of round trips and the network latency doesn't change so locality of having block store or instance store is still a thing so you cannot work around that and then depending on the cloud provider you will have different access classes where in-free connections is cheaper but might have more latency or might only be available after some time those are normally obviously not very good for data stores where you want to have data available very quickly and also with a very high durability so that's probably not what you want to optimize there for but a lot of different cloud providers or data stores in their cloud implementations support some kind of serverless, stateless implementation so for example for Postgres there is Neon which is I think an open source implementation and that is exactly going for a block store in the background Ugobytev, Cockroach, Clickhouse cloud Firebolt cloud and there are many more because the general pattern of outsourcing the storage and state to a 3 is just to appealing to do that and we are to no exception I just to give you an idea of how the elastic search kind of like looks this is the old way of doing stuff and this is like how we see block stores and how we want to use those and this already looks slightly messy but you can see it has almost all the problems of like the plastic distribution model so just to show that the blue error here this is a right and the orange error this is a read so when the data comes in from your client you write the data and then you might have the so-called pop layer this is where you do most of your writes and reads and you can see we have primary and replicas we need to write all the data twice a day for high availability because if that instance here dies and we still have the other copy but you always write all the data twice and then at some point you might say like oh I move this to the code state where I have my data read only so I'm not doing any writes and I do fewer reads I move that to an instance with more density or like larger distance than CPU maybe you have already backed up the data so you only need to see your copy anymore you can move it to frozen that might be backed by an option to store moving through different stages you need replication and like writes are going all over the place this is kind of like the classic approach of how you were accessing data stores and the different tiers with hot, cold, frozen those are highly optional but that was kind of like a performance optimization that here where you do all the writing and reading or most of the reading you want more of the view here you want to add more of this so if the data is increasingly accessed so if the compliance layer where yeah we need to store data for 6 months or 12 months or whatever but nobody is going to search it or at least not on a frequent basis so for that every now and then theory it can be quite slow you can do some optimization here the idea of serverless or stateless is then again you can see we have the right part is all on this side and the read part is all on the other side and the only thing in the middle there and then you can also split up the writing or indexing here and the searching here because this one here writes and all the writes go through here if you have more writes happening you can scale up this layer independently of the other one the same if you have more searches you can independently scale up the search here and go through here you don't even need replication because the assumption here is that you write the index state but also the transaction log or bin log or whatever it's called in your data store even that is going to the blob store so you have really no state that is only on an instance so you don't need to have any replication anymore but your replication layer is basically relying on the instance store so as I said before state is still a thing but it's somebody else's problem and you're just relying on them doing a good enough job but this basically allows you to scale different keys independently you don't need that replication that we've had before so you potentially only need to index half as much data anymore because you don't need to do it twice but only once and that should make it much easier for you to operate and ideally also much cheaper because only writing the data once and scaling reading and writing independently whereas up here you can see if I have more reads I need to scale the same components or the same thing for ingestion whereas here I can have a more fine grained approach and do that there are still a couple of challenges around that these are slightly extra specific for example we have these masternodes that manage the state we also want to make those stateless because otherwise scaling down to zero for example is not really a thing but you always need to have these components that runs the data and we have 10 minutes left yes you have the transaction log or BIN log or whatever it's called in your data store that you need to keep somewhere like our approach again is that you can still put that on a tree and it will perform well enough to actually also solve that state we have one thing where gets for example are real-time and how to implement that but those are like more very specific implementation you get so I skip over those we've recently done some benchmarks where we have in terms of ingestion it is much faster also because we only need to do it like once and not twice anymore and that also needs a lot less CPU because you don't need the coordination between a primary and a replica but you can only just write it once go to the blob store and then it's the blob store it's a problem and if you have optimized like how much data you put together and write in one operation since you normally paid for API requests and data transfer and everything it can be a lot cheaper to run such a system if your access patterns are bad it might still be very expensive or even more expensive because for the extra overhead from a free-throttle so to wrap up Stateless and serverless are a thing I know that everybody wants to call everything cloud native nowadays so maybe this is a cloud native data store because some providers are very keen calling us as we or cloud native and others are not and it's always a question of like what does that really mean and I think it's kind of like as our industry progresses this stateless and serverless approach of outsourcing the work to somebody else is very appealing and this will work well there one thing that sometimes comes up is like isn't all of this just auto scaling and I hope we've made it clear that this is more auto scaling that you just dynamically add more instances but you can never go down to zero with auto scaling and that you still manage everything in terms of state yourself that serverless and status is more than just auto scaling even though it might solve parts of the same problem it's really only it's probably less than half of the solution just because you need to think about state very differently in a serverless and stateless environment and with that that's it do you have any questions I'll try to repeat the questions for the recording ah yes please but you talked about using the most storage in the right pattern right I guess that means that you should try for example send bigger chunks of data here how are you going to do this reliability and the process it's kind of getting the data from yes so the question was basically about the right access pattern for writing data and like the tradeoff between having large enough data chunks and then writing the data frequently enough so part of the solution here is this transaction lockdown the basic assumption at least for us I think that's quite similar for most others is that the right comes in you batch a couple of transactions together and write that transaction lock and only then you acknowledge it back the data then is still kept on that indexing here before being written to that index door but the guarantee is basically in a transaction lock so if this node whatever node here indexes your data dies you can always get out of the transaction lock so for the transaction lock you might have the tradeoff that you write smaller chunks of data but for before you write you create bigger chunks here that are then also easier to retrieve so you don't have like a gazillion very small files but you have larger chunks that ideally you prepare depending on your access pattern so for example if you have I don't know let's say anything that is close to a time series like locks and anything with a timestamp you can then just sort the data in that and you retrieve like a spend of time and you have like those data always together so the retrieval is much simpler in a transaction lock you might have to have the tradeoff that you do smaller and more requests just with your ability but that is the tradeoff that writing of the actual data which then goes into the searching and many scenarios that you will search way more often than you actually write the data and then you can optimize for that so there is a bit of a tradeoff in the access patterns our assumption I think and general assumption is that data is being read more than it's written so you want to batch that together in some format that optimizes for retrieving like not too many data chunks but fewer data chunks than larger ones rather than having these tiny reads the access pattern from a local SSD is totally different than with these block stores so that's kind of the way to go there I hope that answered the question any other questions yes please so the question was hotspotting with a year month date pattern for writing so your writes always go to the same so it's not historic data that you're processing but it's basically today's data so all your writes are basically going to today's bucket I mean I mean I mean I don't think it's great in terms of distribution because you have this hotspotting and depending on your cloud provider you will find that in the documentation so I only picked the GCP examples which was here but there is similar documentation for S3 and Azure as well or maybe you need to talk to support we've also spoken to support maybe they don't make that super public in any every scenario you don't feel it but if you try to push the limit you might but hotspotting is generally a thing I think S3 also has a caching layer that for reading can kind of like offset that problem if you have to basically exploit locality that you always read the same bucket that it will cache the data but that's very S3 specific the amount of data that you rename I think our fear is that if we have a very large client and they write to the same bucket because we have incremental buckets or whatever and all their writes always go to the same bucket that we might create a hotspot or that maybe we put three noisy customers close together and that we might not end well I think as long as it's working it's probably not a problem but hotspotting is still a thing, it's a classic anti-problem and I think even though it's abstracted behind five layers at this point it's not really going away and on the negative side you see there's hotspotting on the positive side you say we exploit locality so I think it's a bit of a trade-off again what makes sense and how it combines well anything else otherwise thanks a lot for joining I'm going to raise some speakers so I don't have to carry them home and thanks a lot for joining Hello everybody, my name is Callie Doffy and I am a data scientist in the open source program office at Ratat and we'll be talking today about community metrics and what to measure and why so the three different parts of this presentation we'll first be going into the value of community metrics why do you even want to spend time doing this the methodology behind community metrics and the major point that I would like you to leave with is the analysis lessons learned so what can strong community metrics enable first I'm not going to tell you that data analysis or automation is going to be the one thing that informs all of your community decisions it's actually the opposite this is to build off of your own open source community knowledge incorporate others and potential biases and perspectives not considered this allows you to be able to integrate people's special knowledge around communities for example there might be one person in your community who is really informed about events and how that can impact community activity and maybe on the co-base maybe that be in your matrix channel and another person might be able to know a lot about PR behavior and what tendencies might go a positive trend for your community and what might be a little bit of an indicator that things might not be going as well another thing is this can help you stay informed in a sustainable way we all have a thousand and one things to keep up with and it never feels like we have enough time in the day if an answer about your community takes 10-15 hours to get you aren't going to do it regularly if ever and the last thing is there is so much data around repositories in every aspect of communities how do you start to work through it there is so much pressure around being data driven and you can go the angle of getting a metrics on the slide to just try to validate your expertise but how can we start to pick through this needle and data haystack and what to look at to actually inform your decisions the first thing when thinking about a metric is about the perspective you want to receive or gain and there's a few things to start considering here is your main goal to gain information or is it to influence action whenever there is an area of your community that is not understood or are you trying to take your first step to getting there or is there initiative that is already in place that you're trying to decide of whether you're already deciding to decide or are you trying to decide on the initiative yourself next thing you want to consider is are you trying to expose area of your own improvement or are you trying to highlight strength there are times when you want to hype up your community and show how great it is especially when you're trying to show business impact or advocate for your community to people outside of it but whenever it comes to informing yourself about your community most of the time in identifying shortcomings is where you're going to have the true value there's no problem in highlighting strength but there's a bit of a time and a place don't use metrics and visualization just as a yes man just to tell you how great you are last thing you want to consider whether you're looking for measuring community impact or business impact the language that many businesses speaks is numbers and data that can make it incredibly difficult to advocate for your community in that landscape and make sure that people actually listen to your messaging before at instead of actually just trying to figure out if you have a valid point to make so whenever you want to do this you want to start speaking in their own language which is numbers and data then on the flip side potentially you're trying to look at the community impact maybe this is looking at open source impact overall within your community or to the entire open source ecosystem these aren't always like an either or situation but this framing helps kind of focus in on what type of metric you want to make so when talking about general data science and machine learning work some version of this graph is what you're going to see for this presentation we're going to be mainly focusing on this first step the codifying problems and metrics and a little bit into the collection and cleaning from a data science perspective this presentation can be looked as a case study of this step and this step is also the one that's usually overlooked it's not the fun or exciting one everyone wants to go to training models and doing the fancy stuff but this is usually where the true value of your analysis comes from you don't just wake up one day and know exactly what you want to measure and what's going to bring value so let's start to really hone in on the goal of our time here which is codifying problems and metrics the tooling debate is for another time or you can come find me after this talk and I will rant about it all day and so first this starts with truly figuring out what you want to know what data you have and how to get us to the true goal of thoughtful execution of data analysis so we're going to be going into a couple of different analysis angles and breaking down these different scenarios and these are just the main examples for this talk and a lot of these points can be applied just generally throughout this first scenario is building upon current data analysis so say that you're already starting to go down the path metrics and you already know what is generally useful for your community and you want to start to build off of it and make it better the idea here is to start off of your traditional community analysis and it's like commits overtime that's cool and it's something we always see but what does that actually tell you about your community so let's look through a few examples here first let's think about numbers of contributors over time it's great to have on the slide that for example you have had 120 different contributors but how can we take it the slight step further to actually start to inform ourselves so we can look at maybe active versus drifting contributors this is the same exact data but you can start to see the breakdown between okay how many people total have been involved in our community how many people have say been involved in the last three months and is that again starting to go down are people leaving the community and you can start to investigate why another thing you can break down into is looking at repeat or fly by contributors so are there contributors that are just coming in and opening a single issue maybe patching a bug and leaving or is this contributor base very solid and staying consistent and active overtime another example is the classic commits overtime break you know that you had 200 commits last month and 100 commits this month what else can you learn from this the depth of commits can be a really important factor maybe those 200 commits just came from a hackathon and a bunch of college students decided to put a period and a bunch of different places to bump up their score versus that 100 commits might have been a huge overall in the entire code base and so that's a completely different perspective which you wouldn't have gotten just from the 100 commits overtime another portion of this is commits by subsets of contributors this is actually something that I'm really focused in on right now and starting to figure out okay what factors or what portions of your code base is dependent on one maybe two people how many people have touched these certain areas of the code and for how long if those people leave what happens you probably want to know this before those people leave because if that happens then all of that knowledge has been lost versus if you kind of look ahead on this you can start to have that knowledge share so it's not some disaster that happens in your community so then we're going to be looking into scenario two which is community campaign impact management that's a lot of words let's break it down a little bit let's think meet up conferences or any community initiative how do you start to measure text and establish goals these two steps actually feed into one another whenever you establish your campaign goals you should also be considering how to actually measure and detect this impact a lot of times people get really excited for actually going forward and applying and doing things within the community and that's not a bad thing but a lot of times you can get six months in a year out and it can feel very unfulfilling the way of determining if anything actually changed from all of the effort that you went into it and so a lot of times these two steps kind of feed into each other over time and a really good example of this is actually some work that I've been doing with the Fedora community and their goal of doubling their contributors by 2028 we have been discussing what actually what does it mean to be a contributor what is a contribution whenever it comes to the counting of these metrics and what repos sites different chat channels what is being what do we need to look at to be able to be measured by discussing what we're looking at to measure you can start to make more targeted goals and what you want to do to be able to impact these numbers and so we're going to be now going to scenario three which is where we're going to spend the majority of our time for this presentation the prior examples can be viewed as different portions of this cycle and it's pretty much living you can you're going to be hopping from the end back to the beginning over time as you start to develop more of your visualizations and for me I'm honestly a lot more of an examples person we're first going to go through the entire methodology of it and then do an entire example to see how this would apply so the first step in this process is to break down your focus area and the perspectives and this is where you take into account those perspectives that we discussed in the earlier slides and I like to think about this in three different parts first let's think of a magic eight ball whenever I was a kid I had that little plastic toy and you could shake it up and ask it anything for example I could shake it up right now and ask if Max Verstappen is going to win or get pulled tonight for the Canadian Grand Prix and it's probably going to say certainly it is anybody who watches Formula One gets that reference so this can be applied to your analysis area if you could get any answer about your community right now no limitations what would it be the next step with that is to talk about the data from that magic eight ball question what are your data sources that could potentially have anything to do with that question if you want to be limited gather anything that you could possibly could have possibly applied now with the context of the data your magic eight ball question in your focus area what questions could be answered that could bring you closer to your goal most of the time you cannot ask the direct question that you want but you can start to get pieces of it but be careful because it can be very easy to want to take those pieces and bring it all back together so if you have a question there's usually a lot of assumptions that can come into play there so you just need to be careful about how you aggregate your different analysis points and sometimes you just want to take them piece by piece next step is converting this question into a metric and so this is going to be you're going to go through these steps each time for that final part of the step one so that first thing you want to do is to get the specific data points that you would need maybe you looked at okay I want all of the data around these specific repos and then once you get here you realize that you just want to look specifically at the data around the pull requests the next thing you want to do is considering what type of visualizations are going to best represent this data to answer the questions that you're looking at is it going to be a pie chart or is it going to be a bar chart with different layered plots maybe a line chart over the top of that and then next you're going to want to look at the insights and actions that is able to come from this what hypothesize what would be the impact of this information how you would incorporate into your community then you want to go straight into the collaborative portion of this and you want to go to the most skeptical person you know in your community to bring this to and tell you every single reason why you're wrong usually this is the step where all of the magic happened the deep like the best analysis and different things that I've done have always come from this portion of it you're not going to come up with your best ideas in the silo and that skeptical person is who you need to come in and bring you back down to earth and so next we'll be looking at the analysis and action not in every single scenarios we'll go through every single step in this but we'll start to walk through it now that you have this hypothetical visualization or metric the first thing you want to consider is if this aligns to the prior knowledge around your community if it does align with your prior knowledge make sure to step back and see if there was any assumptions that you made to kind of cater those results to what you already thought existed and if it doesn't match then go back and check to see if there was a data or calculation issue be very thorough here or if this was just something that was priorly misunderstood part of your community from there we can start to look at community and initiatives that can be impact or could be implemented from this analysis and so we can inform those initiatives from the analysis and we want to make sure they are geared to be measurable in some way and then from here is where we would start to observe these community initiatives in this scenario where we are not actually able to observe anything is it a scenario where you're not measuring the right things are you looking for activity from commits and PRs whenever the actual activity that's being impacted is people in a matrix channel now starting to communicate more to new users or does the actual initiative strategy need to change so there was a whole lot of information now let's go more into a concrete example so let's say that my focus area is to analyze new contributors and what their actions are for the first time so my magic 8 ball question here is are people having an experience in the community that convert them into being consistent contributors now I want to consider what data that I have into this analysis area and magic 8 ball question and for here I would like to look at individual contributor activity with repose and their repose and with time stamps so so now that we have our data and our magic 8 ball question let's break it down into a couple of different sub-components and take them all the way to the end so if we're looking at this from like the prior step examples we're going to go from step 2 for each question because if we try to hop around it just gets really confusing and at the end of this presentation I'll actually be showing what these visualizations look like implemented so first sub question is how are people coming into my community I want to be looking at new contributors and seeing what their first action is and so the specific data that I'm going to be looking at is the contributions issues, PRs, comments and all of the above by contributors over time and only looking at that very first action and the visualization that I choose to look at this is taking first time contributions and breaking them down over the quarter and so now that we have that visualization we now can look at the extension of okay what can we start to look at next let's say I start to talk to different people and we start realizing that we want to take this visualization one step further and breaking it down by quarter and looking at if the contributor is going to be a repeat contributor or if the actions point to being a flyby contributor and so this breakdown we can start to look at if there's any trends between that first action and whether somebody is going to make an issue, maybe do a blog and leave or become an active member of the community and so let's start to think about potential actions that could be informed by knowing the activity for flyby or repeat contributors. We can start to consider if our current documentation supports contributors for that first contribution that points to being a consistent member of the community. Is there a way that we can support new communities more in these actions? Is there a consistent contribution that points more towards that repeat contributor but it's not happening very often in that case. We can say that PR could be a common for repeat contributors but overall most people don't do that for their first activity. Maybe something we can do as a community is starting to label first new issues and put that apart of our contribution documentation or start to create a PR body system for new members of the community. Second sub question that we can look at in this area is what is the conversion rate from first time contributor to an active or repeat contributor and the data we are going to be using is the same one as before and we're going to be looking at the visualization I decided to look at the visualization of a pie chart of looking at how many, what percent are converted and which ones are not. Some of the questions that we can start to ask and be answered around these visualizations is is that number or percent starting to go down? Is there something we're doing differently in the community or is there something that's kind of fallen off the table to start bringing down that conversion number of people starting to leave or first time contributors are not having as positive an experience and is there a little bit of a trend for ones that say this is another way that we can answer some of the similar questions from the prior visualization and a lot of times it's really helpful to have one question that can be potentially answered by multiple different visualizations because it provides different perspectives and start to inform an overall answer. The last sub question of this is is our code base really dependent on fly by contributors and this is a visualization we can look at here is total contributions and breaking it down by repeat or fly by contributors and some questions that we can start asking is this a ratio that we like? Is there a lot being done by fly by contributors and maybe that's an underutilized resource and we're not doing our part to making it a welcoming community and bringing it together? Now we're going to go into analysis lessons learned if you take anything from this please let it be the last couple of slides. So might be a shock to come from a data scientist, numbers and data analysis are not facts no matter what people say you can make them say anything that you want to and your internal skeptic should be very alive and well when you start to build your own visualizations the iterative process and bringing different people in to question what you're doing is what's going to bring the true value and you don't want your analysis just to be a yes man to what you were already thinking before. Take time to take a step back and evaluate the assumptions that you are making through this process and if a metric just points you in a new direction to investigate that's a huge win you can't look at everything every single aspect of the community so if there is just some anomalies on the graph that you can start to look at to investigate more that's a huge win you would have never known to look into those areas and it could really just be a conversation starter to take some ideas to a completely new place and sometimes exactly what you want to measure is not going to be there but it can bring in valuable pieces to a puzzle but just like if you try to put those two puzzle pieces together and it kind of cracks it whenever you try to force an answer and solutions that are not there it can start to bring you down a very dangerous path leaving room for the goals of analysis to change can actually lead you a lot of times to a better place than you thought that you were going to before another thing to consider is that data analysis is the start each scenario that we went through today is just starting at different points of the same cycle that first example is that second go around of analysis after you made that first time pass the second example is looking at whenever the goals and scope was established to an extent of a community initiative and starting to feedback loop of what that initiative is and how to measure it and the third example is coming in with a brand new idea starting from scratch in your data analysis you should never stop asking if more context is needed or if it's truly answering the questions that you want most of the time it's usually a yes and situation and sometimes you need to ask yourself if you're actually asking the correct question the scenario and there's just so much going on whether that be around data your community just trying to be a human if you can cut down the amount of time it takes to get information or create an easy system to check things at a regular cadence that is a huge win if something is going to take you 15 20 hours you're not going to do it regularly but if you can create a system that takes 10 to 15 minutes a week to check that is something that you can stay consistent on so with the closing thoughts data is a tool it's not an answer but it can bring together insight information and individual knowledge to make it more accessible to everyone that wouldn't have been possible otherwise and the methodology is so vital to the success of the overall analysis you have to get comfortable with breaking down what you want to know into manageable chunks and start to build off of that taking a step back open source community data analysis is a great example of the care that needs to be taken with all data science you must take in the nuance of the subject matter and the topic area and open source communities are about as nuanced as it gets the process of working through what needs to be asked and what is the answer and breaking it down at the very beginning is something that's often overlooked people want to go straight into a new visualization playing with the data before asking what they want to do in the first place knowing what to ask can truly be the hardest part and when you're coming up with something and when you can come up with something insightful and innovative after spending that time thinking about it yourself bringing in other people that's when you're going to truly bring something exciting along so if you're a community member with no data science experience just looking for a place to start I hope you see from this presentation how vital your perspective is even if you don't write a line of code implement any visualization yourself that insight of the community and the different perspectives and nuance specifically around whatever community you're barred of is extremely important and you are the one that brings the insights not necessarily just the data and I hope you can see all the places you can be involved in this and if you're a data scientist or somebody in your community that is applying the analysis or visualizations you need to listen to the voices around you even if you're also actively a member of this community and so thank you I'll be taking questions and also I'll pull up the graph that you were talking about a little earlier and this is for the conveyor community the concept to the legs curl the legs curl directly now are you interested in doing it if they're all get based repositories we can put them into our graphs and see what happens all of the visualizations that we're building right now as long as it's a get based structure we can use it building this is actually called the application here it's actually called 808 we're just using the data we'll do most of our code just repeat the question so how is Red Hat using these visualizations there's been a couple of different examples these specific visualizations we haven't used as much like directly within Red Hat some people within the open source community our community managers have been looking at these visualizations for how I guess that is working with Red Hat so people within the open so whenever they've been working with their communities have been using this to look at the activity around new contributors there's been other visualizations whenever we have there's an assumption that's been put forward by people who are involved in the community whenever it's going in to talk about how that impacts Red Hat products we have been able to use these visualizations to kind of take it away from being like does this actually exist and a lot of times if you don't have proof of something especially from a business perspective people want to stay there and just discuss the validity and these graphs have been allowing people to go away from being able to say is this actually happening to okay now what are we going to do about it so that's been something that's been really exciting but I didn't, whenever we started on this path on this project I really didn't get how valuable it would be to be able to just say this is what's happening we have this is shown in the activity evidence to be able to take away the time of just going back and forth and actually starting to discuss solutions sometimes community managers or I guess the question is where do these requests come sometimes it's community managers sometimes it's solution architects on the product side who are starting to hear things from customers and they're starting to go down the rabbit hole what's going on in the community that's starting to get it to the point where our customers are not necessarily getting exactly what they want and so we're actually able to start going back and solving these problems from a community side not just worry about it from all the way on the product and dealing with the customer side Right now we're working mainly with get hub data this is all all these graphs are supplied by a project called auger which takes all the data you could possibly want around a repository puts it into a relational database and that's what we plug in with the dashboard that we built but we do want to extend outside of get hub as this project grows There's nothing else then we're good to go So Hello and I hope that you like this morning that you had a lunch and everything but right now I want to share one great ideas with you but I am afraid that for that I will need some app developer Is there please any app developer here? Hello Eric Mr. Petra Thank you Yeah I have a good idea Do you have some time? No time, okay I will pay you my name Maybe, yeah it's okay, right? So please what is my idea When I am travelling to the office or to the pub or somewhere I saw the people on the streets listening to music they are singing, they are dancing they listen to music on the mobile phone and I was thinking if they enjoyed so much what about they will pay for them what do you think about it they can pay us so I need to have an app for that for listening to music, what do you think about it? Yeah I think it's a great idea and how do you want to have it Yeah it should be cool it should be simple it should be very interesting music there and it should be working everywhere we have no connection we have connection we have 5G connection and it shouldn't be like that what about the features I don't have time, bye bye you are the expert please do your work bye bye okay Hello Mr. Petra how are you here I am here so what's happening, show me oh nice it's so cool I called it Listenify wow that's a great name I like it and we have a lot of features like play and stop button wait is that anybody who downloaded this app please yeah come on please tell us your user experience and Mr. Petra hello hello Mr. Petra how are you so what do you think about it it's great right yeah it's a master piece I don't have sound it's like having a personal DJ who missed the beat can you see my review even oh don't use it I am afraid I don't have time bye bye I have two buttons it's useful is the stop button like I don't want to listen to songs from medieval times backpipes you know like who wants to listen to that it's like random music playing I can stop it that's useful I cannot search for the music I have no suggestions oh by the way thank you you saved me ten dollars yeah and the product owner but they call the product something front owner he's the guy who can fix application I know him I had similar reviews on some applications and then Wild Andre came and fixed everything I think you need him but you don't want it like why do you want it I spent money for this I need to go to work now no don't worry I spent two dollars man like what I believe what I believe don't worry they come cool water ok Mr. product company how can we like what are your products here have you ever like try to see this from a user perspective of course not ok we can call the guy the big money you know I like that I like money ok hello Mr. Pettor yes who is that hi Mr. Pettor I'm Andre nice to meet you product something yes that's it what do you need I was reading your reviews and things the deeper I didn't buy your app that's pity I like money yeah I just read it have you ever tried to use your app of course I see I imagine you spend a lot of money here like four dollars that's a gold it is better than a check you shouldn't tell you just have it maybe so can we want to try again you want to fix it remove don't use it and you will write it it's the best app ever we will do better ok I have ten minutes we can discuss it ok prepare for this so can you explain me the idea the idea it's easy when I'm traveling to work I see people on the streets and they are listening why not I don't touch them they are listening to music but they have phones and I am saying why then why they don't pay me for the listening ok let's do it once so why do you speak can we take some notes yeah please make some notes imagine that we are talking about topics and try to split your idea into topics you are the user you imagine the app I should imagine that I am a user with money ok I can so it's about songs I like some of them I don't like some of them and I want to have it in my playlist one playlist for traveling to disco one playlist for traveling to the job one for divorce different moods share later on with me yeah yeah music search it's very important and then as I said it should be about money different people have different profiles so I know if they paid me or not users it's a fluffy name yeah users money what else socializing to judge the people buy their playlists that's nice that will be great that's exactly what I wanted from you really but I tell you you didn't understand it just enjoy the show so if I took notes and it's right that's it what you want in song management there are playlists yes production means money yes that's nice don't go remember the disaster in 10 minutes you made that disaster if you spend a little more time we can drag it so let's do it right and let's start with the most basic function control I like it master likes it ok master play back control again I know it's hard but money you are the user how do you see the functions if I open the app but it is not crashing because yesterday it crashed to the floor with the phone users will pay don't worry I want to choose the song I want to hit the button and it will play ok so if we start with play the user wait you use so many papers for that the company is paying for this ok users will pay ok we have a lot of them so that's it you open the app hit play and listen to me what else pause it's very important stop listen to music somebody calls me I need to stop the music and I need to answer the call I am a business man so user opens the app pretty much the same and stops don't worry they are paying for that and I have a solution the pause if I for example am listening to music and I need to pause it but I want to buy a new Lamborghini and then I can continue you fine I try to do something with this tea can we repeat the play because we have a lot of grains and the other ones so you play the song and stop and what else next sometimes I listen for example the radio in the car I need to hit next and it is skipping to something more what I like if you don't want to be judged fine you are playing does it work it works but I think the next button shouldn't be visible why how much this situation what should the button play if I just if I am not for example in the morning if I don't know and I don't play but next it should play the next one from my playlist and if it is just Bieber next again so if we have a next another one and it is a different a little bit different yes thank you thank you we have all the things for the playback control but if I ask the developer can you implement this everything at once but one thing I need should we prioritize the things because we are doing it in release so what do you want first what should come first repeat because sometimes there is Adele she is so lovely I want to listen it again repeat again repeat listen I think I got the idea we explored but to repeat something should be playing right maybe that's true so maybe the playing is a little more priority than a repeat just a little bit I could agree ok after all the discussion we have skip the repeat but we have the next so we don't need to deal with Justin Bieber people will not judge you and we have play, stop and next ok in the end we have this but I have something to tell you we need to do that for the others remember our disaster money we can type ok let's do it we can now play ok Patrick we did everything for all the other features and we selected what comes in the first release and the other things are they go to the product backlog and we listen to the users we select that and we are releasing stuff I see free 3R in the MVP it will be for free no it's about addiction you know something about addiction addiction to money it will be socializing creating addiction because they start spreading the word of listening that's a good idea and later on we can start it will be so addicted that we start playing and then the money will flow ok slowly can I talk to the yeah please talk no wait a minute you will do it right that's a war you need money we are you are not engineer you do nothing what do you think bye bye you're paying me ok engineer are you ok with that? so you don't need me right great because now you will deal with him and with customers I need you you need my share I don't want to deal so fast no ok so ok thank you oh you want to share ok so I'll pay you yeah no money later hello Mr. Big Money could you come over and discuss our result I am here ok so this color again why? what are the features here features I don't care about what about the user experience do they start it to be addicted? do we have anybody? do you remember the guy the T-Board yeah somebody who is not stupid we are friends now ok I will call him yeah you can hey yo T-Board can you come over yeah thank you hello hello I like the new release of the application really? great job I own this app thank you it was a great job I think that this app finally knows me like it suggests we were in a flower party and we were listening to the repeat we were dancing it was such a great party yeah it started to be addicted my life I like the feature of trending music like I didn't know in such baby shark it's really long I was so quiet dealing with him I was definitely happy to play this library with this hard button because I really love it I will listen to it of course I will just play it and go and earn some money to spend it on this app something Mr. Big Money are people getting addicted? yeah it's nice we will very soon release another thing and that will be later so do you see that I am not an engineer and maybe you are not an engineer but you really made this software engineer who was working he finally created my thoughts which was great they started understanding you right? I would say that you helped me so you deserve it thank you so much good job Mr. Big Money bye he is not kidding I am not kidding it's my money so Q&A so do you have any questions? it was a disaster what you just see so no? no questions no comments I just want to say that it was like very talented thank you because I must have okay there is no questions thank you I have a question please I am a product owner and I don't have any money what are you doing? the same we are basically the messages we are the bridge between the customers the final users and we have kind of translated to the developers to the development teams and we try to help to facilitate that so we don't waste time we don't waste money and we implement whatever the users will use or they will take advantage of to improve and help the product owners don't actually have a contact with the customer I experience the same issue and do you think that there is a solution? one driver left it for you this is a nice question I heavily engage with my product manager and she is exposing me to the customers because I am asking and willing to do that so a very nice guy called Dimitri Paul told me that if you have the business you are developing to or you have a partnership with company A but in company A there are customers or users inside it so we kind of need to trust past this company and reach to the final users and kind of listen because sometimes they have ideas just like the one we presented and now just do it because our people will love it she is trying to expose me as much as she can but it's time consuming and it's hard to go to those people they are understanding the world the things that they should put me in front of if you need to listen to these people and try to get the things and understand the scenarios and come back to us and we are doing some attempts it's getting there yes I have a question so I'm going to then call Dimitri Paul a big customer so how do we the type of each family from one big customer let me see if I can go to your question so you are asking how do we manage for a feature and we try to understand if that feature belongs to our product to all the customers versus pleasing a single customer is that the question fun fact I'm dealing with that right now so some customers they are pursuing some or like they are really trying hard to get some features that don't belong to our product so first your product should have a vision that our product should do otherwise you will create something that you try to tackle so many problems and it's like a Frankenstein ok what is this supposed to do and you need to be focused it's not so easy but if you have the statement or my product should does this this bottle should carry water and keep things fresh I would not add like solid things here because if someone asks this product doesn't solve this problem maybe another one it's hard but if you have people on your side and understanding that it makes things easier and if you have anything to add it's my thing that it's not something to add it's not what we are to both what we are trying to do thanks for your question um yeah sorry something was passing that's great there are some mechanisms to try to try that but none of them we replace the final user the end user access in that so there are some companies that they just put like one percent they launch a new feature and they just put this like one percent of our customers will have access to that and we see how is the damage we test internally of course we go through like the user stories we understand ok this feature is doing that so we are it's actually doing what we expected because I try to clarify after the product owner try to clarify as much as he could and try to understand the user journey so there is a lot of work before the final product before it reach to the development team we kind of go there and try to understand ok this, this, this, this and we try to come up with different solutions it's good to have technical people in those discussions I called product trio it's a product owner a developer lead and someone from UX so us three together we kind of speak about it and everyone brings their point of view I listen to the customers I know the technical feasibility of that and I kind of try to understand from the user the final user perspective so we together have more chances to do something that will please the customer or achieve the goals but it doesn't prevent some disasters but we try our best we have the testing we understand that we are doing what the user stories are supposed to do and once we before launching or in the launching phase we can do some A-B testing ok half of the developer half of the customers will go through the new path the other half will go through the old path and we can try to understand what is happening some companies just put one percent of the final users in touch with the new flow or the new feature and they try to collect things there are beta testers you are like about the addiction we were playing that but oh you are a beta tester user you have access to this before other people you feel special so we are dealing with emotions as well all those things and we kind of get a review especially we are selecting us to review this before others so if I can maybe just comment a little it's always about the beginning when product owner is telling it to the team what they will do it's really about to specify very precisely and think about it in this time how the test should look like if we don't know we create a spike for that we have more understanding and better results then there is a testing process which is funded so with this test that it is doing and also to heal the product owner can check that the the regular requirements are met and then yeah the beta phase and testing with the partners and so on yeah all those stuff is there in your position is there with your product who doesn't have any knowledge of what's engineering about it can you get a solution yes yes if you want to say no no yes because sorry I feel this is yours big money this is about knowing I should know my customer I should know their needs it's a benefit if I understand the technical stuff but it is also when I know technical stuff too much and I try to find a solution and not doing my job I should be there and I should have a voice of customers I shouldn't be a when I want to solve it I collect my team we have a meeting I am trying to say what we need and they are providing the solutions but I really trust them they have the knowledge they are the engineers I am not the engineer by the way so yeah because I was an engineer for 10 years I was a solutions architect and I moved it to a product owner role and sometimes I really want to go there and dictate what is the solution how it should be the solution will be this you will build this way it's not my job I can't touch that I just need to say what was the question there it was my question was how much do you think the automated behavior of an analyst is something like that basically a great place to be role these are things that try to use the behavior of an application or product I saw a memo in the internet that the customers are people they really should know what they should ask about so if we have those conversations to understand and dig further what the person really means so in the beginning of the presentation he had something in his head but he was not able to translate that so if he was able to write all the specifications in the right way that Eric would understand and he is sure about what he wants yes but I don't think it's the case yep sorry I think this person was close there was one so always you have known before you have become a person sorry what do you wish you have known about the work before this? that would be so fun yes I enjoy what I do but there was a question what would you like to know before joining this role what I wanted to know I wanted to have a training before I did my training during my pal there was a lot of trainings but at the beginning it was towards some processes it's not about processes it's about communication and about the tooling this theater maybe was a little you know but the way how Andrei was asking me I am calling it a tool it's very important to know how to properly ask the question to get the answer so that's what I wanted to know at the beginning yeah I have a question about project tour how often it can be adapted or changed just one every time it's needed no every time it's needed product development should be in contact with product management we are and recently we updated in the beginning of the year and I think that before the end of the year we should check if it is still valid I try to replicate something personal so your life change so it's the same as the product today it makes sense to go this way but maybe tomorrow something happens and ok we should start driving in that direction so in maximum once a year but you should keep that handy ok this is my product that should carry water but maybe in one year I can change it or it doesn't make sense or the competitors are doing something different so the users are asking for something different I found a different niche I found a different opportunity and we started talking about it and crafting the vision it's important to have more people in the room that we discuss different point of views so when there are some recipes to go through it and it's really nice exercise that's one person your first question in your scene you have great engineering in our presence by eye because you said thank you for the talk here it is, it's delivered no problem but what could happen is now I'm not going to touch it because it's working for years it doesn't make sense plus there is a super corner case I want to fix right now because I have to fix it how do you deal we are caring about priorities and there is a section there is one another person who was hidden it's a scrum master I really have one and if that is this type of things it's about a matter of evaluation so how much this corner case is important are you releasing a single customer or like replacing a lot of customers oh a lot of customers once in a while just like this jupiter crosses setter and all that happens but how important is against other things that customers don't want to beneficiate other people so it's about there how long is in the backlog oh it's here for five years otherwise we would have done that before is there a nice way how to tackle technical bad or like long term why should there be like the staff that needs some technical preparation do you I have an answer that is I am right now walking with a project which is a very long project I would say 20 years and we have a technical depth and we have also the features which are very important and other stuff and we started with sprints and when we started with sprints we have a one goal to finish everything in the sprint we should plan in the way that we finish everything what is in the sprint if we don't finish what is in the sprint it is going with technical depth and I call this type of backlog we just touch it and we didn't do that properly and every time when I have a sprint running again I am looking for this very small backlog and I am asking myself do we still need it or this is out of our scope and we can close it at all the other technical depth which maybe you are talking about is the old stuff which we didn't do in the past if it is important I have it in the priority that is my answer can I reply or do we have time one minute so some people have some strategies to deal with that and keep it alive so maybe dedicating 20% of the sprint or whatever you are doing or like a rotation every time someone is taking a look at some items in the technical backlog so you can explore that in different ways but the message is you need to look at that and be friendly okay let's try to handle this and bring engineering to decide what is the most important thing so it's time to say thank you it's time to say thank you thank you so hello everyone and welcome to our presentation about Pekit and its maximum integration capability my name is Laura and here are my teammates Maya and Matjo so what have we prepared for you today so even if I see some friendly faces so some of you already definitely know Pekit but so everyone is on the same page we will do a quick introduction of Pekit and its core functionality then Matjo will guide us to the Pekit dancing automation to Fedora after that Maya will talk about the automation for the virtual images and in the end very shortly we will cover the future plans of Pekit and we will answer to any of your questions so what is Pekit project and who is Pekit team Pekit has two main goals and these are validating upstream changes happening in repository in github, github during the time the changes are developed before they get to dancing distributions such as Fedora or LL and the second one is bringing the upstream release to downstream specifically to Fedora and to do this Pekit operates both on github and github so that is the form that we will mostly refer to today Pekit as a service and there is also a CLI tool which you can install on your Fedora distribution and run locally first point validating upstream changes you can do this by multiple so called jobs that you can configure Pekit to do and the most frequently used one is the rpm build so for the rpm build Pekit can react to any pull request commit or release in your upstream repository, take the changes and forward this to the copper build system and then provide feedback in the place where developed change so you can see on the screenshot github UI with the commit status is and you have there a link to the dashboard, you can see the information about the build, the time the works and everything you need then the other also very frequently used job is test and this one is to test your upstream changes in a very similar manner and for this we are using the testing time infrastructure so similarly you will have the feedback about the test in the github UI as you can see and then you can be redirected to the testing time infrastructure and check the logs there there are also other jobs that you can configure such as code scratch build or the vm image build that Maya will guide you through and then there is the second point and then also of the federal releases this I will leave up to Mateo since he very detailedly show you how we do this you can basically see all the services back it interacts with so on the left side we have the github github or githlab then we have the copper testing farm and on the right side there are the federal specific things like distribution, gith, koji bodhi and back it goes everything together and it is the integration of upstream and upstream and if you still don't believe me you can check the icons on the screen here are only some of our users hopefully satisfied with back it and to cover all the all the parties on this slide you can see the back it theme or our github avatar so stay happy and now I will pass over to Mateo ok so how the code gets to the federal Linux I will do a quick make sure how deep do I need to cover the topics I will talk about so raise your hand if you have ever been in the packager group and has done any releases in the github ok so that's like maybe a half so I will go into the details and how many of you have ever used packet ok that's a bit less but still enough ok so how does it work I as a user of my favorite Linux distribution I don't care about any stuff in the between right so there is an upstream someone develop the project I develop my app I do something with it I do some releases and as a user what I want to do I want to run dnf install and upgrade get the latest version or get the application itself in the beginning I don't care about anything in between and that is something that I go through because someone needs to do that so what happens in between as I said in the upstream someone develops and they release versions and then someone needs to get them to the downstream so how do we do that of course there are many Linux distribution each of them has a different way of building the packages and in our case we will talk about RPM based distributions so what do we do we take the local side cache so someone releases the new version we take the sources that we need to build the RPM from and we upload them to the local side cache it keep the sources and that's still not enough right is our sources the only thing I need of course not I need to know how to build them so in this case you can see that we have this git and what do we keep there is we sync our configs and also a current version of the project that around the lab shop and apart from them you can see a spec file which is basically a series of steps that is used to create the RPM package and sources which references the which references the archives in the local side cache and we're still not there right we have sources we have packages that can build them but we still don't have the packages so what do we do we have a build system that's called Koji so you can trigger build there what does it do it takes the spec file it takes the sources it produces an installable RPM package so we can install it and we can use it however we get a package but that's still not in the distribution so what do we do next we have an update system what does it do RPM build and what do we do next we need to get it to the user so we create the update and what happens next is someone going through testing stage I can install it via the instruction in the update I will show them in more detail later so you test it you can provide any feedback on it you can give karma and if you find bugs you can postpone the release to the stable branch if you are satisfied you can give the positive karma and the release can get into the stable and then we get there the user can install it and he's happy ok so I went pretty quickly through those steps but I will go one by one with a real life demonstration and how can we help with that so today I've released a new version of our package and also a warning to photo sensitive people I will be swapping a lot between slides and browser so be careful and what do we have, we have a release, it's a pull request so what can we see there we can see there is a change log and we also give a spec file in the upstream so we can see changed version and also there is a new entry in the spec file so what happens next, it gets merged I have a prepared release I called it the Defcon release because it's for the demonstration and I released this an hour ago I wanted to do this live but we have a lot of releases for Fedora, for Apple so I didn't want to take chances of waiting 10 minutes for the PR to propagate this gift so I've done this beforehand and we can see them right here so those are the pull requests that we create there might be a question why do we create the pull requests and the reason is that maintainer has the last word he has to decide if it meets the quality requirements if it meets the packaging guidelines so we create those pull requests, you need to review it and then you can merge it so for the live demonstration I will probably pick one to the latest Fedora and we can see the changed files and I also have enough time to run the zoom pipeline and what do we have we can see the version of the package that created this pull request, we can see the changed version, we can see the changed change log and we have also uploaded the we have also uploaded the new source to the local side cache so we can use it for the build so what am I going to do I am going to merge it so we can talk about our integration while the service does its job so we have seen a lot of PRs and the reason is that we have two instances we have one for testing, we have one for production and at the same time we test both proposed downstream and pull from upstream so a lot of PRs is like proposed downstream pull from upstream twice times two pair instances and times each truth so there is a lot of rules and of course I mentioned the code rebuilds where we build the RPMs so we can automate it too and we can also automate the bogey updates so as you could see the PRs there is upstream to downstream we have two jobs one is proposed downstream and the second one is pulled from upstream I will describe the difference between those and as I said what we do, we just take the archives upload them and we create the pull request with the needed changes so what does the proposed downstream do you can see a pretty simple definition what's the job, it gets triggered on the release and where do we propagate the changes so what does happen proposed downstream lives in the upstream repository so it's very good for the maintainers that have control also over the upstream repository so you create a config, you enable the github app and well we listen for releases and if you have it configured, we do our job and the other one is pulled from upstream and there was a demand for this feature because not every upstream likes to have the distribution specific packaging files in their upstream repository so what can you do, you can switch to pull from upstream you define the job in the disk gith which is directly in the Fedora and what we will do we will just listen to different service that provides the notification for release and you don't even need to touch the upstream at all okay so we also have advanced configuration, there are actions so you can configure the way we create a pull request and also the change logs if you want to see some files you can configure it and by this time hopefully it got merged so there should be running build and we don't have video like that unless I missed it I missed it okay so there is no build yet so I will carry on and this is basically what I just said so we can skip this and the cogibilt so what has happened here we have the changes included in the downstream what do I need to do I just basically need to build the rpm right so in this case we have a cogibilt job we listen only for our commits and our pull request and the reason is that there are happening misery builds and we would trigger the builds for those and we don't really want to do that and also other use case is rebuilding cogisite I won't go into those details so we don't want to run for everything just what we know that is ours and makes sense for us to go and that way we get to body updates so after the cogibilt is done what do we do we create a new update in the body you can review it you should test it of course before pushing it to stable and then after it gets to stable user is happy and can use it so let's have a look if we got yeah we have a one build running and I'm not really sure if that build will make it till the end of my section and I will give the floor to Maya that will describe the upstream code into VM changes now we talk about how we can take our upstream changes and get it into a complete operating system to be able to test it into the system and first of all we need it changes and we for example we will use this simple project we added back it and the change it is just changing the color of the hello world sentence and after this we should put this changes in a RPM but this RPM should be shared with others the release and the way to do this is to put it into a compare shareable repository and last step is to use the image builder from console.redact and we inspect it oh yeah and the last thing the final image can be any of this so there are many different kinds of images and we inspect it can help you with these different steps how you should create these two jobs inside this package yellow and yeah the first job is really simple it's the comparable job you are telling us how we should create the compare builder for which target the second job is a bit more complicated but most of these data are needed to customize your image so are needed for the image builder yeah we are saying here that we want an Amazon Web Service image and also that we want to install in the final image the yellow work package and we are also sharing the image with our Amazon idea Amazon Web Service idea and this is almost it just one more small thing by default if you don't specify anything package will create a temporary project copper project where to put your custom RPMs and the image builder job will use that compare repository in the case you want to customize your copper project you need to put these keys on your end project both in the copper ps job and the image builder job yeah we already said that we can share the final image with our Amazon idea and final step when you are ready when you want to test your change then you can comment your full request with this comment we don't do this automatically to save some resources we think that it's not always work to test the last change we have done since the an image is something big that will let you tell the system when you need it yeah what you get at the end is this link it's a link to the Amazon work services that brings you to your image and there is also another way in which you can do the same thing but without using the service instance you can do this by your command line and this time you need to create a token to be able to assess the image in the service you need to put this token into the dot call feed back into the Amazon you have done this you can simply build your sources through this common into the image builder through the image builder this time you will get a link to the image builder service and into the image builder service you will find the Amazon work services link and in both ways you can arrive here you can launch your Amazon work service instance and you can test that really will change the color in the lower work and yeah that is all for behavioral machine ok so if you are interested in any other functionality that was mentioned here is our documentation where you should be able to find what you need and now let's talk a little bit about the present and future of package so in package we are still developing new features but of course we are focusing on something more like on the others currently we are trying to make the dancing automation more and fitting multiple use cases of dancing packages and then a big focus is on the VM image build where at the moment the functionality is really simple as you could see but we are definitely trying to make it more complex and customizable so these are all the steps we have been through the only thing left is the Q&A but before that if you want to get in touch with us there are multiple ways you can get in touch with us on matrix, in flag we have a postal account and then if you have an issue you can also check our links based on github and we will definitely answer you there and although if you don't know what to do next on the conference we are already having a package that we will upload today and tomorrow in the e-blog and then there are some interesting talks you could see so we have mentioned in the beginning the testing farm so testing farm is having two talks and then if you want to hear about how our users use packet you can go to the journey of automation talk and that's it so now it's time for questions, if you have any yeah, go on I have a question well how could it be so so we have a midfielder about pitching from the software yeah some are here some are, yeah, thanks well in life packet monitors have to deal with many patches many rebasing patches and so on and so forth I understand that it's near impossible to automate rebasing patches but are there any best practices for using packets in such real life integration thanks yeah sure, so the question is when you have a lot of patches in the downstream how should you use a packet with that and automate it we realized that there were some issues with the patches and if you have a very lot of patches we had a source grid initiative and that was led by our former colleague so the idea was to basically unpack the sources and have like mirrored repo that acts like the unpacked archives so you have the whole commit history and on top of that you have the patches and you were able to locally rebase it and we also tried to automate it and I'm not really sure in what state it is so you could probably reach to us at the book and talk about it in more detail thanks and maybe just to add like to cover mostly the cases where the packet is very simple and automate these type of packages any other questions yes, no? yeah in general we were talking about if we didn't build the machine here I would just wonder if you're planning to add the built-in container-based even for example to that and how is it related with both manuals if there's any plan in such process because you're the machine so I'm thinking about the container-based yeah you want to answer? yeah yeah cool yeah so like in fact if you can keep the you get the VM view and you can view whatever you want to be able to have the three to be three you know you can have the VM so we can you can see on there of course it would be not but if you want to run it you can tell it's a regular tool if you want to run it you can tell it's a regular tool yeah so you need to there will be a problem that you can't be able to because if you then you need to pass over the slide to see if you're able to register it and upload it yourself actually arrival is building or testing part of their own VM view so we have not really seen the content okay so the answer came from Mira from the testing farm and just to add so yeah in the testing farm you can either use the copper build build by us or you can keep completely this step and then I also forgot to mention that in our future plans we would definitely like to integrate the building of the VM images with the testing farm that's something any other questions yeah let's see so the coaching build is done and we should have yeah so we have the update there created 9 minutes ago and we see the changes and somewhere there should be also instructions how to install it and you can test it, provide karma and if you are satisfied it can get shipped to stable I think we have a zoo and in body we don't have the tests but we also have a testing farm for this so why would we test here okay what do you use the reference for the it can be used for debugging if the production we move the production once a week so we should be able to tell which version ran the job but in the case of stage it's tracking the main branch so as we merge it gets deployed to stage it makes it easier to check when the bug will be introduced so in general if you are using the github or github integration and you want to verify the changes upstream you will have the configuration in the root of the upstream github but if you want to spec it only for the automating of the releasing to Fedora you can only place the configuration in the github and don't touch upstream at all so to sum it up on the upstream you can run copper builds and also the upstream cogibules and tests and on the downstream you can define just this part so it gets released, you create a public request once it gets merged we build in Koji and then create in Koji I have a question are there any plans in both automations it's a topic that we discussed multiple times and currently there is some ongoing discussion with an outside contributor and we might consider it but it's not our primary plan ok anything else if not then thank you for attending this talk and see you at the booth maybe if you have anything else to answer welcome everyone today I'm presenting what is about being an open leader in the real life I'm Stefano I'm long story short I'm your manager and also manager of the community of practices of public leadership so before starting talking about open leadership let's set what is a leader in this context at least a leader is not something that is endowed by formal authority is more everyone company organization that could do one or more of those functions provide vision and strategic direction clarify the values establish or reinforce connection create and maintain momentum mentoring and coaching basically what normally we we think leadership from a general perspective is not necessary in a company and this is important because again is nothing related to hierarchy is nothing related to any formal or invested authority but before going we are talking about open leadership so I would ask you if you wanted to play just a second with me with mentee there is a code if you can use it or use mentee.com with that code and well open leader has five well known characteristics defined formally by a document on what is open basically that helps an open organization to grow and to act as an open organization but before going I would like to see if you can put one or more single word that define the characteristic and see what do you think an open leader should do or should be most probably they will not be strictly what the final document is a start for our discussion then I will present you the five characteristics and since I know perfectly that there is art to remember I try to help you to remember them open minded self reflective mentor listening inspiring accepted open of course inspiration trust trust and transparency the most voted of the movement let's try with mentee.com and the code that is easier so let's see mentor and mentor is more like the same caring interesting one direction direction should pose trustworthy transparent transparency again open minded trust the most voted one so the five characteristics while you are still writing transparency inclusivity adaptability collaboration community those are the five characteristics as defined from open source let's move back to my slide and as said that you are pretty hard to remember let me help you with some animals easier to remember images and animals than words so transparency see behind and see inside basically this is the idea but what is transparency in this context please note that I'm trying to speak with you and to talk about what is being an open leader in the real life so what are the challenging more than the characteristics of course this is my favorite one is for me the foundation of being an open leader be transparent is the most important for me and I found everything on that also the other four characteristics but every characteristic is interconnected as we would see so feedback is a gift you need trust you need to make open decision create safe space and see failures as opportunities this is the way to be transparent an open leadership but where is the challenge here the feedback is always challenging both if you give or if you receive feedback but is really the foundation of the transparency because if you don't ask proactively feedback you will never get as a leader because everyone look at you especially if the leadership for me I'm also manager not only a leader and this is even more challenging because people reporting you is hard for them give you a feedback so you need to ask proactively always and create a safe space for them to give you feedback and safe space for me and me not only that they feel safe no reaction but also that you will apply their feedback on your day by day job and this is very important and the same is on open decision and together with feedback make trust because when you get a decision that has an impact to someone you need to be transparent with team order and everyone else because all the transparent decision create trust and trust is very important is a challenge especially when you cannot be transparent or people expect that you would be transparent and they think you weren't recently unfortunately we had a few radio a lot of radio and when I am now if my team has not been I said I didn't share anything with you because I didn't know because the most important part transparency is to keep the trust of the team and also when you know something and that's true I didn't know but also when you know something and you cannot share because there is a situation where you know and you cannot share transparently saying I cannot share this information with you not at the moment be patient and there is good reason because of that and if you can explain the reasons not to the topic not to the content because you cannot but explain the reasons because you are not sharing and when hopefully you can share and at the last point see failures as opportunity your own failure not only the failure of the team be transparent on your failure I am always transparent with my team when I fail because it is important for them to understand that failure is not an issue failure is an opportunity an opportunity to learn and we will return on that on another characteristics inclusivity inclusivity we are is more than diversity equity and inclusion is not just that that's important it's part of but being inclusive with your team is something more is is know and share that you need others perspective and tell them that you cannot have all the view all the perspective because different cultures different positions and maybe different views on things concur to have the team opinion and the team direction and this is to be a really conclusive other perspective and other means that you don't have all the the large view that you need to have to take the decision and so on my team is distributed in I think nine countries and this is a challenge of course a challenge for culture a challenge for time zones a challenge for many reasons but is also a challenge from inclusivity perspective because there is a I'm jumping to the third point because you need to be aware of different cultures, different biases my biases maybe is different from my colleague or maybe is the same but has a slightly different level and so you need to reconcile and go over the biases and the same is inviting others to play their role in your development and it's sometimes challenging because you have people that is maybe very closer or not not really to step in any discussion but inviting everyone to participate is part of the role of an open leader even when it seems forcing a bit them but it's not forcing you you need to be gentle of course but it is important about and probably one of the most challenging sharing my personal person don't be essential always make a step back has been very challenging for me becoming a leader to step back and don't think to be essential well I'm ambitious I work hard I try to work smart but still I want to be part of the team still I want to be part of the action but being essential if everyone depends on you is a big deal is a big deal to everyone to the team and to yourself also you need to step back a bit don't be essential and only in this way you are here and observing what happening that you observe and listen is too different and very difficult classes that you can bring in the team because if you talk too much and if you give all the directions you are not inclusive and probably many people will not talk, will not bring their perspective and so on so try to step back especially if I endorse the new role of manager like you step back because the opinion of the manager is as always a different weight but should not adaptability the adaptability is is another important characteristic and because you know we all that our organization is flexible our organization is resilient is able to change and adapt to the market and have new challenge always and react to this challenge but if the leaders are not flexible first time and the leaders are stick on their decision or on their biases the whole organization the team and then up and up and up to the organization is not flexible is not resilient and here the feedback again how can you not you need to know what you did what I better add and so you need feedback always and again if you don't ask feedback you will not get feedback as team probably for sure as leader and feedback is very important for them they was very fast failure I extended this very fast learn faster and adapt quickly to improve again transmit your team that feeling is is fine is a learning process but of course you need to fail as fast as possible because if you fail too late speak more than as usual and there is nothing anymore to learn or to adapt while you need to document what is happening all the things is basically your role as leader and create affection inside the team discuss and perspectives for example to bring you a story in my team we are so used to have perspectives that we have at a different level not at the release date but with a cadence that is monthly or quarterly depends on the project we do have perspective of the project what happened during the quarter what happened during the month and what we can do better in a different way something that you work on perfectly and so we want to keep on but we bring this discussion also at a personal level when I have one-to-one reports I always try to have perspective of their goals which is not about assess their goals we need to do that with the perspective of why they work in a way instead of another and what is the goal they reached on the goal where they didn't work at all and why and see if we can adjust something and the last point is very connected with previous characteristics is to adapt to different people and cultures how is it hard to adapt your style and your leadership and maybe your needs or common needs to different people but especially different cultures if you have a Brazilian guy and a German person working with you you learn that the culture is so much different so it is not about commitment or things like that they are both very committed very accountable both but the point is that what you say then will be taken in a different way the feedback has different weight for them so you need to adapt to different cultures and the best the best way I found in the years to do that is to have a common culture and our common culture is our company culture and our company values so this is the common ground there is even if they have the background and so on he knows that we act with those principles and the values of that the four values collaboration the first sentence has been copied and pasted from the open leadership definition and I think it is the most important sentence of the slide because the key of an effective collaboration is achieving mutually beneficial outcomes for everyone who collaborate as person and of course need a lot of negotiation negotiation is my day by day job but not only me negotiating with other other for a proficiency negotiation with other people or between people in the team and so the second point is that my role is the role of the coach I am coaching my team to be collaborative they are great engineers but sometimes great engineers tends to be very personal and very so much focus to their task that they use the idea of collaboration I ate single person project when I got the team basically everyone has a project by him or her and with very little collaboration I mix everything because it is important to contaminate others with ideas with fresh ideas and the other point the open decision basically the open decision framework consider that you as leader or manager has me at some point you will know that today's decision in the end is part of your responsibility but is respectful to involve as much as possible people involved by these decisions in the decision process not only being transparent on the decision and why you took but involving him or her or them much before why the decision process started again being transparent as much as you can there is a situation where you cannot be transparent of course but try to limit as much as possible those situations and involve them in the decision because that decision will be much better accepted and it will be a much better decision also because most probably you don't have the full context to take the decision for someone else the one affected by the decision is probably the person that has more context than anyone else and celebrate collaboration success this is a great point pretty hard to distribute quite frankly my story is pretty hard but we try to celebrate not the success of the team the success of the project but when the collaboration draws to this success there is a likely difference but very important to try to have a more cohesive team finally community who else is better than the language for communities an organization a team, a group is really open only when the acts as a community this is the the basic milestone what is doing a community a group or a company well the most important part is having shared values and culture work a lot on that whatever it is the company culture is already written maybe for you or you can create your own culture but try to somehow document somehow make it explicit for others it's just common practices but common practices are not culture from common practices you can build culture but culture is something more something more formal somehow even if it comes from the formal and so on that facilitate the great coach and other whoever comes to the team, to the organization need to be coached and to a bigger to the culture I listen and talk today I'm going to talk and in the end the speakers said that they select people also trying to understand if they can get to the group or the culture which is a great approach it doesn't work always interview is an interview but still still you need also to to adapt the culture sometimes to to the people that form your group or your organization it's important again and we connect as you see community connect to adapt to inclusivity, to transparency everyone is interconnected it's important to be an example like the culture of the group or the team otherwise no one believe you no one trust you when you say we have this culture but you do the opposite we have the culture of transparency and you are not transparent no one will trust you and no one will be transparent that's it I'm happy to have 8 minutes instead of 5 first one yeah so I have a question regarding culture so as we are working and you actually need people in different countries so how to maintain culture and how it works how it works so I think could be challenging could be challenging to obtain culture for the people I'm more important that for the people because that doesn't matter if they work at home or in the Canadian office they are in Canada again being an example and applying the culture that you define for the group or for the company in our case we have a strong company culture to everything in the group and ask to do the same and try to be very clear that that's the only common culture we have I have 17 17 people to in direct and in direct and only 2 are in the same country everyone else is in different countries in different time zones in different geographical areas and so on it's very challenging and I always say them we cannot rely on our background because it's so much different that we will not understand each other try to rely on the company and the group culture and then not the others did I answer I have a question for the presentation my question would be how do you define between having everyone understand that they are welcome in this process of making decisions but also while avoiding having people that feel for to be involved in that decision making process so that they sometimes decision making processes and having not so constructive conversations because some people don't give constructive feedback it's kind of important to something they don't really care yeah sure this is of course a challenge and well the very first point is to share everything by default and share it early so basically my meetings all my meetings have a background document that needed to be shared 24 hours or 12 at least before the meeting and when I say all my meetings is all meeting of my team not only driven by me I'm pretty lucky having other leaders that work with me and I'm very happy to give them the space to lead but the contract is no document no this is important because it gives time especially to interact to digest the topic maybe to give their contribution offline I have a great engineer who gave in the past and still work with us and give us great feedback online because she is very and that's fine and she knows that's fine because I told her that's fine don't you want to talk no problem but having both way is important otherwise she will because I know that she will never take the mic and say I don't agree it's part of her culture by the way to return to the culture to return to the culture is something that I didn't mention is important to know that you have different culture as leader and try to adapt the way you communicate to permit different cultures maybe for the place where they grow because of personal reasons to contribute to the discussion their way, not your way this is an example but there is many other way to do the same is the way I found in my team I need to work because it's too large I need something else but the important point is give them the space to express themselves is it the follow up question is it a shared understanding within the team so is the whole team aware of all of these differences how do you coach the team and these differences so that no one will feel left out I meet them once a week my partner is pretty crazy one-to-one and then I have no length every three weeks no less and I always try to coach them during the one-to-ones and during the whole length basically I set the culture so I try to explain anything that happened at my level or apparently in a transparent way and doing that I try to set the culture and I see that sometimes it's hard I have just two or three people taking the mic in this whole length and this is another challenge of course when you have many people and you want every perspective you also need to limit a bit the most extra work it's a balance always did I answer alright everybody we made it thank you so much to be patient so here today it's me I'm a senior agile practitioner for the for agile system engineering I've been working at Red Hat for three years more than three years and I worked within OpenShift and now I'm working within Proud that's why today with me I have Mirena she's one of the engineers in the Red Hat team that I work with good afternoon and nice to meet you all I'm Mirena Diaz in the Red Hat team and we are responsible for Red Hat as well as for the IMP so in this talk we're going to try to explain how we will manage to create a successful environment so that the team can get better how our specific team transformation will and at the end we'll finish with a couple of points that we should take away from this talk alright I think we all know that change is hard right and to make an environment to be successful we really need to try to face to handle the pace of change people deal with change in different ways and we need to take that into consideration people go through different phases and I want to show you here an example I will be using the fabulous Ross model and explaining this with a real life example I think each of us here probably had the experience this situation we basically had a key team member leaving the team while back it was a high performing individual and we had it was a great loss for us basically so I want to show you this with this example the first things that happened for us were that we were all in shock so we were all surprised about what was happening and the second phase is the denial phase so here is when we are actually trying to look for evidence that that didn't happen so I remember personally talking to someone in the team and saying no I cannot believe this is happening is really true then people usually move to the frustration phase and this is even the phase where people are angry so they usually try to blame themselves or they try to blame someone else then the next phase is the one of the depression where there is a lot of mood low energy, lack in productivity and from this point on we move on to actually more positive phases so we start with the experiment here is where people try start to deal with this new situation and they start to be more positive about that and then we go through the decision phase here people they start really accepting that things changed and they feel more comfortable and positive about the whole situation and the last phase is the phase of integration here is basically the change is now a status quo and you know people go back to the normal life basically so people so I was saying that we really need to handle the phase of because as I said people go through those phases and some of them they just go through few of those phases other people maybe go through all of them one by one and people move backwards and forward between those phases so people really need time to actually deal with the change that is going on so I wanted to share with you what I believe is the foundation of a culture that is needed to create a successful environment for a team transformation the first thing is creating conditions where the team can actually start new behaviors and they do that through new habits when we talk about new habits new habits are basically reinforced patterns and they need to be consistent consistent in bad days and good days so really consistency here is the key to enforce new habits so they can change behavior secondly it's creating a goal a goal that is inspiring and scary at the same time and that it goes behind comfort zone for people a goal that is going to sparkle imagination that is going to engage heart and mind for people so it's going to be it's going to bring all together so that you can actually go through the silos break the silos that you have within the team or even within an organization first thing is to actually start practicing transparency and open communication so basically people really need to understand the why behind the decision or at least they need to understand how the decision was made people I think we can all say that we use at least to say that that the sorry we use to say that you know the change is needs to be it's difficult for people and that's why we need open communication within the team the next thing is about creating an idea of meritocracy so great ideas comes to people attention to the leaders attention on different ways it can be because they come directly from the leader from the leaders or because those idea comes from the people within the team or an organization so they can come from different levels that you really need to make an environment and a culture where people are not afraid to speak up to share feedback and you know to be open there is always someone that is going to merge within a team so you need to try to create as well an environment where people where a leader can emerge in that so that's really important as well last but not least I haven't mentioned this word yet so maybe you were expecting it it's agile principles and values so the agile has a lot to offer in terms of transformation if we think about limiting the work in progress or if we think about creating self-organizing teams so if we think about prioritizing the work so that people can actually focus on delivering value for their customers stakeholders and we have as well the values that are really relevant like commitment, courage openness, respect and so on okay so I think I'm going to go now to Irena to talk about the team so I think our team is six people which are the developers we come from different backgrounds there are people who have been in red hat for more than 10 years and so on others like me are more new team and so on our team is fully remote that is something that causes some issues because we are not used to dealing with people we don't know each other and sometimes there is some kind of context that is lost you cannot yet simply go to someone's desk conversation with them in order to find a common ground that is something that is lost being fully remote so do you have to make the most of the meetings that you have scheduled and so on in order to be productive and so on another thing is that the team has different familiarity with agile there are people who have been working for agile others like me we studied it at the university but this is our first job with the framework so we had to learn it again and of course the team is still growing because we are building problems in an emerging market so the team is still growing as we try to identify which are the areas that need most of the focus regarding to our scrumping we have all the roles that you would expect we have our fantastic is from Matthew which is Eleni over here the product owner and the rest of the team are the developers so let me talk to you based on the Tecman's status of team development what happened with our team so Tecman says that there are like four different stages performing storming performing stages and there are a couple of behaviors that are more common in those stages for instance in the performing stage everybody in the team is feeling very excited because I mean there are new things that we want to work for I mean we want to work on but that excitement is common but also there might be some anxiety going on because we don't know what the future might hold and overall we can say that there is a low task accomplishment because we are getting used to the processes the team and everything then we would get to the storming phase where we start to understand the goals for velocity and yeah we some disagreements might arise because of the processes and because we are trying to understand velocity and some IELTS concepts and so on and yeah then we get to the something in the storming phase we also have to take into account that people have different personalities so we need to try to identify which personalities work best with each other so that is something that we also need to take into account finally we reach the enormous stage where we can see that the goals of the team are better understood and we know what are the things and the processes that we need to do in order to so overall we can say that there is an emphasis on team goals and we try to help each other because the goal is to perform as a team and then the final stage is the performance stage well each member in the team understands what are the strengths and weaknesses of each team member and then we can plan and we can try to make the most of our team so this team formed in 2021 and since then I mean this process that you see here is not linear so in our case we had new team members something on the left and so on so on one stage we were in the performing stage but due to these changes I would say that right now we are between the norming and the performing stages ok so now we want to share with you the challenges that we faced during our team transformation so I remember being in some of the meetings that the team was having and they were discussing about can we actually move this user story to them so that's where is this happening with anyone in here? Perfect, yes that's normal so I actually helped them and guided them through the facilitating the discussion about the definition of DUN so that they were all aligned on what DUN means for the team then we had some other meetings where they were actually trying to estimate user stories and something else that happened was I was hearing them saying should we go for a three or a five let's go for a five because it's safer is this happening to anyone in here? ok cool and another thing was sorry so I actually did some training with them about using story points and you know help them to understand how to estimate so we aligned them as well on this topic this is not happening anymore right? maybe we need to do another section then then we have we had they had a lot of things to work with and there was a problem that like they had too many things to handle so we really needed to understand which one were the priorities so I helped the managers the managers, the leaders and so on to talk about the priorities and then we actually talked to the team about this and then we have the capacity planning that's another challenge but I'm going to tell you more in a second about this so I'm going to show you now where velocity chart so here we are at sprint 31 to start with basically they are already having a change here because this was a team that was basically a bigger team so it was they were two teams together at that point they were one but sprint 31 was the sprint where the last sprint before they actually went they split it so the teams were going in different directions so they decided actually to split between them and so that's why we see those high bars in here because it was even way more people working together so if we look at sprint 32 that's where they start working together as a team here we were just before Christmas holidays people were like yeah but don't worry we can do everything right we're going to be able to do a lot of things and so we see that they actually committed to story points and they were able to deliver only eight but you know it was Christmas so it's alright so we go back in January and you know we still had people doing the onboarding as well because we had new people joining but still the team was like yeah we are all back from holidays now we are all here we can do a lot of things together so they actually committed to do one more 48 and they delivered only eight points again alright so a spring 34 it's when the team started to feel a bit more confident the onboarding was going better for everyone and yeah they were like yeah we are a team now cool we can still do a lot of things right so they committed to 43 and they deliver 10 a bit better okay so here we start seeing some challenges first it's about the capacity we couldn't really understand the capacity of the team at this point because they were carrying over stories sprint by sprint is it happening to anyone yes yeah a lot of things so we did try a thing so we decided to actually go back to our definition of and review it and it was specifically a point that was saying we are going to move a user story is going to be moved to them when the PR is going to be approved and merged right cool yeah that's what we were doing until now but we had to review that because of this carrying over the stories sprint by sprint so we changed that to we are going to split our story into two different a story and a task basically the story was the implementation work for the team and they were like yeah I did my part right it's all done from my side but then they needed to wait for approval from upstream or from external teams and so on so they were like yeah I done my part I can move it to them so we did this we split the two cards we had the implementation one and they could move that to them with the story points they estimated at the beginning and we opened a new task basically there was a tracker just for the PR with zero story points we were having to hit there just to remind there is still going on so let's see what happened things are going better now right 43 committed 30 deliver so that's a big change I would say now for them happy times but still there is still a problem here that actually we are over committing how many of you come on I expect a lot of fans here about other committing all the time okay good so we said you know what let's try something else let's try to actually take a step back and commit to only one card each so we tried this and we were 29 that we committed so less than usual and they were actually to finish everything plus that they committed plus something else so they deliver 34 at this point so you know we tried different things and so on so we experimented and we saw that this was helping the situation some of those things were helping others that we tried maybe they were not helping so we actually adapted and changed and try new things okay so from my point of view while there was this transformation going on I was actually transitioning from scrum master role to an agile practitioner role so I was there at the beginning with the team actually giving them solutions suggesting things and I had to move to actually guide them to come to find solutions themselves without actually imposing my ideas the other important point here for me was that I wanted them to get into a mindset of continuous improvement so they they were trying experiment all the time I mean every four weeks we were trying an experiment but at least to get them into this mindset of continuous improvement and that's thinking about finding a sustainable place for the team right because this way we knew what we could commit to and we wouldn't get to out that's what we don't want from team members yeah and from an engineering point of view I mean you have to understand that some of the messages my eyes which are for instance back that affect our clients so no matter how perfect the planning was according to three points and so on there are times where you must drop everything and take the feedback which is a task that doesn't have a lot of story points but you must do it and finish it as soon as possible the other thing that you have to take into account is that we are an operating system team so although we don't work designers and there are a lot of processes that we don't control so even though we said from our point of view that the work was finished there is still the dogs testing and other things that have to meet approvals to our task and that is why speaking the cards worked for us because it also helped us get to that mindset where we were accomplishing something because we saw story points moving to them and finally, yeah from our point of view the story might be finished but we cannot seek something that is not tested and that hasn't been thoroughly documented so we need to focusing on one task allow us to help those other things in order to get the information that they needed as soon as possible so that everything was finished as soon as possible and that story points could be looked to down so does agile really feel engineers for better so personally I come from an academic background and we work with the waterfalls development environment and I find that agile of course is more flexible and it does not only allow developers to define their processes better but there are specific like meanings where you can look back and try to think what is what is working, what is not working because with the waterfalls development environment you have that psychological bad situation where you think that okay I haven't finished this deliverable time or I'm running out of time to do my testing so the idea allows us to identify what is working what is not and what we can do about it so we wanted to share with you some of the lessons learned from us for this transformation it is still going on I didn't finish so first thing is deconstruct speed change into smaller steps change fatigue I really believe it is now a thing and we need to try to people go through the different stages that we saw so we need to take into consideration that they need the time to adapt to the changes so don't do changes just fire changes one by one they need to have the time and that's why don't come over like oh we are going to change everything from today to tomorrow that's actually deconstructing big change in smaller steps next thing is about changing the behaviors of the team through their habits and especially very important I said this before is try to get the team driving actually the continuous improvement themselves within the team third thing is to try to build a team with enjoyment of practice enjoyment should be part of our daily life but when we talk about our professional life we usually forget about this and that's very important so try to schedule some team building activities or even give the space to people to actually talk about their passions together and last but not least focusing on building the right things so we are here to satisfy the needs of our customer or stakeholders so we don't just have to build things but we need to actually build the right things yeah so we are at the end what did we learn today here first we created an environment that encouraged and enable the teams to develop and mature and that's how you build high performance second this is our journey so we shared our journey that is not like other teams but that's definitely unique to us so you may or may not be sharing the same challenges and third we were successful building the team and our performing team but as Irene said earlier we weren't at that stage of high performing and now we went back and now things happened and now we are trying to get back to be an high performing team so yeah change his heart right okay so we are going to leave you with this sentence to reflect about the transformation takes a long time and value realization typically takes even longer thank you any questions yeah the first one is very first come what do you mean no yeah no we started with Trum and we continue with us yeah in this case what do you think and the second one more probably but did you think about the changing environment which is great and you did a great job yeah how did you design the changing environment and is an experience yeah changes the problem yeah so it's okay oh yeah sorry so he was basically asking how we deal with change fatigue nowadays right so as I said this is a thing going on and what you really need to try is not to have too many changes all at once as I said before you need to really understand that people need to go through all the phases that I explained at the beginning so you need to give people the time to go through this process right so right there is a change going on let's try this we are going to do that but give them actually the time and the space to be comfortable with that change and then you move on to the next step so this way you are going to try you are going to avoid the change fatigue because if you are going to have all the changes all one by one after the other one then it's not going to work that's when we arrive at the change fatigue phase right yeah and the meaning point to you your team needs to communicate as soon as possible and if you haven't set up the environment for free communication then you have time as an again for the change sorry I'm very glad that what she said because it connects to what I was trying to say so I work mostly in upstream so my team is a couple of people from Google a couple of people from Amazon whatever and again communication is basically the main tool that we have we don't have sprint but we try to have some kind of periodic call to do this but I cannot tell people what to do organize work cost companies because my period is not there and my manager is not there especially so what are some kind of agile concepts that I can learn about and apply in this completely different situation I think that's collaboration in here yeah sorry so here he was asking about cross collaboration basically between colorated teams and different teams working together so I think it would be all about collaboration together and try to communicate so try to find at least some times for example with them we couldn't really find the time that would suit everyone but we settled in the end of finding two hours in the week on a Tuesday and a Thursday to actually meet all together just for one hour to start with and try to do all the things all the things that we needed to in our case we have one hour every two weeks hopefully with some people doing it at 6am because it goes from Pacific Coast to China yeah I see as well we use Slack for example and some of the communication are over there as well for the teams it's not the same thing I know but yeah try to find some time even 30 minutes maybe 3 times per week to gather all together okay and one of your suggestions of splitting the cards I think is something that happens with upstream so I think doing that with upstream to get over that point because they won't fit into your sprint schedule if you try it just will never work so I think it is to split the cards and let that happen because otherwise you can't it's a different schedule it's just a meeting you have to you have to you have to yeah so there was one good question what happens when you are lack of senior developers and you guys feel like a lot of hard topics and you cannot split them into senior developers you have maybe criminal issues and then you have a lot of hard topics a lot of senior developers and you cannot be given that because it doesn't seem to happen senior developers yeah so basically he's asking how do you deal with the fact that senior developers and engineers don't really want to split cards or lose time doing that all these actually came from them they saw the challenges from them they are here they actually saw the challenges and they started the discussion just helped them facilitate the discussions and they were like do you want to try to actually improve the situation what we can do about it let's talk about that so it's not something I did in post to them it's something that came from themselves and that's I think it's a very important thing because if you go there and say okay we're gonna do this and now you're gonna do it like this you're not gonna have people happy about doing that but if you give them the time and the space as well to try to understand what are the challenges and let them see as well what are the challenges then it's gonna be different and they can start having those discussions together and they were really worried about the velocity why it's like this it didn't come from them we found that in a lot of cases with the senior developers getting in without the power of the smaller ones and helping them actually better help the junior developers and actually help their own understanding of the problem that we're trying to solve exactly for us anything else yeah you said that the change was made on the team basically how many people did not survive the change in terms of how many people are affected because of the change we are all here we didn't lose anyone and just the guy that decided to move on just for personal reasons it wasn't nothing really related to the team so yeah we have all of them here they are over there if you want to talk to them later yeah cool alright thank you so much thank you hey everybody today we are here for herding cats standardizing engineering processes so what we want to talk about today is standardization and looking at this many people in the room you might be in the wrong room because nobody likes standard at this launch we totally agree standards can be really annoying and limiting but we want to change your mind today and show you some best practices on how to actually use standards to benefit your teams rather than add burden to them we use the term herding cats but before getting kicked off to make a really boring topic a little more humorous we wanted to use a fun title but wanted to explain it a little bit more too so herding cats and to put it to life want everybody for a second picture every person in this room as a cat there's no quiz at the end we're not going to call anyone out and say oh my neighbor's grumpy cat or anything like that so don't worry who you're picturing is that picture all of you cats being asked to come to this room it would be chaotic there would be a lot of screaming a lot of pissing, a lot of meowing and very few cats in this room and that's really what herding cats is about is trying to drive people and teams that have different mindsets and different beliefs into a common goal and that's really what standardization can be like at times so if it's so tough why the heck are the three of us here talking about it and why do we continuously subject our teams to new standards and new ideas a little bit about me is that I joined Red Hat a little over two years ago in the Red Hat in-vehicle operating system team when I joined that program was in its wild wild lesson days we'll talk a little bit more about that in our next talk but really we had a blank slate to start with we didn't have any processes we didn't have any standards and we were free to find the best ways for our teams to work to enable them to thrive and at that same time I joined the technical enabler pillar and I was learning all of the awesome things that they were doing across the entire portfolio and delivery team in RHEL and beyond and wanted to find synergies so we could actually collaborate and find ways for our teams to work together and really that's how the three of us started working together Hi everybody so I used to work with RHELP like engineering teams in RHELP like in vehicle operating system we were here for first time so you can imagine we had all those processes in place but we not always understood the processes that were in place or knew why they are in place while now we're a different team with different dynamics the dynamic in RHELP was steady as 30 all together with different mindset we are trying to bring all those people together while best and steady as 30 process with RHELP so it's a challenge and here we have also Marcella I'm not so much like that but I think that I've had for 17 years and most of the time I was working with RHELP in various roles like project management let's say agile coach and I was looking at RHELP and how to work with them better and we were told that I think making them more accessible to the audience alright so we already introduced the topic but a little bit more about what we're going to introduce today and it's really the fine line of interactive standards and standards that actually help your team to reduce the burden and make their job easier not only within your small team but across the portfolio as well and then we'll give you a sneak peek into some of the things that the three of us have collaborated on and done to to really show the light of how standards can help but I want to ask everybody here who likes being told what to do oh we have one person who's going to leave you guys are all set oh no we need them actually please join us who here likes being told what to do and how to do it probably even less people wow Clara loves that Neil likes some certainty in his life not a lot of people like that and a lot of people like being told what to do especially when there's not really a reason to be told what to do if they're not understanding the purpose behind it it gets annoying it feels restricted, it feels like you're not trusted and it breaks down every rule that Lucia presented in her talk yesterday about trust, respect and humility but still we want to subject you to a fake statement about itself I'll let Lucia go against all of her rules alright so you can standardize almost anything but for the sake of this presentational goal which is workforce because they are easily joined as you can see on the screen this workforce consists of three statuses new in progress them and I would like you to raise your hand if you think this workforce would work for you and your team pick up all of your hands alright LeFontaineot everybody I saw maybe three hands and it's a common situation if you show something that you are thinking about there is not a lot of buying LeFontaineot at the first place so if we move to the next slide we don't want to start with anyone as a persona so we are using proto-persona it means made up persona but they definitely come from experience so here I present you Edel, Desi and Ada those are our three proto-personas for today with all valuable reasoning for extending the workflow we presented earlier so Edel, his software engineer works upstream and sometimes the issues upstream can get in progress even sooner then you start working on them or vice versa so he would like to actually have a status to call down that something in upstream is already in progress not in progress in general so he would like to have that specific status available right? Desi, she is a project manager and she goes through a lot of approvals and pre-approvals so she would really benefit from adding pending approval status like it would make her life super easy and Ada the third proto-persona that we have she is a product owner and before she puts anything to the team's backlog she would like for them to work on she would like to have some mock-up for stakeholders with you reasonable alright, so we if we take all of these suggestions in mind we have interesting workflow in place so those of you who raised the hand or those of you who hesitated to raise your hand would this workflow work for you? alright, so I see some skills facing the whole thing out I count for every corner case it's not realistic you are actually creating right here we would create artificial complexity at its best nobody would understand the hell out of this it's completely mind-blowing so would stakeholder review be the second state or only the last one and what does the afters mean everything would picture the type of definition differently once you start making those little tweaks to the standard for corner cases it's not the standard anymore but you might ask the question but you don't have to make everybody for the one standard and put them on one line okay, so take two let them have their own workflow alright, we could do that under one premise we don't need them to collaborate okay if we don't need them to collaborate and they will never talk to one another it's okay they don't have to know each other's workflow but in case we want them to work together or there is a high probability that folks will work on different projects or switch teams imagine you are an engineer and going from one thing to another from closer to closer or collaborating with someone else and you come to a new team new workflow is that really that different are those different workflows that different to do are they really that different so if we really wanted the teams to collaborate together there should be some similarity otherwise it's great yeah, that's me so I came to those four workflows and I was trying to create some reporting some dashboard which will represent all those data so, not easy because I had to always ask those teams what does it mean, what's approved does it mean it's also delivered or is it done or is it done on the upstream what does it mean so that was very confusing because it's saying for all managers or those who need to manage those data and represent them somehow to high levels and it's also confusing for teams as Lucy mentioned when you are doing even if you are working in one team and you should report something into another team and they have completely different workflow that it might be hard and it means that you need to communicate about data which should be readable and doesn't need more explanation but they need to be explained again so something is wrong also in my case a lot of stuff is not documented because it's sort of right on knowledge we are doing it for 20 years or more so it's clear, right it wasn't so we are trying to find a balance with it we discovered for a long time how to do the standard and that we can't compromise too much when we are creating them because then it's not standard because you will lose all the requirements you get from teams it's just something which is not useful to anyone it's just force to the team and fitting with the standard is not easy because the balance between complete freedom do whatever you want just give me some status what we are doing and provide some guidelines to our team so we go to the next slide and talk about what we try to do we really need to understand more of this stuff firstly we need to understand teams so we recommend to work with teams directly the best thing is to be part of the team and work with them every day so you can observe how does it work so you can propose something meaningful you can even see bottlenecks and propose how to remove them and then you have some business needs you need to report from the team of them so you have some requirements that you should make them easy for the team because team would like to create a code and not to fill some fields and reports so we must be useful for both sides and then third one we really can do a little complicated process that we need to understand tooling we are using so we can make it easier for people in our case we use all the mentioned tooling around data visualization and the processes we propose are easy to adopt and they can be scaled to bigger teams or different teams we have more best practices so we try to implement teams with teams and we also get some requirements from programs program usually comes to team and asks them could be just another field because you need this definitely for reporting but it ends up like the team is not interested so much they don't want to fill just another third field when they don't see any connection to them and why they should be doing it so let's try to implement something which doesn't bother team too much and if you are trying to go for standard don't do it just because there has to be a reason there's really a reason why the standard has to be in place I'm an adult practitioner so I live and try under improvement culture so ideally the standard shouldn't make people like worse but better so just if you remember this super complex workflow with artificial complexity don't implement standards like that the standard that is in place or you are trying to book something down introduce overhead or unnecessary complexity for the team nor program it's easier said than done but there has to be a reason for a standard and then Marcella had touched on a couple of these earlier on and first of all she had touched on it's important to be involved with the teams and being engaged in the teams but it's not enough to just be engaged with the team as you are collaborating on a new standard idea to keep them engaged throughout the process so once it's implemented you have to maintain that communication and maintain that documentation so the teams are aware hey this is what's going on this is how other teams are using it this is why it's important because the moment a team member feels like they understand the why they are a lot more likely to buy into the have so we find it very important to communicate and continue revisiting these standards you know Lucy touched on agile continuous improvement is a huge thing for us and for everybody in this room I would assume so it's important for us to continue adapting our standards as things change and keep the documentation up to date because up to date documentation means it's easy for people to find what they need and I didn't mean to flash laser pointer on somebody else so I apologize that's a bit of a way we saw you getting bored sorry so in our team we discussed what we can do similarly and also say some of our time to reuse our work but it wasn't so easy in the end we standardised on Gira set up because we really should end up with unified backlog where the whole team will see what they should be doing next what has the highest priority what is stuck in progress so we also have similar screens and consistent fields I have one example we are tracking in our Bakery class architecture where the bug appeared and we had three fields quoted so it was very confusing once it was architecture going hard there so going from one project to another it was hard to find a field but we are doing better now then there is traceability process if you want to also figure out what you fix and how you test it later on in the process it's good to have some process for it so we standardised on this one and I will use in all projects Gira automation rules I like my relationship towards Gira is rather high and the same with automation rules because you know there is so much that we can do with them and I was fortunate enough to work with Alison who is great with documentation and her automation rules just triggered another line of thoughts that what we could do in our team since we had already similar hierarchy, similar work flows I could just by two clicks implement really easy automation rules for us and picture this you move simple tasks or stories but you forgot to move the same like ethics one level a lot that we have ethics and it's taxing to do and the whole reporting system doesn't make any sense and it's manual work nobody likes to do paper work for Gira why not have automation to do it since Alison already standardised to process into one very simple it's not that simple to implement her work it was just few clicks for me to do the same for my team we already had similar hierarchy we already had similar work flow in a matter of few minutes I was able to for them to adopt and no longer care about switching the status on ethics if they change status on stories or one level below it's really that simple so these are the benefits that come from the standardization not the automation itself but what the automation gives to you ideally time or that you don't have to care about the manual stuff another one that is interesting and Marcel already test on this when she was test to have one unified dashboard if you are with big product or big program ideally you want them to want themes to see on the dashboard I know I know for the sake of teamness it's really good to have a team dashboard but at some point you have to see from the back and from the whole scale no matter if you are a manager or if you are an engineer you want to have the overall what's going on on the entire product or program for that sake the standard could really help you to build program the support and you can subtract the data also for the team one you can have two but really the building both are similarity and similarity in what you are using and then another thing that we had worked together to implement is a standard status report mechanism so back when I joined the tech field Marcel had really really long hair that was before she was tasked with moving status reports into a standard tool since then we've seen a lot of new hairstyles I kid but really it was a complex to ask there were google docs there were emails there were smart sheets there were google forms there was status being provided to managers in about nine hundred and sixty seven different places and that is an actual number but it was really it was complicated so we wanted to find a way that would not only benefit the management team who is consuming the status but also the individual contributors that were actually submitting the status so we all got together and we came up with this status reporting mechanism that actually leverages just about everything else that's mentioned on the slide here where everybody was using a standard workflow standard fields and then because of that we could implement Jira Automation Rules on top of it we could then have a standard dashboard that can be used across the entire enterprise so the VP of Linux Engineering the QE manager the automotive manager you know director everybody can look at a single dashboard and see the roll up of data and get a high level overview of what's going on in a really quick way and then also for the individual contributors just put your status in a Jira card forget it, Jira Automation does the rest and it's been really really helpful we'll talk a little bit more about the impacts in another slide here and last but not least a partner management process this is something that's really important in Red Hat, especially being an open source company we have partners that are logging directly into our Jira instance they are logging bugs, viewing bugs viewing feature requires all of that alongside our engineering team so it's really important that we have a process in place that will protect our data and their data so with NDAs being really really important we can't leak sensitive information so together we've been collaborating on a new partner management process which means the same fields need to be selected in the same manner across all the different products and programs to make a bug or a Jira issue be visible and by doing that we're reducing the risk of data leaks because engineer in product one doesn't have to remember a different process for product one as they do in product two, three, four and five so this has been really really helpful and it's still in the process of being rolled out and adopted but we're really really excited about this one so the impact we talk a lot about what we've done but what does this actually mean to the team what does this mean to the company and what does this mean to management well I talked about status reports status reporting mechanisms has actually reduced the time it takes managers to put together, consume and roll off the status report by 67% how do we get this number it used to take hours for managers to go through those 967 different data sources to figure out what story they needed to tell what was most important but now they go to a JIRA dashboard and they can produce a status that's worth rolling up to the VP level and beyond within minutes so that's been a huge huge win and more teams that started adopting this the better because that means less and less places that need to be done or need to be done need to be reviewed to actually pick up on the story then also this is another one Marcella had touched on a little bit but a 63% reduction in number of fields on screens for the programs that we are working with we've aligned on different fields like architecture and I think 80 other fields to figure out a standard use case for these fields to figure out what was important about them, why are they being used and then being able to trim the number of fields that are on those JIRA screens so alright great you reduced the number of fields why does anybody care well you think about the guy in the back that shrugged his shoulders when he asked about if he liked to work full that was simple people like it simple they don't want to look at 87 different fields that they're never going to use they only want to put the fields in place that are or have the field visible that they're actually going to use so this reduces clutter on the screen and makes creating JIRA issues far easier which means more accurate data in the long run the less bothered people are to use the more likely they are to do it right we have here what we would like to see in the future and I leave the camera for a minute so we have seen the whole the whole field in the world here and it should be on story and epic level provided something to be done and maybe some video or something that they have their own and then we have a project called where we can see features the features can overlap more so it's really important that we have the same subsystem and some addition fields depends on what we need to report on so we can see the progress and then the feature will be delivered and I must know that redhead is doing awesome features but you usually don't call them like it's already written and then there might be some a little bit more because it's so forth which can overlap over the years so even more standardization is needed but we hope it will be something like this so not there yet not there yet but the standards are helping us get there so what's next unified backlog is part of it but what's next for each and every person in this room we want you guys to go home and you don't have to do it today because it is the weekend and you're already learning too much but think about the teams that you're working in think about the processes that you have in place and think about what you can do to collaborate with similar teams with similar programs and with similar engineers around you to standardize processes and we're not asking you to go standards crazy and implement a standard for every single thing you do because like Lucy had mentioned that's not helpful either you have to pick and choose what's important to standardize and it's similar to if anybody resists here with Delania's presentation it's very similar to that focus on what matters and focus on the processes that are having the most burden on the teams or that are the biggest problems so think about that think about ways that you can collaborate and take something totally chaotic and make it meaningful and learn it on engineers and make every baddie's day more enjoyable and really we want the sky to be the one that's for you guys go back home, think about this and imagine what's possible to make your team's days better that's all we have thank you all very much any questions? I won't point under with the laser frame for your comments that's great so you decided to develop a community-like process or standardizing the same for different teams now I'm pushing how does this translate when you have an external kind of community that's part of the thing a part of the community are they involved in the work or are they continuing to do it so the question is what we've talked about has been internal only has any of these processes been extended for external partners or for the community and at this point these are all internal only but trying to roll in the external is actually really the face we're at right now the partner management process is a great example of that Marcel has spent a lot of time working with these external partners to make sure that the onboarding process matches their needs and that when they're logging in they're having a consistent process that they're used to not only from their existing tooling but from what they are expected to see in the field that they need to fill in and if there's anything else that you want to add to that Marcel it's a community the expected people that will be back to looking into our chair and stuff like that we have we have some of the mission in place to be able to sync to get from Github to Jira and the idea is that they give you pretty much only and if you get it done I feel like it's great to be creating a Jira let's not get this back to where we think the way Michael does it is that if you want to go through one of these you can do the same thing in family in Jira and at another time you can do the same and therefore we are able to work particularly both to work on Github but really the mission is where you get the credit and you get the code and you get the confidence to work on Github and you get someone to report to in Jira what is the purpose of that or can they come to find the mission or not for the record the engineer in the back is saying that they have some automatic solution which translates Github issues to Jira I'm aware of it but it's different things some things needed for planning, some things need to be done from Github and some things from the community impact much weaker so in a multiple of the people with thousands of communities so we need to pick and choose which one makes sense for the general makes sense a lot of time makes a lot of sense to say but others may not so we cannot forward the work to work our way the point is certain things work for certain teams and we can't force the world to work our way which if we could we'd all be really happy our lives would be easy three of us wouldn't be up here presenting we probably wouldn't have a job it's really important to know you can't force a square peg into a round hole you have to find the processes that are worth adopting that actually yield a benefit to the online to clean that collaboration maybe I would like to point out that once we go for a standard it requires some level of maintenance so for example the integration we have in JIRA there is some work in the background that the automation can get broken the way how we do it it can get broken there is some maintenance cost so each time we go for a standard keeping in mind that there are some costs to it as well and it doesn't always make sense to standardize everything as we think from the audience there are thousands of communities if we attempted to standardize them all why would we the costs were just too high too high, unrealistic and we had an interesting question I'm only interesting one we have a boring question we'll take those in the hallway I can follow up to your question because the Arcade Hub is a very large community environment and we really struggle to translate the issues reported by people into our 5-0 and the way we have to do it is we have community managers who are going through all the issues putting them into one-year projects which is open to the public so we can take in external developers who are very, very active on the beginning of and they can put them into one-year projects if it's external and then the developers can fix them and then bring them to these internal projects for actual development you can break the communication on some of the issues on the outside and all the other stuff they're very, very difficult but it's a way to have help open the issues down so the developers can only focus on the priority path rather than having to go through really thousands of issues constantly on the other side so for the people on the stream that was a boring question we just get half this mentioned all the time and how to connect it to projects it's not easy each of them is doing it differently because there is different data as you said on the project it's possible but there is no unified solution which will be good for everyone and it's not easy to set up it's breaking the law so some non-github related question what does it mean for you when you introduce the standard when we say documentation before the rule or yeah sure so what it means for us is a lot of communication and the standards that we're implementing shouldn't come as a surprise to teams the thing is that we aren't embedded in the teams we're working with and the reason for the standardization usually bubbles up for the teams themselves expressing a concern so that means we're engaging with them all along the way so when it's implemented we usually do a small pilot make sure things are working and document it very well I think we are out of time here but we can talk a little bit more about that offline if you'd like thank you all okay hi everybody alright hello thank you all for coming today today's topic is less is more or large-scale scrum through the eyes of the red hat in vehicle operating system in other words welcome to the least technical automotive talk here at DEF CONF before we get kicked off I want to take a second to introduce ourselves my name is Allison King and I am a technical project manager within the red hat in vehicle operating system team I joined the team a little over two years ago when the program was really just beginning it's Wild Wild West Face and I'll talk a little more about that in a little while but I am a JIRA love hater just like Lucy had mentioned in the last presentation I think better ways for teams to work together and to help them thrive I'm also a sucker for a good dad joke so I apologize in advance for any puns in this presentation they're definitely intended and do not feel obligated to laugh you can cringe, I'm used to it turn it over to Sabina I'm Sabina Fogel I also joined a little more than two years ago actually one week after Allison so she helped me onboarding I'm working as a senior agile practitioner in the automotive initiative in the red hat in vehicle operating system and my passion is to support people to become great team members and contributing and growing so I love to help people and I think the co-creation and collaboration is the key to success and as part of my job making myself redundant is the most important thing so it's not about me, it's always about the team and the team members I'm working with so talking about the team members we work with together with some other project managers including Satya who is here in the audience today program managers and agile practitioners Sabina and I are members of the auto pilots and the auto pilots are the agile focused cross functional program team within the red hat in vehicle operating system we chose that name pretty deliberately early on to not call ourselves a team of scrum masters because our roles go far beyond that our roles and responsibilities are very fluid and they change as the program changes and that's really been key for us in helping to provide a culture of collaboration and continuous improvement across the organization so I'm going to ask everybody to talk real quick we'll take you on a journey today first it's the program inception the why and how of the red hat in vehicle operating system from here we'll take you through the wild wild west and talk a little bit about how things started and why it was important to find structure and then once we found that structure how we've used that to get to a bit of a more steady state within the automotive program from here we'll give you a sneak peek into some key activities and decisions that we feel are really important regardless of the team that you're supporting or the framework or structure you're using and we'll send you home with some road trip goodies in the form of learnings so our goal at the end of this presentation is one that everybody is still awake and two that regardless if you realize that less is best for your team that you understand there are other frameworks out there and there's already patterns that exist in the industry that are really well documented to help you wrangle your teams into a more steady state so if you're looking at this presentation thinking I like some of it but not all of it don't feel like it's all or nothing you can choose pick and choose what you want and maybe find another framework that fits you a little bit better so let's hop in our DeLorean why automotive back in 2021 the automotive the red hat leadership team found a market need and they thought that a linux based operating system would be really well received in the automotive industry because linux and cars that was the only reason I joke they really had a vision and by vision I mean there was a business objective and that was to provide a functional safe certified linux operating system based on rail so it was a really really great idea and we're continuing to develop toward it today but when that vision came out there was no structure to achieve it and that's where the autopilots came in you know when I say there was no structure when Sabina and I started there were 200 engineers that were coming in from different programs, different companies different industries and this team of engineers was tasked with building something that had never been done before and that was to develop a new operating system based on linux that was functional safe certified and you know I want you guys to picture what this looked like for us if you think about all the people that were on the tram with you on the way in today just think about that tram hundreds of people on there and somebody coming on and saying hey this is now your team you guys are tasked with building a new one before and just go ahead and do it, get started pretty uncomfortable probably a lot of questions and you're probably yearning for some structure and that's really what we came in looking at it was pure chaos and we really wanted to find a way to get to a more steady state so how did we start we started with some principles we knew it was important to establish a collaborative culture and we knew that we needed to have a set of principles that would guide us there so everything that we talk about here today is based on these principles that were assumed over two years ago now but still lay the foundation for everything we do right and then the next layer was okay so what is the guardrails that would lead us and also provide a structure and clarity but still be flexible enough for us to scale and try things out and use different tools so just the guardrails not predefined of course that that was important and then as we expand out to the next layer on this we get to guides so building a complex operating system we needed a set of guides some tips tricks tools to help people it's really similar to a map I know people don't really use those anymore but back in the day we used to print those out the pictures and the words and tell you how to get there we've adopted we've moved forward in time we used GPS but we wanted to give our teams a GPS we wanted to give them the tools they needed to succeed and to make their day to day easier yeah and in in order to scale and make it really long live we wanted to create an environment for improvement exploration and innovation and so that is the thing that keeps us driving and it continues learning and we've heard that in several sessions already and yeah that's also our goal of course so a couple of slides ago I had mentioned our principles you know as the auto pilots had started their journey through the Tuckman's five stages we were storming norming and we were spending daily meetings observing what the teams were doing observing some of the issues that the teams were trying to solve and trying to find ways for them to collaborate better and in doing so we were able to establish a core set of principles the first one being we needed a way to bring people together to collaborate being 200 people from all over the place being brought together we needed to foster an environment of collaboration help to find that team feeling help people feel comfortable and ready to talk to each other ready to communicate and really ready to work together we also knew that 200 people were not always going to be 200 people we knew this team was going to grow and we needed to find a way to scale with intention also I had mentioned functional safety a couple of times we knew that there couldn't be silos functional safety relies on a culture of safety it's not just a single team safety mindset has to run through the blood of the organization so for this reason plus the reason of collaboration we knew we couldn't have any silos in the organization and last but not least being a first to market product putting the customer at the forefront of what we were doing early and often was more important than ever we needed to find ways to build confidence in the customers and help deliver early and often and let them see what we were doing so we can iterate and adapt and then that's really where it left us to take this road towards less exactly so we were facing from our deliverables and we wanted to understand we understood what the needs were so we were looking out for a framework there are plenty of frameworks out there from very complex ones to very simple ones and most important for us was the flexibility so it needed to be flexible then also would left so would left provides us a very customer centered view that is important as Allison said before then it does post a continuous improvement and in tried system thinking and a whole product view so it's not just one segment it's really the whole picture that's the next bullet and we can scale it with an attention so less is scale scrum and therefore you can just use scrum for one team but you can then build your teams up and Allison said we on voted 200 people not from the very beginning we had fewer people but then we had to create the structure in order to be able to scale but you ensure that when you start you don't put the whole blown framework over an organization that might not be existing yet so you build it up piece by piece so that was very easy with left and it helped us also especially since we're using featured teams in the organization so we could scale less is very flexible it's very easy to understand because many many teams already know scrum and therefore it was easy to adapt so we want to take you to something a little more tangible now so what Sabina had just covered is why we chose less and some of the benefits but what does this actually look like in practice so in less there is a chief product owner a lot of times on the framework diagrams will be referred to as a CPO C3PO Star Wars I do know though they bear a lot of similarities the chief product owner is responsible for translating the customer's desired outcomes into meaningful work for the engineering teams and the chief product owner collaborates with the area product owners so as Sabina mentioned we are a feature area based organization each feature area is responsible for delivering a specific set of features to the customer each of these feature areas has their own dedicated product owner who collaborates with that chief product owner to develop their feature set and be able to deliver what's meaningful at the right time for the customer the feature areas are then further broken down into feature teams these feature teams there's usually two to six within a specific feature area but what's really really cool about these feature teams is they are inclusive of all the disciplines that are necessary to deliver an end-to-end increment of software at the end of a screen a lot of words what does that mean? That means that QE DOCS PerfinScale, ToolChain, etc all of these cross-collaboration teams that are really responsible for the end-to-end integration are embedded within the feature teams so at the end of a sprint a demo, that team can actually demo a working piece of software and that's really really important for us because now we can deliver to our customer early in the office Alison already mentioned our feature teams not all of them but we provide you an insight into our organization but don't tell anyone so we have seven feature teams and you can call them randomly mouse or whatever you like or herding cats so that's a cat team, that's a dog's team so that really isn't important but you might have heard already some talks here from containers and wheels people and as Alison mentioned these feature teams do have a product owner of their own and we do have the teams comprised of all these cross-functional layers as well so we are not just five people in a team, we are more people so we have people also especially from documentation in the teams so delivering end-to-end but also documenting the pieces right away and then on the left-hand side we have the chief product owner who interacts with all of the product owners of those feature teams the cross-collaboration teams also have product owners that might be a specialty but this is how we set it up just to ensure that these teams are also aligned across the teams then on the right-hand side we put the autopilot so it's Alison and my team to ensure that we work with all of the teams we really support tool development in enablement and onboarding so basically everything what's needed and again that's why we didn't call ourselves Scrum Masters because that would be just too located for a team and we really helped onboarding those teams and setting them up in the beginning so that they would get into the cadence of how Scrum works how the organization works we also have a community layer who ensures that we work with the community and we have the automotive management structure also and they ensure that they take care of the team individuals helping them to overcome obstacles and remove roadblocks so they're not necessarily members of the teams because we want to ensure that the teams do take ownership and become self-sufficient working teams and the prioritization is also done by the product owners and the chief product owner right so I'm click happy it's the Czech coffee it's a lot stronger than American coffee I can't stop clicking so I apologize the infrastructure we are talking about here we're starting first and foremost with a unified backlog this unified backlog is really the culmination of the conversations the chief product owner has with the customer as well as the feature areas and the feature area for product owners what we talk about here today this is our hierarchy and we're using JIRA as our issue management tracking system but I do want to highlight that less is very flexible it actually does not specify which tools you can use so what we share can be tweaked and it could also translate to any other issue tracking center so if you're using github or anything you can take this with you as well like I said we're using the unified backlog great we understand what the strategy is we understand what the customer wants but how are we going to deliver at what are we going to deliver specifically what we are going to deliver are the features to the customer the features are the point of collaboration across our feature areas and from here the feature areas are able to break down the work into epics and the point of collaboration within a feature area so whereas the feature spans across the entire organization epics tend to be a little more focused on a specific feature area and an epic is a unit of work that's usually one to two sprints in normal people terms it's two to four weeks and that's really where the collaboration happens within a team the teams are then able to break down the epics further in the stories a task in story is a single delivery owner within the feature area that is defined in the epic and those can usually be delivered within a sprint one to two tasks for stories per sprint depending on your team's capacity and how your story pointing well actually not really how your story pointing because that doesn't matter I don't know why I said that to be totally honest so I'll turn it to Sabita thank you now we've taken you on the journey it was how do we on board the first team people how do we scale how do we understand what is needed pick a framework put that in place bring people into the structure show the structure how we set up also as teams grow and I forgot to mention that before but if a feature area grows I mean that means you're not growing the team indefinitely right because then you need to break them out into smaller teams within that feature area so you can decide whether you want to have a product owner of their own or it's that main product owner and you have sub teams within that area so that's a matter of scaling right and then you have in scrum the scrum scrum a scrum what we have in less here with the chief product owner so we have two layers the team layers and the product owners and we have the product owners creating a team of themselves right but in order to ensure that it's long lived it's scrum and also in less we have the continuous improvement cycle so at the end of the sprint we do have a retrospective and since we do have two weeks sprint we have many opportunities during a year to improve ourselves but to make sure that we only take tiny steps that's important also what Ilania said with don't over don't stress the people out with too many changes at once don't change big items every two weeks but make sure to tackle the ones that are really needed so the first item that was like the most important thing is to have a stop having items tracked in multiple places and you probably know that as well right so you start documenting things and different people have different preferences so one person would start documenting in a d-doc other person would start documenting and so you have multiple places and data documents get outdated and there's nothing worse than outdated documentation so that's like the fourth thing stop doing this define and agree on one single source that's also why we initiated ATERA so that's the documentation piece then when you continue starting if we hadn't already started with a team of autopilots we definitely would have started one because it's really helpful to ensure that we are talking in a consistent and in one voice to the organization the organization has multiple people to go to in different time zones so you have a continuous flow also to interact with the people and to collaborate and support the team moving on and as I said before it's the unified backlog because you have so many different areas it's important to have one source so that you are not losing the team where they're going towards and it also provides a very good overview for the product owners in their conversations for prioritization because what happens in our area is that engineers from other feature teams need to help out in a certain area where you have a lot of demand right away so balancing this and knowing that there is a need then the other product owners can decide okay so we're going to help out and we're going to take work on to get you through that bottleneck so that we then all can go back into our areas and continue working continuously growing with the automotive team across functions and organizations as you have new people onboarding you need to reevaluate what you're doing and if something doesn't work you need to fix it and continue the improvement going to your only it's the same thing new people you constantly need to to onboard people you need to explain why you're doing things the way you do and if you experience well the way we do it doesn't really make sense then you need to also be open and change it so that it does also meet the the ongoing needs same with team empowerment whatever it's not working you need to listen to your team members you need to engage them otherwise any change will fail because the team will be resistant and the last bullet point is about onboarding documentation and keeping up with the changes that's very critical and continues to improve our documentation I think that's all right so now we would like to present you with our journey suite for your potential journey and it's basically a summary of all what I said before so when you get into the situation of introducing a new framework or a new type of work so please make sure you do understand your needs that you really take time to look at what there is and not just pick a framework because you're heard from somebody it's nice or cool or that's the state of the art right now it did work for a different company a different culture so please make sure that you start smell from your own framework so we did pick less and we also provided your link to less and keep in mind to make sure that you're working only on one backlog so it's either single backlog called or unified backlog whatever you want to call it but just make sure it's one source and everybody knows where it is and that they contributed to it have a standard method for tracking in our case it was Jira if you have another tool that you really love and know that it's working for you you can do that and last but not least we are really advocating for Autopilot as servant leaders to the organization because we receive positive feedback that it really helps the organization grow and become stable and that was it thank you very much for listening go ahead I have a question what are the current challenges that you're facing in Autopilot okay so the question was what are the current challenges like in the Autopilot or in the organization like in the Autopilot and how are you working with them okay so the challenges in the Autopilot do you want to start so there's always challenges with growth and change so in automotive like we had mentioned the teams are continuously growing and as they grow new feature teams are broken out and trying to make sure that new people are understanding the framework and understanding the value the framework brings is something that is going to be a continuous challenge documentation helps and having regular sync ups with the teams and having a presence within those teams so we don't just throw this framework everybody is saying here you go fit in and do it we are embedded in the team so I think that does help with it but it still does remain a challenge is that scaling yeah and what we also learned that it was very important for us as a team to also function as a team and that also needed a lot of time and a lot of communication and in the beginning we had more time and we definitely had more time in becoming a team as well now we are very much engaged into all the different teams so we still have daily planning meetings for our team in the morning so that does help us to align and synchronize but we only have time for an hour per week as a working really working meeting and then we are using Slack and other asynchronous communication channels to align and also make sure that we are not diverting as a team as well any change that's introduced from one team we bring it to the team we discuss it and then we make a proposal putting it out there to the teams and then have the teams decide what they want to do and I think Sabina here on something really important there is the importance of the auto pilots team being a team we're not just out there doing things on our own in ad hoc we really function as a unit and we had to build that team we had to build the trust within the team we had to find our own norms and all of that to work together but really eating your own dog food is a good idiom for it we need to practice what we preach and we preach agile and we need to hold ourselves to it for these daily meetings we need to make time to document our work we need to make time to plan our work and work on what's most important so I think holding ourselves to that high standard is something it is a challenge with the time constraints so the question is what is the resistance with people onboarding or that are part of the program do you want to take it or okay so there's always hope to be resistance if there wasn't resistance you wouldn't have a team I mean people are people people have their own mindsets so how you deal with that resistance is definitely key and I think it goes back to understanding and helping the teams how you and what they're doing you know a lot of what we do there's different segments some teams are far more R&D and it's trying to figure out oh could this work what's the best way to do this other teams it's far more repetitive tasks so we do find some resistance on either spectrum of this is too restricted for us or that would be on this side or oh this is just flexibility this is repetitive I need to know what I'm doing every day so trying to find a happy medium and helping teams understand that the framework is actually flexible can work for them not against them another struggle to follow up on the other question yeah and if I may add to that in the previous session people were asked how many people were left after that form formation we are about 200 people I know that we had three or four changes so far within the last one and a half to two years so I think that is a very low attrition rate and yeah so surprised I think people are no matter all the uncertainty and ambiguity I think we did manage it quite well so far and I hope that we're getting more stable of course one thing you didn't talk about which because my email was working in the last framework we find it really powerful with the six a week demo cycle and the goal setting that's part of the framework I wondered if you wanted to talk about that because I think it helps with the resistance people see this as almost immediately whereas some of the frameworks maybe don't give that so the question was we didn't emphasize on the six weeks demo and second layer sprint right correct we didn't do that so we didn't lay out the less framework in total here we thought it would be too long and probably a bit more complicated but yeah that's exactly the point you have the flexibility of building it and there you have in our case it's three spins so it's every six weeks you have a demo of the organization so to say so every two weeks you have your sprint and your demo at the end in the future teams individually and then every six weeks you bring it together so that really helps the organization also to see what the other teams are doing and linking it back to the unified backlog you know what is it that we are working on yeah that is very powerful so thank you very much for mentioning that and to build on Sabina's response there too is bringing everybody together to see first of all what the other teams are working on it does foster collaboration it breaks down silos but on top of that it really does help give meaning to the work so I think Sabina touched on the low attrition rate teams being able to see what their work is doing and how it interacts in the bigger picture this demo is recorded to our product management team every six weeks and for them to be able to see their work rolling off into what's going to be a release is really meaningful and keeping them engaged in the morale of through those demos is something that we find extreme value in Thank you for the honor it's often recorded as meaning to be low and there is a lot of stuff put on so it was digital so I wonder what are the outcomes from what the company has and how the company supports in the course of autonomy is it making overall support okay so the question is how, so the chief product owner in some cases is referred to as a mini CEL so how are we engaging the chief product owner in the teams to get their feedback and to really create that environment and give them but still giving the teams the autonomy and freedom to do what they are doing is that correct I would like to ask and I believe there is a lot to say when it comes to digital so I was a team and I talked about information for our authority to this role so how does it work in the open environment have you always structured to get this person the autonomy or are there some people from the company on them So the question is how do we give the chief product owner the autonomy they need and get the teams engaged well the first off is that 6 weeks demo is really really important to showcasing what has been done but the chief product owner in our case is very very close to the customer there are multiple meanings per week with different engineering teams in different levels of program management and architects with the customer themselves so they understand at the very level what needs to be done and how it needs to be done so by having that credibility it makes it a lot easier for them to come back to the teams and say hey the customer would like to see this and then on top of that having a certification that we're working toward there's a lot of requirements that they're coming to us with that are just hard requirements so we try and give them the freedom and let them be involved in the process but very very happy to have their feedback and have their close tie in with the customer so we're out of time thank you everybody this one is very loud so is it just that everyone else is going to be quiet no don't worry I'll start shouting should we start? should we start? okay so let me start so this presentation would be about estimators of the lost arc you can see the Indiana Jones reference and it's the story of the advanced reconnaissance group my name is Michal Koneczny I'm the mage from that is my official role now joking my official role is senior software engineer that had and who are you again? I've never met this man before in my life no my name is Ethan I'm the product owner for the community platform engineering team which I work with Michal and other mages other sorcerers in the team as well and yeah we're going to take you through how we came about creating this team within a team with the founding member sitting right here so I need to be very accurate so today's presentation like I said is going to take you through the origins of the arc team where it came from where the concept came from where the acronym comes from because that works on what the purpose of the team is for what we use the team for within the wider cp team it has its own heroes and villains as every good story does and then we have live inaction tales from the quests that this team embarks on I will tell you through all the battle stories and then we will unveil our chest of treasure that we always find at the end but first I'm going to take you through the origins of the arc team so at the beginning there was only a very small spark this spark broke into a fire and the fire started and people were terrified of this fire they really didn't like it at all but they started warming up to it and realised it mightn't be that scary and it could be a little bit helpful and the fire is absolutely a euphemism for agile so our team started in this agile transformation journey and we wanted to start planning some project work and delivery timelines we needed the requirements and we needed deliverables and we needed success criteria and we were going to get all this stuff and then I was going to come into the team and we were going to pass it to the delivery team Michael and others and we were going to have what people wanted at the end but something like that was going to happen like what else is needed I tell you what I want and then I say great I'll give you what you want and then you don't care what you want so this was our idea that we were striving for holy grail and we were going to get there why wouldn't we get there it was really really straight forward simple also we thought so the problem that we had was while we were getting these requirements and these asks from requesters and they were there we had our success criteria we knew what people wanted we just hadn't a notion how to actually deliver it to them we had lots of we knew several ways to deliver the thing and we couldn't really decide what is the best way forward you may be familiar with when you're tasked with something there is some indecision as to what technology is best to use and in the in the project like Fedora that is so complex and wide and cool but there's a lot going on if you don't choose the right tech stack at the start it's very tricky to know how many other people's projects are impacting if it tends to be something that's not compatible with dependency issues or whether you're going to choose something that will lock you in in three, six, nine months down the line when you've developed something else so our team was wasting an awful lot of time figuring out what the best path forward to even begin the development was and that was causing you know causing a lot of upset would be a nice way to put it within the team because I wasn't sure what they were asking me for to begin with I told them that's what they wanted they wanted a red button what do you mean there's different shapes of buttons give me one yeah pretty much so there was a lot going on and nobody knew what was going on we were making it up as we went and we were delivering things but it was not a comfortable process and this was happening for a couple of projects happening for a couple of months and we said enough we have too much going on we need to cut a break here and I was fortunate enough to have Hingo as a friend and also a team member who was saying what the heck is going on with the project they haven't started yet I don't know what to do so we were discussing the problems and he did come up with the idea of is it the technical pre-scoping of the problem yes I don't have anyone to let this with I can vet it against business requirements and benefits to people but I have nobody to actually vet that this is not going to cause a dependency in the library over here that's upholding all the release engineering I don't know I didn't have that context so we decided we were chatting and he said why not propose a solution the solution being we pre-scoped work technically so we still get the requirements from the requestor we still get all that stuff we can't even say yes or no we pass it to a subset of our team and they investigate whether this will actually even work and if it doesn't work good luck but if it does work then they're going to look into like how we do it the that has trade-offs like you know you're going to grab the golden idol and you're going to have to put something in its place the trade-off was that I would need people from my team to spend time doing that and that means taking them out of existing project work so the bar here was we dropped projects we had been taking on like I think there was four projects running plus our sustaining team as well so we had five subsets of our team running which is full capacity nobody would left over you took a holiday things stopped I'd like to tell you it's completely different now but it can be a little bit like that still but we really had zero capacity for innovation or thinking things through we dropped projects dropped everything by one and put more people into our infrared energy team and they will be able to then take some time and pre-scope this work and work with me once it comes from the requester to us and we'll work together and we'll better technically give out some options and then it's ready and then the development team can go with it so that was the solution send out comments to the managers send out comments to whoever needed to know and they said yeah okay cool so we did it and here we are to talk about it because it worked and we had a lot of fun then with the acronyms so like there was a good few that came up obviously the advanced recon crew was the one that actually can be publicized but like there's a few there that was that was a hot contender the angry rabbits committee nearly ran nearly won smooth ones with either rating children very good we said let's keep a professional for God's sake let's just try and be like adults so we called it the angry advanced recon we stuck Indiana Jones so that's how the hot set started that was our problem that's what we were facing that's the solution we came up with and it has been by far the best decision I think we've ever made in terms of product delivery within the season 18 and I'm going to fight anyone who disagrees and Mikhail we'll take you now through the rest okay not through the rest but through few slides okay so first every good story needs a hero they emerge from the battle I'm scared they were heroes that everybody talked about they were the ones whose enemies tremble when they hear them coming who they are really and it comes to me that these are the heroes of the history the arc is actually made from people from in front of the launch team and they are the heroes you can see some of them here unfortunately we don't have these trade cards for everybody and the team yet so what we will need to thank design team, federal design team making tools so few people here we usually just try to find the best suite for the investigation for the organization and it depends on the thing we need to investigate and every good story needs good willing the people were scared when they saw them approaching they were nasty group and didn't bring anything good but they were still imposing figures and who are the villains oh the ghost okay so one of them was having the clear requirements we usually get some request with some requirements but those are not technical or they are just somebody wants so one of the villain is the requirements we need to actually get to him and defeat him the next time is time and time is always a villain every time and in this case we didn't had clear estimate we were struggling with starting project and when we thought it will be ready it was halfway through so we actually trying to do this during the arch investigation to estimate time how long this will take it's usually a good guess or good estimate because the people are trying to work on proof of concept trying to do the work that is we are trying to investigate so it's usually good estimate and we are delivering on times which says a lot and third one is technical options this means that we can see in the arch investigation the options that we can think of and the team is actually trying to explore them just trying to look at them and see what is viable and we prepare for the next project so first we get the underpants all the underpants we can find then we do magic and then we have profit we have profitable business at the end and this is where we do it no I'm joking but we would we would rather hold the underpants quickly if we do that so we decided to go other way first we will choose where this is the arch investigation we actually want to do and we will we will actually explain why this is needed what is the story behind it why we need it then we choose the right heroes as I said before the heroes are chosen usually on the arch investigation if this is some old project we have and we want to write it we actually want to have somebody in the arch team who knows the project who actually at least have some knowledge about the project and if it is some specific technology we want to try we want to ask people to be in the arch team who have some knowledge about the technology so we have somebody who actually is doing in the arch investigation and then we go to the adventure and they left for a great adventure looking forward to see what they will found what new beauties they encounter what happy moments they experience but it will not always be a positive situation so we had a plenty of investigation I in this case open up the let's just open it up I'm not sure this is why I'm wearing the glasses let me just do it a little bigger so these are actually the arch nodes this is the output what we do this is the output of the adventure these are the nodes the journey of the heroes and here you can see what we actually what initiative we have and what we actually did for the investigation we have plenty of completed that are not implemented yet we have plenty of implemented as well and as was said before we don't always have everything is not always great so we actually ankle that few of the things that we needed some time and it was great that this was actually found out in the arch investigation not with the project actually working on something so for example what do people use FMN for when we try to rewrite the federal messaging notification system we are looking at first what is there then there were plenty of customers for applications and we wanted to ask community first what they actually use and what could be useful so we don't run in the same case like we have database for the plenty of customers that are used by one person so we want to share them we got some examples we still even now when the FMN new FMN is deployed we still have some request for some messages that weren't there next thing is how could we scope badges properly this was a big one because the badges are here for long time we don't want to take the maintenance of badges your team we want to leave this on community but the badges needs to be rewritten and we want to help with that so in this case it was how to scope the badges properly and how to actually make it work so what we can do how we can design it to be better and the last thing is is for the activity count script that is new and there is still ongoing discussion what should be marked as contribution in Fedora because how you can recognize contributor by contribution but if they are not clearly defined it's hard I can actually show some of the other stories for example good example not everything that is bad webhook to FedMessage when we did the investigation we finished it we had some design of hold the webhook to FedMessage should work and then we look at the get up to FedMessage and found out that we actually designed it in the webhook so that was a win the second investigation was just writing all this should be implemented like this next thing that I can talk about how before you do can I just point out as well if you don't have somebody looking into that and you pass the development team get up to FedMessage and they do the work and then you say okay next we are going to do webhook to FedMessage and they go same thing it's a waste of everyone's time so highlighting that prior to this art process was just huge because we were like great that saved a huge amount of time was it possible without somebody doing that free scoping work from a technical perspective? I would say second thing about this or second example at the end the communication is now implemented it was stuck at the first time for a long time at some law issues mainly on the GDPR stuff and this was something we couldn't do anything about and it was actually something we did art investigation for but we didn't look at the law requirements and this was actually met when the federal legal team was actually started looking in it so yeah not everything could be catched by the art and the last thing I would like to talk about is that we have always more than one way so the art team is usually looking I can open one of the investigations so we can see it was a fun revelation to art and software with software engineers to be like yeah but there's loads of different ways to do it you can see a list of the ideas that actually we're looking up in the case of Datanome Datagrave all of those were looked up by the art and said no this is not how it should work or yeah this is a good way to go forward and there are conclusions and recommendations so the team who is actually starting on it something to work on work with and I will give it back yeah fortune and glory kid that is what we were doing it for we were looking forward to the reward that was waiting at the end because we were going to be guaranteeing some technical clarity like who doesn't want that we were going to have guardrails for development teams to pick up the project and work within not rules but like rails that they wouldn't be worried about but they would have the confidence and they would have the surety of doing some development within without going over or making ridiculous choices that they didn't understand the ramifications of that far so yeah fortune and glory but there was there's a hell of a lot more to it than that the real fortune and glory that we have actually seen within the team is and hopefully as well I would like to think of the core project as well anything that the CPE team has put out in the last maybe two plus years since we started using this ARC concept is better services and we have upgraded the through our messaging notification system based on ARC which is going to see your through our messaging notifications be a lot less laggy you might get them on the same day that you have that the thing happened which is an improvement we would have a we did some work updating Bodie we have done nomadetic refer there was there's loads of services that we have updated based on the outputs from the team as this is the best recommendation so it's better services back into the community all right our development team was getting more clarity on their work because it can be painful to do some of this heavy development stuff so it's nice to be nice and if we can be nice to the development team by giving them a little bit more clarity on their structure then they'll be happier to do it and you don't want to piss off developers but you don't want to piss off their engineers because people deliver the projects treat people nicely and the projects will be nice you're going to get less confusion from your development team you're going to get a lot of less time wasted at that up front trying to figure out what path is best because people have already looked at a couple of different paths they've already been able to advise you on this is probably your best shop here this is an agnostic enough service we don't want to do a lot of damage we roll back etc so it's easier to just pick up and go and as Michelle mentioned like we have had successful deliveries of all of the projects that we've picked up anything that we've actioned we have delivered on they've been successful and we've also kept them to a very good timeline which has made it easier to plan around as well so for people sitting in that project planning management that kind of sphere it's wonderful to be able to give a very good guesstimate of how long something is going to take how long you're going to not have team capacity for or how long you're going to need capacity for as well and you can plan it better and you can start building out like short term medium term long term road mass which will get a little bit fuzzier the longer you go but you stand a hell of a lot of better chance planning properly when you have that clarity already on the work in your backlog speaking of backlog you get one properly with delivery plans whether in our case it is initiatives, they're projects it may be like ethics in your backlog but by having a team within a team like an Arc team doing those investigation spikes pre-scoping the work for getting your backlog slowly scoped and it's getting delivery plans already built in so that by the time your team has capacity to start picking up that work you're ready to go and choose from at the very most you're spending maybe, I think we've spent maybe a week five days just reviewing the Arc documentation and onboarding into the project but that is it we've gone from like three, four weeks of who should we be talking to about the service to just pick them up going that still makes sense and away they go and very importantly we have an opportunity for community feedback in the projects before which is extremely valuable obviously for the community platform engineering team I referenced the FNN project most recently and I'll credit Ryan Lurch for doing the bulk of this work before we started the Arc investigation we were compiling the requirements and the use cases were so vast and so many to try and cover Ryan just mailed out to the community and said okay what do you use the service for and because people were coming back to them with their use cases he was able to compile kind of like but this all fits in this general bucket and that should be covered by this rules we don't need all of these custom rules that they've made to virtually get the same thing these are a bit more use case corner casey we should probably check if they'll be covered by something else or whether we need to keep the custom rules but it meant that like we weren't spending development time stripping out all of these rules from a service only to launch the service and have a load of people going what the hell I have my thing configured to this and now I don't need more I'm like I don't know if that actually makes sense it did because we had already pre-checked it so that was extremely valuable to have so yeah it was great it was great to have that feedback loop and as you saw the documentation it's open you know you can love on you can see you can read through all of the other ones the ones that aren't implemented please feel free to read them through and submit yours or message us and say like you know actually there's a better way of doing that we're open to that that's why they're there they're not implemented if they're even if there's a draft throw in your thoughts and ideas we want that we want the feedback we are always open to new acronyms yeah you can open PR for that as well okay and we have some numbers because I can read numbers we have 70 in finished ARC investigations we have 7 projects that were actually delivered with clear requirements and on time and we have 8 acronyms suggested for ARC and this could still grow so yeah I think okay and this are some links so we actually got some issue uploading the slides so hopefully we will solve this and here are some links so initiative report this is where we have the initiatives ARC investigations this is the document I showed earlier and ARC report is the sources for the documentation that is all thank you you have a couple of minutes so you have a lot of people on your team how can the same concept be applied to smaller teams where you might have only 3 people working on like 3 or 4 different projects and you have to vet those projects so you're not wasting so much time on small teams do you want me to take that my suggestion would be you need buy-in from your stakeholders and your management team to actually not separate your team maybe there is one or two people off that 3 or 4 that can do that free scoping work if you really are only a 3 person team I would try and get buy-in from your management or stakeholders we just call it a spring to zero to set aside some time to investigate that properly it depends on the size of the work some of our investigations have been finished in like a week and half some of them can go on maybe like 5-6 weeks you don't try and keep them any later than that because otherwise it's a project and that's not the point but you do need buy-in for that free scoping and if you're a small team it's probably going to fall to you and you need to be pushing the wasted time that you're spending or build that into your actual delivery plan all together there could be a lot of campaigns and they're all hyper-focused on their specific path and there's not much overlap between them so maybe there needs to be more overlap if they're all delivering up maybe you do need to pull something from each of them and get them into a room just get them to hack it out and then come back and say okay deliver this yeah so that's one of the first team to be able to do is trying to be styled yeah in my position the team made a small team to go basically styled the imperfections and the engineering and the application and the first team to be able to move all people together and you can try to find out the ARC team is never the same people all the time so if you're understood that you're just one team but if you're many teams in it kind of a bigger team but just separate you probably do need to look at forming a standalone team like that we were fortunate to be able to have our infrastructure and reliance team that supports the staining work but that's where we pull our investigation team from and they're never the same person so it's definitely like there's a request to come in of all different skillsets but you need a variety and you need different people talking about this stuff and just to make it clear we have in Fun Ranch team people from Centos infrastructure from Fedora infrastructure and from Fedora Release Engineering so we have people from all over the place and they work and developers yeah the mix is good a variety of things there was a sort of reason to approach the in my relationship with this is the idea of the ARC team is something we do but if there is a fire coming up in some of the infrastructure that team can stop and go fight the fire you're not already finishing any of the projects that has been worked about do you set kind of guard rails for the ARC team can you mention a few group concepts and I find it very easy for your group concept to sort of just become the concept so do you kind of have rules of that where you stop the ARC team they stop short of implementation no they're usually usually we have only a few weeks so it will take four weeks we are in some dead end and we shouldn't go this way but there is never only one option so there is a rule that we don't just look at one way we do the team does investigate at least two if not three ways some will be automatically ruled out but they will still be locked out I think we were looking at replacing the off-lib library and there was one that was absolutely not going to work but we are going to look into it to be sure to be sure but they were both looked at both put forward and then you recommend so when something does take around five weeks or something like that does that mean it's three people full time for five weeks no matter because there are other things that they are pulled into it things that have taken around the five week mark has been things like the FNN project where there was time needed to get people to come back to us with answers on what the use cases are the badges project has also pulled into that as well that people need time to think about what they use these things for and why it's important to come back with this we found that important enough to give that extra space because we didn't want to then start looking at our options with point of view or with only half the information we try and get as much information from before we start out so that we are looking at it in the most holistic fashion possible and we are okay to keep it a little bit extra from this we also have a new way to doing it and that is if we are working on a ticket, on a regular ticket that we get just in the federal and taking too long we actually just fill in the ARC investigation and leave this to our team, dedicated team that will work on that so anything that comes in that's too big that's like this should be a project that needs more people, I don't know where we are going to take this go then to that side otherwise and we pick it up and look into it and say okay that's big enough we need to projectize this in this case the ARC investigation is actually done by somebody who tried to finish it not at least started that's worth a lot of time if you have to leave any other questions before could you say goodbye thanks Pingo for creating it team work, team work yeah, but anyway yeah so that's the ARC project if you have any follow up questions or anything you can reach Nea McCall, our emails are at the bottom you can find me at the bar thank you very much