 Great, so I'm Eric Johnson. I'm the VP of engineering here at GitLab And I'm with Dalia Havens one of my directors of engineering on the development side of the house as well as my quality leader Mech who's engineering manager of quality and they're about to take me through some new metrics System we call throughput. So this is a way for us to become more serious about using metrics to measure our productivity And one of the key things about GitLab is that we want to keep things light and fast And so we try to focus on passive measures of productivity The that's in opposed to you know, heavyweight planning and estimation meetings. So this is our first iteration They're gonna walk us through what this looks like and then we'll talk about what future iterations might look like So take it away Dalia All right. Thanks Eric. So this is kind of a pretty informal walk through but Just a brief description on throughput because it is a distinction between a lot of the velocity and estimation that a lot It seems are used to This metric captures work that's been done and the idea of throughput is that you're measuring units of work Within a certain timeline what we decided to go with for GitLab is that we want to measure a number of MRs per week We're not so much interested in the raw number But we're trying to build a trend that we can look at and identify the ups and downs and what bottlenecks we may See from the chart So that's kind of the focus of our first iteration the next thing that we also were able to add Which was a really nice plus from the first iteration was categorization So we wanted to see what a level of investment we're doing by team in each of these categories and the big categories For us, I'll take you through them as we look at the charts But a big heavy on you know, we have community contributions. We have Security we have bugs we have feature work and so on so if we didn't want to clutter the graph with like every single label that We have so we really were methodical to identify those five categories We're also tracking undefined just because we want to capture all the work done Even if it's not labeled properly and that's actually a really good way to kind of go back as an engineering manager looking at your team And seeing hey this work was not labeled and therefore I have little data to go on You know making certain Educated guesses on our investment. So those are the two aspects of the graph that will go through So what are some of your questions? How do you want to start? Do you want to look at a? Kind of a graph from a team and we can pick at random or what's a good way to walk through this? Yeah, I think making it situational if possible is good I mean we don't have a lot of data yet, but something that's more example-driven would be would be great at this point Okay, let me go ahead and share my screen All right, so this is Quality dashboard and a big big thanks and kind of the ownership of this comes from the inside team So Max team drives a lot of like pretty much the implementation of this with It's been really great to see this put together just want to make a shout out to them So let's see I'll pick a team. I'll pick on configure. I really didn't like I'm not so much You know trying to put them on the spot But if you look at the configure graph, this is what throughput looks like week to week and one of the things that we're doing for First iteration is we're capping it to 12 weeks. So it's long enough to give us, you know, 12 weeks of data It's not so long that the graph is cluttered and you can't really nail like get down to What each week looks like if you hover over the graph you start to see the breakdown So we have these labels that tell you our feature investment how many bugs and so on and again Remember, these are number of MRs not number of issues that we're tracking here The other thing to remember and was really really awesome to see On the pages that were we're displaying the associated Projects that we're tracking. So one of the things that some teams are impacted by for instance secure Contributes to projects that are not yet indexed here. So their throughput would not be reflective of their full capacity So this is a really good way to kind of gauge how how we're tracking throughput based on what projects and so on and We're looking to add these projects as like additional projects. So hope the data should encompass all work We're just not quite there yet So, you know, it's it's somewhat typical of our of what we would expect You see the the November 5th being one of our tall bars here It is close to feature-free so it is expected that a lot of MRs would go During that time, but it's really nice to see that we don't lose momentum on the other weeks either So we're able to see, you know merge requests going in week after week And that's what I want to see I want to start to see A bit of a closer consistency in delivery week to week as opposed to just spikes every four weeks or so on So I'll pause here So yeah, good time for questions. I was going to say what was the thinking about using merge requests versus issues As the unit of work to track Yeah, we had a bit of a conversation like we had a good discussion and I can point you to the issue But the big driver here is that MRs are kind of the lowest unit that that you can measure Which means that we have more flexibility If we start to try to drive the smallest change and make that an issue You'll start to define issues that don't really focus on value Like it's not implementing the full thing whereas an MR could be a building block toward building this, you know, bigger vision So the issue is an easier way from a tracking perspective To make it the full use case and then the MRs can start can be broken down to what is the smallest deliverable Ideally, I do want developers to get to a place where they're delivering on a daily basis And sometimes that's not going to be implementing the full thing that the you know, either pm or or the bug addresses But it's a really good, you know Definition of done small piece that we can add on day to day It is one of the camera Positively mapped to an issue in other words So if if there's one issue that's strategic from product management's perspective And it's put in the code based on the course of five MRs is each one of those MRs traceable to this issue Or is that something we're going to have to Track and see what the compliance is and if we have that traceability The the MRs do tie into issue. Okay. I'll pause and let Mike go So this is one of the challenges I want to help chime in is that some of our senior and staff engineers Sometimes just work directly in MRs and they don't necessarily have an issue associated with it This is why we can't use issues as a as a unit and also one issue can also spin up into two to three MRs So ideally MRs is the the lowest unit that we can measure here Yeah, right so what I what I'm getting at is that we have this sort of separate discussion about what we're prioritizing which is uh loosely coupled to what we're actually executing, right? and If it's okay that they're different in fact, it's it's ideal that they're different because I like what you said Dalia about the smallest unit of work and tracking that But it would be nice, uh, or it's I guess is a question. I wish you'd be iterative and say, uh, it would be nice if, um Uh We have line of sight to being able to do that because at the end of the day If you feel we're not doing enough tact at or if we're not doing enough security issues or recently in the discussion We were doing too many security issues and we should change our SLAs That might be more Actionable at the issue level opposed to MR level. So it'd be nice to be able to operate it at both We we still have what like I think To max point we should probably try to drive more that start with an issue get an MR But but to your point Eric about Kind of the ratio of investment. I feel that MR is the right the right way to go because Feature development tends to be a bigger block of work and if you only track it as one you're gonna You're not going to be very accurate and you're not going to also be able to get to Consistently smaller units of work across all of your deliverables And that's what I'm trying to achieve if I'm going to stack them against each other I don't want a wait system to basically inflate the feature work To try to kind of predict that that's more of a chunk of work than a bug and so on So getting to the MR level gives us a bit more consistency than if we were tracking at the issue level Okay, cool and what about You mentioned the uh, I think you called out November 5th because it was ready before the feature freeze and therefore was naturally higher How does this translate into a future world where yeah, we're still doing monthly releases In terms of being Is there a customer facing for self-managed and open source versions? but uh But for comm especially we're doing continuous delivery and deploying, you know, 30 times a day that sort of thing yeah, I mean The seventh shouldn't should start to not necessarily be a date that we care about and that's something we have to figure out Because we want to maintain the the milestone delivery With these consistent, you know, like we're just always shipping. We're always getting stuff out to production um I don't know that we've had this conversation of how we're how we want a package or when do we want to stop for a release Or what we would consider to be This is the deadline, but we're seeing a lot of like that that date has been actually driving a lot of stress and um, You know, not necessarily the behavior that we want to see so I I'd like to see that that week is actually a non A non show in our graph It doesn't it doesn't change our throughput because it's an activity that we're able to do in the background based on You know, this is the time when we cut we package we test it and we get it out Uh, and then the rate of delivery is consistently You know throughput is high, you know, we're seeing the that trend stabilized week after week basically Okay, cool Uh, and so if you mentioned this was the configure team So if I were a Dylan the manager of the configure team and I'm looking at this 12 week moving window What does this tell me about my about my uh, my people collectively? What kind of decisions would you make from looking at this? Yeah, I I think it would be you know, we're still seeing a lot of activity on on the week of delivery That might be something to talk about You see some undefined so maybe labeling mr's could be helpful in categorization The other thing I forgot to mention is that we're actually not able to see security In any of our graphs there's It shows up in the verify for instance, but that's only because we've made those issues public So security being able to track the work that's being done on that front would be important For Dylan what I'd like him to focus on is have being able to look across his investment Look across the trend see and why are we still you know merging heavily, you know This is not too bad, but like are we heavily trending toward merging on the week before freeze? Or can we normalize this? There's not so much a huge dip So we can have conversation about you know What was going on on this week and why it was throughput down? This is a two-person team though So I I can predict what the answer might be if it's a week where someone took a vacation that dip is pretty obvious One of the things i'm starting to do with my manager's actually is reviewed this chart Weekly and we're in our one-on-one We look at this and we take notes and I've picked up this behavior from having done this before But it's really helpful because you can start to mirror the data to what is happening with the team So if you're taking notes every week on like the state of things You can go back and say oh this dip was You know corresponds to people taking time off or corresponds to when we were really struggling with tests Or whatever it is that may be impacting the ups and downs in your graph Got it Cool. Um, what else comes to mind anything jumps at you from this we can look at another team too and see the The variables here What are these these other charts? Regarding misdeliverables and regressions of those relevant to throughput or is that just elements of the team dashboard? Hey, Tommy Cool, um, I'm sorry Eric. I think I caught some of that but I got distracted That's okay So I was saying there there were other charts that I could see in the view and it looked like there was a A chart for regressions I wasn't sure if that was strictly part of throughput or just something that's worth talking about No, I can turn it over to mech. That's that's a lot of things that he's been focused on and I know that I Wanted Dash through but by quality. How are we doing on quality? How are we doing on incidents in production and so on we've talked a bit about like how to combine these different views Yeah, if you can it can leave the screen the screen on it'll be great so The charts on the regressions and misdeliverables are the overall measurement that we're trying to do for our engineering productivity so We're going to marry these two graphs into one milestone where in one milestone How many regressions do you have and then how many misdeliverables you have and you can look at that data and compare to the throughput For during that time period. We're also going to add another assumed out chart That's not only per week, but per milestone for throughput So you can actually correlate now like what happened in this month the next month and the next month going forward as well So those are the next improvements that we can add to to the team view In addition to that we're also going to implement a assumed assumed level view At the engineering level where if you can scroll down Dalia the the weekly throughput Graphs we're going to provide this this data at the whole engineering level as well So it's like a collate of all teams so that we can see the speed of our of our engineering velocity overall Nice Okay So what's um What's the current state of this is this in the hands of every team right now have people have been trained on it Sorry, I was muted. Uh, it's in the hands of every team. I I'm more active with my managers. I'll touch base with Tommy and get a get an idea of Maybe the dev team and and touch base with tim as well Front end and back end is interesting because this is a group level, right? It is tracking everyone whether they're contributing at a front end or back end And we've discussed like this came up before if we wanted to have Metrics specific to back end versus front end and then we may want to do that for debugging purposes like trying to You know get a little bit down if we hit a bottleneck We can it'll help us identify which side it's on for instance and and all of that That's a good that's a really good point So for people that don't understand the difference at gila between a group and a team a team is a collection of individual Individual contributors they report to a manager in our case We have separate front end and back end teams. However together they would form what we call groups In other words, we have a feature set related to one stage in the DevOps lifecycle called monitor And there's a front end and a back end team for that So it's certainly really relevant to look at well, what's the velocity of our Monitor feature set, which is the whole group in aggregate But obviously managers want to uh separate from one another understand the performance of their teams So that's a I think a really useful Slice and the group is probably our external interface like product management and and sit It's going to want to know that Internal team is going to be really really interesting. So I think that would be a good a good next iteration for sure Yeah, the other iteration and mech is our mech's team is already Creating an issue for that is to also be able to click here and get the list of issues So you can start to see where is this data coming from so if something isn't anomaly It's a quick filter to show you This is the issue that doesn't have a label or this is you know, where feature proposal is coming from So that's going to be a really great tool for debugging as well The other future iteration is the trend line. So we talk a lot about You know with metrics you want to be careful not to focus on the wrong number and really focus on the trend So having a trend line with this graph is going to help, you know, drive that point across Because that's what we want to you know, we want managers to be having the conversation not about number of mrs But you know the trend over time Yeah, so that's uh mech and I have been talking about trend lines and um This became relevant because prior to this that this dashboard There was a spreadsheet that was meaningly maintained and one of the things we looked at was like the average number of mrs per developer Not individual developers, but average across our developers And there's actually a difference between a trend line and what we end up using which was a rolling average The trend line is almost like sort of just a naive line with a single slope. It's like a classifier The rolling average ended up being more useful kind of like okay Well, here's 12 weeks. Let's take the rolling average of three or four weeks or something like that You get kind of like a smooth Smooth line And so I don't know if we've been talking about that level of detail But uh the extent you need head start I would say a rolling average is going to end up being more useful than uh Than an actual trend line at least this microsoft excel defines that trend line Okay, I I can spend some time here to show to share my screen as well to show you the sneak peek So one second here. Yeah, go ahead Can you see my screen? Yeah Great. So these are the the next future items that's in an epic. Um, I grouped this under engineering factory metrics So the first one I was talking about is this is the zoomed out level view where we would have There's a similar sharp that dally has right now But now the time slices on milestones instead and that's going to be next to the the chart where you have your Your regressions and also missed the liberal. So you see that slice in time What's going on with that release? Or milestone for team and then you have granular day-to-day activities the the current one that we have at the weekly schedule We're also working on hygiene insurance. So If if no labels are added To categorize the types of throughput we're going to nag people to to automatically add these labels or pick one And then the the good lab charge is going to automatically Add the other team label on an on an MR. So if someone opens an MR They forgot to add a team level Every night leave on the bus going to go and make sure that that teams member Team label is added to the MR. So it's correctly added And the last one we are currently going to propose renaming feature proposal to feature So it's concise and easy to understand and the terminology is going to be used across all MRs and also issues as well So the the the terms are the same And this is I believe what we were talking about. So this is the the snippet from the google Sheet dashboard that we are using or we're using We are gonna we have made some progress. So i'm going to show a review app Which is a git lab feature on the quality dashboard and these are under git lab orc There's a new page now called developer productivity metrics And this is the first iteration that's currently in progress for migrating all the google sheet data into into this dashboard And we're going to add the rolling average trend line after this. This is still in progress It's in a feature. It's in a rewrap for the dashboard and mark pletcher is currently working on the the rolling average line It would be nice to add This is great. By the way, in addition to team only community only it would be nice to have aggregate Um, if we don't already meaning the combined got a project um Great. Well, maybe we could um a dial you mentioned maybe looking at another team besides configure Just i'm curious if we see a smoothing effect at a larger scale. Maybe a team of six or eight people Okay, uh dial you want me to throw back the screen share, uh, sure I I was gonna try to do it before I share my screen, but I can share my screen and we can look at it together Okay, uh, let me do that Um, so let's my teams tend to be on the smaller side. So I'm Like if we look at verify that's a team of four There is a little bit of smooth. This is this week So we can't like think of this as a full week yet So that's not concerning But this is a team that very much of embrace smaller mrs and there is seeing a lot of value in it Um, no, but you can still see like we have a huge impact of that day Um, you know the date of the 7th does drive a lot of you know getting things to the finish line um so We'll see I I am watching that that trend and seeing how quickly we can move away from it but I would say let's let's definitely do ourselves a favor and um either Exclude that week in progress or I should say exclude the week in progress from the rolling trend line And even maybe visually denote the the week in progress here from the other weeks Because one thing we noticed uh in the uh dashboard and you saw it briefly when mech shared a screen of it When you include the week in the week in progress or the milestone in progress in the rolling average The rolling average always has this downward angle to the slope and visually people check it and there's this sort of freak out moment They have oh my gosh we're crashing and it's like no no that that week is always in progress So we should do something either. I don't know Uh Visually denote it from the completed weeks or not included in the trend line or something just to prevent that initial like yeah shock factor Um, yeah, I know I can understand that because this is the third time I've had to remind The person i'm talking to that this is the current week and it's only wednesday Yeah, it is So yeah, I know I get it I like it in the chart because as you're going day to day you can start to see like You can also optimize on is everyone waiting till friday? Or is there a pressure from you know, we're not actually getting that daily delivery that we're hoping for um, but maybe we can denote it in some way that Can you know highlight that it's the current week So i'm just looking at you know across some of the different teams. This was planned that we were looking at Create like we're still seeing a focus on november 5th Which which I think will continue to be the case till we get in the habit of just Delivering without worrying so much that this is when everything For now it's most of the team graphs that we're seeing So this is I mean this is interesting this trend does not have the peak that we saw But more of a peak, you know this past week Ignoring the last week on the graph here So that is interesting and and the security one I assume was one that was done like was converted to a public issue and why it's showing up here So i'm gonna make that quote from you Yeah, well, are there any other um either you're having having used systems like this before are there any other edge cases or benefits You'd want to highlight that we'll potentially get or any Any things that are further out than the next few aggressions. They're kind of like, you know You're in a vanda state or stuff you really want us to get to Yeah, I mean with throughput. This is this is pretty good A big part of it is keeping it simple making sure like there is this This drive of I want to track every label and you really want to you know get away from being so specific because that leads to inconsistencies and Just you won't get the benefit of having a chart like this So I think we're in a pretty good state for throughput with the things that we discussed Being able to have these debugging tools for managers is going to be helpful and even for the team So they they themselves can have a view into what's going on week to week Along with that I've used cycle time. So cycle time is a great way to pair up throughput and be able to to understand What is that time to delivery? So that helps when you're having conversations about bottlenecks is to you know pair up cycle time with it Quality is really important if your throughput is to the roof, but you know, your production is going down We need to do something, you know, like this you're we're not focusing on the right thing and Focusing on quality in that case would be more important. So definitely like I have I have an epic defined with multiple Additional graphs if you will and those all compliment each other when you're having those conversation at the team level cool Great. Well, any anything else or any closing thoughts from you Dalia or you Mac? I just want to say that we have a follow-up issue to address the Depending or current milestone and current week. Um, I know that this feedback has been given to us before But I'll make sure there's an explicit issue to to tackle this Okay that I'm Looking at this screen. Sorry to interrupt Dalia We we talked about group versus team group is sort of our slice for any generic Arrangement of people at the company or within the open source community And team is a very specific thing, but I see when I blow here associated project little views I see gilab dash ct team. Yeah. Yeah. Yeah, we should probably just rename these groups or make sure it's terminology These are projects too. So we I don't know mech how you feel about it But maybe just drop team out of it because because basically the info we need is this first part I agree. This is confusing because these aren't teams. These are projects. So Yeah, exactly that action is cool. Cool. And we made it up here as well Yeah, yeah, so thanks for overall awesome work that the last coming up group versus team is just a little bit of cleanup, but But yeah, really awesome progress. It's nice to see this See this come together and how about eventually getting this into the product or project itself If it's useful to us, is it useful to our users or our customers and will we eventually see this in in gilab itself I am a huge fan of that idea. Like this is something that i'm planning to drive. I've had conversations with victor We've exchanged some notes on issues and the epics that I mentioned and if you need links to them, let me know But yeah, no, absolutely. This is something that I feel as an engineering leader You you will find to be like a great tool to have conversations with your team. That's not I have a feeling that we're not as productive as we need to be That's never like the greatest way to open a conversation with the team and instead if you if you build out that habits that You have engineering metrics that you review weekly with the team and it is a healthy way of saying What is what is going on here? Is there a bottleneck we should discuss? Are you frustrated with something? So i've really seen a shift in conversations with metrics then when you're kind of flying blind without them And I'm assuming since we're such a transparent company that these dashboards are open to The entire company. It's not open to the public Mac Leaving that one to you. I I I just pulled up another data point here So the last updated on the groups of the function the feature charts was in december 3rd And I think the team if you can allow me to share my screen one second. This is the latest and greatest so far So I think the team has decided on a charting library that we're going to use to implement this into GitLab.com as a feature And this is the that that's what it's going to look like When we implement it as a feature in GitLab Okay, but these uh with regard to transparency and openness like these these charts regardless of how they're implemented Will be open to the world if or or to everybody within the company Yes, the dashboard is public and we okay, and then the the code is also open source and public as well Great because I was thinking as you were making your point I uh, my hope is is that if these are useful enough Um Individual team members will be looking at them and they'll be raising potential problems or Celebrations to their managers or their teammates as opposed to just the manager, you know kind of top down saying hey everybody Let's look at the dashboards. I hope this is useful enough for everybody just kind of organically starts using it to learn about The what the the company's doing and what their team is doing. Yeah, absolutely. Yep. No, it's not a You know secret manager tool or anything Big brother, right? Yeah, we want to avoid that for sure for sure Great. Okay. Well, if there isn't anything else, uh, thanks so much for the overview. I appreciate it Are you both good with uh hitting the upload to youtube button? Yeah, I think so Okay, that's great. Thanks so much great progress. Talk soon. Bye. Bye