 All right. So I forgot to add a doc to the meeting since it got scheduled last minute. Let me spin something up and then if somebody doesn't mind taking notes. Can we just use the issue? Well sure if somebody wants to pull open a comment in the issue and take notes as we go that would be fine too. Do I I'm the worst note-staker so I think Dau or Sean are my candidates. My candidate would be Victor because he said it said quiet anyway so then he's in the best place to just be typing the whole time. Victor are you okay with that? I was planning to do something else but I was sorry Victor I didn't. Victor you can just nominate somebody else that's what we're doing. I'll take the notes. Okay thanks Sean. I was about to say we've got like eight other people where this chain can go through. It's sort of like a white elephant Christmas thing. Anyway so I wanted to pull everybody together and have a quick conversation because I did feel like we were sort of talking past each other in a few different degrees. Since I did call the meeting I feel like that gives me the right to share what I was trying to communicate first and then we can go from there. So just as kind of a restatement for me of what I was trying to get across in the issue I mean I do think we have a problem here. I do think we need to find a better way to deal with these problems and I do think a solution I don't know if it's the solution that will end up implementing and I don't think it's a solution that we can implement right now given where we are on staffing but I do think you know there is a solution and it's a viable one that we should talk about is creating a team that is responsible for these kind of cross vertical problems. So when I'm you know say cross vertical obviously I'm talking about this isn't just something we can put on manage or create or discuss or anything like that or discuss isn't a team anymore plan like it touches all different areas of the team and there are concerns about that great that we would need to navigate we don't want to create a team of architects in the sense that we have people who are the only people who can work on those topics right we do still need to take into account that product needs to be driving prioritization which is our current process and how does product drive prioritization if there's not a product manager that's thinking about those issues right so like I think there are a lot of other dots that we need to connect and one of the big ones that I threw out there was the fact that we're not able to hire to our plan right now even for the things that we know we need to do to drive the product forward in order to make get lab as successful as we want it to be. We know we're having difficulty even pulling the you know I say roughly three engineers because it's two full-time and two part-time engineers for release management right now without causing a whole bunch of consternation where are we going to get the people to create a team that's going to focus on this stuff right now and I don't think we can that's not to say we won't in the future but I think right now we need to say if we want to solve this problem we have to come up with a different approach and to me I think this is one of those things where you know like in like I mentioned I think in the issue that spawned this conversation which I do want to be sensitive to the fact that that's a confidential issue but in that issue you know what happened was we had an engineering stakeholder reach out to an engineering manager specifically and say hey can you help me out with this and what I would really like to see us do is move more towards the format where we say okay the engineering stakeholder now goes to product and says what are you going to do with this issue and it may be that they prioritize it for their team it may be that they take it to the other product managers and say this is something we need to do I don't know where it fits it may be that any other number of things happen in response to that but I think that needs to be the first line of engagement as we need to say this is something we think needs to be prioritized do you agree yes no maybe and if yes or maybe what are you going to do to try to make sure that we get it prioritized in such a way that it gets worked on knowing that it's not something that doesn't fit into the the area of the specific team so that's what I was that's where I was trying to come from in that conversation I don't know sorry just for my notes I lost could you sort of restate that last sort of paragraph that you said um because I made a complete mess of typing out can I remember the last paragraph that I said um yeah I think um yeah I think what what I'm trying to propose is that we have this um we have this process of identifying a responsible person and product and not for getting it prioritized with their team but for making sure that product is taking it into account um and so this is where we avoid the bystander effect which was a concern that Dawa had raised but then we also need to trust that for product there may not be a clear person because it doesn't fit in a clear product area so we may just need to pick someone and say hey uh you know and maybe Sean the person you always pick is Victor if you run into these issues because you work with Victor directly I don't know but um we need to pick someone and put it on their plate and say like we need to kind of hold you accountable for lack of a better way to put it for making sure that this gets talked about and prioritized appropriately against the other concerns of the product organization at large tell me I'm I'm curious it sound like you said two different things there the first one was that that we want product to be the first person to get the information and act on it and then you just said Sean should take information to product but I mean they might be different examples that you're referring to but yeah I think those are different examples right I think the problems that we're talking about are problems where we have an engineering stakeholder who recognizes something that is a cross vertical concern and what do they do with that right uh in the case of the issue that spawned this conversation it was Caffe insecurity who recognized a problem and what I'm saying is in that case they should go to a product manager right so if in my example that I just shared about Sean if Sean has an issue he's worried about you know like the Rails 5 upgrade is one that got brought up as well as something that should have been um that could have been on any number of teams and it wound up on uh you know what was discussion given the nature of the problem but if Sean felt like that was something we needed to prioritize better you know pick a PM to talk to you about it and that may as well be victor like I don't care how we pick the person um as long as we have a person in product that we're communicating with so my concern there is that you know that kind of is what happened but then I picked Victor and then it's like easiest for my team to do it because we're already talking to each other but like that still depends on the engineering manager going to the right place so if I pick James Ramsey for example does that mean Dowers team ends up doing it and I know you said no but it feels like naturally that will end up like if I just pick James Ramsey it means why don't you have to deal with it Dowers team get to or if I pick um Daniel and like Dylan's team gets it like you know I'm not sure that solves the problem um sorry Dylan it didn't address the fact that an engineer went to talk to Sean about it so then your proposal was that we should encourage senior engineers in our company that have good ideas about engineering to take that straight to product managers before they talk to engineering about it so that like so that I don't have the mental burden of figuring out whether or not I can I can have the capacity for this right now I I should encourage the senior engineers I work with to speak to product about engineering architectural decisions or code reuse strategies or this kind of stuff but it doesn't seem practical to me what what doesn't seem practical well it doesn't seem practical that you'll have to like drop this on someone else to think about this and then that someone else is going to be talking with five different people to decide who doesn't want to pick this and it goes into that spiral of throwing from one person to another like it doesn't seem practical to just have a random product manager go and and and discuss this with product managers yeah um I think if it's technical that that is local to the product area it's relatively easy to convince a product manager this is something we should be doing because it will make it easier to like build on top of whatever thing is currently a mess but if you are talking about something like Rails 5 or GraphQL or whatever the security issue you're talking about what's talking about that we're not going to say because it's recorded then it becomes the kind of thing where any PM will think I don't want to give up like a full-time resource for a month or a couple of weeks to work on this thing if they could also be working on things that will actually immediately or down the line see benefit for you know for users of my product area so that's where kind of the bystander effect comes in that especially if this goes straight to PMs who are already probably less familiar with why this is even the thing we should be doing and less motivated to push hard for it I think it's just going to end up nowhere because what PM is going to say if I can choose between shipping a feature or fixing local technical debt that will allow me to ship a feature next month why would I do this thing that might make engineering happier but any other team could do it as well so so Dawei I promised Tommy I wouldn't talk but I want to jump in and I think the the conversation about mentioning who and who and who I think we already do that at GitLab so I don't think we're going to solve that anytime soon about like spamming too many people but in terms of responsibility the case which you brought up Dawei I would say Sean should ping me which he is doing but he can say clearly to me Victor I want to know a response from the product team about when how are you going to handle this and then so I think he can set the expectation for me and then I can represent the product team to get an answer back to him so he cares about it because it's an engineering thing and then I can go back to him oh like how will this impact the plan side can you tell me about that so I can bring this conversation to product but between me and Sean I can we can have that mutual responsibility an expectation because he cares about it and so I owe him a schedule and so I can bring that to the plan team and if we drop the ball you know the plan team drops the product team drops the ball in terms of figuring out an adequate schedule for this thing then I failed in the relationship with my one-on-one with Sean and so that that's to me that's a way to catch this problem and then or similarly you will bring things and then and if he drops the ball then there's like two strikes so like there's enough you know safety net there that I think that process still works it's we're just elevating it to the next higher level and if if this is like a seriously royal like big thing like rails five then you know Tommy and Dahlia would know about it anyways and then Yobin know about it anyways and Mark would know about it so I don't see it as as a problem as long as that the one-on-one relationship is still maintained so that we avoid the bystander effect so so just just to be clear are we saying that the product team will be deciding which engineering team does the work based on the product priorities of each area I think that should that should be a start that should definitely be a start especially if it's cross product area that should be a start to say that you know the product team gets together and say like oh you know there's these three product areas that will be impacted if we don't do rails five upgrade you know and then we get together and then we look at our priorities and we say then we come back to the engineering team as a whole and say you know all this could be then we sink of course and we say that what do you think about this what do you think about that you know the plan team does the first two parts of this and then they create team wraps it up etc etc so then it's still up to the product team to prioritize have that proposal and then work in everything and the engineering is supporting with saying that this makes sense this doesn't make sense the expertise does not lie here this would be more efficient and so on and so forth so we have done this already and that put us in a problem object storage is my example in this case which was really important I went to josh said to josh can you prioritize that through others that was done it was delayed multiple times between different teams that were working on it we now have two almost close implementations that barely touch each other or touch each other like in places oh that sounded awful sorry didn't mean it like that but the implementations are overlapping partially but not completely and now there is a need to clean this up because there is a large chance of things going wrong so the cleanup is also now not clear to whom do i go to like do i go back to shon and say discussion team needs to do this or do i go back to camille and say cd team needs to do this like where do we turn next and how do we prioritize that but that's one example so yes to victor yes yeah we just make Victor fix it um so the question that i i have in response to that and i don't know if we have a clear answer on this yet um i agree that was a failure in the process was that a failure because it was the wrong process or is that a failure because the process was not executed right um and i don't know enough about the specifics of what went on there to say one way or the other but that's that's just kind of my natural question there is okay we did this and it didn't work does that mean it was the wrong thing to do or does that mean that like we hadn't communicated effectively about how to do it and and created those systems or maybe this is one of those things where um you know we we do need to lean in a little bit more and and stay a little bit more on top of okay well how is product thinking through these problems the first few times to bring the big ones not because we don't trust product to do their job but because we need to make sure that everybody's well informed and we need to over communicate about how these issues are being handled yeah and marina i didn't really quickly that i think definitely we need to retro on that like tomi said and the the the point where i would like to retro is you said it was a failure and but from my perspective it was a success because i was annoyed that the planting had to do some things over the past like year uh it was annoying but you know we went through it and there's like no impact to me now in my day-to-day and so i i'm saying this you know not to be like mass or anything but just to like my incentives as a product manager was aligned through this process so it was totally successful from me as a plant product manager but it's clearly not successful from you from your perspective and what you want to get done so so maybe there's some miscommunication here like like tomi said like was executed well so that everybody can can reach their goals or there's clear expectations but i wanted to qualify that from my perspective it was it was a success so if that's of any value i know it's not much to you but from my perspective it was a success is there a a third suggestion here so obviously tomi you you kind of intrude and i think we've covered in the fair bit of detail the going to product for prioritization i think down on a few others have spoken about the concerns of the bystander effect this the second option is more long-term in terms of resourcing it to a dedicated team for those of you who are concerned with the first option is there's something else that we're considering you asked that question in dylan smirked i don't know if he's gonna yeah but i think i think we have discussed proposals before which violate a particular assumption that product prioritizes all work and so this is basically how tomi introed this is the current expectation at gitlab and has been stated in several places is product management will prioritize all work there are a bunch of competing ideas about different approaches that may be more effective but they violate that one core assumption which seems to be a topic that it seems to be an assertion that we maybe can't move past as in order to propose a third option which like the maybe i shouldn't say this but i don't feel like we do follow exactly in plan um like you know i will tell victor like i'm putting this on the milestone and then he can take it off but he really does like you know yeah so well i am still prioritizing in that at the end of a mouse or at the beginning of a mouse you can come in which one is relatively important oh no not even that i'm just in the scope of a milestone not even within a milestone but across milestones i look at the scope of a milestone and i'm responsible for that and i'm responsible to my boss for that so the the detailed dynamic is that you know shan is just putting crap as far as i'm concerned because there's like so much i don't care but then you know i i'd like to think that our relationship is good enough that he's not going to put a crazy amount of stuff there so i trust him and then that he's doing his job fine and so maybe that's atypical because we've been here at gitlab both of us for so long and that's worked well but from my perspective that's working very well because i can trust shan actively he's doing a lot to put stuff there and you know maybe he doesn't know but behind the scenes i just trust him and and so it's fine to me so it seems like he's prioritizing but from like a you know at the end of the day when sid says product is responsible for a prioritization um and if shan you know gets grilled for something because of doing something first if one before they're like ordering i'm on the i'm on the hook for that right shan is not on the hook for that i'm on the hook for that shan's on the hook for delivering that but ultimately i'm on the hook for deciding that went first but the the fact that he proposed it is just between me and him and that's that's cool with us so yeah like i mean i guess marin and josh is the other case where the product manager and the engineering manager have been the same for a particular area for a really long time so maybe that's not the best example but yeah i don't i don't know i don't see that like product owned prioritization as meaning like literally everything that you as an engineering manager want to schedule has to go up to the product manager then they have to put it on the milestone and they have to decide how important it is like you know you could shortcut that right as long as everybody's everybody understood like everybody put by which i mean everybody in that stage group understands what's going on and why and like why we're doing these things then i don't think it really matters like you know exactly how you follow that process like we just said as long as the pm and the em are comfortable with that then i think like that so in in create i have similar relationships with james in which i also like i'm free to put technical depth into the milestone if i think and my team thinks it's high priority james will mostly just be fine with that sometimes we'll ask for clarification but i you know i wouldn't get challenged on that because he knows that i'm on top of that i'm the expert in that stuff but now you're saying that's for the engineering wide or the back end wide and the technical depth and engineering recognition is we should go through the committee of products who don't actually know why this is important have don't have the context on why engineering is so convinced it should be done so if i throw something up to victor or to james you would and james has to kind of fight for that issue in the committee of product managers he's not going to do as good of a job as i would and honestly if i and my fellow engineering managers agree this should be done i don't think it needs to be discussed in the committee of product managers it should just go into you know the next quarter or whatever if i decide and the engineering managers decide it's that important which is similar to how it goes at a product area level where james will just mostly take my word for it if i said we really should do this this month yeah for me the key issue there is ownership like because the examples we've got are things that i don't particularly care about owning but if i don't particularly like it's the bystander effect you mentioned if i don't particularly care about owning them then like how am i supposed to convince a product manager care about owning them like you know exactly then no stuff the stuff i care about it's super easy for me to do because like you know if i need to make a stronger case to victor i can do that and if i can't make a stronger case to victor then maybe it's not that important um but the the stuff that i recognize that someone should care about but i don't particularly care about is where i think this falls down a little bit especially if it doesn't belong to my particular area either like if it's something i don't really care about belongs to plan then that still is definitely on me if it's something that i don't really care about and could belong to any team that's the hard one right so let's let's step let's step out of that particular problem for just a second because i think if what we're saying is we don't have a clear advocate for it on the engineering side yeah that's a problem and we need to build better tracking for that and i think i mentioned in the issue dolly and kathy and mech and i have been having conversations we're trying to put together a proposal for how we can track those issues in a way where we can advocate for it from a metric standpoint so that as long as those issues are there they get attention whether somebody from engineering is specifically saying we need to work on this or not i think the second issue here is something that i want to be clear on i'm not advocating for us to have two separate processes where one is kind of an informal talk to your pm whatever that process looks like whether you just get to throw things in and victor i think you probably had a great relationship with shon until you called the stuff he put on the backlog crap um i think y'all are going to have to have that crucial conversation later um the stuff he puts on the backlog is crap as well so it's just it's fine well now we're just now we're just in fisticuffs over here um but like i think you can follow the same thing with the picker issues too right and we've done that in some cases where we said hey like i think we need to do the real spy upgrade or whatever and okay sure put it on the backlog where i think so i don't think there's any process anyone's proposing where it's like oh well this is a bigger issue so it has to go through the whole committee process but what i'm saying is more if james agrees in your example dowan that this is something that needs to be done but he doesn't feel like he can get it capacity for it on his team he should feel free to talk to victor or talk to josh or talk to any other pm and say like look this is an important thing either i understand why it's important to explain it to you or i'm going to ask you to trust that i trust dowan in his opinion about why it's important or hey like let's talk to dowan real quick about why this is important and how it needs to be done and see if we can do some horse trading or something else to get it prioritized sooner than i would be able to on my personal team right like i don't want to i don't want to create a like we have to go up the product mountain and convince the seven states that we need to get this done kind of environment what i think you're hearing here and it's also times back into what i didn't say explicitly is that i personally i'm definitely not sold in the idea that product should be responsible for prioritization and if you listen to what shanair saying for how we have been dealing with engineering the initiatives i also wouldn't characterize that as product being responsible for our prioritization but your like one of your comments on the issue says we also have come to a very clear understanding that product needs to be responsible for prioritizing or engineering work i don't know how we came to that clear understanding because i definitely don't have that clear understanding and these three examples of teams having figured out a way to make it work all seem to be the opposite of that so then why would we try to stick to that more strictly at a engineering white level if in my and shan's team everything works fine and we're not actually doing that i dowi i i i think like i i'd like to see the difference in responsible there and like processes if you can read responsible right like victor is responsible in the sense that exactly yep it belongs to him i'm responsible in the sense that i actually do some of it so i think that's maybe an ambiguity there right so do we end up with a situation where engineering managers together would kind of decide these this is this is what we think our top priority technical depth ever for this period and then product for the most part would just not challenge it and make sure it gets done over the quarter in the certain order that the msg right is definitely yeah yeah there has to be a single there has to be a single responsible individual and so what's happening in this case the relationship that you and shan are describing that you have with your product product managers is that the way that you advocate for issues is you put them in the backlog and then if they are if they don't agree with your advocacy or if they feel like there's something else they need to do then they push it back out that doesn't happen a lot because you built a long-term relationship but what we're still saying in that situation is product is responsible for the prioritization of the backlog and the way that they have decided to work with you in your advocacy for the technical issues is that they have a very receptive a very permissive approach to that and i think that's fine and i think that's great but at the end of the day it's still product is the are the people are responsible for prioritizing this and if they can't get it prioritized for your team they are responsible for taking it making sure it gets prioritized somewhere else if it's important work and i know i know at one point which is when the the comparison is within the team i'm arguing with myself to say i can do this tech that thing or i can do this feature thing i'm i'm not going to do feature i'm going to do tech that at the product level i'm no longer arguing with myself i'm arguing with my peers and so i'm in all likely circumstances i just say screw it i don't care because i don't want to lose sleep over it and i don't want to ruin my relationship with james and i'll just say you know you'll figure this out tell me what you want to do and so like to me that's it's an easy job for product in my opinion as long as you know engineering as a as a team says that this is the most important thing that we need to do tech that wise there's no clear owner for it we need to do it and if product is not responsible like i said then you should be escalating to tommy to dalia to to sid or whoever right and this is exactly what sid said on one of these calls about product owning prioritization if product is not responsible the escalation should just go up and so i think this will still work i'm somewhat but i do like mary bringing up the the counter example and i'd love to dig into details where where failed in that perspective so i think that kind of the proposal we're coming to is the first sentence of rachel or comment which is perhaps one person needs to own prioritizing the list of back and quiet issues pulling in few points from different teams contributors and that person would kind of contribute it or present that now i'm talking again i'm not coding anymore that person would present it to you know product as these are our top technical debt priorities for the quarter and then product would figure it out mostly what i worry about a little bit in that case is that we end up with a hot potato effect where no individual em has like very strong personal feelings about the issue it would end up with a random p.m. and then maybe it will turn out big much bigger than expected and every month you know for multiple months in a row both p.m. and em would be like oh i'm still working on this thing i don't really care you don't really care why are you still working on this and the developer will realize that it will get rushed out and then you end up with the upload object storage situation and again i mean again i think that's that's a case that could happen i think if that's something that is going to happen then there have already been several communication and relationship failures along the way um so i mean i feel like that's an iterate and that's an iteration that we're trying to get to right we want to be able to look at our technical debt issues and have them categorized with priority and severity and we want to be able to go to product and say this is what we understand is our definition of what it means for us to be managing technical debt in a healthy way will you commit to taking this into consideration when we're doing prioritization and roadmap planning right i don't think our priority labels are very useful for that because at the moment one mean p1 means do it immediately p1 means p2 means do it next month and then you have two labels left for any option that isn't those two uh and i understand and this is why i say something like that but something that's actually applicable to technical debt right like we have to come up with a whole new classification scheme which is why i'm saying we're talking about it and trying to build a proposal not we you know we pushed a merge request three weeks ago and it's been approved and we're using it um so i think you know we need to try to figure that out and that's going to be a big help but until we get there like let's focus on the ones where we do have engineering managers that care about it let's focus on the ones where we can make those advocacy cases and let's at least not leave those issues alone or in danger of getting dropped while we work on building that next that next median point yeah true and the issue where it is all started which is that security issue does have someone advocating for it so we you know this could probably work through Kathy prioritizing it through product in general and it'll just get done because she'll keep pushing for it right and then i guess to kind of make a dent into the entire the whole list of technical debt issues that are not also assigned to a team label we as engineering managers i guess kind of have to band together and decide what it's just stuff that we want to tackle this quarter for example yeah okay uh we are at time um so i want to thank everybody for participating i hope this was a productive conversation it's quickly outcomes so we're just gonna stick with what we've got at the moment but with the understanding that engineering managers can talk to each other about like talking to pms about this stuff so it's not just like i have to own this because like you know it's the plan team and i'm the plan engineering manager i can talk to dara i can talk to marina i can talk to liam i can talk to rachel or delin or whoever um for the specific issue the security issue kathy can own that and take it to the product team it doesn't really matter which product manager it is but she can she can take it to them and work with them to prioritize it and um i can't remember if that was a third one oh um marin could you follow up somewhere he's gone for his stuff separately oh fine i'll mention it in the mode yeah throw it in the issue um and then the other action item which is actually already in flight but um i'm going to start in a last dolly to start as well pulling the engineering managers into that broader conversation of how should we can't uh classify technical depth what is a good system for that look like that we'd be comfortable taking the product and saying you know help us maintain these these health metrics yeah is there a way for us to be aware that those conversations are happening without it having you know coming up in an issue and you mentioning it or has this been something that it wasn't even a case in like the last last deaf back-end staff meeting uh it's relatively nascent at this point still so and i mean like i said it's already on our deaf back-end staff meeting agenda i put it on earlier this week that like we need to talk about this so um yeah it's just it's an ongoing conversation that just got started we're getting to the point where we're talking about a specific proposal i was about to enroll you all anyway in that conversation and and here we are with a specific issue so uh accident of timing i think in this case cool all right thanks everybody i will publish the call and post it in the issue but i think this is a productive chat and i'll talk with everyone later