 Yeah, so hi So Niels and I began this project about I don't know three quarters of a years ago to try to improve the culture of the debut mentors mailing list and mostly I'm just going to talk about what we did and why and Afterwards in Q&A. I can talk a bit more about how it applies to other projects or what we were inspired by so First I hopefully you all know what you've gotten yourself into By coming here if you want to reach me You should email me as a shisha at a shisha org or I'm Paul Proteus on IRC and at Debian and this is me I want to know a little bit about you guys. I guess can I get a mic runner? like For you folks are Can I just get some introduction as to why you came to this talk or your relationship with the debut mentors list? Someone give me a hand. I Know who I am. You know who he is great. I wouldn't go so far as to say I know who I am, but I'm actually organizing a sponsors buff Tomorrow at 5 so I'm trying to think What to minimize overlap or somehow? Oh, yeah, I'll be there right so so you asked why I'm here So I'm sorry. It's a boring answer, but I'm here to make sure that our two things go together. Well, cool anyone else like Yeah Hi, I'm a rico Vini from from desk and I think Mentors is an very important part of joining Debian and so I'm happy to be here cool I'm Jimmy Kaplow. It's and Enough of my And I know enough people who write software that they may want to get into Debian and I'm trying to just get a sense of the current and Desired future State of Debian mentors to figure out a good way to maybe induce them to be involved directly cool, so I'll take a second to say what led me to begin this effort in fact where I came from in Debian in general So my history with Debian was that I started out by lurking on the Debian mentors list six years ago And I was really excited about Debian. I thought after a few years of using after like five six years at that point using Debian and More and some number of years of using Ubuntu I thought I want to become a Debian developer and I think now I can so I made some packages I emailed the Debian mentors mailing list and I got some pretty helpful responses one of my packages had a Conflict that I hadn't declared with a different package and somebody helped me with that Nobody actually sponsored my package, but they did review it So I found people off-list to help with that and it wasn't so I was sort of Slowly getting involved. It took another year for me to find somebody who was consistently willing to sponsor my uploads and review my work and And I think that this process for me is a really important piece of the motivation of why I wanted to Look at the way that I've been mentors list work. It seems like a lot of the time Well, it seems like it doesn't always quite work. I had to sort of go outside it to find somebody and when people When people sponsored my packages, they didn't actually email the list back to say so so Then a year ago. I ran this brids of a feather session, which I'm running again soon called Debian for shy people and It was a general discussion of cultural problems in Debian and how we can get more contributions from people and So Nils and I talked for a while after that and we thought what are we going to do? Well, maybe we can just fix the entire mentorship process as a whole not just the mailing list but connecting people and Getting people to communicate with people who want to review their work and We decided that was a bit of a lot to do so we thought we would just focus on the Debian mentors mailing list so Then Nils PMed me a lot during July and August to try to follow up with me and I ignored his pms Because I wasn't sure I had time to fix all the mentorship and I wasn't even doing much sponsorship or review on the list at all and So I thought about what the problem was and I realized that for me for me I read the Debian mentors mailing list and I saw Things like this where there's almost 300 mailing and messages every month. That's good There's a lot of activity but 10% of those messages are unanswered threads They're messages where someone says I did this great work for Debian I tried my best here it is would anyone even take a look and if it's good, maybe upload it into Debian and about one of these messages a day comes through that nobody ever responds to and for me that was a big problem because It wasn't just that I I wasn't just unhappy that we were failing the new contributors and the Debian was slowing down in growth I was thinking more about how those people are Trying their best and we're just ignoring them and That's When you think about one person a day of practically showing up having done work for Debian and then us Saying nothing that just made me feel so bad times 28 people a month So I didn't even want to do packaging on the Debian mentors list because it's such a sad place and in addition In addition, I know that if I just do one or two or three extra reviews a month It's not going to have a huge impact in this 28 un-reviewed mail un-reviewed RFS's per month so I sent an email to the Debian mentors list explaining what I just told you and I figured that Neil's and I discussed this and we figured that Even if we just reply to people Even if we just reply saying sorry Nobody answered your mail, but it's not because we don't care about you. It's just because we're all busy That would probably be nicer than hearing nothing and So we set ourselves this goal that we would do that within four days that every message would get a reply within four days Who know who cares what the content to the reply is Hopefully it's a review But if it's not at least we're not leaving people on the floor so that's what we set out to do and We got some positive responses on the mailing list David DeLoe is somebody who on the Debian mentors list he only he submits a package once and if he Hopes as a reviewer and if not he just waits until there's new upstream release at which point he submits another package Hoping to get review Other people have figured out that you can submit packages multiple times So on the list you often see the second try third try thing But there's categories of people like David that just don't do that And so we're missing out on their contribution to Debian There were some enthusiastic responses like this one by Holger which basically says Go ahead What the way that we structured this was that Niels and I said that we were going to set this goal for ourselves we hope we'll succeed and If you want to help out that's cool, but this is really about us working on the list We're not saying everybody you must reply within four days and There were some other males that suggested that the problem was real This person is now a DM But at that point He suffered the same he was one of the people I was rightfully worried about I guess So one question you might have at this point is how we came up how we came up with this number of four days That's just sort of the average between two things that seem too small and too large To expect a reply one day seems too fast to expect or to get here nothing for a week seems like too long so for and and so I'll tell you now that it worked and then I'll tell you about five minutes how it didn't work but There's something we saw some really good progress so during November The number of unanswered messages was less than 1% which is a big change from 10% and crucially is only four over the whole month and This is especially remarkable because I actually didn't do any Review during November even though I said that I would sorry deals Leaving Niels holding the bag but also other people picked up from the hopeful enthusiasm that we set on the list There were lots of other people doing reviews and there always are lots of other people doing reviews But it just seemed like there's more reason to be realistically hopeful if you submit a package and So the number of messages also doubled and that stayed through that stayed true through December basically so about 2% of the messages went unanswered and much fewer than one per day and So I like to use this slide when I talk about this publicly outside of debcon, of course, we're being videoed So it's all over now But yeah, so we double the number of messages that unanswered threads went down 90% like success right and so I gave this talk at the Python conference and talked about how great this worked and Somebody actually left a comment on the video saying That he really appreciated My work doing this. This is actually the person who left who left a positive note in the beginning I realized recently And you know, it really did work that we got more package review and We like I I don't have a good way of tracking I would really like to know if more people during those successful months Have become DMs and DDs than on an average month because sort of that's the real metric of progress Not just how many packages get reviewed But the other thing is that a lot of other people on the mailing list showed up with their own ideas about how to improve the Debbie mentors culture and That was a good thing and the other thing that I actually did eventually by December start doing more package review So crucially for me, I solved the problem for myself Like I was scared of this list and I wasn't anymore So there were a bunch of naysayers on the mailing list Some people talked about how The The wall tabs in particular said that we need more people willing to do review We can't just solve the surface level problem of people being scared of the mailing list Michael Towschnig, I think is how you pronounce his name talked about how We need to get we can use the Debian mentees themselves as reviewers and that was a really good strategy and actually worked really well He's been apparently the one there's been this huge surge in review traffic recently on the list and that's from his mentees But I do want to cover some of the ways in which I failed so the I talked about how great it was that we decreased unanswered threads by 90% but By December it was going pretty well by January It's back up to almost one per day and actually in February it's even worse than it was in September 40 unanswered messages We got more traffic, but somehow lots of unanswered mail still and yeah, that continues through March I Think that the big reason that we that this We lost the momentum that we gained was that well there were two one of that we stopped saying we would succeed And the others that we didn't actually succeed I think that if we had if we set ourselves this four days goal again and Try to and hold ourselves to it and announce it and maybe maybe make it part of the fact that You should ping the list again when your mails don't get an answer in four days and mentors You should make sure that people get an answer in four days. I think we can go back to the November and December figures so the That's how he failed the things that I learned was that what I sort of already knew years ago But I forgot a lot of sponsorship happens off the list So a lot of the reason I'm worrying about these contributors whose mails don't get replied to is that for a handful of them They get reviews off list. They get uploads off list and nobody tells the list But it's great for that Mentee and great for that sponsor and great for the for Debbie as a whole You have it's bad for me because I look at the list and I worry about all the people not getting answers to their mails The other thing is that it did in fact take a little more effort than I thought I mean surely when I put in no effort during November that was bad, but The other thing is that we set up a script to Neal's wrote a script. I think to monitor which threads had not been responded to yet and the number of days that they have been waiting and Then I deployed the script, but then I failed to maintain it properly. So I kept showing results from the from the previous month and So that's the kind of thing where It would be nicer to have concerted effort and maybe more of a team So thanks Yeah, so That's sort of all I have for you in terms of how that went I can talk more about I Can talk more outside of the slides about where I got the idea to just Run this but I think that's pretty clear at this point What I really want to do I hope and maybe I can get a few more people involved in this in the next few days here at DevConf is to Create some sort of team of people who want to maybe trade off Making sure that people get answers to these mails so that it's not just Neal's and me holding the line But maybe it's Neal's and me for two weeks and then it's two other people for two different weeks So I think I want to end here and ask if any of you have questions or if you want to change other parts of the culture Or if you want to help me So thank you Well first, thank you for your effort. I mean, that's really appreciated. I think I Personally have problems with the Debian mentors made mailing list because it has too high traffic I mean it's more than 15 to 20 mails to read a day And that's I think something like eight to ten packages that have to be reviewed every day So it's it's huge and I I wonder about the idea of Refactoring the whole mentoring process because you said that when someone comes to the Debian Mentors mailing list with a package and with some work. That's work for Debian But I wonder and and if that's really work for Debian We all know that having more leave packages that are only maintained by one person for personal reasons Doesn't help Debian much so I wonder if we should not refactor the mentoring process to to Also have people avoid making work for nothing So because if we know from the start that well packaging This is not useful any in any case or you should do this in a team in which you can get sponsorship For your package that helps much even before that the work on the package has started and the problem of the Debian Mentors Mailing list is that you get the work once it's almost completely done and this filtering process and Directlying people to teams and maybe just tell them well you should not package this or Something more polite Yeah, and this work happens off-list or doesn't happen. So I mean I mean I think we should fix that Well, I guess I don't want it totally still for your buff So you should certainly bring that up tomorrow at the Debian Mentors buff, but just talk. Okay. Okay, cool Yeah, just to know a little more Approximately how many How many okay answers per month? Did you had to do in the golden days of November and December? So just to evaluate How okay how how much load the this project put on you like between two and five emails a month or something Maybe a few more, but sorry between two and ten. Okay emails a month and more over how many I'm I must admit I feel a bit guilty because I never helped the Debian Mentors mailing list and maybe somehow Someday I'll do but how many of the packages that That are Okay, that we have requests for sponsorships for in Debian Mentors mailing list could be Ham Okay moved to other specific mailing lists because in my opinion, this is an important issue too if we manage to Okay, so parade the RFS by Category by team work is More efficient and finds more people interested into so I'd like to know with which percentage in your opinion could Could be optimized with this idea. Thank you So I think I think the question with optimization is how What percent of the list could you move to other lists? Basically? At least 20% probably more You do a matter to that in particular Okay, well then the other thing I wanted to say is that a lot of the in the like 30 or total mails I sent in this four days effort, which you know, I've probably spent more time Preparing for this talk than I did it actually typing emails. It wasn't actually very much investment. It was more about the commitment the a Lot of those males were sending people to other mailing lists So what I find really surprising about well, what strikes me about the Debian Mentors is that you are You need some kind of systematic process to avoid having some mail getting lost and the way you implemented Fourth day's limit Basically is by distributing that process so that every reviewer look at the males and hopes that doesn't leave one goes through And I think that's the kind of stuff where we really you really need to move the process to somewhere else like a website With a really nice workflow, so you can track with responding to women. What's the status for each? Package and so on and I'm really surprised you don't mention expo because yeah That's exactly what you are doing with expo Debian net and I think that's moving Most of the content of Debian Mentors to that to that website. I would be well really change The way we deal with that. Yeah, I can quickly show that you should just talk about it for two minutes to give an idea of what people yeah Well, I'll quickly summarize what the expo is It's a product that was created about three years ago in summer of code and the summer of code project didn't quite finish successfully But it's been sort of half revived Well, the point of the summer of code project is to make a new code base for the mentor of that devine net website so in the beautiful future you'll be able to when you upload your source package the lintine output will be there on the web and You can Maybe do things like deb diffs between different revisions of the same package You can leave you can leave comments on the web not just on the mailing list Which might help traffic be more relevant on the mailing list Well, so basically shortly after this effort I started reviving the deb expo and We it's ready for use now So I can quickly show it But all it really is in a more a more colorful version of the mentors out Demi net site it does have a few new features like you can leave comments on a package on the web and The lintine output is on the web and there's a few other quality assurance checks But the other exciting thing is that the code is actually maintainable So it's really easy for people to add new quality assurance checks that are part of the package upload process And we could if we want to move a lot of package review to that and you could just leave comments on the web We could even have the unpacked source tar ball be viewable on the web so that you don't have to de-get at all you just To review you just read what's there I don't know if people want to alter the mentors workflow that dramatically, but maybe they do And the other thing we could do is yes Anyway, then then we could actually use a mentor's mailing list for questions not for a request for sponsorship basically So first of all thanks a lot because this kind of activities I find a really brilliant So as I mentioned sponsoring is often the first thing that someone encounter when he wants to work for that So giving an experience which is not frustrating or which is not like you know You're doing stuff that nobody wants. It's very very important for us That said to change the culture you essentially seems to have been doing that up to now the part Which is visible to applicants which is of course the first part because we have already given the experience to them that in Four days someone will will reply to them and someone will you know let them know that their work is not being ignored But then you need a second part So right now as you said you've done that only in two person and need to scale because otherwise we will be back at the same problem I know one years from now and what you have been doing in some sense reminded me of the use net days where you know Newsgroups used to you know set their own FAQ which were for all the participants So I've seen you have an FAQ on for the mentors at least it's mentioned on the website on the mailing list page But it seems it's mostly oriented to our You know applicants and not our mentors So maybe what you need to do is actually you know generalize this FAQ saying okay If you want to be a good citizen on the mentors list Well, you are every now and then please keep track of those mails so that all together we make it You know we we collaborate and we make the process less painful for everybody including mentors Just to make sure I understand it sounds like what you're saying is that you're suggesting that We optimize the mentors FAQ to help mentors not just mentees In addition. Yeah Okay a few questions from IRC The first one is regarding the static statistics you showed earlier The static statistics were on answers to messages not ERIFUS Receiving review, right? Did that happen or was the reduction in unanswered RFCs just Through sorry, I can't help you right now, but thanks anyway messages. I think most One of the things that Don Armstrong brought up in that thread was that in the initial four-day thread where I Proposed the idea with Niels was that it's important that we have a way to tell Perspective packages and mentors is good. You did this work, but we don't actually want that package, which is kind of like what you were saying and So to actually answer the question from IRC, I think that most of those emails were Some it's hard to tell which which ones You know in September there were what 26 or so unanswered threads and then in November there were three or four or five or something Which so then there's like 120 threads which one of those do I point at and say these are the ones that my four days effort caused the reply to There very very few of the total number of messages though were about us being too busy Most of it was I think like one or two or three were like that. Most of them were actually review or Saying this you your email your email will be better off on the other list Okay Another question. Well, actually it will be two questions baked into one Observing the current mentors culture I see a large technical variance on requirements Sponsors require from them from their sponsors and the time varying probability of getting sponsored There is a chance a good package which shows up at a wrong time Not getting sponsored while others occasionally perhaps of not so much a good quality Benefit from a mess fly by sponsoring which seems to happen every once in a while How to address this problem to provide everyone the same chance and Related to this There is a varying Well, there are many sponsors and they all have different standards on what they accept For example, some sponsors require depth five formats copyright files while others just Suggested Is there perhaps a way to arrive at Something common. Well, you can do it to answer you can use a second question to partially answer the first So if you just you know, hypothetically if you want the maximum chance of sponsorship You should comply with the most stringent guidelines mostly People are happy to sponsor DH seven or eight packages that like check all the boxes Depth five readable copyright Clean linchin I try to remember what the other Uncommon requirements are by sponsors. I don't think there's that much divergence in the requirements It's just sort of how stringent are they mostly is that they don't disagree. Is that a fair assessment? Well, the other thing about well, you should have to move them like this direction, but the other thing about Getting a fair chance like you said, there's so many emails. I don't know A second and third try emails are probably the best way So I think the idea of peer review is great I mean, I think that's got to be part of the culture or it will collapse But it is already no, I mean not I mean that that people submitting packages to be sponsored should also Yeah, helping with reviewing but unfortunately It's not It also adds I'm noise is not the right word, but it makes it harder for mentees to know what are the real requirements and What are the things that other mentees are repeating? That you know the sort of the rage of the newly converted So in fact the advice from mentees tends to be even stricter Then then then I would give myself. Yeah, you know, they tend to interpret every suggestion as policy must and So so I mean, I don't know what the right answer is it but it but it's part of what's going on here the the feeling of you know a Plethora of requirements that that mentees are seeing so I can give you a Let me tell you about how a different project handles this in a way that Debian probably will never accept But works pretty well for them So memo my mo memo They have an extra's repository and in that repository anyone can submit packages your They have a review process though like sort of like what we do There's though is very clear so When you submit a package there are some clear rules as to when it will migrate into the actual extras repository And out of the staging ground those rules are it has to have been there for ten days It has to have a karma of ten or higher and a karma means a karma is a review of somebody a user actually trying the program trying to resolve the deb Running the program making sure the program actually works and that's not a package quality quality required check so much it is a well, it's not about The details of the package quality so much as that the package at least works And you have to get at least ten upvotes to meet that jump and the other is that You have to have three people who are actually on the QA team of my mo extras to give you a plus one so the I've met the one of people who's involved in my mo and In October and he was super enthusiastic about this really clear mechanism and of course we have nothing like that So yeah, obviously we should just say if your package has been on mentors that I mean not net for ten days And it's had ten positive votes and at least three developers say this seems okay You should automatically move in unstable, right? maybe not so the Debian Mentors mailing list is is really for review of Packages packaging or is it better done elsewhere? I mean the mailing list doesn't seem like a great place for coordinating Reviews in an efficient manner. If I was going to review a package. It might take me a day or two and In that time somebody else might be reviewing it as well Maybe it would be better if we were both reviewing different packages and we wanted some coordination between us and is Does is that sort of thing a problem? I? Don't really think so. I mean sometimes people say I'll review this and then they're viewed in a day or two, right? But what do you guys think? Sorry the latency is so high with the mic So essentially you're making a point of the risk of some race condition, but actually if you have two concurrent reviews It's not necessarily bad I mean You're right You're right I'm gonna confuse the video team dramatically by so this is Mike also on That is an ambience Mike just Noise and it only goes to the stream not the speakers then I wonder if people can share my headset Mike and then we can have two simultaneous Mike things okay, we can move the mic around, but the other question in terms of your statistics Yeah, when You were saying males were unanswered. I assume that you cut off four days prior to the end of the month. I I sort of eyeballed it like I Looked at the last blob of the month, and then I checked the following month and didn't count them if they have replies okay, I Just took a look at the our EVU Tool from Ubuntu do we have some first-hand information about this because it just seems like Mentors that they've ended net on steroids. I mean You can access and an unpacked source version you can have access to the DSC the changes file you can comment on So and you can mark the package as reviewed once as advocated etc So I wonder if we could just or additionally adopt this tool for us or for our needs Because it just seems like it's collaborative review and you can add comments specifically on packages and files That's right. So I wonder if we have some more information on the fact on if it's doable or not and by whom Danny you have first-hand information. Otherwise, I can relay second-hand information in practice. I guess you may also You may also use review Okay, right So review doesn't work very well in practice because in Ubuntu we don't have enough people reviewing stuff on review and People some people are very persistent and keep coming on to IRC is saying can you please review my package and often? It's a package that we really don't want in the archive We're all it might be okay and Debian But there's there's no particular reason to get it into Ubuntu where no one's ever going to maintain it in the future So these packages go with our review for years at a time and it's probably pretty demotivating to everyone involved on the other hand if there's a package that does need to get into the Ubuntu archive say it's something related to it's an indicator or something Ubuntu specific and the reviewer with the Mentory can persuade a Ubuntu developer that this should get into the archive and that personal come back and repeatedly review it Then I think it works quite well because you can see all the history of the review you can see that every change that was made and It's a bunch of developers have a button. They can press to advocate. It's a little shield will appear saying this was advocated So that that echoes some second some things that I heard from other Ubuntu people earlier at the conference and But at the one of the key things that they were telling me is that? Ubuntu doesn't really want to keep using review for new sort of leaf packages Unless it's crucial to Ubuntu. Yeah, I'm gonna get her stuff We've been talking about putting a banner on it saying please don't use this unless you really need to yeah something else about it because the comments on it are Restricted to the website and aren't on a mailing list You just don't notice when your packages appear unless you go looking for them It might be really useful to bridge website comments with the mailing list somehow so So DebExpo does about half of that you can see that at expo.debian.net I Can show you it's gonna be kind of wobbly and Let's see Very exciting Pardon me. Okay good Yeah, oh unless I'm not very online And this is the part where my Privacy later proxy settings slow me down There yeah The look is likely to change But the difference between this and mentors.debian.net in particular is that this is maintainable so we can add features like that to it What I've wanted to do is bring this up to the point where we can deploy it to replace mentors.debian.net And it's basically at that point now. I need to finish figuring out exactly what server it'll be on But after that point, I kind of want to step back and let other people Mostly be patching it and deciding what features it should have so if you want if you like writing Python and making web apps then Really really we need help with this and I'm also very willing to tell people Exactly what lines to change or what part of the code to change and I will deploy patches quickly Just to look at a package for example If dim and keeper on IRC is watching then hi and also thanks for your work on expo too This is his package and you can see there's the linty not put at least and It does things like check the watch file to make sure that The newest version is the one we're using and actually in this case there's a new upstream so I imagine he'll be giving me a new package to upload soon and You can leave comments There's no state changing like the review app has for like when a developer marks it as advocating for the package But of course that would be easy to add Was there anything else you wanted me to point out about this Lucas? Anything else in particular to say about expo or just to show it Besides it looks really promising. I don't have anything else Yeah Lucas also proposed a feature which I really hope somebody is willing to implement and if you are then I will find you Want to talk about it. You're going to talk about when you talk about the okay, so the first thing that I'd like is Currently we have something like debian mentors on one side and teams on the other side and most teams Reviewing process is suboptimal means people Ask for sponsoring and then get Get forgotten even on teams and it would be nice if actually team could teams could use expo To manage their own sponsorship queue. That is if you could propose a package on expo and have a box that says okay this Package is probably something that the you as a ruby team could sponsor and then the team could have a special Special expo view where all the package of interest to that team would be listed We could even do something crazy like have it automatically email the RFS to that team list Do you want that would be fine? Okay to get notified in some way or? And the other thing that's the one that I wanted that thing is awesome because that'll move a lot of traffic and the other things that I interested I'm interested in on is Turning this into a kind of social website where every user can review as a package get Karma points if they do good reviews and stuff like that just want to reach Also none did these to do reviews and get their packages ranked higher because they are listed as a more Experienced more experienced package or and stuff like that. I think there's a huge potential once we have a clean code base and I Guess in brother grudge the Ubuntu experience for review It seems like you had this tool that did a bunch of the things that Lucas described except for notifying teams But you still didn't that people didn't end up using the web tool to do a package review Is that because people don't like filling in web forms or is it? For some other is just because the packages are of stupid leaf things that it doesn't want Yeah, he just said that people were using review for review, but it was hard to know to come back again Apparently I have a minute but you all okay one one of the idea had thought for the Review process that I'm not going to subscribe to the mentors mainly because as it said way too much mail But I'm willing to do a review now and then so in spite of these Dating websites. I saw that actually we could have some come kind of automated met match system They would match a package pending for review and some Gds that works in similar fields and Yeah, and thanks and Rico has shown shown me a script that is doing that similarly for NM and I'm writing it right now So probably I could have something to have to expose that could automatically send emails to Gds I could would be interested in doing some reviewing like being bugged bugged a little bit, but That's one idea there to encourage the process. Yeah While walk them while you walk the mic to pads. I hope you will unless we're out of time You can imagine on expo signing up to say I'm willing to be randomly assigned one thing per week And then it'll email you in particular So so Jeremy's idea was basically one of the reasons that their bexbow was started So we could add that sort of thing to mentors Which wasn't maintainable Right and perhaps is one of the people who has been working on expo here and there. So thanks to him too All right. Thank you. Sheesh Thanks