 Right, in best LSF tradition, I don't have slides. So as this is more designed, or the intention was to have it more as a discussion topic. So what happened was that, or still happens, that there are patch sets which have been proposed and for some reason really have not been applied or have been rejected. Occasionally, at a really late stage, meaning after, say, revision 14, suddenly we'll decide, oh no, the patch set itself is, no, we're not going to do this. Whereas others, which are nearly as intrusive, will be accepted without any problems. And that makes it really hard for companies who are not that involved in the entire development process to figure out, right, we are going to propose this, what will happen? Are we going to spend lots and lots and lots of manpower trying to implement something which at the end of the day will be rejected? And so I really haven't been able to come to a consensus or some guidelines, so which award is accepted, what is not acceptable. Occasionally, it seems to me really, I mean, if you happen to, well, get in touch with them at the main time at the wrong time, you'll be rejected and if you're lucky, you get your patches in. So really, some more discussion topics. Is there anything we can do to improve this process such that maybe if there's something which we are not ready to accept, we really should put in the brakes rather early such as not to spend lots and lots and lots of development manpower. So I cared very deeply about this subject and this is something that we've tried really hard to address inside of ButterFS specifically and if you look at our patch counts, like pretty consistently, 300, 400 patches every six months, we're up to like 800 now because we got really, really good at this addressing how and when patches go and this really takes a concerted effort on the communities part, the maintainer and the tech leads in that community to give signal really early on that like, okay, this idea itself is worth doing. We are going to put effort into making sure that you can shepherd it along and I find that in having worked in this area for too long, this isn't the case, right? Like a lot of maintainers just sort of like view it as their own fiefdom and that is like, this is the thing that they get to do and they don't care, they're not bothered, they don't, like if it's not what they're working on, they don't care and so by the time you actually get their attention, like they're just gonna shit on it or whatever or you have like random drive-bys, especially for new people, they don't know who they're supposed to pay attention to or who they're not supposed to pay attention to so you get drive-bys and then you go and re-architect your thing and for a suggestion that should have just been ignored because the person who said it didn't talk about, like didn't know what they were talking about, this to me is a fundamental problem in this community that the way maintainers behave is erratic and inconsistent. Like we, this is like, we are all grown ass adults in this room, we ought to be able to work together a lot better than we have been and I think that starts with having a more clear definition of the role of a maintainer. It's not just I take patches, it is I give clear indications of like newcomers or whatever or I give clear, descriptive like feedback of like this idea itself is okay and you need to get reviews or I give clear feedback of this idea is not good for these reasons, right? Instead of it, you know, you get to this point where you're 15, 16 reviews in and suddenly somebody comes out of the woodwork and is like, oh no, this is terrible. Now there's other cases where this happens that are different, I think Willie can attest to this where it's like, you're doing something that's very core and touches a lot of things and some of the people aren't paying attention, right? That I don't have a good answer for, right? I think that for ButterFS, again, this is a smaller part, one of the things that we did was just talk to each other more, right? Like we get, we just had one an hour ago, we get in the same room, we say, okay, this is the patch that I'm working on, this is what I need or this is what I'm doing, this is where I'm going and then I can say, Christoph, I need you to look at this part or you know, whoever I need you to look at this and so it's easier to merge these things that are like a little bit bigger. We don't have that for the big things. I don't know what the answer is for that but I think that those are the two core problems that I see. This is actually very related to something we're doing in the tab. I'm leading an effort right now, I've had help with Dan Williams and others on the tab we're coming up with a communication style documents. There's only two of them. Well, one of the communication style documents is basically how maintainers are to talk to other people and it's just, it's not gonna be like, it's not gonna be like telling you, it's basically here's guidelines and it's mostly about getting a point of views of what the other person's doing. There's also going to be another document that's gonna be how contributors talk to maintainers and a lot of it, like Linus is- All of this, sorry to interrupt you. All of this sort of implies that they talk and that is the main problem we're having. The thing is here is that the reason why we want to make a document and put it into like the tree is that when something happens you could say, please read this document. So when something breaks out. So we try to get away from the, so we want to document everything of the issues that we see. So I would want to talk to you about this because this is exactly the type of thing that we're trying to address. So I just want to bring that up. And I think this is a good thing and it's not only like communication, right? Cause like you guys know me, I swear a whole lot but you'll have noticed that on mailing lists I purposely don't do that because in mailing list I don't know how I'm going to be perceived. Here I can be pretty sure that you guys understand that I'm not angry or upset with your calling you names but on list that you may not, right? And also there's other people on list that haven't been in this room with me that don't know me and that is can be very off-putting and very like aggressive and it can make people not want to join. So the way I conduct myself on list is very different from what I'm standing up here, right? And I think this is sort of what I'm getting at, right? Is that it's not just the small group of people that we see all of the time. There are more and more people coming into this community all the time and it may seem silly but the smallest things can turn people away. Me being me on list can turn people away and I have recognized that and I have changed my behavior. If I can do it, all of us can do it. And then I think that's really good for the communication style but the next part is like the actual responsibility of a maintainer, like you're maintaining code. What I really, really, really, really hate is like we still complain and arbitrate topics that are settled. BPF, system D. You will see snide shitty remarks from maintainers about technologies that are in the kernel, currently being used as if they're still up for debate. We are all in this together. We're all trying to move in the same direction. Let's try to pretend like we like each other and pretend like we are actually all in this together and not just fighting with each other and maintainers need to take that more collaborative, more like understanding of, okay, I see what you're trying to do here. It's not the right way. Here's how you do it. Or, you know, it's not like every idea's good, right? Straight up rejection and why? And alternatives, that sort of thing. Sorry, I didn't mean to hijack your shot. No, no, no, that's perfectly fine. But then my key objection to that one is that this whole of this entire thing is more due to a lack of communication. If they react, then we can get started. Then we have some idea of what's going on. But typically what happens is that relevant parties get involved rather late and then say, all right, no, I'm not going to do it. Oh, thank you. You could have said that earlier. Or rather, having a flat refusal for just because is also not really going to help. Because that sort of leaves you with no middle ground. Right, okay, so we don't like it, but what is it exactly that you don't like? So is there, or rather, is there anything I can do to change, to get to a similar result, allowing me to further my use case or is the entire use case completely bonkers? Exactly. Constructive criticism, as Jeff Layton just said. Right, and this is the thing, is like maintainers, your job isn't just to take patches, your job is to steer the direction of your projects. I am maintainer in title only for ButterFS because Sturba does all of the work, he's fucking fantastic. But I take my role as kind of guiding people very seriously. And if you're going to be a maintainer, especially for a core part of the kernel, like your job, unfortunately, is to look at the random patch set from somebody that you may not recognize and give some sort of feedback. And the flat refusal thing that I see a lot is just not helpful. Again, we are all trying to make this thing better. Somebody has a use case, they didn't just sit down like write patches and send it to a mailing list for a thing that they're not actively involved in for fucking fun. This isn't fun, right? This is a job. And so if they're trying to accomplish something, the least a maintainer should be able to do is understand why and if it's bonkers, fine, it's bonkers, but understand the use case and provide some sort of way forward. I think part of your responsibility as a maintainer too is to offer timely review, like you're saying. The last thing you want to do is churn out a dozen sets of patches only to find it rejection later. It's incumbent upon a maintainer to step in and at least have a look at what someone's trying to do and steer the direction. I think you're out. I think it's really important to consider that this needs to be a team effort because if you make it be the statement that the maintainer is responsible for a timely review, you're just gonna burn out maintainers, right? I cannot review every single patch that goes across the list. I am very, very dependent on other senior members of the community stepping in and doing the reviews. So for example, in EECS-D4, Yon Cara has just done amazing works in reviewing other patches. And so I think we can talk about maintainers trying to help recruit people to do that review, but I think we need to be a little bit careful about demanding SLA agreements that all patches must be reviewed within a week or responded to within three days, right? Now, if I'm at work and I have a P1 bug, I have SLA requirements. And I think we need to also be understanding of the fact that very few maintainers actually are privileged enough to do it as a full-time job or even are allowed by their employer, 50% of their time to do that work. And so, yes, long-term, the maintainer should be trying to recruit more people into the community so that they're not overloaded. But I think this is why I think one of the reasons why it's been so hard to make categorical statements that are accepted of the form, it's the maintainer's responsibility to do X, it's the maintainer's responsibility to do Y, because the reality is, if you try to pile that all on the maintainer, you're just gonna burn them out. It's just not possible. And by the way, one of the things I actually choose to do is if I see a newcomer, I may actually spend more time with them, but that burns a huge amount of time, right? There are certain companies that bring, ask their new engineers to fix syspot bugs, and the amount of time I need to spend trying to bring them up to speed is a huge time suck. And what I generally tend to do is I will make judgments about whether this person is someone who will become a long-term part of the community. And after a couple of tries, I may give up on the person, right? Or just, you know, it's like, okay, this is a drive-by person, you know, this is, yeah, yeah. I think one of the, there's code review and then there's code review. What people are asking for is early feedback, but I see a lot of maintainers or code reviewers who take the attitude that all code review has to be super detailed, check every T and cross every I code review. And if you're not getting any feedback until you get that, then that is gonna end up waiting a week, two weeks, because that is mentally intensive and demanding. And then that's what gets people frustrated if they don't get anything before that. One of the things I do in BcashFS is I try to make myself available to everyone to at least discuss patches and ideas before people are working on them. And as soon as people have code that they can throw up and get, I will glance at it and say, does this, is this the right approach or not? Before we do the detailed code review and before it goes through CI. Taking a more conversational approach, I feel it always helps. I think that's a good point. Oftentimes it feels like we do, there are some patch sets where you really need to do deep code review because it's just so complicated. It takes a very long time, but I think that just doesn't scale. Like you don't, you won't have a bunch of, 10, 20 experts that can review all the code in sufficient detail. So I think the consequences that sometimes we need to be fine with accepting code that isn't perfect. Yeah, it's like, you will hear the bell. It's, it's real fun, it's very odd. One thing I'll say too is I think it's, the Linux community has evolved over time and perhaps it's a bit more difficult for folks who have been in companies where maintainers have been something that's, they've been doing this for a long time, but perhaps now as new companies are starting to hire maintainers, it's important for companies to realize that this is becoming a challenge and also that they need to consider this as part of that role when they are hiring maintainers. So I will say for instance, I think that Samsung does give me good leeway to understand that this is part of my job, you know. I will do my best to try to take time to review patches very carefully from new maintainers as you were saying. I mean, sorry, new contributors, but perhaps it's much more difficult in companies where this is already an instant student thing where they already have maintainers, but if new employers start thinking about these things and come to these conferences, they might get it. So it's important that when companies hiring new maintainers, they consider that into their workflow for the future. And I think this is, you know, this is a miscommunication on my part, right? Like so clearly like Sturba is the maintainer of ButterFS, but Sturba is not the one that does all the reviewing, right? The entire community is reviewing. And Sturba and I and Chris and the other core developers, we collaboratively decide what's going to happen like on big changes and that sort of thing. So I like, I keep saying maintainer and that's a misnomer, right? Clearly I'm not expecting one guy to do all of this stuff. It's a community. And the other thing is like, there's a big difference between Ted, me, Christian, and like random one-off things, right? So like obviously the load for smaller projects is gonna be different. But I think like we're all talking about big subsystems that touch a lot of things. And this is a problem for the bigger subsystems. Now I think that most of the people in this room don't have that problem generally, because I mean, you know, Jens is really good with Blocklayer. I feel like ButterFS is pretty good. EXT4 is pretty good. Like I think we've have a, most of the people in this room have maintainers that they flow to that are better about this, but we're not the only part, right? I feel we've, what would be nice for example, if for a start we would see more act and reviewed buys on patches. Like if people see a patch and think, okay, this looks same to me and I have a rough understanding what this is about, it's fine to act. But I feel there's a certain restraint in doing this because people are afraid that it might look bad if later on someone comes in, reviews it and finds a bug report. And, but that's totally fine I think. You know, it's totally fine for someone to later to show up and it's like, oh, here's something that you missed, that doesn't mean like you didn't do your job as a reviewer, you might have missed something. That's fine in my opinion. But I feel like that's a lot why we don't see acts and reviewed buys on patches. And that's annoying. Yeah, but I think we're all managing our reputations to some degree and that makes people hesitant. I do agree, violently agree with you about maintainers need to engage. But there's definitely a role to play for contributors giving maintainers grace. But also maintainers giving them self grace. Like one of the kind of interesting conversations we had in the tab, we just happened to start talking about maintainer issues. And I think it was, Jakob said like, why don't we talk more as maintainers between each other? It's kind of like we're all on our own fiefdoms. Like if I feel like I'm drowning in my substance, I can't really like raise the bat signal, say, hey other maintainers please come help me. Like I don't think we, I think there's more room for us to help each other. And another thing I would say that has seen it start to work really well is these subsystems that have regular phone calls where you can get on and say, hey, did you see my patch? Or hey, what do you think about this idea? Because sometimes, because the list is permanent, public and that reduces some communication. So part of this is like, hey, can we have other channels to get like, I think all experienced maintainers on the route where people developers, they have their back channels. They have their way to be like, hey, but like contributors are coming and they don't have that network. So I think, yeah, I think there's things we can do here. I think, well, I also wanna go back to, there's one thing about the newcomers, but I think there's another piece, which is the company developers who have put together a 10 patch series, they're actually trying to solve a real business reason. Business, there's a real business justification for that effort and they may not know when is the best time to approach the list. But the other thing which I think is really important is very often I find that I end up having to push them on why are you doing this, right? And that's actually a really, really good question to ask. People may have noticed I did that one or time here. It's like, what's the business case? Why are you doing this? Because if I understand the reason why, I can often give them a better way of solving the actual underlying problem as opposed to this mysterious 10 patch series that is making a whole bunch of changes without any explanation of what the high level goal is. Because at the end of the day, yeah, they're doing it because they have a real business need, they're not doing it for fun. But understanding what that reason is to the extent that they are willing to disclose it can very often help with that conversation. So that's a tip to maintainers. Feel free to ask why are you doing this thing? And this may be the other part of, if we write these documents with the tab about giving suggestions for people who are trying to approach an open source community, tell them, give advice to those companies, tell them what it is that they're trying to achieve. And very often, if those companies can also give an assurance that this is not a drive-by patch, they will actually stick around, then it's a lot easier for the community to work with them as opposed to, you know, hi, I'm a company, I wanna solve this business problem. I'm going to drive by a set of patches and then I will never be seen from again. And again, making that on the community because we have to support this work because we are maintainers, right? I think there really is a two-way street here and setting expectations in both directions is probably a really good idea. The question I'd like to put to the room is do you think we don't encourage reviews enough? Because I remember back 10 years ago, we used to have huge problems with people who reviewed code wrongly and they'd give the wrong advice and we spent ages trying to climb down on this. But now the big problem is we don't have enough reviewers and is it because we've overcorrected from the previous problem of all of the bad reviewers? So perhaps one of the things we should be doing is actually trying to build up a stable of reviewers. I think you only actually get reviewers if you see someone, this is my experience, at least if you see someone who's doing good work, who's commented on patches a couple of times really well, you write them, maybe off-list. It's what I tend to do and tell them, it's really good what you're doing, keep it up. If you need any help, I can try and give you some advice and that's how you keep people. It's really crucial, I think, sometimes to just point out, hey, what you're doing is good, you're doing good. We're really good at saying, this is wrong, this is not great, but we rarely say stuff like, oh, this is a great patch, this is a good idea, this is a nifty change, whatever. If you tell this, I think encouraging people is usually the best way to get them involved into the community for a long time, in my experience. Maybe even adding those people to the maintainers, as official reviewers, would put a little bit more skin in the game for those people. Maybe you can have several layers of reviewers, like experienced reviewers and junior reviewers. Yeah, I think that a wholeheartedly agree, Christian, this is something that we've kind of worked on at Butterfest, like encouraging people, thanking people, because it makes them feel good and it makes them feel included, right? As for the reviewer thing, I think that's really important, right? For me, I'm the worst at this, right? I don't look at FSDevel unless I absolutely have to, right? I'm stuck inside Butterfest, but I know that I could review stuff if I had the time, and Willie sent me stuff specifically, like hey, Joseph, will you look at this? Yes, of course, I will look at it, right? But that's basically the only way that I will step outside of Butterfest, is like if I'm purposefully asked to go do it. Right, and that's also part of the maintainer's job, as Christian just said, is that to reach out to people that they think would be reasonable to review the thing. Personally, I find, this is a whole other topic, but I find part of the problem is just tooling, right? Like we only have so much bandwidth for stuff and I only have so much bandwidth to look at stuff. I'm gonna look at the Butterfest list, I'm not gonna look at anything else. If we had better ways to be like, okay, this is our outstanding patch sets, or these are things that, whatever, but I think that's a whole other. One question. So, we do have this R tag in the maintainer's file. What is the pathway for each of your subsystems to getting a new person than that? Do you even know what it is? I have no idea what it is. No? What? What is the question? What was the question? The question is, for each of your subsystems, we now in the maintainer's file have the R tag for reviewers. How does somebody get an R tag in your subsystem? What do they have to do? You know, so you were talking about. I reviewed, I was reviewing. So, do you ever go up to anybody and say, could I put your name in this, or do they have to ask as in? I know one person was reviewing FNOTify and he requested to be in the maintainer's reviewer and John merged that patch. Right, but perhaps we should be more proactive in tapping people on the shoulder and saying, hey, you've reviewed these patches three times. Can I add your name to the R list? Yeah, okay, you just said it. You basically just said it. I think one good way of dealing with this is if someone is around for longer amounts of time, is providing consistent review and is interested, then why couldn't you just go to them and ask like, hey, do you want to be added as a reviewer? Do you see yourself having time for this in the long run? That's part of it. It's basically the same thing as saying, thank you, you're doing good work. It's also a plus. It's really hard. You want some responsibility. And it's really hard to keep track on infrastructure changes within the kernel, especially the tooling changes because as you've said here, one has only so much bandwidth and most of that is eaten up by the actual work and following the maintenance which you care about. Linux kernel is not one of those. We tooling is definitely a part of this. If you're dealing with larger amounts of patch series, then keeping track of them in some way is really crucial. And for example, we lack infrastructure for this pretty much. So it's easy to just get lost in the amounts of patches even with all of the new tooling that we have. I just wanted to come from the other side as someone who recently became a reviewer. To say it was a very encouraging to take that step and I did end up asking for it. But you really feel like a part of the subsystem a lot more once you're here. But it still requires you to ask for it. Yeah. It would be nice to have a clarity on that. You first have to have the guts to actually ask it. Yeah. We should definitely be more proactive about this. One thing I wanted to raise as well is that sometimes with the patch series that kind of falls down the gaps. You put it forward and no one responds to it. It can be hard to know the polite way of pushing that forward. And then I guess that comes into the whole humans talking to each other thing. But it'd be good if there's like clarity on, I mean obviously maintainers you can't have an SLA on that but how to push it forward politely or the procedure would be pretty helpful. Yeah. I think this is a good thing for the tab to take up because it's like, I think that for me I like I maintain stupid shit like NBD. And so like I just forget about it sometimes and then I come back to like guy just saying ping and for me that's completely fine. Like I'm an idiot. I let it go. I'll review it and push it along. But like other people I would probably be really upset about that. So like having some sort of like way to encourage. Yeah. I don't want to talk. Maybe the exact thing about the maintainer guy. Yeah. We do actually have in our kernel documentation. We do have kernel documentation by the way. They maintain our profile section where you can actually put in the description of your subsystem including things like expectations for patch response times and that sort of stuff. Very few subsystems have done this. But it would be I think very helpful if more of them did. Yeah. Absolutely. Yeah. One of the expectations when we started this was like herding cast trying to get maintainers to agree. So we said, okay, we're not going to try to get people to agree. Everybody kind of write their own document. But now we do have two good documents. We have X86 and NetDev. And we're like, hey, maybe it's time to start up leveling to have the common set. But yeah, but that's the place to put your expectations about yeah, ping me every 48 hours or no, ping me once a week. Like that's where that information can go. Okay. All right, we're running over guys. Appreciate it.