 So thank you all for attending this buff session called Bajakana Maintainers. I'm going to make a quick announcement first. There's going to be no slides for this talk. So I feel very ashamed not to have submitted my slides by the Friday deadline, so I decided to go with those slides in the end. So to avoid troubling you with unishing visual, that's going to be the background we'll have. I can do that. Absolutely. OK. So I'd like to set the goals for this session. The first goal is that I want you to speak and not listen to me. So we'll see how that works. If I'm the one doing the talking, it's going to be kind of a failure, even though I've prepared a few topics that I can use to start a conversation. This is about reporting problems. You might probably have experience when doing Linux kind of development and trying to get batches upstream or interacted with the Linux kind of community. This is not about calling people's names, so there's no need unless you want to state the name of a particle and maintainer that you would like to bash. I'm more interested in having feedback of what doesn't work, of the problems you have encountered, and discussing possible solutions. So hopefully the outcome of this session will be a summary of all kind of problems you will have encountered. Discussions, solutions we will have discussed, but it's really about a problem summary. I plan to then make a summary of that that I will try to anonymize as best as possible. So please feel free to speak out. I want to quote your name. This doesn't mean that you're anonymous in this room. You might notice there are a few people around you, so if you decide to speak out, if you see, I can't guarantee that nobody will remember when to step out of the door. But it's not going to be recorded by me. It could be recorded on video, though. OK. I'm also going to make a readout of the session at the kernel maintainer summits later this week. And again, I won't quote any name during that presentation. Right. The format for this, what I would like to do is to start with discussing problems first, because I know that a lot of you have a lot of ideas on how we can improve the process. But I would like the possible solutions that we discussed to be based on actual problems that people experience. So if you have a problem, please report it. If you think of solutions for your problem, or if you think for a solution for a problem that someone has reported, let's discuss that. But if you have a solution for something that, well, you think might be a problem, but that you haven't experienced yourself or don't know anyone who have experienced that, that's a bit of low interest in the context of this session. So I have a second microphone here that hopefully should work. I might need to push a button somewhere. Yeah, that works. So does anyone want to start? Yes. Thank you. There is a bit of an opaque world as to what the expectations are for various parts of the kernel and various maintainers. Each maintainer has their own way of accepting or rejecting patches and notifying people of that. What their time frame is, some maintainers, you have to comment on patches that are there going to them within a week or two, or you have no chance to reject them or say they should be different. Other maintainers you could expect a month or two before the patch actually goes in. So it's more a matter of documentation of what these practices are by area or by maintainer. Right, that's actually something I had in the notes I prepared in case nobody had any questions, wanted to start. So yeah, the first part that you mentioned is the time issue, what's the deadline? It goes both ways. How long do you have if there's a patch that's being sent to a mailing list and you want to review that? My issue isn't the timing, my issue is knowing what the timing is for each maintainer. Yes, okay. Yes, indeed the time differs between different maintainers and different subsystems. I think that's pretty much a given. I don't really think we'll have a big influence on that because we have a very diverse style of maintainers and maintenance ship. We have people who pay to work full time on maintaining a subsystem, we have people doing that as a hobby. I think we should strive to improve the process, to improve the community, to possibly move that's another topic that we might mention to a group maintenance ship for instance, to try to make things faster and smoother. But I don't think that setting indeed a timeline or deadline that works for everybody will be my thing. But documenting that, yes, that's something we can do. So we can't really enforce the amount of time, but documenting it is probably something that's useful, yes. Is there anyone in this room who is a new kernel contributor who has submitted just a few patches from time to time and who was unsure whether the patches were correctly received, if it was lack of feedback and not knowing when to ping the maintainer, whether it was appropriate to do that. Yeah, I see your hand over there. Okay. Yeah, thank you. I need to stand up. Actually, I'm really not a big submitter. I probably have six or seven patch in that. Perfect, perfect target for this talk. And you know, when you get it right, you know when you do little patches very often in one liner, two liner, a mathematical function that you just change. So most of the time those patch are right actually and it's a black hole, nobody says anything. And you actually learn in the next release six months later or three months later, depending where you are, that they got merged, but yeah, there's no feedback. Okay. So on that topic, documentation is indeed one of them for the delays, but that doesn't really solve the black hole problem. Doesn't really solve the fact that you don't know whether your patch has been accepted or not. And I believe, I mean, I've been doing kernel develop for quite a long time, so I have absolutely no issue going to maintainer and say what's going on. But I assume that's a bit more difficult for newcomers. And that's one issue that I also have identified indeed. So documentation, one of them. And how could we improve this black hole problem? Anyone has suffered from the same issue? Yes, right over there. Do you want the microphone? Well, I have a sort of similar problem with submitting patches when you're doing work for a client and you say, okay, I've got a few patches for just general fixes that the client is never gonna complain about, so you send them off. They don't get looked at or they get looked at late and by the time they are even reviewed, you've moved on to a different piece of work and you don't have time or you don't have the hardware to go back and look at these. I think that's another problem, is there some way of getting somebody else who could look at that or? Yeah, I think that largely depends on what kind of patches you're sending, what subsystems are possibly as well. If you're sending few cleanups or general patches for a very specific driver that has very few developers or users, there's a high chance indeed that if you don't do the work, nobody's gonna pick it up. If you send patches for code of a subsystem, there could be a higher chance that someone could say, okay, this patch has been sent, it hasn't been picked up, it has been reviewed, changes were requested, but the submitter lost interest or had to move to something else that doesn't have time. So in some cases, we could probably do something about that, but it requires actually people to go over the meeting list, basically using tools like patchwork or other ones, to pick those patches up and try to get them upstream. That could be an interesting junior job for, like we have lots of newcomers to kernel development, going through kernel newbies, we have outreachy people starting working on a kernel and there's been lots of complaints before of maintainers who are not happy to receive check patch style patches out of the outreachy program. So, yes? We're not supposed to be talking about solutions first, okay, so you are kind of doing what you said. No, no, no, I said I want to talk about a solution if there's no problem first. Yeah, okay, but just to finish that, and then I'll move to Hans who got a microphone, it could be an option, for instance, so probably outreachy instead of running check patch to go to meeting list and try to see what patches have been sent, got reviewed, but had never got a V2 and tried to get a V2 of that, okay? But I think that one thing I heard is that they might not even have hardware by the time, so that would be, even if somebody picks it up. No, absolutely, I totally agree with that. If there's no hardware availability, you can't test that, but I don't think that all patches would fall in that category. I'm sure that some of them would apply to cursive subsystem code, for instance. I think the important part, if you have post patches and you know you are losing access to your hardware in the next month, tell us, we don't know, right? But if you tell me, if you send me an email, hey, what about these patches? I can, now I still have the hardware in the months I don't have it, then I will prioritize them. Otherwise, they go on the big pile and every so often you go through them and you process all the patches. And because if I don't know that you lose access to the hardware, I can't prioritize. So it's, we are not telepathic, luckily, probably. So if you don't tell us, we don't know. We are not, generally I think if you post a patch and you don't hear anything about it, ideally if you have patchwork, that is great because especially if you can delegate patches to someone and if you see in patchwork that your patch is delegated, then you know someone will look at it. And if it's in patchwork, you have a good chance that it won't be forgotten. I really like the tool. But you need to tell us if there's something special about it and just send an email. What's the worst that can happen that we don't reply? That only works for the media patches because for everybody else, nobody goes back to old patches and applies them. That's the only subsystem that I've had that I've sent patches to. Didn't hear anything for a year and then they got applied. Either I hear something within a few months or they are dead and I have to resend them. Media is the one subsystem where people actually go back to ancient patches and then suddenly they're in. Well, the key thing I heard from Hans is I agree with that communication. Sometimes what happens in my case is I maintain case health tests and then I have a bunch of patches come in for tests and sometimes I wait for the test author to review. So I sometimes lose track of them, honestly. And then I really appreciate that somebody pings me and I get back to them. So communication, just don't hesitate saying politely pinging and in saying what's happening. But not every week. Otherwise, other people will get irritated with you. And that goes back to documenting the time scale. Some maintainers, Mark, you mentioned that you don't want to be pinging every week. Some maintainers might want to be pinging more frequently than others so that's something that could be documented as well, yes. So I haven't seen this happen for a long time, but years ago I would maintain a certain amount of drivers directly. I'd be doing refactoring on them. And then someone submits a cleanup patch to LKML and it breaks my series and we kind of submit at the same time. But for some reason, since it got all this visibility in LKML, it gets priority. Again, it hasn't happened in a long time, but it's very frustrating. That's all. Okay, trying to write it down as well. So any other problem you might have encountered? Yes. Just can share my experience. Send patches, wait three months. Pink. Go to angry email, wait one year. Pink. And then, yes, it works. Yes, that's quite long. Well, way too long obviously. From my personal experience, I'm not a subsystem maintainer, but I definitely maintain drivers. And things fall through the cracks, depending how busy I am in the given period of time. And just on a side note, you can bash me as well. Maybe that's what you were doing. Just to not say the name. So, yes. As Shua mentioned, I appreciate it being pinked. Obviously, if you do that every day, I'm going to reply politely that, okay, please wait one week. But if I haven't reviewed a patch after one month, there's a chance that it did fall through the cracks and that you will not go back to it. Because it gets buried in my inbox and that should not happen in the first place. That's something that I should try to fix. But you should not hesitate pinging, I believe, at least in my opinion. There's nothing violent in pinging a maintainer or pinging a reviewer because a patch hasn't gotten any feedback. Obviously, it depends on the tone of your email. But as long as people remain polite, I don't think that's a problem at all. I definitely would say the same thing for Armstrong. We are sometimes really late in picking up patches. And if I get an email saying, well, it's been two weeks, why haven't you applied any of the patches? That would actually be... That might actually be enough to get me to do all the work and not wait another week. Although if it has been a month or something, then maybe just resending rather than just pinging can work better. Because there is a good chance that the patch has actually gone completely able. So if you do get a reply, it will be, I don't know, please send me the patch again. I don't know what this is. And again, that's different per maintainer. So for us, pinging would be better than resending. We are doing political... It's also time-dependent. It's a situation thing. But consider resending, not just pinging. So if there's been a merge window in between, please do resend. Yeah, if it's a really long time, obviously, if it has been three, four, five kind of releases, the patch might not apply anymore. So if that's the case, it's, I think, better to do to resend it. So moving on to a somewhat different topic here. Oh, thank you. So we've been talking mainly about patches being forgotten. But there are other cases where you submit patches solving a specific problem that you have. The maintainer is not really happy with the solution you're proposing, but does not make any other proposal that you can use to rework your patches or make a better proposal. And things stay as they are. You still have your problem, nothing gets merged. And that can stay this way for months without the thing moving forward. The maintainer doesn't make up his mind on how to solve that problem. The problem is still there. Nobody has came up with a better solution, so it stayed there forever. I think that's actually a very common problem. I've seen that myself being on actually both sides. Personally, when I review patches and see something I don't like, I try to at least give a direction that could be experimented. I don't know if it happens to anyone in this room that you get in a patch series and you review that and say, okay, that's not right. But I'm not really sure why, but I really feel it's not right. Okay, good, it's not just me. That feeling from a maintainer is fine, but what you expect as a contributor is some kind of, okay, that's the direction. There are some maintainers that don't basically drive their subsystem. They're just taking patches, but as soon as there is a problem that requires more reflection, more thinking, well, there's nobody that really drives the subsystem and tells, okay, that's the way I want to follow to solve that specific problem, and so the problem remains open forever. Right. And some of the problems are really hard. As a maintainer, you don't know the solution. As a contributor, you don't know either. You really have to get in the room possibly with three other people and find a solution, and that's fundamentally a hard thing. The best solution for these things is come to a conference and schedule a meeting. But not all of those are hard. I mean, sometimes it's really just a maintainer being lazy and hoping the person goes away and does something else. Yeah, and I hope that the outcome of this session will be that we'll try to keep people in the development community. So if they are lazy, they will just take the patches and apply them, wouldn't they? That's too much work. Of course, it's always easier to apply a patch than to reject it, but if you know it's wrong, then you know it's going to come back to you. Yeah, exactly. You're always going to come back to you and that is going to be much... lots of work later. Trying to get it in staging is an option sometimes. So if the problem is... depending on the problem, right, if that is like a very isolated one where there's only one person complaining, it may not appear as an important thing, right? But then if... so my approach would be to just make it more apparent, like, you know, find somebody else with a problem and so on, and then try to appear... try to make it appear as a common one. Yep. I'm sure you can understand that it's frustrating if you have an actual problem. It takes months to build a community that has the same problem and basically fight against the maintainer to prove that the problem really exists on multiple platforms or multiple situations. That can be very frustrating for some type of problems that are not that complicated. Not that complicated. It's going to be a bit of a gray area sometimes, but yes, we still don't have detail overlays or request API. I don't want to depress people. Okay? Any other topic? Any other problem? I can't believe that... Yeah, I just... did you have something compared to what Thomas was saying? So one of the things is when the maintainer doesn't give a clear view of how we are supposed to fix things but also in my developer hat this time, one of the things that really frustrates me is that when you have quite simple solution that looks reasonable but maybe not to the maintainer, which is like totally fine. And then the actual solution is several order of magnitude more complicated and involve much more time. And so basically in this case, most of the time I'm just like giving up and not fixing actually the problem because I'm just not going to do it. Right. So yeah, that's one of the issues as well, I guess. Yeah, I've seen Danny a few times before where you submit a small fix to something and then you get told, yeah, but can't you fix the whole world? Yeah, I've done that. Yes. People have sent me very simple patches but just wrong. Then we're trying to figure out how to do it right and ended up getting bigger and bigger and bigger and ending up the whole subsystem. In the end, sometimes it's actually better to have merged that heck in the first place. That's a lazy approach. And of course, sometimes all it takes is for the person to go, no, actually I really don't have time and you'll apply the patch anyway. Yeah, possibly having a middle ground solution where you can say, okay, the best solution would be that you could come up with the partial solution that's a bit less hacky than the first one. Yeah. So I just want to assume it on this as well. If I send a patch and then some of the core developers in the subsystem say that, oh yeah, I see what you did there but you should implement it in this way and then you send your v2 and at that point another core developer from the same subsystem steps in and say, yeah, I see what you did there but you should do it the way you did it the first time and then you just have to iterate it all over again. I see that recently, yes, rings a bell, thank you. I also have seen those before and it gets worse when you have five maintenance coming up with seven solutions. Yeah. So how could we improve that? Ricardo? What? Yeah, so sometimes you have to send a patch that is going to affect some area that is not of your expertise or for example, you need to change, you want to make an IOC deal that is one architecture, that IOC deal has to be in all the architectures and you cannot cross compile for all those architectures easily in your computer or there are some tests that you cannot run in your computer because you don't have the static analyzer that, you don't have appropriate static analyzers that the one that runs a kernel every week. So yeah, it will be nice to have something that automatically checks, that you will entirely send to like a demon or like a bot that will check your patch against all the cosy nail, against all the static analyzers, against all the architectures before you send it to the maintainers just to make sure they are not going to call you stupid when you send it because it's not stupid. Well, they should never call you stupid in the first place. They can certainly say the patch that you have sent is not right for these reasons but they should never call you stupid. Actually, recently, that may not be known to everybody but there is a zero day service that actually, if you post the patch to say the LKML, it will just pick it up, apply it, compile it and complain if it doesn't and send a fix to you. So yeah, and actually, it runs an array of static analyzers and things and so on and even sometimes boot tries to boot it on some hardware, so yeah. So you want to send it privately because you're scared of losing face if you send it publicly? But I think that's an attitude problem that we might have then among maintainers in the community because whatever you send to me, you should not lose face. Nobody should call you stupid and it's... I'm not saying it's wrong that you have a feeling in the first place but if you have that feeling in the first place then there's something that we need to fix. We don't want to have an abusive behavior as maintainers. Yes? Oh, I was just going to add to what Rafael said. Sometimes I don't even... Even if I don't send... Zero days is very aggressive in some ways. I just upload it to a... to one of my GITs that I use and it just pulls it in. So I haven't even sent it to the mailing list. Which is good actually, I kind of like that. I sometimes do that. To catch stuff. So you're absolutely right that we should be building a positive internal community and you should never be tolerated but there's something to be said for making maintainers' lives easier for the simple checking of the examples of crops compiling. So I think maybe it is sufficient to have just a test at LKML or something where you can just send and say, send your patches here they'll churn through the CI and nobody ever has to look at them and you'll eventually either get back a report or nothing. So it's not even the fact about saving face. I'd probably say it's more making sure the maintainer doesn't actually start to do the review until it's done the bare minimum of stuff. So that could be an example. It's basically low on your traffic in the main English so that's humans who have less CPU time than the computers wouldn't be bothered with that. The only issue I have with that is that if you send a patch like that to a bot and get a report saying that there's lots of front of view patch newcomers might say, okay I can't really fix that myself. I won't bother. It's too hard. While if they send it publicly they could be given a review and even if they decide to give up they could be someone picking up the patch or picking up the fix if it's an interesting one. So having it in the open in first place also has some value. Now what the balance between the two is I can't really tell but that's my opinion at least. Two things. There are some maintainers very high up who are certainly not nice to newcomers and we all know that. Yes we do. Secondly, for the sending patches for automated testing we have a tag like RFC, RFT having one which just means I'm not sure of this patch please automatically test it I'll always look at it. It might be worthwhile. Yeah that's a good idea. It would be public, it would be stored in mailing these archives but maintainers could definitely skip over that. But if you send the patch to this automatic tester and when you get the result you don't know how to fix it you can send it to Kernel News instead of sending it to LKLAM and it's your decision. You can do that, yes but my point was maybe I'm worrying about something that is not proud in practice because we don't have that infrastructure in the first place. Over here. I was just going to say that that automated bot system might actually encourage newcomers so it might give them way to have an in. That might be worthwhile. Yes. It would lower the barrier to entry people might be less care to contribute so that's a good point as well. Whether we should be doing more to call out aggressive behavior like sometimes I notice a developer or maintainer replying to a patch in an inappropriate way but then I also don't feel like I should step up because then I know that person is going to come after me. Yeah, but I think it's our responsibility collectively to do that. Everybody is responsible for the whole community in this case because nobody does it. I think that if you call out that kind of behavior and the person comes after you you'll get lots of support. That's probably true. I don't expect newcomers to do that but certainly developers can And that can not only be extended to replies to patches but sometimes commit messages fixing bugs also contain pretty aggressive language like how could have been done so stupidly and I think also this is very that's not appropriate. Some people might be smaller and some people who did it before might be in a hurry or whatever but they probably did not do this intentionally bad. That's a good point. As a relatively new contributor I can tell that even if you don't have a face initially you always are afraid of losing that. I'm sending something that is stupid and sometimes you refrain from doing that. So I don't know if there's something that it's a problem that is general or personal or widespread but I think that something that helps like this is my first patch I will look at that like a tag or something and or run then through automatic test infrastructure or something could help people getting more confident in sending patches probably. Well to answer your first question there who here wasn't scared when sending the first patch to the kernel? So it will be general. I wasn't really scared but the patches were really really bad. But yeah I think that the problem of confidence for newcomers is an important one. So a comment on sending a first patch so if you tag it as RFC which means or stands for request for comments then yeah then probably so first of all maintainers are not very likely to look at that and second you you might actually get some more like warm reception for that because people will treat it as experimental code essentially and experimental code might be bad or something nobody would complain. It's expected to be bad. It's expected that way. Our model is based on sending patches to mailing lists and we have jobs and we work with GitHub and we send pull requests and there's CI and gives us a green check mark. As a maintainer I wish that my job reviewing patches coming in could be as easy as looking at a graphical UI seeing the check mark and then reviewing the code. I wish. I don't know that that's the best thing but it's something I miss when I go back to Linux open source work after I leave and it's not all day long. It's great having that CI and we just basically newcomers they don't feel comfortable sending pull requests and I don't really think any newcomers do and it can actually make the maintainers work easier. The maintainers would probably not really accept that in most cases. Many won't but I'm talking about the trend. We haven't moved forward. It's been patches on a mailing list for the past many years. We haven't discussed quite extensively actually the past year a couple of years. When my personal experience with UIs for that is that you'll have a graphical UI with a button that you can press after you finish the review and most of the time you'll press the button without reviewing it. Don't get me started on Garrett for instance. Sorry? Yeah. We can probably improve tooling and change like that. Well, that would be very disruptive and I'm not sure what the outcome would be whether it would be a better place or not but that's something that's open. Yeah. Zeroed it to give you some level of CI and there's been lots of efforts recently to improve CI in the kernel in general and people at Billy Bray for instance have been working on understanding kernel CI to do more than boot testing and I think that's a trend in the Linux kernel community at a moment that we're really trying to improve the test infrastructures to improve CI and to improve the quality in the end of the product. Yeah. Where's the microphone? Okay. Thank you. Just first to go back to what was discussed about people sending things and not being sure it's correct somehow. I don't even like to get a nasty message from the zero day bot. I mean, it's like, you forgot to compile your... I mean, even if it's not human it's still sort of a reproach. Yes, if it says have you forgotten to compile that... No, I mean, it's not the phrasing. It's just the fact that the message comes. Okay. One might prefer to have control over when this detection happens and so on. It's like once you've sent it off somehow you have justified and then getting it back is a bit unnerving. Do you think that for newcomers it would be better to have a human who can use a softer tone to guide people instead of having something automated which by definition is going to be cold? I'm not sending into the mailing list to the subsystem maintenance as well. Yeah, so I don't know if that was clear but there was a suggestion that maybe the message would only go to the contributor and not to everyone and now reduce embarrassment. Oh, right. So that the zero day report would be private and not go to the mailing list, for instance. So actually my question though was about cross-tree commits Yes. And I think it's very unclear where they should go and who's going to pick them up and how they should be cut up and so on. And I think maybe this discourages sending such changes. So maybe we could have somehow overall better code quality if this process were more clear. Yes. How are we doing from a time point of view by the way? When do I have to stop? Sorry? Sorry? Okay, final clock. Okay, then, well that's it. Yeah. I hate you. No, I see people are queuing already for the next session. So thank you very much for attending. Thank you for the feedback. You've been great.