 So, we'll go ahead and get started. As you may have guessed from the schedule or from my intro slide, I'm Frank Rowand. I worked for Sony. I've worked for Sony around 13 or 14 years now and quite a few other embedded links companies before that and proprietary companies. I have several roles in Sony and one of those roles led to a real-life story which I'm going to talk about today, the impetus for these slides. I work with different groups inside Sony who are learning how to work with open source and how to interact, how to submit code, things like that. And I was working with a team that was creating a brand new driver for an ASIC, a TV tuner. And we gave them some advice on how to get their patches in the kernel, reviewed their patches and helped them out a bit. And so they send in your typical patch series. And as usual, the last patch in the series included some housekeeping things, things like make files, but most importantly, adding an entry into the maintainer's file. So here we find that we've added a Sony engineer for a set of files that encompass this driver. So just by getting this patch series accepted, magic, this engineer is now a kernel maintainer. And you don't really think about that so often. If you're just putting in a new driver and just adding your name to a few files in the maintainer's file, but you really are a maintainer at this point. So what happens next? You wait a few months, maybe four or five months. And an email appears in my inbox from this development team. And if you look at the bottom of this email, someone has sent them a patch. So this development team is forwarding me this patch and saying, what the heck am I supposed to do with this? Should I be replying to them that this is a confirmed fix? Or should I give them some comments? What should I do? So this exposed a gap in our internal training. It turns out that we taught people how to submit a driver, but we never talked to them about what it was gonna be like to be a maintainer. It was an area that was totally unexplored. So that leaves the question open of what role do they have in any patches that later get submitted to their software that they put into the tree? And there are a couple of plausible paths that patches and fixes can go through. Mostly I'll be talking about the, well, I'll be talking halfway about the likely path and also possible path. There are two different directions here. The easiest way is when someone sends a fix, the developer sends a fix up to the typical subsystem maintainer totally bypassing you, the driver maintainer. And the subsystem maintainer worries about getting the patch up to cornell.org being Linus and exposure in the Linux next tree for testing. Another possible path that you could choose to follow is that a fix would go directly to the driver maintainer, the person who created the driver. And that person would either interact with the subsystem maintainer or directly with Linus and Linux next. And about half my talk is going to be about those possible interactions. So you should work with your subsystem maintainer and make sure that you're on a common ground about what role you're going to be taking just so that you're on the same page. So there are five different possible roles that you can take. And I'll talk about these each in detail. But the first is just to totally ignore your role. By the way, my slides are already upstream on the sked.org page and the enix.org wiki if you want to copy the slides. And if you have any questions, interrupt me at any point and I'll answer questions along the way. So the second role you can take is you can provide advice to your upstream maintainer, advise them on the patches as they come in. The third is that you can directly review patches, explicitly review them. The fourth is you can actually become a sub-maintenor. And that was that less likely path I was showing where patches would flow through you. And there are probably other roles you can take off that I didn't even think about. But I think these are the main ways you can get involved. So the first way you can ignore your role totally. The amount of work you have to do for this is absolutely no work. This is the easiest path to take. But there's a consequence to this. The next time that you have a feature or a driver that you want to submit, if the maintainer is aware of you taking this do nothing role, they're going to be less likely to accept your new feature. When you send a new feature and the maintainer wants you to be there to help maintain it over time. If you want to follow this path, please, please, please send a patch up and remove your name from the maintainer's file. That way everyone's clear on what the story is. A slightly more involved process is you can provide advice to the upstream maintainer. When you see a patch email on the mail list or if it's directed to you, which it should be because you're in the maintainer's file, you can add an act by tag. This is somewhat a controversial statement on my part. If you look at the actual definition of an act by tag, it also implies that you've reviewed the patch. Actual practice in reality varies as to whether you've actually reviewed it when you send an act to tag. My perception, which we'll get debated by other maintainers, is that when you add your act by tag, you're telling the person who the patch is going to flow through that you're okay with them accepting that patch. So that person may be your subsystem maintainer. It may be an entirely different subsystem if a patch series covers multiple subsystems. And that's really commonly where act by tags get used when multiple maintainers are trying to coordinate with each other. If you do see problems with the patch, you should at least reply to them and mention. And I'll get to the next role, which is actually officially reviewing. This is kind of an informal, you're not claiming to be a reviewer. You're just trying to provide helpful advice at this point. But the minimum you should be reading and analyzing the patch, you should test the patch. And this is really important. You might be the only person in the community who has the actual hardware that this patch can be tested on. Especially if it's very expensive hardware or very niche hardware. There's a way you can get around that. You can provide your hardware to your sub maintainer. And that's highly recommended for many reasons. And that will allow other people to test the patches. You can also make your test lab available on the network so that people could test your hardware via the network. So the amount of work for the first option of doing nothing was no work. This step, taking on a slightly larger role, it's still a small amount of work. And the important thing is that you're putting a burden on your upstream maintainer. One thing that you want to do is make life easier for your maintainer. That makes your maintainers happy. And Greg Cage gives lots of talks about trying to make your maintainer happy. So just go reference his stuff. A slightly more expanded role, you can officially start reviewing the patches. This means that you do everything you did at the lower level plus, you now explicitly review every patch that comes to you. And comes to the mail list. You should be providing good feedback. And this is kind of the art of being a maintainer. You need to let people know what is good about the patch. You need to let them know what's not quite good enough and what needs to be changed. And you can consider providing them suggestions on how to change it. Sometimes it's enough simply to say this has to change and it's obvious and they'll figure it out. Sometimes you need to be a little bit more involved. And when something isn't good enough and not acceptable, you have to be explicit about that. And there are a lot of talks where people will tell you one of the most important roles of a maintainer is saying no. And it's also one of the most difficult roles to do it effectively in and in a positive way. Once you've reviewed the patch, you then need to add your reviewed by tag, which is simply a reply email. And this is a little bit more work than the previous level of maintainership. The amount of work really varies based on your subsystem and based on your actual patch, based on how active the fixes are. So it's hard to predict how much work this will actually be. Talk a little bit about reviewed by. This is taken straight from the kernel documentation. I'm not going to read this line by line, but when you do reviewed by tag, I suggest that you read this and really understand what you're saying is reviewed by. But to summarize it, point A is saying basically I've done a technical review. And is this patch appropriate and is it ready? Point B is saying, if I found problems, I communicated to the person who submitted the patch. And I am happy or satisfied with their response to my comments. Point C is basically saying, it's possible that this patch could be better, but it will make a positive contribution in its current form. And basically it's good enough and there aren't any critical issues. Point D is basically saying, I'm not guaranteeing the usefulness or validity of this patch at all. And then this all gets summarized at the bottom of that list in the documentation. Essentially saying, this tag is your opinion that this patch is an appropriate modification of the kernel, and there are not any remaining serious issues. There still might be minor issues. There still may be additional work that needs to get added in a future patch. But it's good enough to go in for right now. If you want to move up yet one more level, and most people probably will not move to this level, you can actually become a sub-maintainer. And this is going to require more conversations with your maintainer. And everything, all the other levels of maintainer are plus you help the maintainer out, essentially. You have some more responsibilities. The maintainer, depending on the relationship you have with them, they may want you to take on some of their specific tasks and help them out. They may like it when you start reviewing other drivers that are similar to yours. You're an expert in a certain technical area, things that are similar to your driver. So your knowledge and expertise can be very helpful in providing good reviews for other similar drivers. And again, the amount of work involved in this depends on your driver on the subsystem, what your maintainer wants you to do, what you want to do. And it's all negotiable. Some other tasks that you need to do is you need to think in a broader way of looking at the kernel as a whole and at your subsystem as a whole. You can't just look at your narrow use case. You need to be thinking about overall architecture. You need to make sure that that architecture is maintained when things get added. Architecture may evolve and change as a result of patches coming in, but you need to be aware that that change is happening if it is. You need to make sure that you're not just solving one small use case at the detriment to other use cases. Linux scales incredibly well from very small embedded devices to supercomputers. The top 100 supercomputer list contains mostly Linux. And it can be difficult to encompass all those potential use cases. You need to make sure that the patches as they get added don't hurt maintainability of the subsystem. It's always a balancing act. You want to be adding functionality, and sometimes adding that functionality is going to make it harder to maintain. And you'll see that if you look at the Linux kernel from 2000 and look at it today, the complexity has grown immensely. And there's some parts of the kernel that consequently are very, very difficult to modify, fix, add on to. So it's a balancing act for a maintainer, keeping that at all in balance. You need to make sure that the person submitting the patches explains clearly what their use case is or what their problem is. A lot of times you can help that person adjust their statement just a little bit, and it might solve a much broader problem than they're trying to fix. Or if you can help them look at it in a different way, they may realize there's maybe a different approach to solving the problem. You need to make sure that testing was done. You need to make sure that static checks were also done, which is another type of testing. Compatibility is important. Performance, some patches talk about improving performance or solving a performance problem. Some patches explicitly say, this is a really, really important feature, and it's going to hurt performance. But this patch is so critical that we should accept the performance hit. A really good example of that are the recent spectra meltdown, all those other patches, where we were taking a big performance hit in the kernel to fix those security issues. But it was felt that it was critical to solve those. And over time, they've been trying to reduce the performance hit from that specific set of patches. Another task that you have is you're there to help your developers. When they have questions, issues, concerns, you're a resource for them. I mentioned that you should be talking with your upstream maintainer. You need to make sure you're on the same page. And if you want to assume different levels of responsibility, there's a lot of freedom to move back and forth between the levels. There's no need to commit to a three-year role as a top-level sub-maintenor, where you're devoting 20 hours a week every week to it. Negotiate with your sub-maintenor. So to summarize all of that, all those different roles you could take, you have various levels that you can get involved in that flow of that patch coming from developers into the Linux kernel, from a very minor involvement to a very heavy involvement. It's really important to try to make your upstream maintainer's job easier. I say that selfishly as a maintainer. And again, your role is to help the contributors below you. And this is true of all maintainers. You're there to help the people above you. So people who are top-level maintainers still are there to make Linux's life easier. And they're always trying to help people at the bottom come into the kernel and start working on it and understand it better. So changing the tags here a little bit. This is kind of a strange jump, but going to tags. Tags and patch emails. It's an interesting way to provide some other insights just looking at tags. So what tags should you require that come in emails to you? And when you process a patch in an email, what tags will you be adding as a maintainer? Signed off, every patch that's sent to you or sent to your sub-maintainer, every patch that's going into Linux's tree must have a signed off by tag on it or it will not be accepted. If you take a patch email and then you forward that patch email to your sub-maintainer, that patch is flowing through you, then you would add your signed off tag. If you're simply letting the patch go to your sub-maintainer bypassing you, your sub-maintainer will add their signed off by tag. And this is one of the meanings behind signed off tag, paragraph C in the documentation, just simply saying, somebody gave me this contribution directly and they certified that they were allowed to submit it and that's what A, B, and C are. They're basically providing assurances that they have the right to submit this patch to you. So you're making sure that chain of custody of the patch is maintained as it comes into the kernel. This one is not a tag. This is a, it still fits in the patch comment area. It's not an email CC, it's in the patch comment. If you CC the stable people, it looks like an email CC. It's something that they'll notice when a commit gets accepted into the Linux kernel and that will be a clue to them that this is a patch that fixes something that's in older released kernels and maybe they should add this into the stable trees and then they'll make that determination. If they're ignoring your patch and you really think it goes in there, you can talk to Greg KH, he has other ways of getting patches into stable. Fix this tag. This one I strongly encourage. If you're fixing a bug, it has a lot of useful features. If you're a reviewer and you're trying to figure out whether a patch fixes a bug, having a pointer to the actual code that's broken really makes it a whole lot easier to analyze whether the fix is effective. If you're testing a fix, it's a whole lot easier to be able to go to the version of the kernel right before the commit that created the bug, verify that the problem doesn't show up in your test case, add that one commit, make sure that the problem does show up, continue forward, add the fix, make sure the bug is gone again through your test case. Next tag tested by, how much should patches be tested? There's not a black and white set of rules that every single patch goes through and there's a lot of variation between the subsystems and a lot of personal preferences by maintainers. So this is my personal take on it, things that I think should be mandatory. The first one almost everyone will agree with, your patch should have been checked against check patch. If it fails that, you didn't do, even the beginning of due diligence. The next, you should make sure that when you compile the kernel after adding your patches, no new compile warnings occur. Generally, the kernel day to day does not generate any compile warnings for most configurations. When I get a new major release, occasionally I'll see one or two new warnings and sometimes it's because of a specific configuration that I have that maybe no one else tested. But the community as a whole is very good about maintaining lack of warnings with no special diagnostics turns on. And I'll talk about a little bit stricter compile warnings on the next slide. This may sound really stupid, but if your patch is fixing a bug, test it and make sure that the bug really does go away after applying your patch. You may have to try a few different configurations to make sure it's not configuration specific that it fixes that it fixes a broad set of configurations. And that's just a judgment call of how do you figure out what all to test. Your subsystem may have a test suite. Some of the file systems specifically have suites that they expect you to run a device tree, which is what I'm a maintainer of. We have built-in tests. So just find out what your subsystem expects and follow that. If you are adding a new feature, make sure that the feature by the time you finish it actually matches what the documentation is. A lot of times your feature will evolve over time and it's really easy for the documentation to get out of sync with the actual implementation. And one that again sounds really stupid, but you'd be surprised. Make sure your kernel still boots after you apply the patch. And sometimes again, this might be a matter of trying a few different configurations, maybe trying a different architecture. Most people will not try building and booting a different architecture. Some people are really good about that and religious and they'll try three or four or five different architectures and they use virtual machines typically to do that. Or they may have a large board farm available. That's a really easy problem to hit if you don't test different architectures. Okay, mention that for compile warnings, you can check a little bit more stringently. And when you do a kernel build, you can specify w equals one, two, three or any combination of those. And each of those levels of warnings is a different set of warnings. For make w equals one in general, you should not create any new warnings. And kernel is a whole that's pretty well done. Once you get to w equals two or three, then you get a little bit more esoteric warnings. And sometimes they're justifiable. Sometimes you just can't avoid them. There are a lot of warnings that come out of macros in header files. So if that problem is already there in the header file, you're stuck unless you're fixing someone else's problem that's preexisting. It may be incentive that you may want to start going fixing header files. But you can always say, hey, not my problem. Someone created this problem ahead and it's already there. It's already hitting everybody. The other static tools, sparse and smatcher, I recommend them very highly. I would say, I don't have any justification for this, but I would say most people actually don't run these tools. If you run these tools and catch things, you will stand out as being a really good person, a really good developer. And again, I mentioned that if you're going to impact performance, you should be doing performance measurements and providing data for that. And that's such a wide open field. I can't even give a hint as to specifically what type of data in any given case. I mean, there's so many different types of performance data that matter in the kernel and different subsystems. The last tag is reviewed by, and I talked about that earlier, time expectations. How are you personally gonna get hit by this whole role? You're going to need to allocate some time in your personal work schedule. You'll want to make sure your manager knows that you can't just ignore the community for three weeks to the next window when your schedule opens up. By the time that three weeks goes by, someone else will have accepted the patch or whatever. Kind of a side note on that, if you're going to be on vacation or otherwise unavailable for a while, it's really nice if you can find someone else to help fill in your role. Especially if your driver is for hardware where not all the documentation might be publicly available. You might be able to share hints from your documentation to other developers as they hit specific cases. And so if you have someone else in your team who has access to those documents, that can be extremely helpful. And they also may understand the driver and be able to provide good review. For bonus points, if you want to be really, really a good maintainer, monitor or skim your subsystem mail list for a couple of reasons. One is your subsystem is changing over time. And what it looked like when you first developed your driver may not look like what it is today. So just keeping track of how the subsystem is evolving will help you as a reviewer. Also, people who submit patches are supposed to use getMaintenor, it should find your name in the maintainers file. You should get CC'd or have a two on the mail for a patch. But that doesn't always happen. I'd say about 5% of patches in my subsystem I don't get directly emailed. If you want to keep up with even broader trends, so that was subsystem, if you wanna look at what's going on in the entire kernel, there's one more email list you might want to consider subscribing to. And that's the case summit discuss mail list. And this is where the top level maintainers have all these theoretical esoteric and actually practical pragmatic discussions on how our processes works, how it's going to evolve. And interestingly, a lot of these processes that I've talked about today have been discussed over the last month or two on this list coming up into our next summit. And Portugal in a couple of weeks. And that's one place where you see people don't necessarily agree what act by means or reviewed by or kind of interesting. So to summarize what I've said so far today, once your name is in the maintainers file, you automatically, you now are a maintainer, congratulations. You can choose how involved you want to be in that maintainer role, work out with your maintainer what you want that to be. I discussed some of the different tasks that you'll be responsible for. And I wanted to point out one more time when you need help, your upstream maintainer is there to help you just like you're there to help people submitting things to you. Thank you for coming, questions, thoughts, suggestions. While people are thinking of questions, I did mention one unlikely path. If you become the path that patches flow through, then you'll be sending pull requests to your sub maintainer and you need a repository and a key. And that's all described on this website in a wiki. So if you do get in that position, just go look at that. And the administrators will help you out. Sorry, the slides are online so. Okay, I'll let you take a picture. Okay, questions, cheers. No hecklers, my front row is missing today. Well, thank you for coming, I'll be here all week. You can also reach out to me by email if you have questions and thoughts. And thanks for coming today.