 Thank you for being here and thank you to the Linux Foundation for all that organization and corralling to get me here and the support and I really appreciate it. So this is my talk, second place is the first runner. We'll get into what that means a bit later and hopefully by the end of this you have a better idea of how to grow your influence in the Linux kernel community. But first let's get started with the agenda where we're going to go over today. We'll start off with a career path disclaimer. Everybody's career path is a little bit different. So the examples in this presentation are painfully mine but there's more than one way to go about this in the Linux kernel. Next I'll actually explain what second place is the first winner means, where that comes from and what it relates to and it's something in real life that kind of mirrors Linux kernel growth and development. The meet will be a discussion about how to kernel, how to be a part of the kernel community and grow there with an emphasis on reviewing and maintenance. And then finally we'll go back and review like why is it worth it to spend time here and persevere in the community with what's supposed to gain. But first we're speaking of career paths, I don't know if you recognize this person, this is a HPA, H. Peter Anvin, and he's been involved with Linux from the beginning. He purchased a Linux, one of his first computers. He's been probably, I think, one of the third, I think the third contributor to Linux. He's been there from the beginning. And beyond being wicked smart, he's also been there to, at the point where Linux, there was a lot of greenfield in Linux and he grew with Linux. So his expertise is rooted in the fact that he wrote the foundation for a lot of these things. Literally, he wrote bootloaders to get Linux off the ground. So he's been there for a long time. And then there's people like me, who came a bit later to the show. And decently intelligent, but I'm no Peter Anvin, but I have grown in the Linux community and my path is different. But it has arisen to a similar level of influence in some areas of the kernel. So find your style in maybe somewhere between Peter Anvin and Dan Williams, or somewhere completely different, but there's multiple ways to get to growth in the kernel community. Okay, second place is the first winner. What do I mean by this? So this is back in 2007, 2008, I participated in something called the Arizona Treasure Hunt. This is something organized by the IEEE Society of the Phoenix Chapter of the IEEE. And it is an annual event where engineers set up a treasure hunt in the desert, in North Phoenix. And a good idea of the scale of what this is, this is the entire map of the playing field of the treasure hunt. And from one corner of this map to another, it's maybe about a 30-minute drive. And multiple teams sign up, they all rent large vans, they bring their head flashlights because it happens at night, and you drive around to these sites trying to solve clues. If you go to the wrong site, they have a document there to remind you that you've gone off track. And being teams of nerds, you can imagine some of these puzzles. But anyway, the scoring of the event, at the end of the event, you basically scored on how many puzzles you solved, how fast you solved them, and how many emergency envelopes you opened. And I equate that to Glenn's Curl Development in terms of all the wrong turns you can make, getting from one point to the next, and the problems and the scale of problems that need to be solved. But at the end of the scoring, the team with the highest score gets the honor of being the organizers for next year's event. So second place is the first winner, means you achieved everything except the responsibility of having to plan and organize for next year. And this is similar to the path I see some people take towards the maintenance. They arrive at the kernel, and they think their very next thing, like after they get their first thing done, their very next destination is, I want to be a maintainer. And you want to have the fun, but you may not want that responsibility up front. So how do you come in second place in Linux? Like I was saying, if you aim for maintainership, you might actually get it. And that may not be where you want to be, especially early in your career. Kernel maintenance is a support role. It's not an executive role. It's more janitorial than royal. Nothing wrong with janitorial, but the scope of the work is getting into the mess, cleaning things up, and being there for people. The power that some people scribe to maintainership, I get to the side where things go, that the power is actually, there's some power to say no, but there's a lot of responsibility in saying no to saying no. You have to say no in a way that doesn't lose a developer. You have to say no in a way that points out the next steps to get a yes. But more importantly, you have to say no from a maintainer role to people that would use your maintainership in a way that's detrimental to Linux. A maintainer job is to maintain the long-term health of the kernel, not necessarily to push the agenda of their employer. So sometimes you have to say no to your employer, no to asking you to do the work to say no, this is not the way Linux works. And the long-term health of Linux means we have to go a different way. And I didn't mention at the beginning, but at any point in this presentation, feel free to raise your hand, jump in and ask questions about anything you see. I'm happy to go off on tangents on some of these points. Okay. So, review. Review is the centerpiece of this presentation, hoping to make that apparent compared to a maintainership. But before getting to how to review, just went to level set about how to get started. I'm assuming that this audience has gotten started, has found a way to get started, but if you haven't, there's certainly mentor programs, ones organized by the Linux foundation to help you get connected with people that can guide you on this journey. But the most common answer you'll get from any person who's been doing kernel development for any amount of time is when someone says, how do I get started? The person needs to say fix a bug, fix a lot of bugs if you can. And there's multiple reasons, there's a couple reasons for this. Some of them are self-serving for the maintainer because maintainers are overloaded, they don't have time to necessarily track down all the bugs themselves. So when somebody shows up with a bug fix, that's one more thing off their plate. That gift to a maintainer taking something off their plate is currency. You use that currency to basically buy forgiveness for the mistakes you're going to make submitting your own bugs. It's the way to restore, it's a way to initiate a trust relationship. People begin to trust you if you show the fortitude to stick it through and fix a bug. But it can also be used to repair trust where you've made mistakes and want to show, hey, I'm in it for the long term. In general, the community values longevity and staying power and being there to fix mistakes more than they value perfection from the beginning. And so these are ways to get established and even if you are established, repair things and build that trust. But once you graduate from fixing bugs, then you start to look around for maybe scratching an itch, that small features, small things that you'd like to see. I describe it this way, I think the common way, of course, there's people that show up with fully formed ideas about what they want to do with Linux. They come in with a new subsystem. That tends to be more rare and that wasn't the way I came into Linux. I came in fixing small things and then I had an idea to add a new feature. And giving your first significant feature landed after that point is where I think the danger comes in and the advice is to graduate to become a reviewer, not a maintainer. And of course, I did not do this. I went the other way. I think it's funny that, or ironic that as I was preparing for this presentation, this article from LWN came out and there was a review of basically how much does the maintainer's file tell the truth about what's actually happening. And the first example of a subsystem that probably needs to be cleaned up as maintainers that seems to not have had any ongoing active development happens to be the very thing I introduced as I would call my first major feature. This was an asynchronous offload engine support for the RAID subsystem. So things offloading XOR and RAID 6 checksums to an offload engine. And I probably put my name in the maintainers file. But looking back, I didn't get much influence from being an async TX, a subsystem maintainer. There aren't too many other people that are looking to develop code in that space. So what did that actually get me? But I will say that what I got from this experience was not the maintainership that was the relationship with Neil Brown. I was lucky enough to start in a subsystem where the maintainer's working style and my working style were compatible. That's not always the case. Sometimes you end up in situations where the maintainer and the way you work just don't jive. In those situations, I would recommend finding someone that can mediate or help you interpolate with that maintainer. But if you're lucky enough to have that kind of relationship, what Neil gave me was patient feedback. And that tends to be rare. And you tend to have to earn it with trust to get people to take notice, take pity on your mistakes, and set you on a better path. So if you find yourself having a hard time in Linux, double check that it's not just a simple incompatibility with the way you work with your upstream. And either find different upstreams or find people that you can work with. But so the lessons from your first contributions are all the mistakes you make. And in general, to be a good reviewer, you need to be a connoisseur of mistakes. You have to seek out your mistakes on a shelf and remember them. And you have to be actively looking for other people's mistakes. Of course, a lot of the basic ones and fundamental ones are in the documentation. Of course, everybody reads the documentation. But then there's a next level of mistakes that people make that don't get written down. And those happen on the mailing list. And they get documented in email back and forth. But like I said, the first place you're going to get a good source of mistakes is all the ones you make yourself. And of course, you didn't make any mistakes or in the documentation, right? You landed all those perfectly. It's other ones that weren't documented, that tripped you up, of course. But this, I found invaluable, is to look for the mistakes that veteran reviewers point out. So go read Alina's review. Go read Greg's reviews. Go read Kristoff Hellwig's reviews. And see the things that make them grumpy. See the things that other people repeatedly make. So this line here is a search string. You can put into lower. lower.kernel.org. Fire this up. And this is Alina's responding to people pull requests during the merge window. And Alina's will dig in, especially for developers that are newer or don't see as often. You will dig into their pull requests and not only find issues with the pull requests, but you'll find issues in the code. And there's just a lot of good reminders that come from that feedback. So grab a coffee, and when you have free time, check out the mistakes other people are making. Alina's is pointing out at this point. But the biggest one, especially the one that goes to maintaining your trust, building your trust, is not repeating the mistakes. What tends to contribute to the most grumpyness on a mailing list is when people have to repeat themselves. So don't be one of those people that makes them maintain and repeat themselves. But even more bonus points than the points you get from just maintaining those mistakes is actively find ways to preclude other people from making the same one. So documentation contributions, but also looking at how people misuse APIs, maybe it's not their fault. Maybe the API is leading people down a path to make the same mistake over and over again. And do you see a pattern that can be improved by just changing the API? So I put this quote in here in full. This build on the idea, once you have a collection of mistakes, what do you do with it? And when I send out the slides, you can click on this link to read the whole thread. But this was in response to the fact that the XFS maintainers were kind of complaining about a lack of review bandwidth. And this is David Chinner talking about how they get more people to review. And a common refrain for people that are asked to review is saying, hey, I know nothing about XFS. I know nothing about file systems. Or I know nothing about memory management. I don't feel qualified to review these patches. And Dave's point is that that's a defeatist attitude. But moreover, there's actually value in not being a subject matter expert. We don't need a pile of subject matter experts because that's not where the code of view goes wrong. The big concepts are usually easier for a maintainer to spot and address. It's the detail. It's the loop termination conditions. It's these corner cases of small mistakes that people make writing code, even veterans. And they're there to be found. And when you find them, the insight of fixing them helps you get more acclimated to the code. So what he's saying is that don't be afraid to jump into a place where you don't have deep subject matter expertise. Bring your general expertise and general code review skills of finding these mistakes and use those as a window to learn more about the code base. Because as you connect these dots, then how the code works becomes more clear. It's similar to fixing bugs in that you learn how something works by seeing how it's broken. And this thread is worth reading in full. I think it was the most concise explanation of, yes, you can review code that you are not the expert in. This also goes back to us, something that I was kind of alluding to with Peter and Vin. And the fact that Linux is mature, there's not a lot of brand new things to go do in the core. It's a lot of adaptation and optimization and detail work. Like, there's not a big piece of Linux that are missing or suboptimal. It's been around for a while. So a lot of the people that you would point to as long term veterans, they had the benefit of being there as Linux grew up. And they got to write the naive implementations and then watch those implementations hold up and have to go fix them. And new people coming in today are having to deal with code that's been matured, been battle-tested, and battle-hardened. And it's a bit harder to make an impact there. But I would say that there's not a lot of opportunity to change the core. But core changes are still happening. There's a lot of opportunity to review the real changes and get acclimated to things that you're interested in, even if you're not the one writing the feature enabling for it. So the new age, the up and coming Linux kernel developers are people that need to depend much more on review than the natural something naive and watch it grow later. So continuing on with this idea of you don't need suboptimal expertise, I've also found that you have a window where you can use your naivety to your advantage, especially with asking forgiveness from maintainers tend to be a bit more patient with people that haven't been around for a long time. So take advantage of that time to ask deep questions. But also, it's also a benefit for review if you don't quite know what's going on. Change log quality is paramount. I don't know if you've seen any of case cooks talks about security and how long it takes for some of these root holes that can exist in the kernel for years. And asking basic questions, an expert tends to assume that is there. And there's a lot of basic questions that can be caught by leaning into, I'm not sure I understand how this works. Certainly change log documentation. You want the ones you write and the ones others write benefit from, hey, can I explain this to somebody who's not the expert? Does this make sense to a kernel developer of moderate experience? The last thing I would say about not needing to be a subject matter expert to make an impact is an analogy of you're like a little sister or a little brother coming up in a family. If your older sibling is two years older than you, when you're born, the difference between a two-year-old and a newborn is gigantic. But over time, you catch up. And you catch up asymptotically. So even though they're ahead of you, your ability to catch up and be productive is the same as a little sibling catching up with an older sibling. And when they're 47 and you're 45, you really can't tell the difference anymore. OK. So we've covered how to get into reviewing things, especially things that you're not sure you can review. You can review those things. Then the next level of impact and influence you make is by seeking out and destroying the contributions to the platform problem. The platform problem is basically assumption that, hey, this core is pretty well-baked. And we're just going to use it as is. And any kind of warts or gruff edges, I'll just work around those in my driver. Or basically, I just want to get my job done. I use API, take it as is, and go. The people that get recognized and get more influence granted to them are the ones that say, no, this API is actually broken or is not optimal. And I'm going to fix that problem. And I'm going to go and find other people that have avoided this platform problem the same way I have. So the technical debt that, in general, Linux is the whole point of it, the review process, is to not take on technical debt. But the main source that gets into the kernel is where people avoid fixing the platform. And so it's a exponential effect of impact by the long-term health of the kernel to fix these things. And sometimes, the definition of where the platform is and what pain point it's got is not readily apparent until you start reviewing patches from multiple people using the same thing. All the patches in isolation look OK. But then you see patterns of people making mistakes or open coding things multiple times in multiple places where you're like, hey, this could just be refactored and put into a common place. And certainly, if you want to make, I can't imagine any maintainer that's not happy when they see a red diff. A red diff when you turn it on terminal colors for looking at the stats and the minus lines all turn red, and a patch is mostly red lines and mostly deletions, those are so sweet. And those are the kind of patches that you can develop only after seeing the stream of mistakes that other people are making and the things they're avoiding and the opportunities for refactoring. So in summary for review, the influence you gain from doing this work, from showing up at conferences and talking about this work, is that you demonstrate that you are in this for the long haul and you're in this for the long term health of the Linux kernel. And if people trust that for whatever mistakes you make, that you're there to own up for the mistakes of the people that are also that you're helping to mentor yourself, then you're given a wide berth to participate in the conversation. It's readily apparent when people are maybe pushing an agenda from marketing and not engineering to use an analogy. Ian? Yes. Looks like we've got a question here. Sure. They, Roman asks, could you describe what is a platform problem in a more comprehensive way? Sure. I'll give an example. Sorry, I can think of a good example. So I was working on a SCSI driver. And I found, and I forget the exact API. Let's assume it was, I think it was something to do with the de-registration. Yeah, it was a hot plug bug. And I noticed that every time I unplugged my SCSI controller, this, I think my driver would crash. And I looked at the API, it looks like I'm using this right. There was an object reference problem where we call this API at the wrong time. You get a use after free bug. And I'm like, huh. It looks to me like I'm using this API correctly. And I'm like, I'm certain that other people have to get this bug, too. How did this not happen before? And I went and looked at other drivers using the same API. And they had a workaround that said, oh, they actually covered it in the code. They said, oh, by the way, the core, if you do this in the wrong order, if you use the API the way it's documented, it caused this crash. So we're going to take this extra reference here and to prevent that problem. So they found the bug. They root caused the bug in the API. And then they worked around it in the driver. And that was an example of avoiding the platform problem. I will also link in the signs that go out, I'll put a link to an LWN article. But if you search, if you do a Google site search, if you search platform problem, site colon LWN.net, you'll see articles about it as well. John has done a couple of them. And there's some good examples there. But yeah, it's when you see people commenting and working around a core API rather than fixing it. OK, awesome. Thank you so much, Dan. Thank you, no problem. So we tried to talk about keeping the kernel healthy and representing the kernel health out to the community, but also internally to your organization, your employer. But something I didn't realize or become more apparent to me is that review is so critical to your ability to get your job done in the kernel that everybody should be on this path of growing the influence. Should be looking to negotiate review time with the manager to say, hey, I need one day every two weeks or a day a week to say, hey, I'm going to go review something that's not part of what I'm trying to submit. I need to go and build up and keep fresh my contributions to the community. And if more developers did this, I think we'd always have grumpy maintainers. But the community is definitely starting to feel this review ban with crunch. And when people are saying, hey, I can't get my code in, the best way to fix that is to start helping other people get their code in and ask them to reciprocate. And this tends to work. The Andrew Morton, who was a few years ago, basically said, OK, I'm going to stop merging anything that doesn't get a review from somebody else on LinuxMM. I'm not going to be the reviewer of Last Resort. Somebody else has to review these things. And I think some people will ring their hands all night. This is going to slam the brakes on the flow of patches in the MM, but it didn't. And so it's been successfully deployed in MM. XFS had this policy of no author, not accepting patches that they're only signed off by the author. Just a need for more review. And it's to the point where you should be negotiating the manager to have that review time to go help somebody else. OK. So the influence you gain from reviewing is pure gold for your career. What about becoming a maintainer? What does that do for you? Or how do you do it? Then? Yes. There are two questions in the chat, if you want to answer that. Is it possible that the projects we have been envisioning are so close to a real panacea that folk who carefully review it will become near-instant fans and willing to dive in head first years today? Absolutely, so close to you. I'm not sure I understand the question. James feels free to turn your microphone on and ask the question if you elaborate on that. Please. OK, we will be coming to you in a second. I mean, if the comment is about Willa. Yeah, hello. Hello. Yes. Yes, hi. Sorry about that. I had a little technical glitch. Sound familiar? Yeah. Yeah. Well, you know, I've been studying. It's mostly based on health care, right? And the projects we've been envisioning will solve all kinds of things, including COVID-19, OK, and all sorts of different other related problems out in the world these days, OK? But I think that if there were enough people who got to know and understand the nature of the projects we've been proposing, I use I and we, you know, me, myself, and I, OK, so yeah, I think that if my projects were to be looked at closely, OK, and I'm not talking about some weeks on end. I'm saying if some people took, like, a half an hour or an hour to see just exactly what we've been proposing, OK, that they would most likely willingly jump in and try to help the project along in as many ways as possible. Yeah. Thank you, thank you. No worries, yes, thanks for the question. I hear what you're saying. I think in Linux, there is a lot of opportunity for people to see a solution to a problem or see an idea for a solution to a problem. And a lot of people want to kind of present the idea and then hope that the coding and the contributors show up. And in Linux, I think there's been too many times when the idea, once it meets kind of practical deployment, it doesn't pan out. To the point where that happens so often in Linux that we don't jump in on the idea. They jump in on a successful code. So it's really a much more iterative and kind of practical deployment of, yes, you have an idea. Yes, it sounds good. But before people are going to jump in on it, they have to see the code has to be there. And so it's kind of this deeply practical, deeply non-theoretical part of Linux where it's all about the code and showing the idea and not just the idea itself. OK, so can you hear me? Yes. Yes, OK. So then being that I'm not a coder. Right. Yeah, well, where do we go from here and stuff? Yeah, I actually have not seen that. I mean, I've certainly seen people say, hey, I have this idea wouldn't it be great if. And the level of backlog of people's plates of taking your idea and putting into code, the backlog is already full with the things that are already ready to be deployed that it's been the case that they can wait on those ideas, wait for somebody else to wait for those kind of a natural selection of the ideas to filter into somebody who's going to deploy the code. But I don't. But to your question, how do you take your idea and get people excited about it? Current developers really only get excited by patches. And it's a matter of finding maybe a contracting company or somebody that can take a contract to try to see the idea into patches. But that's a good question. I think people see that on the mailing list fly by every once in a while. Good ideas, but no code with it. OK, thank you. That's very helpful. OK, thanks. So the maintenance side of this equation. So I said my experience was develop my first major feature, maintain it. And the maintainers, done. I won Linux. And then realized, wait a minute, I want to contribute more. I want to grow into other areas. There's not too many people praising me for this little corner of the kernel I've invented. And it wasn't much later in my career that I met somebody who I looked up to and asked, hey, what do you do when you send a pool of crystalline? I've never sent a pool of crystalline in my entire career. He'd been doing it five, 10 years longer than I have. I said, really? Yeah, I don't even have my name in the maintainers file. I've just been reviewing and developing and pushing patches to other maintainers. I'm like, oh, that's the way to do it. Why am I resting to maintain something and be burdened with keeping something up versus going around and being a free contractor or a free reviewer in other parts of the kernel? So that really felt like success to me. Of course, I'm not saying copy this pattern, but you can be in Linux a long time without being a maintainer. And today he is in the maintainers file. And it was because he's a sub-maintainer of a subsystem. Because the main maintainers are like, hey, I just can't handle this. And please, please, please, would you put your name here and take this piece of the maintenance from me? And that's really the optimal model. Like, you want to be second place, not to be the first winner. You want to be the person that people would say, I would love if this person would take over some of my responsibilities so I have more time to focus on other things. And I'm sure that this person was on multiple people's list of saying, oh, yeah, if they would take it over, I would hand it over to them. And I think there's a few examples of people like this. I think Christophe Hallig is one of these people that was kind of late to maintainers file. I think Dave Hansen is one of these people that's late to maintainers file. And you see the level of influence they have did not arise from prestigious maintenance, but it arose from being really precise and deep reviewers. The way I summarize this is you want to be somebody that believes other people of their burden and not necessarily somebody. You don't want to be the focal point of other people's burdens as a maintainer. Not to fully throw a maintainership under the bus. That work is really important and really good maintainers are hard to find, especially for a super high volume subsystems. The things that the networking maintainer Dave and Jacob are filling in is just amazing, the amount of patches they get through is amazing. But that's the top level of maintainership. A lot of other subsystems, they're not that high volume. And your influence from just maintaining a small subsystem is not as high as it could be if you were going around trying to review other things that interest you. This kind of goes back to how to be a maintainer. The problem is that it is subsystem specific. And there's certainly documentation. There's people you can follow, but there's no kind of right way to do it. It really depends on the code itself and how it moves and flows. I found it interesting that the person replying to me in this email is Linus. These are one of the messages you can see if you do that lower search that I showed earlier about the feedback. Maintainership people have been doing it a while. Get it wrong. And then if you look at Linus' feedback, he's talked about what he likes to see in merge requests on the mailing list. But there's no document that says, here's how you must do a merge commit if you send one to Linus. So a lot of this is organic and it grows as people are interacting with the kernel and kind of have to be in that position of reviewing other people's submissions. I was actually one of the people that proposed the idea of, well, we have new maintainers. Why don't we write a handbook that helps them? And there's some tips and tricks in the kernel. This work started by other people. I had the idea of making it more comprehensive and more prescriptive. But as I tried to define it, the number of people that said, no, I do it this way. No, I do it that way. And with good reason, too. Some of the differences are arbitrary, but some of them had deeper reasons. I did not succeed at creating the handbook. But one thing I did identify is that the maintainers that are most effective and the ones that developers tend to be able to work with better are the ones that are predictable and you understand they make their rules clear. Towards that, I started this effort called the subsystem profile for people to declare what the rules are for the subsystem. When is the last day you can submit a patch and still be accepted for the next merge window? There's a bunch of details that are subsystem specific. And each maintainer might have a different answer. So if you're looking for how to be a maintainer, there's some guidance in the kernel, but a lot of it is finding a maintainer that's maintaining a subsystem that's a similar patch ball into yours, similar complexity to yours, and try to steal ideas from them. Asking maintainer questions is, I mean, every year we have the maintainer summit and kernel summit mailing lists. Those are good places to, even if you're not going to attend kernel summit or maintainer summit, so many things that are invite only. The mailing list is useful to ask maintainership questions and get answers and also to go back and review the archives of those things as well. But it really comes down to being predictable and setting expectations. That relieves the most pain from contributors. Okay, so why do we do this? Why would you want to grow yourself out of, or grow your influence in the kernel? And you don't necessarily have to. I mean, there are people that are perfectly happy to say, hey, I'm a prolific developer in this one focus area of the kernel, I'm really good at it. I don't really care about other parts of the kernel, and that's totally fine too. So don't think of this as you must grow and go review a bunch of other things to have a successful career. But if you do want to grow your influence and you want to get deeper into this thing, this way is what you have to do. And it's a cycle of influence and trust and impact. You earn trust. And with that trust, you can make bigger impacts. And when you can make bigger impacts, then you can influence the direction of where links is going and then build more trust, make bigger impacts, grow more influence. And it's a self-feeding cycle. And it's also just fun to be part of a global technology development project and growing your influence in that space is super rewarding. But this middle piece is, I think, what Linux needs people to kind of persevere through this pain. We need fly-by contributions and people to just pop up one kernel cycle, get their things fixed, and off they go. We need those people, but we also need people that are willing to stick it out over the long term. People that are comfortable being uncomfortable and sticking their nose in new places and finding things to clean up and go through that pain. But also with the kind of promise of the reward of this great influence, higher trust, and bigger impact on the back end. But with that, that was my content. But yeah, if you have some time now, I'll go and I think we're ending a little bit early with the slides. There's time to go over some patches. Hi Dan, thank you for that talk. Can you hear me? Yes, I can, yeah. Thanks for your talk. Again, my name is Karthik. I have been using Linux for the last six or seven years or so. I've kind of been into operating systems for the last few years. I toured around with OS 161 five years ago. Then I started looking different in front of Linux and I absolutely loved it. The one thing, given the way of electronics and software background, I'd like to know what's the best way to circumvent the complexity, to ramp up the complexity in a way, because a lot of things are quite complex and just to get the foot in the door, do you have any tips? Do you have any resources that we can use or is there anyone who can probably mentor us along the way or is it just diving and you have to figure everything out yourself? There's a lot of self-direction and I might be able to give you a more specific answer if you said what part of the kernel you're talking about. But grep is my friend. I don't know if you know about git grep. Git grep is the... it filters just based on things that are in the kernel file and you can grep with history. Yeah, I think this might be a good topic for another mentoring talk. It's basically... I learn a lot about how to tackle subsystems by watching some of these presentations. Going into the conversations and seeing the presentations about people that present on, hey, here's how I debug this problem. And that's been how I've collected tips for these things over the years. And yeah, certainly you grow your own ways to interrogate the code, just messing with it yourself. But these talks and... I had the most recent plumbers in the before times when we actually met in Portugal for Linux plumbers. There was a talk by Omar Sandoval from Facebook. He presented his tool called Dragon Debugger. But he walked through how he went from this problem report back to the root cause and how he interrogated the kernel. So that looked like a really powerful tool. You tend to see at least one of these talks per conference about somebody who had a hard problem and really figured it out. And the ways that go about it are very different. But if I had to look back and say, where did I pick up some of my debug tips, my code understanding tips, it was from hearing other people's war stories. But you know, there's code on the one side, but there are concepts on the other side as well, which are quite unusual to understanding the functioning of the whole system. I remember Linus once said, do you know C++ in the kernel because the hardware is quite complicated in itself and when you're dealing with Cages and Cages, you don't want to introduce a complexity of the language. So that kind of stuck to me. So to your previous question, you asked subsystems. For example, the networking subsystem, networking subsystem, you've got a whole host of, it's not just the language semantics, right? So it's a whole host of bug allocations and walking through the different layers and all these things. So that, is there an easier way there? Or is there some kind of support that I can maybe get some references or are some tips or some working guidelines or something that could maybe streamline my workflow or make it easier for me to understand things? Yeah, it's still the case that the kernel is complex and it moves fast, but it doesn't move that fast. In the sense that there are some core concepts that stick around. Some of these books, like, Linus Curlman's Revealment by Robert Love, that even though that book only talks up to the 2.6 something kernel, the concepts in there as a framework for understanding some of the more advanced pieces. The concepts there, the base concepts, haven't changed in a long time. So if you haven't read that, Linus Curlman by Robert Love, if you haven't read the Richard Stevens books about advanced programming in the Unix environment and he also had a book about advanced networking. This is the... I think Richard's pretty books anyway, but yes, some of these foundational books can give you the concept that you can use to understand complexity in other parts of the program. Okay, thank you so much. If I could, I'll just have a really quick question if that's okay. Sure. So do you have any tips for how we can set up the technical environment, like not to use a working PC as a test around to kind of mess around with the Unix code and compile a new kernel, but setting it up in a virtual machine or are there any tips for that or can it be emulated and debugging the kernel? There's something that I found really hard back in the day. I'm just trying to debug the kernel apart from using Kprintf. Yeah, I don't know if you're familiar with Brendan Gregg's website. So Brendan is a performance analysis engineer, operation engineer at Netflix. And if you look for Brendan Gregg's VPF or those on Google, it will bring up his site. He has an entire kind of, an entire collection of trace points and how he debugs certain problems. And that's just a great place to get, how to get your kernel to give you more data and more information so you can start investigating a problem. If you haven't checked up Brendan Gregg's tools, that's a Gregg with two Gs at the end. Those collection of tools are invaluable. So Dan, I also posted, Karthik, I also posted a link to yeelinux.org main page in the chat. That's a great resource. They have lots of tips on various kinds of debugging and a lot of information on power management. And it's a good resource to go look for a specific area of the kernel you're interested in and looking for information there. Okay. Thank you so much for that, Karthik. I really appreciate it. Thank you for time. There is a question, Dan, from Julia. Julia, would you like to ask that question now? Hi. So when you talked about reviewing, you talked about it from the point of view of like a person goes to try to find some patches that they could review. I wonder if there could be a space for going the other way around somehow where some kind of centralized system pushes patches out to people saying like, I think you're a qualified person. Maybe you could review this patch. What would you think of something like that? I think that might naturally fit with some of the, like I know Constantine is working on better tooling to allow kernel to be more approachable from the perspective of a lot of people just stumble figuring out how to figure their mail or to send patches. So he's developing a system to, you send a pull request to the bottom and send it to the patches. I think it might be useful. Part of the problem I think with the community is patches go out and everybody assumes somebody else is going to take a look at it. And silence is the worst experience for a developer. Like me personally, I'd rather be being yelled at than, well, to the extent. At least there's a chance to make progress versus like sitting in silence for a long time. So I think there's a lot of merit to the idea of tag your it, please help review this. And yeah, especially when like, especially like I had the experience of, hey, this bug crashes the kernel and hangs the kernel and can we get it fixed? And it kind of sat there for weeks when I think all the maintainers and the people that could review it all just kind of assumed somebody else would do it. Yeah, I think it'd be a fun experiment to try on a subsystem basis and see what happens. Can you imagine some, what would you suggest is the criteria that would be good for sending patches in the subsystem? Honestly, I think it should be based on how many patches you've submitted to that maintainer without reviewing anybody else's. At some point it's like a throttle that says, hey, you seem to be really proficient at sending patches in the subsystem. And we noticed that you haven't reviewed any patches in this subsystem. Maybe there's a chance for you to review this one. Because that's the burden we're trying to relieve is that the maintainer becomes the reviewer of last resort. And so if a bot was out there saying, you've passed the threshold of a number of patches you can send before offering somebody else to review, please consider reviewing. I think that, I mean, yeah, I wouldn't mind seeing that be a trial experiment. Okay, thank you. No problem. Yeah. I'll just highlight further people on the call. Julia wrote a tool called, I'm going to mispronounce it, but Coxnell, that is part of that find a way to help people prevent basically find a way to preclude people from making mistakes. So let's do a finds patterns. And we even, I think it's, I mean, even in some cases we'll send you the patch to fix the mistake you made. And so that's been, it's been incredibly useful. And if you haven't checked that out, and if you don't know Julia, then her tool is really amazing stuff. Plus one on that. I use that a lot as well. I incorporate it in my maintainer workflow as well. Okay. Well, any other questions? It looks like we might, Vincent might have his hand raised. Vincent, did you want to ask a question? You're welcome to unmute. Okay. Well, if Vincent doesn't pop on, and we don't have any other questions, I think we can go ahead and wrap up. Unless Dan or Sheila, you've had any other, any other points you wanted to make. And no, and, um, but, uh, but do, do reach the, kind of Julia's point, there's a lot of, there's a lot of silence on the mailing list. And, and sometimes you can help that along just by declaring what you want to happen with your, with your patch set. I think a lot of people assume magic happens. They just send the page and magically somebody comes along to take a look at it. Uh, being, being more explicit helps. And, um, and, uh, just the same, the same way that maintainers, uh, maybe kind of need to communicate what's going to happen. Um, developers communicating what they would like to happen. It just helps, helps the process along and, and somebody can recognize, oh, I'm the, I'm the one that's being waited on. And, uh, because sometimes it's not clear. And, and there's a lot of assumptions that can be, be violated. Um, but other than that, yeah, I want to thank, thank you for, thank you for, for the help, organizing this and, and crying the questions. It was a, it was a good experience. And, and maybe one of these days we'll have a, we can do this in person again, but, uh, this will have to be, it'll have to do for now. Yes. Yes. Absolutely. Well, thank you so much, Dan. We appreciate you being here and thank you, Shua. And thank you to all our attendees and everybody who participated today. Um, this will be up on the Linux foundation YouTube leader today. And, um, Dan is, uh, going to share this presentation out with us. And that will be on the mentorship website. So, um, you can look for that as, as well. Thank you all so much everyone. Have a great day. Thank you. Bye.