 I work for IBM, so I didn't make slides this year. The topic is MM documentation of Lake Zero. What we had a couple of years ago was Mel Gorman's book, Understanding Linux Memory Management, and a bunch of text files. Now, this day's Mel's book pretty much is like all the Unix text books about operating systems. It covers the basic concept, but details are not there, many things change since then. And some of the very important things that are now present in memory management, like THP, for example, are not mentioned in 15. I think now it's 20 years old book. As for the text files, we did the transition to RST, thanks to Mauro, John, some want me. It's a bit more organized now these days. We have a different section for user-facing documentation and different section for kernel developers and some coverage of internal APIs. But there is a ton of things that could be improved, and I don't have much to say about it. It's more of a process talk. What can we do about the MM reviewing process to improve the documentation? What can we do as a community to encourage people to write more documentation to describe the existing things, to make new features documented? And how can we help maybe newcomers who want to help us with the documentation to move forward? So the idea I had, that's what I'm trying to do at least to review major MM patches from the documentation perspective. So you and Liam probably heard from me. And I know Matthew does as well sometimes. But I don't see other reviewers jumping on the features like MGLRU or MapleTrees for example and saying, hey, it's not documented or something like that. Andrew, you only comment about user-visible effects. Yeah, I know, I reviewed it. But there are other things that I just can't keep up recently because of a lot of stuff going on with me. So any help would be appreciated. And there is also a lot of tribal wisdom that is nowhere in the written form. And I remember I've asked Vlasti Mill several years ago if he has something to share, just some notes random, I might arrange it in a more proper documentation. And well, it never happened. Vlasti Mill didn't share. So I'd be glad to hear suggestions. I'm sure John will be glad to hear suggestions. Probably we also can participate in some Google Summer of Dogs or something like that as a MM community. John wants Mike. Hi. So I have a few minor suggestions. I think, first of all, let me anchor it by saying, think about the experience you get from this two different experiences you can get. One is you're in the code and you're looking at the code and you see the documentation next to it, the per function documentation and the per file documentation. There's never any per file documentation. That's one experience. The other experience is you go in and you read something under capital documentation either the HTML, which is pretty, or the RST. And if you do the second one, you're going to find that it's not as good as it could be because there's some sections that were written narrative style, like here's how such and such works and that's great. It may be aged, but it still gives you a roadmap. It's very nice. But mixed in with that is a whole bunch of noise and horrible stuff, which is generated from stuff that you wrote with good intentions over in the code. Because the system is set up to pick up two of those, those two separate types of documentation and treat them the same because there's an underlying belief that they're the same. So here you get nice narrative style stuff and then a whole bunch of stuff that sphinx formats in a super white space, lots and lots of white space. So it'll say, here's a function, here's its arguments, here's a bunch of boiler plate things like page means struck page that you don't want. And so now you've burned half a screen full of stuff that tells you nothing. And now you're still scrolling. You see what I'm saying? So this is not as good as it could be. And so, I mean, I have a couple of suggestions, but it's easier to identify problems than to actually solve them. But I wanted to point out what the problems are for... That's John to blame. That's good. Right. But I think it's important to separate those two things out. I just want to... I don't know the best way to do that. But you shouldn't be indiscriminately scrounging around the code and finding stuff that was written carelessly and putting it in with the carefully written narrative. That's my main input. Yeah, everything you say is accurate and good. I would use a facing documentation, but I think the documentation directory is good for user-facing documentation. And I try to be diligent to remind people when they had Prokofievs for update, update. But for documenting the code, I think the place to document the code is in the code where we can look at it all the time or where we maintain it. And that's skilful, literate code commenting and code commenting to not tell you what the code does because the code does that and you should tell you why it does it. It should tell me stuff that I cannot learn by looking at the C code. With Memblock, I did an intro documentation section and I tried to keep it up today. So this probably, like, two, three paragraphs about page allocator before the 800 lines, 8,000 lines would be useful as well. And maybe not the RST, TXT or whatever in the documentation directory, but some piece of text. The problem is that code evolves much faster than the text. And we don't really care about it. And even the quotation inside the code, from once, pretty frequently we get a mismatch between kernel documents and the actual code. I think this warning is disabled in the build, right? Build warnings, and so they ignore them. That's something we've been trying to fight for a long time. I don't entirely agree with you, Andrew, on where the documentation should be. A lot of it belongs in the code, but that won't give you a coherent narrative of how you pull the system together as a whole. You can do it, and one thing I was going to suggest is to look at what the DRM people have done with their documentation. They have, in fact, put a lot of it in prose form, in the code, in doc blocks, and then they stitch together in the restructured text so that you can build a nice manual out of it. But all that stuff is there in the code if you want it there. And so you get this nice theory that if it's in the code, then developers will update it when they update the code. It's a nice theory. But let me just say one more thing, and then I'll give this up. Which is that when I inherited the documentation directory, what I got was this incredible unorganized pile of junk with lots of spider webs and all of that. So we've been trying to organize things and all of that. And what we have now is a whole lot of smaller unorganized piles of stuff. And it's an improvement. At least you know which pile to look in. But we suffer from the twin problems of one kind of trying to frantically collect and hoard every bit of documentation we have, because it's been so hard to get. It's amazing when I go through to try to delete some old and obsolete documentation sometimes. People just don't want to let go of it. It's like getting rid of your heirloom coffee cup set or something. You just can't do it. But the bigger thing is there's nobody who's really charged with creating a coherent overall experience with the documentation. And so we still have stuff that's really been thrown together. It's great. You did all that work on a lot of the AMAM docs and all that and it's an improvement. But you don't get a coherent whole that way. You get a collection of stuff. It's still a collection of stuff. And it would be really nice if we had somebody somewhere with the time to really build it together into a document, a manual. Well, for now it was a PhD. So we need at least two PhD students now. So I wanted to use a recent experience to say none of this works because we don't have enough reviewer bandwidth and reviewer time. So the example I'm thinking of is Neil Brown did an absolutely amazing job, put in a lot of work to document the reader head code. And I was clearly the right person to review it. And I was too busy and I didn't do it. And so a large part of this is my fault. He wrote something that was like 90% right. And every time I look at it, I see the 10% wrong and it bugs me, which is of course the wrong thing to think about. It's like, ah, this should never have been merged, but of course that's stupid because it should have been merged because it was documentation, it was 90% right. But the weird thing is, it's this wonderful bit of prose and it completely ignored the other wonderful bit of prose which was actually right that was already in the file. Even having the documentation in there with the code doesn't work. And I just don't know how to fix that because Neil clearly didn't read the documentation that was already there before he wrote the documentation that he wanted to read. And it's time and people and expertise and we just don't have enough of it. I mean, we need more interns, we need more people who can spend time on this and God help us. We need to go back to our employers and say, do we have any spare technical writers who can do a good job of, because you know, tech writing is a skill and many of us are not good at it. And those of us who are don't necessarily have the time to do as much of it as we should and, yeah, and no offense, John, Colonel Doc sucks, I hate it. You knew that. I know you don't love it either. I mean, we've all inherited this awful syntax from Java Doc or GNOME Doc or whatever the hell we did and it forces you into a bad style. What about, let's say, mentoring Google Summer of Docs? I would love to, but Summer is also when Oracle has interns and so I have an intern starting in the summer and I need to spend my time with my intern and I can't spend it with your intern. Michael, me. I mean, the biggest problem with reasonable documentation is that you need to know a lot just to start writing that. I didn't know a lot before I started writing it. Excellent. But it really depends what you want to really document. For example, if we are talking about how our reclaim process work, that's full of really subtle details that is spread over several brains, but you have to put all those people into the same room to force them to put something down. And that's where the primary problem lies, I believe, where you really have to find time for that kind of work. Having a student or somebody not really experienced in that, in the respective area, will not or very likely will not generate a very good documentation. It might force people to, for example, the experience... It will force people to review it, at least. Yeah, for example, with Neil asking all those questions about GFP Flux, because, yeah, that just doesn't make any sense. Yeah, because that's historical, you know, so would it be fair to do it like that? We do not have that many Niels, so... I mean, his approach to documentation is excellent, because he just looks from a different point of view than most memory management look, because he's a user. We tend to think about how things work, but not how people might use that. And, yeah, so it's a cooperation in that respect would be really great, so... We really need those people asking those questions and then putting that in some kind of documentation, but having a coherent and good documentation that somebody can use is really hard. I, at many times, I just refrain from reading the code and just go through LWN archives and then just relearn things from there. As sad as it is, that's... Oh, it's not sad. I don't have much to add to that, it's more or less, everybody has limited time. Actually, there is one job that an intern could do, which is we have an awful lot of kernel doc that we have written, and it's not linked to any RST file anywhere. And if it's a header file, it isn't even being checked. And I have like three patches outstanding at the end of my tree, I never quite get around to submitting in any merge window, which improve that situation a little bit, but it's like, you know, if we can find a summary of docs in turn to just go through and make sure that every piece of kernel doc that we've written is actually linked into an RST file somewhere, that will at least improve the number of warnings. And it will make a few more functions actually documented in the HTML, probably badly, but it's a bigger pile of stuff. It will create some more visibility, probably. Because if this student will be sending patches, we'll look at them and we'll say, hey, probably we should change here and there and there. But in the end, it really is for the most part just a bigger pile of stuff. And that doesn't always necessarily help. You know, if you can get cross references to functions in there and that sort of thing, that's useful. But otherwise, you've just made the pile a little deeper. Just to add one other thing on expertise, a lot of what I know I learned by writing the driver book, I really shouldn't have been trying to write that book when I started it. You know, people can learn if they have the right mindset and the desire to do that. You don't need somebody who's an expert to start. You just have to be willing to go through a painful period until they've figured it out. Linux Device Drivers was the book that I learned Linux on in like 1997. So I mean, I think that's before you got involved with the book. That was just Alessandro, yeah. But I mean, that book has just been the gold standard for so many years. And I still refer to the LDD3 PDF that you have on the LDD1 site. So, you know, thank you. And obviously Alessandro as well, because that's how I learned Linux. So I think that we need to look at how the best document we have in the kernel is handled. Memory barriers. Well, there's expertise, Paul wrote most of it. It's actively maintained. It is to the point, it's very clear. Newcomers, even though like barriers are scary in general, but even a newcomer can come and read the document and get something out of it. I just think that we should look at how that is done. I think one way this works really well is that this document already had like an excellent structure. And then a lot of people who are actually not necessarily good at technical writing can still go in there and update and complete content. And I wonder if we shouldn't be looking for like a unicorn or corn person that's great at writing and great at understanding all the technical parts. Once you have somebody put a structure in there who's good at spinning up the narrative, who's good at editing, I think we actually might find a lot of people who would be happy to supply bits and pieces of the actual content. It's a completely different skill set to say, I know how this code works and I can document it versus I can explain how it connects together with the bigger picture. Probably John will do it sometime. Yeah, set up a document, I'll be happy to fill in stuff. It's a template for page allocator documentation, ABC. And then we have something for VM scan and it covers all the huge part. We should document it before it's obsolete in the next couple of months. Are there some high order offenders of APIs and flows that people get wrong in terms of where the near term focus should be on where documentation needed or you just focus on not making the problem worse by having new stuff be documented? I'm focused on making the documentation better. The goal is to have a life understanding Linux memory management that is up to date and it covers most of the memory management. It's a high level goal. It's a good mission statement but what's the other actionable chunks that you see in the pipeline? The actionable chunks probably for patch sets like SPF, for example, require a general description of the narrative, what SPF does and probably even in RSTL, the laundry objects. Like Liam and you did great. They've edited their chunks of description of what Maple Tree is and how multi-generation LRU works. If we get some another piece of new significant feature, I would like the three viewers that somebody except of me would ask for documentation from time to time. And then I always try to find time to write something more but for the last couple of years it didn't really work. Maybe somebody else will also find a bit of time here and there. I know everybody is very super busy but still it might be possible. And like I mentioned, the mentoring students and please they will pop up some problems that we may address. But to Johannes' point, do you think it's because new patch sets coming in don't see a structure of where they're supposed to add on to the narrative? Is there missing narrative that people are not building on top of or are you just saying that people are just not... I think many patches aren't doing a great job of having new narratives. From what I've seen, many pieces don't have good narratives. And some of narratives we have are not very up to date. That would be my suggestion for an actionable item is to create this structure that Johannes was talking about. Take the collection of documents, the collection of files and turn it into something with a consistent storyline, a semi-consistent voice at least. And then you can see what the gaps are in a way that it's hard to do now because we have something that people can approach. That would be my priority rather than adding more stuff on other subsystems which is important, but I think that's the bigger problem. Documentation a lot during code review. I think that would help a lot. And I think maybe the reason... Yeah. I think the reason that we aren't bringing it up and up is that we just don't have any one person with a coherent view of what the documentation should look like. I think if we all have... It's got to start from somewhere in someone. We were talking earlier about how more things need to have owners. If we have someone who's kind of kicking everyone else, this is what it should look like. And it's easier to take the initiative. So I'm pretty new to the M.M. world compared to most people in the M.M. And as I work through implementing the maple tree, I ran into a lot of functions. I had no idea what it did. I had no idea what functions had one line above it. And it didn't mean it was finding the VMA. It was finding the VMA or the one at a greater address. And so when I did a significant change to that function, I actually documented what it did. And then I looked at how people were using it. And it wasn't matching up to what I wrote down. So it's interesting. There was one line. So cryptic, it did sort of tell you what it was doing. But nobody looked at that when they used it. I assume people were just copying the code from other locations. And just because the name said find VMA, they thought that it was finding the VMA. And so when you're looking at... Naming is hard, right? And it's important when you review or look at anyone's code that if the name's wrong... I mean, it's a hard problem because naming is hard. And all the good ones are taken, right? So when you're looking at someone's code and it's named incorrectly to you, maybe point that out because find VMA is a prime example. And then there's a find VMA brief. And then there's a find VMA intersect. And it just gets worse, right? So I guess one thing that I really... My patches are big. And one reason for that is because as I rewrote the functions, and I had to rewrite a lot, I actually added the documentation to it. And I think that might be a decent step in making something coherent or understandable at least to newcomers is to at least have a block of what the code used to do when it was first written. So I think we have a general issue of which documentation we actually want. We won't use the documentation. We want admin documentation. We might want some high-level description of a subsystem and high-level description of functionality like SPF. But then we also want documentation for explicit functions like find VMA. I think in my view, the level of importance is the most important is the user visibility, user documentation, what user see, what's going into admin guide. The next is a high-level narrative of how the thing works. So now we have a pile of things that explains how very particular things work, but we indeed do not have the collective narrative and some outline story about, here's memory, here's something, here's the reclaim, here's page allocator with the references to the relevant documents. And kernel doc per function documentation I think is the least important. Yeah, and sometimes it's like find VMA and what it should be doing is finding a VMA and not like something else and you shouldn't need documentation, just the naming problem. Right, but on the other hand, kernel doc documentation is easier to do. It's easier to do, but sometimes it's like, I think we heard this, there's a parameter page and then it says, oh yeah, this is the page. There is some redundancy, there is some redundancy. This description says it returns the page that it found and return says it returns the page. Yeah, and actually you care about something that is not what you would expect. Like in the documentation you would then find something that you wouldn't expect, like find VMA, you would document that it has a corner case of something else and not like that, the parameter page is the page which is not so helpful. What I'm actually trying to say is it's hard to document these various, like different views or expectations on documentation maybe in one or two directories. I mean, I think we have the admin guide, I think under documentation admin guide which is supposed to be the user interface. And then I think we have some internal or like internal ABI thingy. We have like three pieces. The admin guide is a user-visible effects of different subsystem inside the M. We have a pile of stuff in the documentation VM and we have some other pile of stuff in the core API. And I think there wasn't even from Rick some website about MM and then Wiki or something like that. I think it tends to be outdated. Yeah, but it's still alive. I mean, I checked it some time ago when he explained something and I like, it was valuable to have to understand the history. But what I'm saying is like when I write documentation, for example, I updated the memory hot block documentation a while ago. I mean, I only cared about the user-visible effects because everything else will bit rot in my humble opinion. And then the question would be like, what else and where should I document it? Like documenting each function, most probably not. So I think we would need some kind of rules or like guidelines. What is actually worthwhile documenting and where would you put it? Because sometimes I have no clue what to document and what not. For memory hot block, I don't know. I would say the process in general so we take a memory block, we check if there are pages that are movable, that we can free the memory block from the pages to later unplug it. We go to the architecture, hooks to, I don't know, actually remove memory and so on. So something like high level and it's entirely up to us to keep it up to date, right? Okay, here we go. I think I'm agreeing with Jonathan more and more. One of the things I was going to say earlier, which was kind of mentioned, is the difference between internal MM documentation and user MM documentation. Documentation that's for drivers to call into or stuff like that. And that's one of the places that I get confused because I'm not really into the MM as much as I want to be but I've done drivers before and then you get people who are calling Find VMA, they don't really know what it does but they think they know what it does and they probably shouldn't be calling it to begin with and so I think organization is important and I don't necessarily agree that we shouldn't document the internal stuff but sometimes when you document that stuff maybe we get over-documented and then we have users finding functions and going oh, this seems to do what I want it to do and then they call it. So yeah, I mean I actually grew with Jonathan as far as taking the K-Doc stuff and having it referenced in the documentation directory that's something that I've done a couple of things on not necessarily for MM but in other spaces and maybe that can help to organize so if you don't find it in your group of valid entry points you know it's not part of your... I don't know, but basically try to get things organized that way so people know what they're looking at and maybe use the documentation directory and references into the K-Docs to maybe help structure that. I don't know, maybe. Okay, so to chart us a path forward I'm going to make a suggestion and then you can throw things at me which is our best book about Linux memory management is of course Mel Gorman's so let's take his structure it's up on kernel.org I just pulled it off of my laptop if you Google Gorman book it'll find it so we take that as our guide basically he's got chapters there isn't one about the page cache so I feel personally offended that there isn't one about the page cache because it means I can't see any material from him but so apparently Johannes and I are going to fight at this conference after all so we set up basically that structure and we'll add a section for page cache but we start organizing our documentation along those lines so we set up an RST file in documentation.fmm that collects these sections in that order because we've got to put them in some order and it works and I'm going to volunteer to do the page cache one or at least maintain the page cache one and accept purchase from people and probably do a bunch of the writing myself because I do actually enjoy writing and then other people can do their area of expertise and that way we're not all trying to work on the same thing all at the same time I'm distressed because I'm seeing a lot of nodding and nobody's thrown anything at me yet and I'm not used to this okay all right then that's what we'll do yeah so totally agree we can start with Melsley out of the chapters and then more as we go I'll set the page cool it's okay I really hope somebody would volunteer so I'm really glad you did could any tools help us? could a new KDOT tag help flag this is intended for drivers this is intended for policies this is entirely manual labor okay never mind and I think we should stop here right? thanks a lot everybody if I could just add anything that involves going through and changing lots and lots of Colonel Dock comments is this a fiend task that you just don't want to get into? thanks