 So, do you hear me again? Yeah. Okay. So, I'm not going to repeat Peter's stunt of switching off the micro. You all hear me like this, even in the back? Yeah. Okay. Fine. Yeah. Yeah. Maybe we should just start. I'm going to start with a short introduction anyway, so if anybody is a few minutes late, they won't miss that much. Welcome all of you to the documentation tutorial here at EuroBSD.com. Let's make manuals more useful, is the motto for today. And I'm going to cover a relatively wide range of topics related to BSD system documentation. Some of the material is drawn from my two BSD Canada presentations three years ago, and this may, but the focus today is more in actually hands-on getting stuff done than like in the BSD Canada talks, more technical stuff like how it was implemented and how it was integrated into systems. So, today we are focusing on how do we really make manuals more useful. I'll get to a kind of table of contents a bit later. What is this? Yeah. If we want to make manuals more useful, then of course, it's the very first thing. We must understand what are requirements for good documentation. One is documentation good. Two things are really completely obvious. Documentation needs to be correct and it needs to be complete. What is sometimes overlooked is that it also needs to be concise, because if it's lengthy and worthy, then you lose a lot of time reading it and sometimes don't even find what you need. Manuals, for example, are a good example of documentation violating that. Another point that is often overlooked is that documentation should ideally be all in one place, easy to find and easy to access. The typical way to violate that is to put your documentation on the web, then people have to search for it first, and even if they find it, they will wonder whether it corresponds to the version of the software they are actually running. So make it easy to find. Also, it is not just plain text. Software documentation typically contains lots of keywords from different languages that in some way should be marked up so that they stand out, that their function becomes clear. It should be easy to read, in particular if everybody adheres to a uniform style for display and markup. It gets easier for users to get used to the style, and also if people would adhere to a uniform style of writing it, it would get easier for people to edit documentation, which is quite important to get it written in the first place. So the formatted documentation should seem easy to use. The language to write it should seem easy to program us. And remember, if there is no documentation, the code is almost unusable, and if it's badly documented, that's about as bad as bad code, because do what you want. From the user's perspective, basically what users see is one tool, the man command they are typing. It finds the manuals in the file system, it transparently calls a formator on them, and it displays the formatted text, typically somewhere in the pager. You may consider that trivial, but actually this is exactly the point where during the last few months there has been considerable progress. Because now, since about August, so since about last month, we have one unified user interface for all of this, where formally several different tools, apropos for searching, man for steering the whole process, mandoc, or nroff for formatting were required. There is no, you can now access all that functionality from just the man command with one set of options, you don't need to learn various other commands. I'll get to that in more detail. Semantic searching, so things like I want to search for cross references or for function names or something like that, is available in production basically since April this year, and the new online interface containing the complete search functionality of the command line tool is online on the first mayor site since July this year. So even though this sounds completely trivial, exactly here progress has been made. From the perspective of manual writers, which are typically programmers, so typically not, especially in open source software, typically not people focusing on writing manuals, it should be easy to write, it should be easy to diff the source code of the manual as opposed to the manual when formatted for the user, should also be easy to read and change, of course it must support semantic markup, it should readily produce various output formats, and it should be easily portable. So I've got a list of requirements, we, that is, both Kristaps and myself, who wrote and maintained the mandoc toolbox and also the OpenBSD developers recommend to use one simple versatile language for all of that, which is the mDoc language, I will get into more detail about that later, what it is, how it works. It is a language with a long tradition. It is based on the rough language that was invented in 1964, 50 years ago. It is the successor to the man traditional manual language that was used since the 70s, since the first version of UNIX actually. It was originally designed beginning of the 90s for 4.4BSD, and nowadays it's mainly implemented both in mandoc and in graph in a very compatible way, so it doesn't really matter which of the two tools you use, you always get the same output. Also it's not really an arcane or niche thing, it's used as the default language by all the mayor BSDs, and since this year also by Illumos, which is quite surprising, they switched in one single commit from not having it at all, not even as an option to using it by default for everything. Now, to get a bit of a taste and a bit of historical background, I'm showing a few historic slides, let's start with a birthday celebration. We have half a century, almost exactly half a century of rough tradition now. It was in 1964 that Jerry Salzer at MIT writing his thesis was tired of typing and retyping on a mechanical typewriter and invented a language to have his thesis typeset on the computer, which was called runoff, later renamed to rough, and the formatting requests, many of the formatting requests he implemented and invented are still used today in modern rough. The machine you see here is the computer he was using, an IBM mainframe. Now, that guy is advertising something that is 50 years old, why do we still use it? Sounds strange. Here you have some source code. Actually, the source code of the next slide I'm going to show. The slides you see here are written in rough. You see, most of it is just text. You can read it almost as if it were just a text file, but in between you have these formatting commands, requests and macros, like here saying it's beginning a list, it has a list item, and also some of it is physical formatting, make it a bit bigger or have a bit of space in between, so that can easily be hand edited. So that's how the former slide looks when it's formatted. It looks unobtrusive. It doesn't hide or model the actual text. It harmonizes very well with diff, quite to the contrary, if you were doing it in XML. It allows high quality output in multiple formats. It works with simple, fast, portable, readily available tools, and there is no heavyweight or cumbersome tool chains anywhere in the vicinity. In particular, you don't need any kind of XML for doing that. In one word, keep it stupidly simple. That's really important. Because neither the people, that's what you always need to keep in mind. Neither the people reading the documentation nor the people writing them want to focus on the documentation. They want to focus on the content or on the programming. Now, I've talked a bit about the raw language, but how did it happen that it became connected to manuals, the documentation? That goes back to the very first version of Research Unix. These are Ken Thompson and Dennis Ritchie with their PDP-11 computer at Bell Labs. And when they prepared the first version, they rewrote rough in PDP-11 assembler and used that to format their manuals. The section headers they introduced, names, synopsis, description, and so on, are still in use today. They were the right thing right from the start. Then in later versions of Research Unix, since V4, they had macros that were a bit like what we have today. And in V7, the main language was actually invented that is still the main language used today for the Linux manual page project. The BSDs are no longer using it. They are a bit more modern. But Linux manual pages are still using the 79 language from Bell Labs. You might wonder, OK, now he's arrived in the 70s. Was there any progress in the last 35 years? And yes, indeed there was. For V4-BSD, the nearly final Berkeley release, a new formatting language was invented mainly to support semantic markup. And Cynthia Livingstone of USENIX rewrote all the BSD manuals from Man to M-Doc, which is the basis of the manuals we are still using today for the basic system utilities. That language has considerable expressive power. We will see some examples later for semantic markup. It works in practice as a standalone language. So when you're writing your manuals with M-Doc, you only need M-Doc on nothing else. By contrast, if you're using the older man language, then you regularly need low-level Roth instructions in between because that language is not complete, not standalone. Consequently, an M-Doc document is much easier on the eye than a Man document. It has a more uniform view to it. And also portability is no longer an issue because even if you are writing documentation for a legacy system that doesn't have an M-Doc support, the Man-Doc toolbox provides a converter. I'll get to more detail later. And of course, the semantic markup you get with this M-Doc language facilitates semantic searching, which will also be one of the main topics of this tutorial. This is some sample M-Doc code. Again, you see most of it is just plain text. But then in between, you have semantic instructions. Like here, it is inserting the name of the utility. Here, it has an internal cross-reference to another section. Here, it has an external cross-reference to another manual page. And there is a prologue we'll get to more detail just to have a first impression. To summarize, regarding the classic documentation formats, three things really, the rough input syntax, the M-Doc semantic markup, and the Man presentational output format have proven timeless. Because they are so simple and so efficient, many people have tried to come up with a better concept. But basically, all of these alternative concepts failed in some way or other. The most prominent failure probably is the info utility. Recently, Eric is Raymond. And people say, Richard Stallman have to agree to retire it and to replace it with something else. Now, given that M-Doc is really a good choice as a language, we need tools for it. Of course, the traditional tools like rough work, but M-Doc has a lot of advantages. Most of them are functional. You have all the functionality in one small binary. It contains all the searching facilities. It contains handling of all the input formats that typically appear in manual pages. It really produces various output formats. It has integrated support for M-Doc to Man conversion, in case you need it. It includes all the various commands, like Man-Doc, Man sends a few months, apropos what is Mac what is. It is completely free. So in particular, there is no GPL code inside. It is lightweight. Particularly, you don't need a C++ compiler. It is portable. It is small, much smaller than graph, actually. And it is very fast. Even graph as a typesetting system is extremely fast. Now, for manuals, you don't need a full typesetting system. So writing that thing in pure C, we were able to get, for typical M-Doc pages, another factor of five in speed. For pages like a shell manual, even on a notebook, you really notice that. Formating with M-Doc is much faster than. The thing those Linux guys are aiming at, they'll probably replace info with an ASCII-Doc DocBook toolchain. The few times that I experimented with that toolchain, it was typically, for the things you, as a developer, you usually have to do after editing manual, it was typically 100 times slower than the typical things you have to do with M-Doc. That's really amazing. All this, the basic functionality of M-Doc, is no longer new. I already presented that three years ago at BSD-CAN. So by now, it's quite mature. And you can really rely on it in production. Yes, so much for the introduction, a bit of motivation, a bit of history. Let me give a brief overview of what my ideas are, what we could do here together today. First, if there is a need, I'll give a small introduction to the M-Doc formatting language. Then I'll show you a bit about how to use that to get manuals for portable software packages. A bit larger part will be about quality control for existing manuals. How do I approach an existing code base and check whether documentation is really OK? And after that, we can have a hands-on working phase where you really can get your fingers dirty experimenting with the stuff. Before I go on to present a bit about searching and displaying manuals, that's the newest stuff we have in this tutorial. And after the coffee break, we go to system integration. How do you use the M-Doc tool set as a documentation formator in a base system or for ports? After that, again, we can work a bit. And then I'll wrap up about the status in the BSD, status in Linux, and what should be the next steps. So I'm hoping that we are through at around 5.30. Good. Before we really get into it, I propose that we have a very brief introduction of ourselves, just about four sentences per person. Just state your name. What do you think your main role is, like whether you consider yourself an application programmer, a technical writer, an operating system developer, a system administrator, a BSD user, or whatever? Your project affiliation, if any. And what is your most important goal for this tutorial? So maybe let me start. My name is Ingo Schwarze. I'm the current Mandoch project lead at M-Doc ML, BSD LV. I'm also an open BSD operating system developer, dabbling here and there in user land, in the C-Lip, in various utilities, wherever there is a need. And my goal for today is to show you the current state of the art in BSD system documentation and to help you with any questions you might have in that area. Now, I planned to say that it's not really a good idea that I start, because the person who should really start this and who actually started this is Christopher Johnson. I think he will arrive at the latest for the working phases, and then maybe he's the guy who wrote Mandoch originally, and he will be around for the working phases for your questions, too. But yeah, let's perhaps just. Part of that is behind writing and understanding manuals can help us in the way of translating the QDI work. Yeah, good. Thanks. My name is Trifang Dain. I'm also a computer programmer, and let's look at the first look to Mandoch as I have no experience with that at the moment. OK. Hi, my name is Ben Lissent. I'm a PC user mainly, but I do want to get into technical writing, system administration, and maybe some programming. And I've actually happened upon the other talks about this subject, and was very interested, and did actually a little bit of reading out on Mandoch. And I'm quite fascinated with how it works. And yeah, so I'd like to deepen my knowledge. Nice. I'm studying and I come on the front of the office, so my boss will work and help to make it work. I'm a user of Photobased, but I'm interested in programming and system administration as well. I once wrote a 10,000-word user manual in Microsoft Word for a work project. So I have an interest in Mandoch for that reason. But also, I think that generally the manual systems are something that we probably take for granted, and we kind of forget that it's there. We just rely on it so much, and I'm just interested to learn more about what's going on behind the scenes. I'm a system administrator, and I'm the editor, who today is to learn how Mandoch's are created and maintained and up. So I'm a part of that. I'm a system administrator. And writing long and complicated shell scripts, and I want to be able to document my work so that the colleagues can, by a man, see what I've done for them. Okay. That's good. I'm a system administrator. Do you have, Steve? I'm primarily a systems administrator. I've been working around with various students for quite a while. I've done quite a bit of technical documentation, including a lot of quite a bit of battle with several tools that didn't go mentioned in the intro. But I've never actually done too much band pages so let's go here for it. Okay. So if I go back to this page, then as far as I understand, the first two points are probably really important for most of you. So it would perhaps make sense that I expand a bit on those. And these two are probably not so important because I understand there are no really free BSD or net BSD operating system developers around. So you are not likely to switch those systems tomorrow to use some different basic tools. But I'll not cut that out completely because many of you said that you are interested in learning how the basics work and how stuff fits together. And of course those are points where you can run a bit. But maybe we should make this phase a bit longer so you can really get used to it. Yeah, okay. Good. So writing manuals. How many of you already edited a manual page at some point in their life? So took an existing manual page and changed something in it. One or two, okay. Three here. Not very many. How many of you already wrote a manual page from scratch? So starting from nothing and wrote a completely new one? Nobody. Okay, so getting the basics is really important, I think. How many of you knew before coming here that there are two completely different languages for writing manuals, MAN and MDoc? Okay, most of... Oh, half of you knew that. Interesting, yeah. Okay. Yeah, so let's get started with the basics of the MDoc language. Of course, there are long introductions, tutorials, manuals about this language. But the basic concepts you need to understand before you can start writing almost fit on one slide. So my goal here is that as far as we can, we get to really write stuff down and try it out and to learn it that way. But what you do need to know before is the distinction of text lines and request macro lines. So the request macro lines starting with a dot. I already mentioned that a few times. If you have a macro line, the macro, in this case, the section header, .sh macro, can have arguments. The arguments follow the macro and are separated by white space. So here you have an example of a macro with one argument and here the last one you have an example with two arguments. The dot, the full stop ending the sentence is here actually put as a separate argument so that the formator understands that it has to put it right after the formated version of the output of this macro. That's quite typical. If you have punctuation, you usually put the punctuation as the last argument on the preceding macro line. In case the arguments to the macro contain white space that would otherwise delimit arguments, you can quote the whole argument and all this will be taken as a single argument here to function macro. So you need to distinguish text lines and macro lines. You need to understand that macros can optionally take arguments, not all do. Some go without arguments and you need to understand that arguments are separated by white space and in case they contain white space you have to quote them. Of course there are lots and lots more details about the syntax of the Roth language. Roth is actually among the more quirky languages. In case you ever need any details about the Roth language you can look them up in the Roth manual but for now that should get you started on the basic syntax. So the next thing to worry about is to how structure your manual page. Every manual page starts with a prologue I'll get into more detail later and then has multiple sections so the content of the manual page is organized into sections. I'll also get to that. The prologue is actually quite easy to grasp because it's always the same macros in the same order. You first put the date of the last change of the manual with the DD document date macro then the document title and the section number this is now the section in the manual like section one, user utility, section two system called section three, library functions and so on. Here the document title is always in all caps such that it looks the same in all manuals. The operating system macro is usually just left blank and filled in by the operating system by default. Then you have the first section header which is always name. So the fourth line in every manual is always exactly the same line, .shname and the name section contains the name of the page again now in proper capitalization, particularly in plural manuals it's important to get the capitalization right here and then the one line description. Here you see a typical example. The cat manual in OpenBSD was last changed July last year. It is section one, so user utility. Here you have the name in capitals. Here you have it as it stands usually and the one line description. What you should remember after writing your prologue count the lines. If you have six lines it's probably complete. If you have less than six lines look into the manual what you have missed. Now after the prologue and after the name section which is always the first section in most manuals you have the synopsis section. The point about the synopsis section is to document only the syntax, never any semantics, just syntax. For user utilities for section one manuals most of the time you can survive with these four macros. You obviously need the name of the utility. You most of the time have some command line flags, options and some command line arguments and usually some of these are optional. So here is a simple synopsis section. What you also see here is that there is no text inside. It's just semantically stating these are the arguments, these are the flags, this is optional, this isn't optional. So you don't describe anything here. You also don't worry about formatting. There is nothing like make this bold or make this italic. It's just specifying the syntax and then the formatting is done completely automatically. I'll take this example to the next slide and explain a bit about the structure of the language. You see here that you actually have two macro calls on the same line. The first as usual with a dot in front of it. The second one without the dot. In that case we say the OP, the optional macro is calling the FL, the flag macro. So the optional macro is a so-called enclosure macro. It opens a scope and then the scope can contain some text, some other macros and at some point the scope closes again a bit like in SGML or XML elements that can nest. The one kind of block macros are those that close implicitly at the end of the line. So the scope of the OP macro just extends to the end of the input line and that's it. One other thing you see here, some macros have default arguments like here the AR, the argument macro is giving without any arguments and in the formatted output you still see file dot dot dot which is what AR produces by default if you don't give arguments. So that's what you need to get started with the synopsis of a user command. If you need anything more, look into the manual I'll go to get to some more detail where you can look there for what. It might be that you are not writing a manual for a user command but for a library function. In that case you need different macros in the synopsis like include files, function declaration, oh hi Chris Debs. I'll include files, function declarations, function arguments, function types, such stuff. This is the list of macros you basically need in each and every function manual page. Here is a typical example and how it will format. One thing you can see here again is a block macro having a scope, this fo. But now it doesn't just extend to the end of the line but there are multiple lines inside giving the various function arguments and it only closes when you specify the explicit companion closure macro. So there are two types of block macros, implicit one extending to the end of the lines and explicit ones going on and on until you close them explicitly. Again a bit like in SGML or XML. In this case the difference is that in XML you have to close everything that you open so you always have the text clobbered with those closure macros. And here in most cases you get away with the implicit macros like this FA that just extends to the end of the line so clobbers the text less. Good. Now we have the prologue, the name section, the synopsis section. Now we are done with the syntax. Now we have to start to talk about semantics and that's what the description section is for. So at the beginning of this, the next section is the description section and at the beginning of the description section the first thing you do is explain the purpose of the utility. What is this for? And then you describe the syntax and the semantics of all the features. So that's really the meat of the manual page, the description section. Be careful to be at the same time complete and concise. So what you should definitely not do at the beginning of the description section is talk about history. Oh, I wrote this utility because I thought it would be useful for these kinds of people. If you want to write that, the proper place is right down here in the history section or somewhere. There it may be interesting for people but right at the top at the description it's just annoying. In case your description gets very long even though you're concise you can split it into multiple sections. The typical example is a shell manual that has lots and lots of sections but for smaller tools you'll probably not need that and you should stick to the standard section names because that makes it easier for users. If you already know which sections to expect when opening a manual page you more easily find your way around it. I'm not going to explain all these sections right now because chances are you won't remember all of them anyway. What you should know is that various information typically goes into various other sections. For example, the return value goes into a specific section and some of these sections only apply to utilities and some only apply to functions but in case you wonder where to put some information for your specific case again just look at the manual it lists all those sections it lists what typically goes into those sections and it also lists which macros to use there so you can really look all that up at the time you need it and then get more used to it. Now we basically have seen a first overview of everything that might be needed the preamble, the name section, the synopsis section, the description and there are some more sections. Now in case we get stuck somewhere where do we look? I will reset the most important resource is the MLOG manual. When you wonder which section to use for some information you want to convey you look into the manual structure section of that manual. When you wonder which macros to use to mark up some specific syntax element you look into the macro overview which is basically just one page where you have all the macros that are not deprecated. I should perhaps show that I said we should focus a bit more on this stuff because it will be more useful to you. So I'm typing man mdoc which will probably show the wrong thing, yes it shows the mono documentation management tool that's not what we want so I call a man 7 mdoc that's the markup language. So here right the second chapter is the manual structure where it tells you which the usual ordering of the sections in a manual page is and also tells you in which kinds of manual pages to use or not use those sections. Then it explains the various sections. For example, right now I talked a bit about the synopsis sections because that's so important that's really the thing people look at first and here even in the manual itself you see some examples to write that. Now skipping all those sections the very next thing you find is the macro overview and here you have all the macros recommended for use. So the most important ones are typically the semantic markup macros and you see the lists are not really that long for command line utilities for function libraries there you can look I need something for this or that which one fits and then if you have a candidate and think that one might be good for what I'm trying to do then you can look at the macro reference right behind how that particular macro you consider using what the syntax is and whether it fits to the examples given there. So that's where you usually get up to speed by trying to write something and looking at the manual to find the right one. Another good way to get started is to just take an open BSD installation and look at the manuals in the base system how did other people write documentation for similar stuff there you will find good examples of macro usage of standard wordings good examples where to put stuff and so on. Once you have written something even if it's not yet quite complete please run the command mandoc-tlint on it no matter whether you are running open BSD, free BSD or net BSD or dragonfly or ilumos it should always be available by default without installing anything. That catches most syntax errors telling you in that line in that column something is weird in the following way please fix it it may even provide some hints on style now and then so up to here it's really the standard tools that you always use below it's more when stuff gets really difficult in case you read the M-Doc manual and still wonder it doesn't seem quite clear then sometimes the graph M-Doc manual just because it has different wording might help to get it straight but usually the standard M-Doc our M-Doc manual is stricter than the graph one so it's more to get a second view a second angle on the same thing that might help what may also help is Christophe's full tutorial at manpages.bsd.lv so if it turns out that you get into the habit of writing man pages you should definitely read that at some point to get a full walk through and if after trying all that questions still remain you are always welcome to ask questions on the discuss at M-Doc ML, BSD, LV mailing list, both Christophe's and myself are always around there and various other developers like Wiz of NetBSD and Ulrich Boerlein of FreeBSD so usually we can help you there so this is a very important slide by the way when we get into the working phase you can download these slides from the URL written on the board there so if anything got lost you can just look it up what is also on that in that directory is a handout that I've written down that is more suited to self-study than a set of slides so in full sentences and it also contains a lot more information it also contains the content of the previous talks in part updated all in one one write up okay so oh yeah that's the next the next chapter any questions so far regarding how to write manual pages the question will probably come up in the hands-on working phase then I'd suggest before I go on to the next much shorter section Christophe's would you mind shortly presenting yourself we had a small round just saying the name the main role and affiliation and what your goal for the tutorial is so that people get to know you that is yeah it is yeah so this is the homepage for part of a mandog if you ever forget it you can just start from the openBSD homepage and right here besides openSSH and LibreSSL you have the link to the to the mDocML page and here below history you have the link to the history of the Unix manuals that Christophe's wrote up where you can see all the all the various version of Roth that existed since the since the 60s who wrote it and ported it to which machines it's very nice stuff you gotta do something well Cynthia is the person who translated the manuals from man to mDoc but I still don't know who implemented the mDoc macros originally it must have been somebody of the computer systems research group I'll probably ask Kirk what he knows about it he's around right now but we are getting a bit off track he is interesting but let's get back to writing manuals because that's what you are here for ok so before we get wrong talk before we get into the working phase yeah a bit behind but not badly very briefly how do you use mDoc and the mDoc tool for portable manuals so imagine you have some small software package of course you are documenting it but you expect that people are using it everywhere on linux and bsds on solaris on commercial unixes wherever so in the past it was a problem to use the mDoc language because some operating systems after 25 years still don't support it that problem is gone just write your manuals in mDoc and then if you prepare a distribution table run mDoc minus tman once on your mDoc manual it produces man output and then you package up both versions and just that at install time the configure script decide if the target system you are just now installing on supports mDoc install mDoc if it doesn't install man it's basically all there is to get it portable when I first mentioned that in a conference I thought there would be a lot to explain about it but actually it all fits almost on on one slide let's look at it just to demonstrate how easy it is in the with the example of the first portable software package that ever did it sudo maintained by Todd Miller in the project makefile there are two maintainer targets one of the target says if you want to get generate a man version of the sudo manual start from the mDoc version of the sudo manual and run the command man doc minus t man on it if you want to generate a preformatted version of the manual start from the mDoc version 2 mind you not from the man version from the original mDoc version run the standard man doc command on it and you get the preformatted version then put all that the mDoc version, the man version and the preformatted version touble and then long break you put it on the website and someone downloads it and wants to install it runs the configs script and then what the configs script does is scan the system is there a man doc binary around somewhere if the configs script finds man doc it knows mDoc is supported so it installs the good mDoc pages if the configs script doesn't find man doc it looks for nroth the traditional tool for formatting manual pages if that isn't there either well probably windows then don't know whatever it installs the preformatted pages chances as people can at least read plain text if it does find nroth it tries to run nroth with the mDoc macro set if that works it can again install the nice mDoc pages and if it doesn't it falls back to the man pages now if you want to provide a bit of sugar you can provide with mDoc or with man options to the configs script so that people can manually override this logic basically all two lines in the project make file as maintainer targets and about a hundred lines also to implement this logic and portable shell code in the configs script and then you can write your manuals in mDoc and they work everywhere even on HP Unix or Solaris 9 or whatever yes depends on your software package so if you use autoconf then you will do all this with autoconf macros there are no standard ones but you can just steal the ones from sudo if you use something else whatever build system you like for example a handwritten configs script which works quite well for example what we are doing in the portable mDoc distribution then you just write those tests yourself by hand as you wish good so that's all about how to prepare portable documentation basically any questions with respect to that no then before getting into the the working phase let's cover one more topic so that you have a bit of choice what you want to work on quality control so far the situation I talked about was we have some software and no documentation yet we are going to write documentation now I am looking at something different I not only have the software but also have the documentation and even documentation written in mDoc already but now I am wondering whether it's any good or whether it needs improvement so which goals do we have in quality controls for manuals the most important first one is we have to make sure there are no fatal errors we have to make sure each manual actually produces output when you run the formator on it it could crash the formator or cause a fatal error even if there is some output we have to make sure that the output actually contains all the intended text there might be syntax errors in the manual source code that drops part of the text so if the user looks at it something is missing, it would be bad we have to catch severe formatting errors like someone mistyped some width and it's all in two columns on the far right it's completely unreadable you don't want that in a manual we should try to catch typos and stylistic glitches obviously we might try to improve portability but please don't overdo that if you try to write your manuals in a way that tools from the 1980s still properly format them then your manual source code will not be quite readable so portability is good but not with tools from 30 years ago you definitely should try to get the code robust so that it doesn't doesn't crash or small changes a quite important thing is to unify the style the manuals are displayed so that users don't get confused because each manual looks completely different than the others you could even try to unify a bit the MLOC or something such that if anybody else is ever editing the manual the way the source code is written looks familiar and people get up to speed easily with your manuals and last thing checking the quality of existing manuals as a by-product often finds bugs in the tools bugs in the mandoc formator so that's also a reason it's a good thing to do report them by the way don't ignore them you see these are quite different goals and quite many so there are various tools to help with them of course some of these goals can't really be automated like finding stylistic problems wording problems in the manuals that's a manual task you really need to use your judgment the important tool is mandoc-tlint how does that work ah ok no one thing one thing before that mandoc-tlint doesn't produce formatted output but it produces error and warning messages or even better if you have a good manual it doesn't produce anything at all now it's important to to really get those warning and error messages right it's obvious you don't want fatal errors because then if the user tries to format the manual there is no output it doesn't work at all so we are trying to minimize that when mandoc started in 2008-2009 we had lots and lots of fatal errors so even on real-world manuals what happened that you typed mandoc this manual and then you got fatal error and no output and that's of course almost as bad as having no manual in the first place that doesn't happen anymore or hardly ever but even if you don't have fatal errors it's important that if something is wrong in a manual it's really reported because otherwise people won't realize and can't fix it on the other hand reporting too many errors and warnings is bad as well because then users get annoyed and just switch the whole thing off and don't use it so it's kind of a small small ridge you can fall down to the one side be too wordy or fall down to the other side and fail to report what is important and it's even worse you can fall down both sides at once fail to report the important stuff but report lots of unimportant things that happened to mandoc in the beginning too but we have done several cleanups again and again improved the messages so by now almost nothing that is reported is false positives some of the the warnings may be ignored in some cases but most are really real and a wide range of problems is caught and by the way all this applies to almost any software so if you have any software reporting warning and errors you always have this problem that you really need to report everything that is wrong but you shouldn't be too wordy and one thing people sometimes do in that respect is to add switches so they invent 20 warning classes that can be individually switched on and off compilers are a great example of that it's a very bad idea because nobody will remember all those switches people will get annoyed will not really use them try to get as few switches as possible that just works that just reports what is relevant in mandoc we have exactly three levels that can be selected select only fatal errors that's the default select errors so something is severely misformatted or information is lost or show all warnings but no more switches so you come in with a large codebase the whole 3DSD operating system and you wonder what could I improve here the first thing is what you should do is run this command tlint w fatal overall manuals and hopefully none of the manuals will trigger an error at that level if any do that is what you should fix first because those are the manuals that can't be looked at at all in case you are just looking at a handful of manuals you can skip that gap those fatal errors will show up in the next step anyway by now there is basically only one class of fatal errors left and all those are related to file inclusion so there are two types of file inclusion one is the unsafe bd file macro which means just include a file verbatim from the operating system that is not supported by mandoc on purpose because it is unsafe we want mandoc to be safe for a system administrator to run in a root shell even when there are bystanders so now if a malicious user writes a manual that contains an instruction please include the master password file here verbatim and tricks the system administrator to run mandoc on that bad manual while looking over his shoulder he will see the encrypted passwords so we don't support bd file and if your manual contains that mandoc will abort similarly the raw.so file inclusion request which is extensively used by the xorg manual is only supported in the way actually needed that is only with relative parts not involving going up to parent directories for the same reason we don't want to include sensitive files so that's basically the only kind of fatal errors left that you might encounter in practice by the way these mountains here are the ones that give the name to this room so this kind of fatal room Pirin next step and that's the most important step in quality control is catching errors an error in mandoc language means either there is a serious risk that content gets lost or there is a serious risk that formatting is completely garbled and looks really bad so you run mandoc tlint in the error mode and almost always you need to fix anything that shows up there in a good quality code base it will be very little in a bad quality code base you can have several errors each on average it really depends the typical types are unencoded non-asky characters in input those are obviously syntax errors and unportable graph by the way doesn't report that it just does whatever happens with it unknown or mistyped macro names very dangerous because usually unknown macros are just ignored including the arguments which is information loss then blocking errors like opening a block never closing it again closing a block that wasn't ever opened and such stuff and severe issues with arguments if there are macros which are lacking essential arguments so that they don't work at all or if there are less arguments that may contain important information but are ignored because the macro doesn't support so many arguments those are errors so if you see errors fix them final step once you have cleaned up all the errors run mandoc tlint again without any minus w we argument then it will report everything it considers problematic in a low quality tree expect lots of output in a good quality tree so for example the open bsd section 2 manuals the system call manuals that's about 150 manuals also have grant total 30 warnings or something like that you really can get a tree warning clean there are sometimes a few false positives so it's usually good to fix most of them but use your judgment do not blindly just like with compiler warnings do not blindly there are various classes it's typically when stuff is used that doesn't completely garble formatting and doesn't lose information but is bad style or not quite portable for various reasons by the way all the messages you can expect are listed in the portable mandoc manual itself so one way to get that and to get explanations of the messages is to just go to the mandoc home page and here look at the mandoc manual if your operating system doesn't install the full manual and here at the bottom you have the list of all the all the warnings wrong one sorry so now on your tree you have looked for fatal errors fixed them right away you have worked through all the usual errors and finally you had to look at warnings and maybe fix a few of them there are a few other tools that I'll just mention briefly in that BSD they have been using the tool M.Doclin written by Thomas Klausner with for a long time it's quite similar to M.Doclin so it also focuses on catching syntax errors it used to have it used to catch additional errors that M.Doclin didn't catch but most of these have been integrated into M.Doclin during the last few months so it's not it may sometimes find a bit more but it also has a much higher noise noise to signal ratio so that's more for if you want to be very careful you can use that too it's written in pearl it should basically work everywhere in that BSD there is a port in open BSD it's in the regression suits to run that one just give the names of the file you want to check on the command line and you usually don't need options except to suppress messages you don't want yeah a quite useful tool comes from free BSD it was written by Warren Block it has a quite different focus it doesn't know so much about syntax but it finds surprisingly many bad spellings and instances of bad styles so it has some kind of very simple artificial intelligence I'd say not really but a bit in that direction he named it Egor because he thinks it's a useful summon like the Egor in Frankenstein it is available both in free BSD and open BSD as a port and the way of using it is exactly the same as for M.Doc just give the file names and if it's too noisy you can shut it up a bit by using command line options so that's really a good thing to run after M.Doc T.Lint if you if you want to yeah and finally well probably most of you won't do that in the M.Doc repository there's a very simple shell script to actually compare the output of M.Doc against the output of graph that's most useful for finding formata bugs but sometimes when you have a lot of experience it can also be used to find constructions that are less portable I'm just mentioning it it's probably more advanced and I think I will skip this one for now because it's even more advanced go on just to the exercises before people start sleeping I have I have sheets of exercises prepared here it's always two sheets of paper you can pass them around the first page is some general remarks that I'll also give orally there is a very large number of exercises the intention is not that you work on all of them there is not enough time the intention is that you choose what interests you specifically so do not waste your time reading all the four pages of instructions just study those instructions you actually want to work on feel free to either work alone or with partners two or three people together if you want to just as you like and very important do not be shy to ask questions either to the people sitting beside you or to anything you remember might know something about the area in particular feel free to come to Kristaps or to me to ask questions what you really can do here and can't do so much at home is communicate you have all the time and you want at home but talking to others really only works here and then in case you run into anything that is repetitive or time consuming just do the first bit of it and skip the rest and go on to the next section such that you have a chance to if there are any problems to find as many problems as possible that you can work out here and if you want to probably not all of you but if you think you find something really interesting for others speak to me you might have a slot of one two or three minutes at the end to present it with your findings but you don't need to nobody will be forced to present anything so there are about seven or eight different exercises all of them are real world exercises so I have prepared no exercises of the style of a classroom style it's always take some real and work on it you can choose any that seems interesting to you but my recommendation is if you are primarily a software developer you might prefer to choose the one writing manual from scratch or translating a manual from Man to MDoc or checking existing manuals depending on what the software you are actually working on has so if it has no manual at all yet like you sit for your shell script system administration that might be good for you if you already maintain a software project that has MDoc manuals maybe this one might be a good one for probably this one is not recommended for people here if you are a system administrator then the things about locations and formats might be interesting but I would say no given your feedback where you come from I would say focusing on these in this working phase is probably better and getting to these in the second working phase when I've presented the things on searching is probably better so for now the exercises 1.1 to 2.2 are probably the best ones to work on because that's on stuff I've already presented and then we make the second phase a bit longer in case you're lost where to find the manuals the example manuals or the software a few of you might possibly talk to the people saying there are system administrators and have actual software to document and team up to work on that so do any of you already have some software in mind that you are using regularly or where you know is any of you using a BSD operating system regularly like free BSD or net BSD so that's also if you have some software you really care about I'd recommend starting with that seeing whether it has manual pages if not work on it if it has manual pages doing quality checks on them if that's not the case you can try with your operating system I'll probably also go around and try to help you find some, yeah? I can make some suggestions of things you use every day where the manual pages are awful that is good to get people's that yes the GD like Thomas Brunneros the X-Files the X-Files are in-base in the medical role what else OpenGL but that's in John Fogg X-Lib with the first one if anybody says mathematics basically anything that has to do with mathematics is bad you know, late back for example for psychiatrists what are the things you use every day I mean just think about the utility you use we can't like math I mean this is bad let me see Lipset is do you have an example function from Lipset? okay so that what is that? that is already M-Doc is it readable or everything? well would we want I'm not doing the exercise right now but if Lipset is which format? it does actually somewhere for the X-Files in M-Doc years ago because I was so angry that they did not have a good X-Files GD is also man right now I think yeah so is the X-Files useful? in base although it's really useful but I wouldn't use it because you have to jump roots and the GD is just was a CGI program I think PHP uses the GD inside but it's manual I mean you have to go to this huge change of the search function in the browser and I cry all the time I think for you it's probably easy you will probably work on your own scripts and get something done there you have an idea? good so that's an advanced topic yeah that's not possible okay that's bad what do I do now? that's the one in OpenBSD works I know it works but you have to read it okay don't look at me I actually have an idea you have an idea? I will take some tool from check her quality okay suggestion some something you should probably pay attention it's bit more difficult in OpenBSD than in other systems because yeah I have something you should probably exclude you should probably not look at the C library and system call manuals those are excellent quality and so what might be useful is something that you use and probably more in the area of system administration or network demons or something like that also the base systems stuff is quite good okay no idea you have an idea too? I'm just having a look at some of the other manpages that I'm using although I've never really observed that they suck don't think about it do you? it's just not as terrible normally I'm just battling with what am I trying to learn how is the quality of this document yeah so which other ones you're looking at? look at the CVS one I guess the SSH one is probably not too good to look at because Damien is probably really strict with that one not really SSH one is but I don't it's a tricky thing because they don't use the potability stuff I've presented yet so they still have an old potability script SSH causes quirks so you could get detracked there okay you have an idea? yeah there is a utility that I like the PC is bigger there are a lot of frequencies and it's not it's not available as an open base I had to think about if I could port it it would be easier and I thought maybe I'll start with the man page because it's compressed and not in my dock great so you already have some ideas? no we have no laptop so it's that's a problem okay write a manual from scratch on paper but maybe you could team up with somebody else so so far I found I found one person who is working on free BSD manual checks one person who is working on open BSD manual checks and you are preparing an amount to M-Dock translation so that's probably the best thing you can you can do if those seemed interesting to you then I can show you who is working on it M-Dock that would be the colleague writing behind you yeah the free BSD I think that was the colleague near the wall word of sailing the colleague near the wall word of sailing is working on free BSD should I ask him? probably I remember rightly that you are working on quality checks on free BSD manuals there is one participant who didn't bring a notebook who is also interested maybe you would like to team up so it's a leak over there that's nice okay you found something to work on? no it's something what we will probably just look on later it's not what we are doing it's more of an interest what do you want here I will read through this and maybe I will ask him if none of the exercises are real to you one useful thing you could also do is look through Christophe's tutorial because that's if your interest is really to learn writing manuals and to see how it's done and what's important and reading that thing in this 20 minutes is also something that makes sense it was read by a few people oh yes anything that is hard to understand or anything where you suspect something might be wrong from the level of typos up to really content errors Christophe's is around and would be very glad to get feedback yeah that is sorry you already have some idea I was just looking at the man page for Node.js oh it's a lot it's actually very brief it's like most of it is just hasn't actually been formatted it's just been left yeah so that is a man page you see here the .sh of the capital so if you run man doc minus t lint on that one you will really get a lot of output right I'm sure I was actually just looking to look over your shoulder oh sure it looks like on OS 10 this is what comes up for MDoc is that okay no that is the graph MDoc manual page written by Werner Lemberg it's also not a bad one if you want to work on OS 10 you don't have man doc installed by default so you would have to to install it it should be just to compile it you don't need to install it relatively easy you download the you download the you download the from MDoc ML probably just with the browser MDoc or start from the OpenBSD yeah there it is back one okay you can just delete the pass after the domain one more write that and then use release turbo you extract that into a temporary directory just run it's like a last okay just run configure and make okay perhaps you speak up when you have the tab all extracted then I'll have again you're grating on the OpenBSD manuals are quite good what you could try if you insist on looking at OpenBSD manuals is installing the Igor port and looking at stuff with Igor chances are you will find typos with that one it's probably not not a thing for working on for a long time but getting a bit of feel of Igor could be interesting okay you're finding something well no problem if you still need time to get that going did you find anything interesting so far so far of the memory is that I run London they have nothing even with all the levels of warnings so now I'm trying to run everything in this directory to see if there's something which is wrong with some non-file okay you don't get any warnings on the FreeBSD manuals okay yeah it's a prize but that's good FreeBSD is right on the point of switching from installing oh wait what did they install no I mean which manuals do they install it's no it's it's unformatted so right now they are formating with graph and they are on the point of switching to formating with mandoc so if they are already in a state where they have almost no warnings that's quite good yeah but still try running on all of them so yes I mean yeah prologue probably and then just going through yes at first I would write the product from scratch and the name section from scratch okay and then probably for the synopsis which is the next thing I would look at the formatted output of the man manual and just write it down from scratch too and after that I would probably look both at the source code of the man and at the at the formatted version of man in the main text it's a bit more tricky you really have to mark up lots of things that are not marked up in man but you also sometimes some mark up in man that you don't want to lose so yeah let's forgetting started what's the command to untar a file it's minus x for extract then you can use if it's compressed use z if you want to see the file names while they are being extracted it's v for verbose and then f and you give the file name the f is can I specify a path or is it just find out it will extract in something like mdoc ml dash version number okay so if you a good practice is to first exchange the x with the t for table of contents then you see where it will extract to and if that pleases you yeah return so that's where it would extract to if that pleases you just do it with an x now you can change into it right and just type dot configure dot slash configure sorry what is that how did you get old table oh wait no that's okay just type make let's see whether that works it's not the current release isn't fully okay type vimac file please with capital M and let's see what we have to do for for mandoc that we don't need not we don't need okay go down here I haven't used vi in a long time oh just move you could also use a different editor as you like build targets you can command that out insert mode and then what you want to insert yes and escape to get out of it again right so you don't need that don't need this one yeah but that's don't need to bother static you don't need that so it should I would hope that it works now w wq it is yeah okay try make again looks like the same what does it complain about Darwin come up a lot so I'm wondering if can you show the beginning of the error messages instead of the end it's strange maybe you should get chris dubs to help chris dubs you do know how to compile m.ml on mac os right yeah it should so it's just a distribution table if you can help the colleague then I can look at something else thank you okay doesn't make sense yes that makes sense but then I tried with the ego and then it falls about this data macro and I tried to put today's data there but I still think that I forgot to say that about ego ego is not at all afraid about false positives so there is a lot of noise in ego I would guess that about half of what ego reports is actual errors and the rest is noise and that's a typical case of noise so oh great thanks yeah no that's a bug in ego yes ego insists on a date format on a specific date format but you shouldn't put that just ignore that particular ego message no what ego complains about is the mdoc date it doesn't know the mdoc date but you mustn't delete that but the other things they are real so also the the don't it's a good idea to not use these contractions so you're very welcome to just prepare a complete patch of that send me a mail and I will commit it and credit you if you like of course it's not a requirement but if you like we can do that I'd like to I will first learn that the game of course I should be aware of the of the tool which I'm writing patch for well if it's just formatting on style errors in a manual you can even fix good yeah so you found the first real bugs in games well I've committed to games multiple times okay so after boot look at it after boot is quite an important page but it is one of the very very few pages that doesn't correspond to a specific utility it's a bit like the help page or it's just documenting what you can do after installing the system that is an open view of the system then just type locate after boot oh you don't have it if you want to you can run in a shell in the background sudo sh slash etc slash weekly if you do that it will rebuild the database and then you can use it but the location it is installed to is user share man man 8 after boot dot 8 so that's the okay so please you have the page yeah no I'm not sure I would if it would be the yes generally strongly in freeway sd they prefer to avoid contractions because it's easier to easier to read and I kind of agree with it if it would be the only thing to fix in a manual page I'm not exactly sure I would commit it but I don't have the back of an author name that was not marked up so it can just be yeah sshd actually I reported that and agreed with djm to not fix it right now because it creates some portability problems with the stuff they are using for they have custom stuff for portability that doesn't handle it yet that is actually a known bug you re-found but right okay so just running the mandoc command on it so mandoc file name and perhaps less after it maybe actually mostly just made it really big for you yes that is actually correct that happens automatically the mandoc language will form the name of the page in the title line in bold in man you have to do that manually so if you write a man manual you have to know that usually the name and the title line is formatted bold so you have to put font instructions for a bold font and obviously the manual author didn't do that that's one of the reasons why writing mdoc is easier okay I answered the wrong question if you the way this boldness is achieved is with backspace escapes so what mandoc actually puts into the file is an A backspace and another A then a W backspace another W if you just put that to the console then you just see the last letter printed to each position the backspace sequence is and shows it in bold yes you can use for example hex dump to actually see how it looks like okay you are starting to translate and to get up to speed so if there are any questions speak to others or ask chris but as long as as you are getting on I'm happy let me see