 Good morning, good afternoon, good evening, whatever time is on you may be in and welcome to my open source Summit Japan presentation on some new currently email tools. I'm Frank Rowland currently working for Sony have been for many, many years. And I've been presenting at open source summit Japan ever since it had several other names. And glad to be back again this year. And looking forward to seeing you all in person next year. Feel free to be asking questions as we go through the presentation. This is being pre recorded. So I will be attending live watching in the probably in the chat room, depending on the platform will either be chats or q amp a tab. Feel free to use either one to ask me questions as we go through the presentation. And I'll try to leave room at the end of the presentation for more questions. We'll see how fast I get through the slides. I have quite a few to cover today, but I'll try and make good progress through them. So what was the reason I chose to give this talk why do we care about this issue. When we make changes to the Linux kernel, the way that we do that is by sending in patch emails. And over the years people have asked, is this process efficient. Is it easy to use. My somewhat sarcastic response is there have been some complaints about ease of use over the years. It actually has been a very efficient process and a scale fairly well. So I'm going to give an overview of what the patch email flow looks like in the next slide. And it is a simplified conceptual view. Don't take it as being precise reality, but it shows how patches created, how it's submitted, and then how it is applied. So we start in the upper left hand corner and you start working with your base files, unmodified. And over time you make changes to the files and now you have a set of modified files. Somehow you create a diff of what the files used to look like and what they look like after you made your changes. And the result of that is stored into patch files. You use an email client to send your patch files and they flow from your email client through an email server. And we're going to take the diagonal down to the right path into the internet. And we'll be coming out into someone else's email server and their email clients going to read your emails. And you need to that person will be using their client to extract the patch files. And then they will start with the same exact unmodified files that you started with. They will use some way of applying the patch files end up with the same modified files that you ended up with after making your changes. You'll note that there's an alternate path through the internet coming from the email server. Your emails potentially also are going to a list server who will archive your emails. And that archive can be accessed via web server and the client for that would be the web browser. And you can use the web browser to then save the patch files locally and again use that same process to apply patches. And this is what the process looked like about two years ago. So what could possibly go wrong in such a simple process. And I'm quoting me as saying it's a simple process. I've been corrected after several years of trying to claim that. And the reality is it actually is a fairly complex process as simple as it looks. There's plenty of room for human errors. It's difficult to use and usability is not what we would call good. So this is a concept I like to use. What goes in may not look like what goes out. So you put data into the process and at various points in the process. It may look like your data went through a blender. And what goes into a blender is the same stuff that comes out, but it may not look the same. So those various transformations throughout that entire flow in the previous chart may or may not be under your control. You may be able to choose what email client you use. You may or may not be able to choose what email server you use. So here's our picture we saw before. Let's get lined up. And let's modify that. Let's see where those various points for data can be manipulated with that blender. Quite a few different places. Email clients and servers, the servers, web servers serving content of an archive. Email servers receiving email. Email clients receiving email. Web browsers retrieving patches from a web archive. Lots of potential places to be corrupting your patch emails. So we accepted those data transformations can occur. So historically what we've done is we've tried to fix up those problems. And that manual fix up and turns out to be extra work. And you'll find that each maintainer has their own scripts and processes to solve the issues from those transformations. So as of 2019, we had some get tools that helped somewhat in solving some of those problems. We've provided several commands on the creation and sending end of that flow. Get format patch to create patches, get send email to replace the email client and get am to apply the received patches. So looking at our picture again. We can see where those get tools came into play. On the creation side, we see get format patch creating patch files, get send email replacing the email client. And once patch files have been received, we see get am applying those patch files. As of June 2018, a new tool came on the scene. The lower male archive and Constantine is announcing in this email that it's been through a bit of development and it's pretty solid now. It's official you can start using that tool. And it's a well supported maintain tool. So here's where we left our diagram. We added another change to it with get lower. Now we have lower is applying our list server archive. Before having lower, we were depending upon third party archives that may come ago at any given point. And we could not count on those. And when Laura was created, the data from those third party archives was gathered together. It's quite an extensive archive now of the history before Laura existed even and moving forward, Laura is capturing the emails as they are created. As of January 2020, this year, a new tool arrived. Get lower inbox. And again, this is Constantine's announcement of it. And at the end of January beginning of this year, he's saying that he wrote a quick helper script. And it uses the message, message it from an email to grab the entire thread containing that email. He grabs a thread from the Lord Colonel Gorg archive. And he saves the result as an inbox file. The cool thing about that is that inbox can then be sent into the get AM command and used to apply those patches. This is an early, early effort. This is not really expected to be production quality yet. And so he Constantine is saying please try it out. Let me know how it works. Let me know what I can do to fix it. He's saying it's still a bit raw around the edges. It is, it's more than a proof of concept. It actually is working, but he's still finding corner cases. And so he wants to hear back from us, learning where are their issues and how we can fix it. And improve it at functionality even. So he was setting expectations. This tool is doing a rather difficult task. It's analyzing human formatted emails and human formatted email subjects. The emails are appearing in an indeterminate order. And some emails may be replying to other emails. The timing of when emails arrive is it can be somewhat random. And those emails are being touched by lots of various email clients and servers who can making the various transformation transformations that were causing us problems before. So the expectation should be that get lower inbox should be having some hiccups. It should have some issues, some problems. It should have some failures with certain email threads. And it's very important to note that the tool author constant is very responsive to anybody reporting issues, anybody reporting bugs, and anybody suggesting how to improve the tool. He has a good history of very, very quickly responding. And long comes March. So two and a half months or one and a half months later. He decided to make some changes. And in the course of making those changes. Part of that was using a new name for the tool to make it a clear delineation. So the tool became before instead of get lower inbox. So just if you read about get lower inbox, just do a transformation in your mind that that really is now the before tool. And it's now a Python project, not just a script. So it's moved into a better form. So here we have our diagram as we last saw it when we had added lower. And the next thing is before has been added. So this last portion of the flow is covered by this new tool. So instead of receiving emails through an email server through an email client. We can now use the before tool to access the email archives through a web browser conceptually through a web browser. And optionally in this left box. We can use before to directly apply through a pipeline into get am to apply the patches that before has retrieved. So before is now covering the end of the chain here, receiving the patch emails, potentially applying those patch emails. And that little corner of the diagram might appear to be a fairly small factor, but that small addition has an immense impact on improving the workflow that we face as developers and maintainers. So don't underestimate how powerful and how useful and how transformative this new tool before is. And it turns out that there's even more that before can do do beyond that but I'm not going to be talking about those extra features today. I mentioned that they exist. So the introduced before the current version is 0.5.2. As of November 2020. It's important to note that before requires Python three currently version greater than equal to 3.6. Originally, I was using Python to and get lower inbox the predecessor to be for and quickly found out that that was problematic. So I can attest personally that you do not want to use an old version of Python. It's just not even worth trying to deal with the, the incompatibilities. And the fact that it just before will just work very poorly for you and must often, or quite frequently fail. So use Python three and you'll be in good shape. Before has an extensive help system. And at the top level is just listing out the various sub commands, which is somewhat similar to the way that get works to say before, and then some sub commands so like before inbox, for example. I'm going to talk specifically about inbox, am. I didn't pull PR. I'll try to fit that in we may run out of time, and if I'm going to punt on although I will mention it. So there are these other commands attest a TT verify PR as I mentioned I'm going to try to get to in T why these are more of interest for maintainers that for submitters. And in this presentation, I'm looking more at how submitters used before, not so much how maintainers are to start with the, the first to command the inbox and am. They're very similar but distinct in their in what they do before inbox will fetch a patch series email thread and put it into an inbox file. The important thing about the inbox command is that email thread includes all the reply emails, so includes the original patch series emails submitted into the into the mail list and all the the review replies all the comment replies. On the other hand, there's a type of here this should not be be am that should be before am fetches fetches that same patch series into an inbox file, but only the patches patch series does not include any replies does not include any common emails. And then separately, the cover letter which is patch zero of the series is fetched into a cover file. So that before am creates two files and inbox file and a cover file before inbox simply creates an inbox file. You know, see some examples of how that works. So starting with before inbox. Again, we're going to fetch an entire patch series email thread into an inbox file, including all the reply emails. This is the top portion of the help for the inbox command. So for any sub command you type the command. So before inbox, followed by the dash dash help, instead of putting the help before the sub command. And one thing to note here is that in your usage, one thing that you can supply is a message ID. It's going to be critical to understanding how to use before, and all of its forms. The full help will be will be in a slide in the supporting materials after the end of the talk so if you don't have access to be for directly. You can look in the end slides to see the, the full set of options for the inbox command. So here's an example of using before inbox. We have, I'm doing a little bit of extra commands just to make it easier to show my work. So I'm creating a temporary directory, changing directory into that. So we'll have an empty directory and we'll see any artifacts that'll be easy to see what gets created by the command around the command before inbox. So here's a magical message ID, and it tells us what it's doing. And it tells us that it saved the result into this mbx file. If I do an LS I see that file is the only thing in my current directory. And you'll note that the name of that inbox file is the message ID, the same thing we said, gave it as input with mbx added on to the end. So where do you get an m in a message ID from? How do you know what that magic extensive value is? I'm going to show an example of an email thread and we'll approach the same email thread several times over. So keep using the same message ID. And I'm going to show a screenshot of the patch thread in my Thunderbird email client. Thunderbird is what I use in my daily work for receiving and sending emails. And in the screenshot, the entire patch series thread, including replies, will be inside of a red box. And there's going to be a random email which is inside a blue box. So we'll be, here's what it's kind of look like. And here's my Thunderbird email client. So everything here in the red box is the entire email thread. So we have the original patch emails. There we have one reply. And I've decided that I'm just going to pick randomly the fourth patch file email. And I'm going to get the message ID from that email. So I can use any email at all in this entire thread. I'm just randomly choosing one that's convenient for me to access. So if I look at the email headers of that specific email. You notice that the subject line matches what we just saw on Thunderbird patch four or four. The message message ID header has this long cryptic value. And that long cryptic value is the message ID. And that's where I used in the before command in the previous slide. I'll just keep her using the same message ID and a bunch of following commands and slides. So there are multiple ways of getting the message ID. I'm not going to tell you for every single different email client how to do it. But since I use Thunderbird, I know a few ways there. And typically I use the few headers all path. Another way actually I'm sorry that's one path to use. I don't like to look at all the headers. I normally use a few message source. And I'm increasingly just using control you control use the same exact thing as few message source. So those second and third options will open a new window showing the entire email, including all of them, the message headers. And then it's easy to cut and paste message ID out of that. I've been told that there's also a copy message ID add on for Thunderbird that can be used to display the message ID in your normal display. Or I'm sorry to essentially do a cut. There's another add on previously that apparently no longer existed could be used to display the message ID in your normal message list display. So those sorts of add ons may come and go over time. That's another common email client. And if you want to have much display, or to show your message ID header line with all the other email headers that it normally shows, modify your dot mud RC and simply add unignore message ID. And then for all of your muddy emails, you'll see the message ID header along with all the other email headers. Okay, so we saw before using before inbox to fetch a patch series email thread. We gave it the command with the message ID and it created an inbox file. And once you have that inbox file, you can use any normal email client or text tools to examine the mailbox file. So you can use that to look at your cover letter email, the patch emails and reply emails that are all in this mailbox file. Typical clients would be things like mud Elm text tools, which are a lot more painful would be things like cat cat or less vi. Let me see some examples here I'm using mud to examine the inbox file. And you'll see the same exact emails that you saw in my original screenshot of my Thunderbird email client. So we see Pat zero the cover letter, you see the patch emails 123 and four, you see a reply email. So before has fetched all of these emails from the lower archive and save them off into an inbox file which I can then examine with my email client. Okay, in contrast to inbox the am command is only fetching the emails that contain the patches and also the cover letter. The patch series is all combined into a single inbox file, and the cover letter is saved off into a separate cover file, and the format of the cover file is simply an inbox file, but it's missing one header line. There are no reply emails on the inbox file, simply the original patches and usefulness of this is you can use it as input to get am a simply what you use it for. Or a few personally just want to examine the patches and don't care about everyone else's commenting about them. So here's the parts of the before am help command that we're interested in today. Again, the message ID is important. One thing I'll be talking about is using the dash Oh, outdoor option. And for the outdoor option. If you use dash as the name of outdoor than that send your output to standard out. And that will find that to be a useful feature. There's a lot more again and the help and the full help is in the supplemental slides after the final slide of the presentation. Here we are an example of touching that same exact patch series using the same message ID we've been using before using the get am command this time to get instead of before inbox. And you'll see that before simply writes the four patches into the inbox file. So separately into the cover file, it writes the cover letter patch zero. So if I do an LS after this command I'll see two files resulting the cover file and the dot MB X file, the mailbox, and instead of using the message ID is the name of the files. In this case, it's used the subject line of patch zero to create that that final name. So it may look kind of long and obscure but but it really does make sense where that that final name came from and you can predict what that's going to be. So again, the result of the before am is just the patch emails, no reply emails, just make that difference really explicit. And that's, again, an inbox file, but the cover letter is in a text file it's not inbox format. So now we can look at those two files. Again, our inbox file we can use an email client to look at our cover file, we have to use normal text tools to look at. So looking at the cover letter. Again, if I try to use, it's not an inbox file so if I try to use a mail client like mud to examine that cover file, but will tell me it's not a mailbox. So instead I simply use Ted here, and you can quickly see why I don't want to use text tools to look at the cover letter because we end up with all these header files. I'm sorry, all these email header lines in the message and I'm typically not really interested in those as just noise to me. So one trick to actually convert that cover file into a standard inbox file is simply add one single header line that's missing. And a trick here that I've been using is to simply copy the first line of the inbox file and use that as my first line of my cover file. So here is simply doing a head dash and one I'm taking the first line of my mailbox file. And you can see it says from mail or demon it gives a date. So using my fun bash skills. If I use the cat command to combine that one header line with the rest of my cover letter. That head command which is simply getting me by one header, header line. This looks like a typo is greater than sign on the second line should not exist. So the, the first argument to cat is is extracting that that first header line the yeah the first email header with the head command. The second argument to cat is the cover letter file. And then we're combining those together and outputting them into a new cover inbox file. And now what can actually examine that cover box file is a standard inbox file. So looking at just the patches in the inbox file that before created using mud. Here we see only the patches. And notice again we're not seeing patch zero because patch zero is the cover letter, because he patches 123 and four. So this is repeat slide we saw this before. This is the example of before being used to create an inbox file and the cover file. And when we run that command, you'll see what's bolded here. I'm going to show on the next slide is in color. It says this get am text. So here is that that copy from the previous slide. So the before am command tells us what get am command specifically to use to apply that mailbox file to our current get tree. So we can simply cut and paste that get am command. So before is making our life easier is trying to help us out. Here's an example of using that. I create a new branch to work on. And there are a couple of lines here I'm using just to make this example easier to show the result. So I'm modifying around the file doing a get add doing a commit. So now I have a place I can tag is my starting point before doing the get am command. So before making any changes, I have a tag this FS ID V2 before. And I'm applying I'm using that get am command that the before am output told me to use. And we see that get am is applying for separate patches creating for commits. And if I do a get log from the starting point through top. I see those four patches show up as for commit as we would expect. So we can avoid a little bit of that complexity and instead of creating an inbox file and then applying it with a get am command. We can directly apply the changes in a single command. So again, I'm doing a little bit of extra work just so I can show the results, creating a new branch to do the work on. Again, I'm editing an arbitrary file doing an add and commit so I can create a tag. So now we have a tag of before making any changes all on our new branch. And then I'm going to apply that that patcher is in a single step a single command. So using the before am command, I'm adding one extra feature in here the dash Q quiet. So we get a little bit less noise in our slide. And here I'm using the dash option for the first time. Again, dash, oh, base dash says send the output of the before am command to standard out, instead of creating an inbox file. We can't get a cover letter file. So go fetch the patch series, matching this message ID, send it to standard out, and then we'll pipe it directly into get am. And we'll see again the messages from get am applying each of those separate emails as a separate commit. So we see our four apply lines. Just to make it more obvious. What we did different on that previous slide it's the new stuff was, we did a before dash Q am dash zero dash. So we're simply saying 70 output to standard out, instead of into a file, and then pipe it into get am. So this is really all that you need to know that the previous slide was a lot of extra extraneous stuff. And after having done that single step this is the result we can examine with again get log. And once again as expected, we see separate poor separate commits one commit for each of the patches in the patch series. So no surprises there. There are some people from the old days myself included who like to use the quilt tool in addition to or instead of get. And it can be quite useful when you're managing a bunch of different series of patches at the same time. It's at a very high level conceptually quotas based on the concept of each change is an individual patch file, just like we've been seeing individual emails containing patches. So you can think of those emails as being a quilt patch file. And then quote also has a series of aisle, which lists the actual patch files in the order they get applied. So I'm not going to go into detail to explain how to use quilt. So if you don't know you don't know quilt. Most of this won't really make much sense to you but if you know quilt. This should be pretty straightforward. To create quilt compatible files with your before a command you simply specify the dash q or dash dash quilt ready option and before will fetch the patch series just like it was before and create individual patch files and create a service file listing the patch files. Instead of create create well in addition to create me a box file before will still create the inbox file in addition to this. And you can use the individual patch files on the series file as input for quilt push, or as is the data used by quilt push essentially. So here's an example of using the quilt compatibility. In this case I'm using the dash q option. And you can see that before is telling us that it's creating a cover file is creating patches. And if we do an LS, we see the cover file just like before, we see the inbox file just like before. And then we see a patches directory. This is actually directory not a file. And if we go into that directory to an LS will see four individual files that are patches. Each one of those corresponds to one of the patch emails, and we'll see a series file. And if we look at the contents of the series file, we'll see that it lists each of the patches that needs to be applied in a specific order that they'll be applied. Again matching the actual original patch emails. So, explaining how to combine those artifacts into an existing quilt project is a little bit beyond the scope of this presentation but if you if you know how to use quilt. It's pretty straightforward and should be pretty obvious how to how to take those and add them into your quilt project. The next command I'm going to talk about is the PR sub command. And the reason this can be useful as a submitter is asking questions of did my maintainers pull request get into mainline. So when I send patches to my maintainer, he'll pull them into history and then I'll send Linda's a pull request, which incorporates my, my patches. And the second question that few bit sorry before PR can answer is what actually is inside any specific maintainers pull request. So here's the PR command help options. As usual, we expect the message ID to be an important argument to the command. This message ID is going to be a little bit different than what we've been thinking about conceptually, or in some ways the same. The message ID in this case is coming from the maintainers pull request. The maintainer will send a single email to Linus telling Linus exactly what tree to pull from. And so, when you submit patches to your maintainer, your maintainer will accept your patches. Add them as commits into history. The message ID and the poor request is totally unrelated to the message ID and your original patch series. And that's the difference in how to think about message ID. There's no direct way to go back to your original patch series from the message ID and the maintainers pull request, you actually have to go a little bit deeper and start unpeeling things. So here's an example of running before PR simply checking was a maintainers pull request accepted and did it get into Linus's tree. And this message ID says at bogus that it really is a true message ID. It is valid. And before it looks it up, finds a thread, looks at the pull request, and checks to see whether it's in the get tree in the directory that I'm currently located in. And before finds that the pull request actually was accepted, it actually does exist inside my master branch. So the pull request was processed by Linus and it did arrive up in mainline. There's another command before diff and what before diff implements is a range diff, a get range diff, and I don't really have time to discuss that. So I'm going to instead suggest you go look at Constantine's animated example. This is a really good example. It's just a few minutes long. So go to this URL and just take a look at it and see how it works. And I also provide a little bit more information about this command in the resource end slides. So you can, when you download my slides, you can look there. How can you structure your patch emails to be before friendly? The most important thing is follow the patch email subject rules. They're, they're a little bit loose and arbitrary, but they are fairly standardized also. So make sure that you're using the proper versioning syntax, P2, P3, etc. Make sure that you're using the proper sequencing number for each patch within an email series. So the cover letter zero of three, the first patch is one of three, the second patch is two of three, third is three of three. If you do new versions, make sure that you have the appropriate series numbers for the number of patches you have. For a given patch series, make sure that you start with patch zero as being a top level email, and each of the individual patches is a reply to patch email zero. It's useful to not start a new version of the patch series as a reply to a previous version, trying to start each new version of the patch series as an entirely new thread, not replying to a previous version. How can you make before better? You don't actually have to submit patches to make it better. This is a whole lot easier. Simply report your issues to the tools mail list, and you can see the archive of the tools mail list at the URL list there. As I mentioned before, Constantine has been very, very responsive. Typically, he responds the same day when you report an issue and has a solution. I guess I'm putting some pressure on him to never take a vacation. So yeah, you can still take vacations. Don't worry, Constantine will be understanding. And again, he's been very, very active in this last year. He's added a lot of features. And as soon as a problem appears, he's been there to fix it and make the tool work better. So what I've talked about before is giving an interaction to before and why it's your friend. The problems it tries to deal with are email transformations. There's some places that it cannot handle. It's just, it's not within that part of the process path. So it does not fix issues in the email send path. So the client, email client and the email server that the class talking to any issues there before cannot handle. I sort of hesitate when I'm saying that some of those transformations are solved, but I think they're handled more by get them by before. It does avoid the receive path. So before avoids all the problems that get created in the receive path of receiving client emails. I'm sorry, receiving email server and receiving email client before makes it a lot easier to fetch the patch email and to save them into a useful file. It's much easier to fetch just the patch series with no replies, no comments. It's much easier to apply that patch series into your repository. And again, there are other more advanced features that before provides for us. And I suggest using the help command to explore the various options that are provided because I've ignored essentially all the options. Just the end of the talk. And again, I should be in the chat and or question tabs of the presentation tool. Thank you for attending today. And ask questions. And as I noted before, there are resources here. Let me come back first to show you how to get a copy of the slides. You can send me an email. If you go to the Linux Foundation events page, there should be a link to the various conferences. And if this is 2023 when you're seeing this, you should be able to look back to the 2020 event and find all the slides from that. If you're in the current day of 2020, it's get.org. If you look on the schedule, click on the actual presentation and the slides will have a link from there. And I did note a couple of typos when I was talking today, so I'll attempt to fix those typos before I upload the slides. But you should be able to access those slides before the conference begins. Since we have a couple minutes left, while you're asking me questions, I'll point out some of the resources at the end of the slide deck just so you're aware of them. The lower mail list archives where they're located. The, the mail list you care about has been the archive for before to work. So you have to, you can check which, which mail lists are actually archived. The mail list you care about is not archived. You can request the tickets added to lore. And then the, the final URL is just more interesting information about lore. If you use before you need to get it. So there are two ways to get before. One is to download the get repo that contains it. And this is the instructions on that. And the other way, if you're a Python person, you're probably familiar with the PIP method of installing. This is information on that. If you have bug reports and issues, this is where you send those, those twos I mentioned before. And then these are all the, the help pages for before. And then a little bit more information about before diff from one of the emails constant and set. So it's a pleasure talking with you all today and look forward to seeing you in future years and in Tokyo.