 Good morning. Good afternoon. Good evening. Whatever your time zone may be. Welcome to my talk on kernel email tools today. I'm Frank Rowland. I've worked for Sony. I've been working on the kernel for quite a few years. So the kernel patch submission processes is pretty important to me as it is to most of you probably who are listening today. If you have questions, I should be in the live chat and hopefully can interact with you through the talk. So feel free to send questions in. Why is this talk an interesting talk? When we send changes to the Linux terminal source, we send them via patch emails. And you might ask the question or you might already know the answer. Is this process efficient? Is it easy to use? And the sarcastic answer I have is, yeah, it's efficient, but there have been many complaints over the years about ease of use. So I'm going to start with an overview of what the patch flow looks like. This is a really simplified conceptual view. Don't hold me to his total correctness in the technical sense. And show how patches are created, submitted, and then applied once they're received. And we'll be seeing this picture multiple times through the presentation. And I'll be adding on layers as we go through the talk. So you'll see this many times. You can see the chart and the top left and you're working in a source tree with your base files. You modify those files. And somehow you create a difference of those which results in a set of patch files, as we see here. And those patch files that need to be sent via an email client to an email server. You can go into the internet and at the internet, they can take two different paths. And typically, for the kernel you will, the patch series will take both paths. And one direction it'll go through an email server to an email client to various specific people like maintainers, or people you may have to see it on your patch. And people who receive the emails, then need to save those emails, create patch files, which they will then do some magic patch tool applied to base files, and then put the same model files, modified files that you have created. The alternate path, once you hit the internet with a series of emails, is your emails go to a list server, like when external mail and list LKML, and they'll sit in an archive. And then people can query that archive with the web server, pull the patch emails down through their web browser, and then rejoin the path, create patch files, and patch their environment. So what could possibly go wrong with such a simple process. I've been accused of saying it's a simple process and for years and years I did say so. And finally, I've relented, and I admit that in reality it's a very complex process. There's plenty of room for human errors. It is difficult to use. And usability is not especially good. So in a more a different way of looking at what can go wrong. All those emails are going into these various boxes. And what goes into any of these boxes may not look like what comes out that data may be transformed at the various points in that flow that is showing the previous diagram. And you may or may not have any ability to control how those transformations take place. Your IT department may control servers. Of course, the internet is totally out of your control in general. And even your own email client may or may not be adjustable in your own world, your IT department may or may not restrict you. So here's that original diagram, and I'm going to put the blender on some of those various points where data can be mangled. And hopefully this is a conservative estimate. Hopefully there are not more places, but I wouldn't be surprised if there actually were more places data could get mangled. So it's pretty extensive. If you think about it as kind of a scary world out there. Given that those data transformations do occur, we know that's going to happen. What do we do to react to that? We can sometimes fix those transform patches. An example of that is a patch may be converted from plain text into a base 64 representation, and there are various tools that can undo that. But that manual fix up is extra work. And over the years, the different maintainers and developers have developed their own set of scripts and processes to solve these issues. So we do not do not have consistency. We do not have scaling. There are some solutions that have evolved over the years. And as of 2019, there were get tools, and they had existed for quite a while before that point. But when we got to 2019, the tools that we were using to deal with some of these issues were get format patch, get send email, and get AM. So going back to our favorite diagram, we have our starting point, and then we'll see where we use those get tools. And you can see that get format patch and get send email, provide us useful tools to create and email patches. And the receiving end of that flow, the get AM command allows us to take patch files and actually apply them to a tree, typically that's coming in an inbox form or mailbox form. Then in June 2018, no, I am going backwards around from 2019 to 2018. We see an email coming from Constantin at the Linux Foundation. He's provided a new service, the Lore mail archive. And he's announcing here in June of 2018 that it's up, it's working, it's now solid. And he provides a URL, providing some more information about it. And at the end of my slides set, after my final slide, I have a set of resource slides. And there'll be some more information about Lore there where you can learn some more information about things like what's archived on Lore. So I refer you to that but I'm not kind of talking detail about Lore at this point. So going back to our diagram. This looked like before we added Lore. And we had Lore in the top right corner. And now we have a dependable archive, instead of the various random archive servers that previously existed. So we can start depending upon more is a solid archive, which is a very useful piece of the puzzle which we're going to depend upon a little bit later here. And there was some discussion in late 2019 of kernel workflow trying to prove the workflow and solve some of these email issues. And here we have in January 2020 that Constantin has done some more work he's done some development and come up with a helper script. And he's using the script to grab an email patch thread from this archive that now we can depend upon, and he saves that out as an inbox file. So the cool thing about that is once you have an inbox file, you can just pass it straight to get am, and that gets that then it gets applied to your, your get archive. And this is, it's a quick and dirty script at this point it's not really solid. And he asked for some assistance he says you know try it out. Let me know how it works. And it's a bit raw because it's early development, and it's kind of prototype. And there are a lot of corner cases. This is a rather difficult problem set. So he's looking for how it works for people. And so to set expectations a little bit more on that topic. This really is a difficult task that this tool is trying to deal with. It's analyzing emails were created by humans is trying to deal with email subject lines that are kind of random. We do have standards about what they're supposed to look like. But there's a lot of human variants and what they actually end up looking like. And then those emails as I mentioned before go through various email systems clients and servers get transformed in various ways. So the expectation at this point is that before is going to have hiccups. I'm going to confuse you when I say before. Because here I said the tool is get Lorm inbox. It's not called before yet but it's going to become before in a slide or two. So I'm saying that the get Lorm inbox and then before it's going to have issues as that's just the way the world is it's a complex world. The good news is that Constantine is very, very responsive when people report issues, when people report bugs, and he seeks out suggestions for improvements. So as getting ahead of myself on the previous slide by mentioning before, along in March of 2020 a couple months later, we get another email from Constantine. And he's decided that maybe this stuff is important enough to become an actual Python project. And so he's he's packed things up a little bit better. And he's renamed the project from get Lorm inbox. And now it's called before. So you may in the future, or in historic references, you may see references to get Lorm inbox. In the rest of this talk, I'll be talking about before, which is the new name for the same tool and how it's evolved since then and grown and expanded. So once again, back to our diagram. And this is what it looked like when we last saw it, we're going to add one more box or two boxes and down on the lower left. This is showing where before now fits into the picture. There are two ways that before it can be used. If you look in the box here, which is the web browser to patch files transition area before is going to be downloading patches from the lower archive. You could conceptually extend this box maybe with it further up into the right. And it will then create patch files in the form of an inbox. Now you can tie before together with get and passing the output of before through a pipe to get a you can then go further and do your repository updates and changes. So it looks like a small addition is just down that little lower left hand corner just to move to new boxes. But that little addition has an immense impact on improving the workflow. It makes things work much better much easier. I've shown in that diagram where what before is doing is replacing the email server in the email client. But before actually has many other features, but I will not be talking about those in the talk today. So there's there's more to be explored in future talks. So here's before and all this glory. And you'll see the version I'm working with this latest top of tree 0.5.2. As of, I guess today is late October. One thing to be aware of is that before does require Python three. Currently the version has to be greater than or equal to 3.6. Get lower inbox first came out. I was running on a couple of rather relatively old systems, various vintages. And I can attest that even though it worked in many cases. There were there were strong dependencies where we're often the program just couldn't work with the older versions of Python. So I strongly recommend you don't even try using old releases move to to new releases of your distribution if you want to use before. Otherwise you're just setting yourself up for a lot of pain. I'm going to be showing some various before help output through the next set of slides. And here's the top level of before help. And the thing we see here is that before is as much like get in that it has sub commands. And I've folded out a few of those sub commands and box a m and diff. And those are the sub commands that I'll be talking about later in this in this presentation. If I'll give short drift to a preview that concept that it's going to give you a before shattering the other sub commands a test a test verify pull request and generated a thanks email are more useful for maintainers than for submitters and for reviewers. And given the amount of time I have today, I'm going to ignore those entirely. So starting with the first command before inbox. What this does is acquire a patch series email thread from the lower mail server, and it puts it into an inbox file. So this is the same functionality that was being provided by get lower inbox. And here's the help for the inbox of command. I'm not going to go through this and go into detail. But I'll mention a little bit here. Or maybe not. We will see the outdoor argument and example. What important item, a critical item is the message ID and we'll see how that gets used. So here's a using inbox to to pull in an example thread. I'm going to my normal email client. It's in my case is Thunderbird. And to find the message ID for an email specific email. I have to select an email and then do show message source. And that will show me the email headers. And within the email headers. You'll see a message ID. And I'll show it in the next slide. So I have a message ID that I've acquired from my email client. It can be any of the emails in the thread before smart enough to find the entire thread from any of the emails in it and then pull the entire thread. So I'm doing a few things just for my own convenience. I'm making a temporary directory to work in. Going into that directory. Using the before command to tell before that I want to acquire the thread with this very long message ID. Thank goodness for cut and paste and hate to be typing these things in by hand. And once before has done its work it says it saved the results into this file that ends in a dot MB X. And I paste the name of the file on the message ID that I passed into it. So if you do an LS there I do see the inbox file. Again I mentioned that I got the message ID from an email header line. And it's under bird at the bottom of the slide is is what that looks like the subject date message ID are coming from. A few message source in my Thunderbird email client. If I want to see what that inbox file looks like that I just created in the previous slide. I just used the in my case I'm using a lot email client. So I just tell mutt. Let that chef that file name. And here we can see the list of all the emails in that thread, which include the patch emails and replies also includes the cover letter patch zero of the series. So there's the example that that patch series I just pulled pulled in and we're going to reuse the same message ID value for a bunch of different commands. So anytime we see this orange. Big number that's going to be the same message ID over and over again. So we just saw this a repeat. And here's what I said, I just did mutt dash f of that mailbox that I got created. Okay, I'm sorry, I confused myself here. This slide a couple slides ago. This was my Thunderbird email client. This is what that that email series looks like there. And I just highlighted one of the emails did my show message source and that's where I found my message headers was in there. So once I've run before inbox against that message ID, again it created that inbox file containing the entire thread. And now I can look at that thread with mutt or any other email client. So I did a mutt dash tells it to go look at that specific mailbox file that just created. Here's what it looks like in mutt. And we'll see the same exact messages that we saw in Thunderbird. See the original patches will see the reply. Again, we'll see the patch zero for the cover letter. So once you have an inbox file, you can use any normal email client or you can use text tools to examine that that mbx file. And again, that that thread includes the cover letter email includes the actual patches the patch emails and any reply that's come back in the review process. So examples of these programs that you could use our mutt like I used Elm text, typical standard text things are things like head cat more or less Vi Emacs, your favorite text tools. Going to the next command from before going to before AM. This does a similar parallel process where it acquires that that patch series identified by the message ID, again into an inbox file, but it only includes the patches it does not include any replies, and it also does not include the cover letter. The cover letter is instead placed into a duck cover file. I'm not quite sure why the duck cover file was created in the format it is. It's missing a single header line which would make it an inbox file, and you can actually add that manually if you want to read that cover file with a male client. So once you have the inbox file with just the patches in it. Now you can apply, you can send that as input to get AM and get AM will apply the the actual patches as individual commits into your get tree. So here's the full set of help for before AM. And again I'm not going to go through the details of all these arguments you can see they're quite a few that I'm skipping over to provide useful functionality. And the second page of even more arguments. And then we're going to start actually using before AM an example. Once again from my own convenience I'm creating a temporary directory and going into it. And then issuing the be up before AM command within that that temporary directory. Again I'm passing it the very same message ID. And we see the output from before. So telling us that is going to write into the inbox file the mailbox file, these various patches so found four patches and wrote them into the mailbox file. And then it tells us it created a cover file, which contains patch zero. And if I do an LS indeed I do see the cover and the MB X file. And notice these have a different filing pattern that they're not based on the message it. They're instead based on the subject line. Okay, so, again, this is just the patch files, the patch emails there are no reply emails in this patch file in the inbox file, and the text file is only the cover letter. So again, you can use your favorite email client or text tools to look at the cover letter email file, or the inbox file which contains the patch emails. Here, I give an example of trying to access the cover file with mud. And when I pass it. That is an inbox file, but complain to says, that's not a mailbox. And it's simply because one single header line is missing. And so then I just use the head command to show what what that cover letter looks like. And normally, it's no big deal that you have all these extra email headers showing up. If you do a cat or more whatever you'll still be able to clearly read the cover email. And then if I use mud to access the inbox file. This is that same exact patch series you'll notice that that email reply is missing. All we see in this mailbox is the actual patches. So again, this is something we can send straight to get am to apply these patches as four different commits. So I already showed you the slide where I was creating the inbox file. So what I'm going to talk about before is that. Whoops, sorry, jump to head slide. The before am command at the very bottom it tells you what get am command to use to process this inbox file. So you just cut and paste this get am dot slash v to underscore two zero two zero one zero eight underscore john and cetera, et cetera. And then we'll apply them. So here. I'm condensing that whole previous slide down to just two lines. So it's telling us what that get am command is. And then I'm doing just a few extra get commands to make it more obvious. The results of what I'm doing. So I can verify the validity of it. I created a new branch with a get check out dash B. I'm creating an artificial commit here just to make it easier to do a get log later showing the result of the get am to modify them like make file. Do we get add to a get commit. So now we have one at one new commit on our new branch. Then I create a tag. So the tag is marking before I do the get am. Again, this is just for my own convenience. It's not something that that I would normally be doing. It just made it easier to do my slides. So now I do the get am command. This is. So all the, the previous stuff we can pretty much ignore that that's just kind of stuff to make my life easier today. So I'm going to do the get am command and get am applies those four specific patches as for four specific commits and going to the next slide. If I do a get log showing from the tag I created forward, we see those very forward single commits that we just just created. That's the long way of doing things. You don't actually need to create an inbox file and then process it. You can do a single one line command. And again, I'm doing a lot of extra stuff just to make it easier to show what's going on after the after the fact. I'm getting out doing a checkout creating a new branch again modifying the make files just so I can have it a commit that I can tag and creating a tag before actually do any of the actual useful work. So again, all this is not really needed. And here's the actual work of a patch series as get commits using before. So it's like the command is before I'm using dash Q so it's a little bit quieter, but before am then the magic comes in dash oh space dash. So my output to standard list, instead of sending it into a file. And then at the end of the line I simply pipe it into get am you see before doing its work it tells us what it's doing. And then we see kid am doing its work telling us that it's applying these four different patch emails as separate commits. And again to point out what actually we're doing with the before command, all we're doing is we're saying before am dash oh space dash send everything to standard out, instead of into a work file, then pipe it into get am. So it's very simple very easy to use. And then this is ugly, ugliness of course is the message ID. So again, this is just showing that the once I've done that single command to get log will show those four commits actually did occur. So another useful command, get ranged if and given the time available. I chose not to talk about that. Instead, suggest that you go to an animated example at this URL and constant and walk through the actual commands you'll see them occur in real time showing how before if doesn't get ranged if I also provide some more information again in the resources and slides below the end of the talk, providing a little bit more information about ranged if. So those are very last two slides in the slide deck. And I'm not going to go through. Again, the arguments just to give you a sense that there are a lot of parameters available as part of the before diff command. Okay, so that's what before does for you what can you do for before to make it work better. I mentioned very early on that this is a very difficult world that before has to work in is trying to parse human created input human created data. And there are things you can do to make it easier for before to do its work. There are standard Linux kernel rules for how you format your, your subject line when you submit patches and patch series. Basically, be sure to follow those as much as you can. And some of those rules are unwritten and but just follow the standard rules and try not to do odd weird things. Basically, constant is kind of great efforts to actually deal with odd weird situations and make the tool still work in all kinds of strange corner cases. You also can can help improve before I mean typically people are saying send patches make it better that way, but you don't even need to go to that much work to make before better, simply by reporting any issues that you encounter. Any specific, so not, not just issues but also bugs of course, and you report those at the tools mail list. You can get information about the tools mail list at that URL. And again, as I mentioned, constant in the tool author is very, very responsive to user input has been quite active and enhancing the tool throughout this year. No signs of slowing down on that at this point. So an review before is your friend. It makes it much easier to acquire patch email threads makes it much easier to acquire patch series makes it much easier to actually apply those patch series. And then there are other more advanced features I did not talk about and encourage you to use the help option to explore some of those many options. Some of the things I did not talk about is the review process. Why is it important to be able to to get these easy email email threads it's not just to apply the patches into your repository. It's also to be able to read and review patches. The review before is, is a convenient way to find the contents of that entire series. It, all you have to do is find one single email in the entire thread, and then before we'll find all the others for you, whether your email client does or does not. If you want to see other versions of the same patch, you can tell before. And that can be complex to figure out. You can imagine just as human being when you're trying to find older or newer versions of a patch thread. Did the subject lines change to the number of patches in the series change. Even the title, the subject line of patch zero, the cover letter sometimes changes, making it quite difficult to to go from one version to another and you're searching. So, when I was talking about how to make life easier for before. That's another area to be thinking about making sure you have consistent, consistent values for your, your cover subject line. I'm sorry, the subject line for your cover email and not changing that between different versions of your pet series. And that's the end of my prepared discussion. So, again, I should be hanging out in the chat, answering questions, be glad to help out anyway I can. I'm going to jump ahead for a moment and just show you some of the resource slides that are available. You can email me. So be on the elinx.org website as all the ELC and ELC Europe presentations are and the conference website from events limits foundation.org. Some of those resources I mentioned. I mentioned that there's information on the lore archive. This is the announcement of that and have some useful links. I chose you how to find out which mailing lists are actually archived. Here's some of those links, which mail lists are archived by lower. If you have a mail list that's not already archived that you want to add to it. There's information on how to request that. And just in general, there's more information there that's worth reading. I did not talk about how to get before. It's available in two different ways. One is a. It's a standard get repository and you can clone that or download it. And the various versions are tagged with tags. So once you've downloaded your, your get repository, you want to look and see what the current tags are and check out one of those specific tags. Typically top of tree. I mean, well, most recent tag. And if you invoke before from your get repository. At the top level, there's a file called before that sh. And that's your magic entry point. And that's what I was using in this examples in this talk. And simply created an alias where before invoked. This before that say sh file. The alternative way is using PIP, which if you're a Python person, you'll be familiar with PIP. And here's some information about the details you need for using PIP. Again, these slides will be available. I'm not going to leave them up for you to write to copy them down. Again, if you have issues, here's the place report bugs. And then I have all the help pages for all the commands. And then a little bit more information about before diff. So those resources are there for you to look at after the conference. So back to questions any questions. Feel free to join the chat.