 Good morning, good evening, good afternoon, wherever you are. You have come here to learn how to transfer your idea into a patch in a Linux mainline. I've created the presentation to squeeze 20 years of experience into less than one hour session, showing you an experience in open source and in this particular case in the Linux mainline. Why is it important for me? Because there's a lot of knowledge about doing open source. And some of you may think that getting a patch in the Linux is something that only super developers can do. I was thinking that myself some 20 years ago. But it turned out that it's not true. There is a way to follow. And if you follow the right way, we are going to get there. You may be wondering if I have some data to support that because it looks complicated. And yes, I do. I want to convince you with some data. Let's look into the last corner development series and in the number of developers who contributed code to those recent kind of versions. You have between 1700 and nearly 2000 people per development cycle. And what we can see that from one sixth to one eighth of those people are people who contribute the first kernel patch in that particular kernel version. Imagine that six kernel developers and for each of the group of six, there is one who's doing it for the first time. And it's not just the last kernel. It's here with us for years. I'm showing only five zero series here. And if you look at the data for all the version, it's still the same. And I didn't put five nine because when I was just doing the presentation, the data was not there yet. But it's still the same. You can then try to look at it in a different way. If you're a little shy. You can try to hide yourself in 200 other first time kernel developers. If you are a first time kernel developer. In one kind of version. The next kind of cycle. If you contributed patch, you want to be a first time contributor. So just imagine how many first time and second time contributors. So, let's get started. I told you that I do have a path to show you. And we have a few steps. With techniques that you can learn to succeed in getting a patch. First we'll talk about the prayer cup prayer requirements, the things that you need to do that. Then we'll look on working in the kernel, how to understand what's happening. After that, we'll cover the tools that can help you. And then we'll go to the part that is considered by many very difficult. How to prepare your code, the coding style, how to prepare your patch, how to send it to the right people and how to answer to any comments we're going to get. Then at the end, we're going to cover specific case if your first patch is not the back. It may be something else. And there are a few additional techniques that I would like to tell you about in this case. So, what do you need when you start developing in a kernel? Well, first you need to work on a subject. It may be a back, a new feature on improvement. With the smiley faces, I'm going to get into in a moment. What do I consider a back? You have a warning somewhere? Something is happening differently that you would expect it to be. Or you just get a big back error message somewhere. Then you also can have a new feature. You're adding a new functionality in the code that already exists. Or you may be adding a new driver. You're working on your hardware and you're adding a driver for it. Or even working on your sub-system. Then you can do an improvement. What could it be? You are improving performance in some specific use case that interests you. You can do some refactoring before other changes. Or just because it's an important thing to do. Or you may be adding a new test. Now the smiley faces. All those subjects are good subjects for a first corner patch. There are two green smiley next to the back because I personally think that's something that is the easiest to get into the corner. Improvement can be also quite easy. If you're doing a refactoring or adding tests, for example, with performance it may be a little bit more complicated. When you're getting a new feature in, it is a good way to. However, you need to take into account that it may be a little bit complicated. Because you have to learn all of those things about working in the corner to make it all happen. But still, if you are patient, if you are motivated, you can use that functionality too. You can also do something else, split your feature. And for example, start by integrating refactoring needed for your new feature. It's up to you. Just choose the subject. You will need, of course, Linux source code and then compile with a debugger. And of course, a test system. You need root access on that machine. So it may be a physical machine, it may be a virtual machine, it may be an embedded system. It depends what you're working on. Then you will need the text editor. Any editor for developers is going to work. What you need is root or text mode and the way to control where the white space is in. It's including. In all, and it is, I know you do have a Linux coding style function. So if you are working with the corner, you just activate it and then it's going to be way, way easier for you. Finally, you will need an email client. We have a support of road text mode and the email client will be used to send the patches. Because that's the only way you can submit your patch. It doesn't look that complicated, right? You can already start talking in the, in those elements if you don't have them yet. Then yet again, make sure you do have root rights on your machine because you need it for the kernel installation and for the models installation. You will need to install dependencies that differ between the distributions. The example I give is from my last setup of a devian system. But look for instructions for your distribution. They are usually very easy to find. One difficulty people often run into is that they get no space left on device message somewhere during the installation. Quite often it comes from the fact that the boot partition on the, on many distributions quite small. So watch that out. If you're installing from scratch, you can make, make sure that it's bigger. And then if it's, if it's small, you just look. If you just take it into account and resize if necessary or remove all the counters if necessary. We are getting to the core of the problem. You need Linux counter source. The main tree, the master branch from the nose. That's where you submit your patches to. You always submit a patch for the master branch. You may also need the stable trees, which you can clone from a different place. You may need those if you are investigating a problem that shows up on some production machine. That is of course not running the mainline kernel, but is running some stable version. So in this case, you will need both trees to test your patch for the stable and to the main tree. Depending on what you do, you may also need subsystem trees. You may also need trees that contain the latest changes in kernel subsystems like the networking on fire systems or drivers or architectures. When you start developing, you will learn which tree you should be working on. You may already start getting copied from the Linux main tree. If you have a kernel source tree, I really, really recommend you to get the source code index editor. If you are working online, you can use a text editor and see types. If you are working online, I give you a link to a bootlin online source index editor. We'll get into the use of that in a little while. How to compile it? In a simple way, we start, of course, with getting the Linux kernel. Then we need to do one small thing. Get the configuration for your kernel. You may already know that the kernel can be configured in many ways. You can have options that you activate or deactivate. And the kernel you are running right now has already configuration for your system. What we would like to do is to use your current kernel config as a base of your new kernel so that you have all the drives you need. We'll then copy the kernel config to a file in our kernel directory. And then we use make all the dev config that will add all of the new options. For example, if you're using a newer kernel that your machine is running and add those new options with the default values. So if by default any option is activated, it will get activated in your kernel. This is a safe solution if you want to be sure that the new kernel will work on your system. Then, of course, we build it. And then we install it. And in many cases, that will be it. You may also need to update the bootloader to get a new kernel, look into your system documentation, how to do it, especially if you are running an embedded system. So now it's up to you. I will be giving you small exercises from time to time. You can stop at this time and to the exercise, or you can note it down and do it a little bit later. Download the kernel, change the version name, a small version that you can see when you run a name minus a, it's in the extra version in the kernel make file and compile it, install it, boot it. And when you are back, check if the new name is there. If you can do it on an embedded platform, it's even more interesting. So, you should be able to compile a kernel around. Now, I would like to give you an example of a real kernel issue and of an analysis of a real kernel issue, and show you how we can understand what is happening in the kernel. Not fast. During the analysis, you will need a little bit of data. In the documentation directory in the scanner, you have the documentation, there's a lot of things to look at. You also have the code itself and you have commands in the code. You can also use the source indexer to find out where each function is used. The source is in the mailing list. The discussions that have worked out in the past may be interesting. And the new sites, if you want to learn a new subsystem, then you can look into them for the moment when it was introduced and learn about how it works. Usually, being in the face changes are also well covered by the new sites. Let's look into a real kernel crash. An example from 5.0. I will let you look into this dump fragment for a few seconds. Try to find out what is important in your opinion in this, in this crash. Well, an idea. You can see the most important things in my opinion in red. We do have a bag here. And then it's a null point on the reference. Please note that there are some messages of the kernel before. Then we get the bag and then we get more information about where the problem appears. So, we do have a crash in NVMe parts in a log. So we have the function name, where we have the crash. And we also have information where this function was called from. Now, what we can do. We can look into the function with the crash. And we can just look into the code. We know that we do have a null point at the reference. So we can analyze all of the pointers in there. And try to find out which could be the reference. In this case, we can have the controller or in the line 331, we do have the controller that is using another field. That's being dereferenced. And in fact, if we do have a kernel binary, we have the have a tool that helps us add it to line. You can use it and it will point us exactly to that line. So, we know what is dereferenced. So now, why is it null? There are three possibilities how you can approach it. We have something that is getting allocated but apparently it is not. So we can check all allocation patches. All allocation paths. We can look into the call stack to see how it was called and try to find out what happened. We can also look into the kernel messages just before crash. Maybe there is something interesting in it. In this situation, I would probably use the path three or path one. Because in this case, the function is not used that many times. So it's easy to analyze. No matter which path you're going to follow. At some point, you're going to come to another function and be able to path anything in this case. When you can see the messages that we have seen in the back trace. And then what is interesting that in the case of this, those messages, the function returns. And what happened just after that? The buffer we are searching for gets allocated after these codes returns. So what did we have? The code did exit at the error condition. So we didn't allocate. And there was some kernel path that did try to use that structure. From that point, it's only finding out who the path exactly is. You haven't got called trace. And writing the solution, fixing that location. So what is the size for you? Find the comet fixing it back in a piece of kernel code that interests you. Either ideally if there's a warning or a bug, it's easier. And you look only at the description of the problem. And you try to find out what's happening here. You may list a few ideas or about what's going on here. And then you can verify with the patch itself what the problem really was. Of course, if you can analyze this way, a problem is not fixed yet. You get bonus points. Of course, we can do source code analysis from time to time, but it's not always enough. The kernel offers you many tools to help. I'm going to give a small overview about what we can find the most useful. We talk about tools, but we know all of us that we do debugging by printing out. And you may be surprised that there is no printer in the kernel, but there is hope. You do have printk that you can use. However, just printk is pretty rare. What you usually see, you see the PR functions, PR error, PR info, and others with this family. They practically do the same, but they can be compiled out if needed. If you are in a device driver, you also have another family of functions, device error, device info. It is showing additional information about your device. For example, useful if you have multiple instances of the same device and you do not want to mess. To have a mess of the same messages showing up all around the place and you want to know which device is showing it. An example is you are using it pretty easily. And then finally, you do have dynamic debug in kernel. You can activate some messages in pure debug dynamically. At runtime, when you want it, you activate, you deactivate, like the kernel given you an example. Printing is useful, but of course, we do have other things. We do have kind of, oops, we have the bugs like we have seen before. Warnings that are similar. You do have a kernel debugger in the kernel available. We do have another session about the debugging during this conference, so you can check it out. You can also use F trace, function trace set that allows you to look into what happened between two events. It's a pretty powerful thing. If you are doing performance testing, you would like to look into perf, especially if you are into hardware contests. It could be very useful. Then, we also do have testing frameworks. This is separate that is moving right now, very rapidly. For the kernel itself, you do have two test frameworks. You do have the kernel status framework. That is testing kernel code from the user space. In this framework, you write small user space applications that test the kernel. It is run after boot and you can use kernel modules, kernel test modules if needed. Another way of testing the kernel is K unit. It's a pretty new addition to the kernel. It allows you to add unit tests to the kernel, for example, in a driver. The use is very similar to unit testing frameworks you may have been using in the user space. That's not all because there are many other tools starting from static analyzing and many other things and you do have documentation in the kernel of all of that. You can also go and see presentations about kernel testing. There are a few of them during this conference. Now, you have the tools, you have the problem. Now, we'll get to the formalities, how to prepare your change. Linux kernel coding style is something that may be a little bit scary from the beginning. I simplified it here to one single slide. The basic things you have to learn about. The tabs are eight characters, period. They are pretty white, right? You write one statement by line. And there was a change recently about the number of characters per line for years. It was 80. Now, 80 is a preferred maximum. But if you do have reasons to put more, you can. In kernel, you usually use short names for functions and variables in our case. And I have a small illustration about the braces. I think an example is easier to understand and train to explain it differently. In addition, you do have the complete definition of the coding style and you have a check patch PL tool to verify your file. Take into account that check patch by default expects patch file. If you want to check the whole source file, you need to use minus F option. There are also other options of the check patch that you may look into if you antifreeze it into the database. Now, how to format. You can format and that's the most common way to do it. You format the patch with git format patch. For example, if you want to create a patch file from the last comment git format patch minus one. And it will create a file. You can create a patch from a comment. And there is one more thing to add apart from your fix itself. The comment message. This is a template of one you get us in the first line have a subsystem and the title become it then you put an empty line describe your patch. Do an empty line again, and then write sign off by and put your name and your email address. For example, this is not a real comment message example. There is nothing to show you what it should look like. We have a driver with a fixed timer overflow after 32 minutes. What's important is to have a clear description in the first line on what your patch is doing in this case fixing a problem. Then you have a description. In this case, a developer says that the problem happens if a cat sleeps on a keyboard. I wanted to test it with one of my cats, but they didn't want to. What I can only tell you that there are so many different strange cases that problems happen that I would never be surprised seeing a real message like that. So, you send it in the end, and then we'll get to the studio stuff. Sign off by is a quite important and legal matter in the car. You should only put real names in the car now, and it legally certifies you have the right to submit under an open source license. If you're working for a company, please check your contract and your locale. What is saying about your submitting work on the open source license. You can find the sign off by in the column messages, but you can also see quite many other tags in the column messages. First, you can see act by. This is usually done by the maintenance. They approve the patch. They have reviewed it and they think it's okay. Then you have reviewed by. Usually by non non maintenance. This person has looked into the code. They think that the code is correct and it's ready. And they have communicated all the comments they have to the author for someone who has found an issue. And then the issue was fixed by another person. There's the reported by tag. You can put it the name of a person who found an issue for the first time. And you have the tested by tag. The person who checked out that the patch is working. That may be also one of the first thing you do in the car now you can, you can test by someone else on your system. And then finally, you do have fixes. This one does not refer to people. It gives information about the original comment that was fixed by this one. And it's used to trace the dependencies between the comments. It allows maintainers to grab the change and the fixes that were in there to apply it in another kind of branch, for example. And this is for you. Perform a change in the column of your choice, test it, use check patch, format the patch file, and then we can discuss your patch during the conference. I will be happy to discuss your work during that time. We do have a patch. The time will be to get it accepted, right? But first by whom? Well, you are going to send it. For that use the script get maintainer. For example, if I want to know who is the maintainer of the lib random 32.c file, I run scripts get maintainer minus f. As with the check patch, you have the minus f, a without minus f. And what it shows out. It tells me that they are two maintenance to the two persons. In this case, those are network maintenance. There's a mailing list related to the networking stack. And there is the kernel mailing list. You do send your patch to those two people and to the mailing lists. Don't forget to put the maintenance in copy. Why? Because this way too much traffic on the new scanner mailing list and many other mailing lists. If you send the patch directly to the maintainer, they will see it just on the mailing list. They may just not see it. It's important to use the check patch and just get all of the mail addresses you get there. The checklist for submitting your patch. Make sure the coding style. You need to send the patch as plain text email and inline the patch. No attachment, but inline the patch. In the subject you put patch. And then the first line of your comment message that that's why it's very important. You can use most of the main clients, but it has to be configured correctly. And the current documentation includes the right process for many different email clients. So check it out, follow it. And make sure it's all correct. You can also use git send email. It's especially useful if you are sending assets. So many patches at the same time. As a reminder, the things that do not pass easily. Do not add attachment. No encryption, compression, legal messages, or long signatures. And no GitHub pull requests. Personally, when I'm setting up a new environment, I'm sending a patch to myself. And then I verify it to make sure that everything is just fine. And I recommend you to do exactly the same. You're sending it. And the first truth about the review of current patches is that they really rarely get accepted in the first version. It may happen if you are fixing a typo in the documentation. But if there's some technical work, it's very, very rare to see. Another important thing is that your patch should include an answer to the why question in the patch description describing why you are doing that change. And then in practice count one week to receive comments. Okay. How the review looks like. The reviewers comment, they put quotations in the code, and the comments are often pretty short. They take into account that the reviewers are busy. So that the message can be pretty short. Then you should address the comments you get in a polite way. You may disagree with the reviewer, but if you disagree use facts and address the problem the reviewer looks into. Try to understand why they comment on it and address the facts. Not emotions. And you can always ask for clarifications. For example, a technique I often recommend is to tell to ask, did you miss something like that? And then post the code fragment or pseudo code showing what you understood from the suggestion. That allows you not to rewrite the whole patch before you understand what the comment actually is. And then you have worked out the comments you send the patch version two, and you repeat the whole process. What you can get as a feedback, many, many different things. You may get a request to change the coding style. You may get a request to refactor existing code. You may get a request to go when we are duplicating something that already exists. And the maintenance thing is better to refactor it. You can get a message requesting you to use existing API instead of creating anyone, or the API you just didn't see or didn't know about. You may get suggestions on how to make the patch better. You may get a draft of an alternative solution. Of course, maintenance may ask, why did you do it like that? And sometimes they may explain you why or in which situation your solution is not going to work. It may happen that you wait for a week and there is still no feedback. Then you can resubmit, but before that verify, is your title clear? Does it explain what you are changing? Is the description clear? Does it say why is it important to change that part and do it right now? And important to verify that you send it to the right people. And finally, is it small enough? If your change is big, it takes more time to review. If it's too big, you may consider splitting into a person. Then it may be reviewed faster. I covered the case of a bag and if it's not a bag, you do the same with a few differences. If you are getting a new feature in, I recommend you to communicate as soon as possible. And to show that you know the rules that, of course, your first version should be just into the right coding style. If possible, have tests and be tested by some people. At the beginning of the work for a new feature, what you can do is so-called RFC patch. RFC means Request for Comments. It means this patch is not stable, it's not mature. It doesn't have to be complete. And it's a question, it's a start of a discussion about the functionality, about the functionality itself and about the way you implement it. Quite often, this kind of message is sent only to the subsystem, mainly not to the whole Linux kernel. Of course, if you are adding a new feature, split changes into steps. This way, easier to review. Some suggestions for splitting big changes. So creating a patch. First, one logical change in a patch. Then you have a separate title and description for each patch and they should be quite independent. The kernel should also compile and work correctly after each patch. So you should test it out, make sure that every single patch applies separately. They depend one to each other. So you apply them in the right order. Example. Imagine that you have found two bugs and you add a test to be sure that this bug doesn't show up again. If the bugs are in different files, a suggestion for split from me will be one patch to fix one back. Then another to fix the second back. Then if you have a comment to fix that may lead to people to use the code incorrectly in a different patch and then the test in the fourth one. If you are doing adding new functions in different subsystems, you do the changes in each subsystem separately. You do the necessary refactoring in a separate patch and then you use the new functions later on. So we'll be getting to the end. So today's a lot of information from the corner. Some recommendation from me. And the newbies. And the other links and the challenge that includes exercises on how to start programming into the kernel. You cannot subscribe anymore, but you can still look into the internet for the exercises to them on your own. The WVN has a big long series of kernel articles and you can search for the information you need. Of course, the news kind of mining list. All the files and all the patches. There's a lot of information in there. And at some time reviewing the whole column development process. And I would like to repeat a few things. New developers get the patches in each kernel. So it means that it's possible to do. I recommend you to start simple test your setup, learn the rules in the fragment of the code you want to change. And your first patch doesn't have to be perfect. You don't have to spend a month working on two lines. But it should show that you, you did your homework. So the coding style should be okay. It should work of course. And it should show that you are respectful to the kernel developers. It's something to be fixed in your patch, but it still shows work. It shows that you know the rules. It's going to be way easier. And of course, it's way better to have a patch accepted in version five than never to get it accepted, right? Because we all learn by doing. So it's now up to you. There's a lot of interesting session during the conference, and I will be there during the whole conference. I can ask you questions right now during the conference. Do not hesitate to contact me now or a little bit later. And that is the time for your questions.