 Good afternoon, everyone. It's our pleasure to meet all of you here. Today, our topic is to introduce and elaborate the activities of super long-term support kernel workgroup in several infrastructure platform projects. I'm AC Lin, work for Moxa, a Taiwanese company. I'm also a devian developer and contributing to various open source projects. And this is Pavel. Pavel is a co-operating with Danx, and he is also a kernel hacker. CIFR infrastructure, such as radar system, power plant, and railway system. There are some key challenges in this system. That is industrial grade, sustainability, and security. For industrial grade, some requests such as reliability, functional safety, and real-time capability. For sustainability, there are some demands such as product lifecycle of decades. This is very important. That means we have to maintain and fix this for a long time. The next is backward compatibility and standards. The next one is security. We have to manage the security and vulnerability. And we'll have to do some firmware update once the security found it. And the next one is we have to minimize the risk of regression. So CIP is the solution. CIP established an open source base layer to try to overcome these challenges by using open source software. So as you can see, this is open source base layer. We have CIP kernel and CIP core packages. So this is the figure of the scope in CIP. We have super long-term support kernel and real-time kernel. We have testing automation, building environment, long-term support strategy, security. That means IEC 62443 standard. And CIP core package. The last one is safe and secure updates. In this session, we will address super long-term support kernel world group. This is the first world group in CIP project. And we aim to maintain NINUS kernel for 10-plus years. We also maintain the real-time kernel by applying the pre-emptive RTPatch. So for the policy and progress, this is the current LTS version. As you can see, there are six versions for stable kernel now. And CIP choose two of them for being CIP kernel. That is 4.4 and 4.19. We aim to maintain and support the CIP kernel for 10-plus years. So how could we do that? This is our kernel development. As I mentioned, our goal is to support the NINUS kernel for 10-plus years. So we have the developers and maintainer and mentor. We send the patch to the mainline and review the mainline patch to the LTS kernel. And then we build up our CIP SLTS kernel. For now, we have 4.4196-CIP38 kernel and 4.19, 0.78-CIP12 kernel. Furthermore, we also have the CIP kernel CV tracker and field patch tracker. So this is our main team member. I'm the AC Lin chairperson. And this is Pavel. Our kernel maintainer. Another kernel maintainer is Iwamazu-san. And we also have the mentor. That is Ben. So this is our current SLTS version. We have four versions. The first one is 4.19. The maintainer is Iwamazu-san and Pavel. And it's released... The first time released is 2019, generally. So the latest release is October, three weeks ago. So the current version is 4.19.78-CIP12. And projected EOL is 2029 plus. That means 10 years plus. For 4.19-RT kernel, the maintainer is Pavel. The first release is also in the 2019, January. And the latest release is in the 2019, October. The version number is 4.19.72-CIP-RT3. The projected EOL is 2029 plus. And now the version is 4.4. Which is the first CIP kernel. It's released 2017, two years ago. And the latest release is 2019, October. Version number is 4.496-CIP38. The projected EOL time is 2027. The last kernel version is 4.4-RT kernel. The maintainer is Pavel. And the first release is in 2017, November. And the latest release is 2019, October. The version number is 4.419-CIP36-RT25. So our maintenance policy right down in the Wiki page. So I'll list some portion of that. The first one is we follow the stable kernel development rule. So every page should be followed the stable kernel rule. The second is validation will be done by our testing infrastructure and our members. The third is we also accept the feature backpotted, which is not allowed in the stable kernel. But we only accept the feature backpotted from CIP members. And the patches should be in the mainline kernel first. That means we have the upstream first policy. The last one is we use the NINUS Foundation developer certificate of origin. That is DCO. Regarding the out-of-tree drivers, in general, we don't support it. So if anyone want to use the out-of-tree drivers, you can do it by yourself, but we don't support it. Because we have the upstream first policy. This is our development figure. As you can see, we use the stable kernel, which was the mainline kernel. And as I mentioned, we accept the feature backpotted. So once the patch merge in the mainline, it can be here, to the backpotted patches. And it will be merged by maintenance to 4.4 or 4.19 kernel. So we will take over the kernel maintenance after the EOL from the stable kernel. For now, we maintain 2 kernel, just I mentioned, 4.4 and 4.19. So basically, the CIP patches come from two ways. The first one is stable patches. Stable kernel rule lists that it only accepts security fix and bug fix. So we create another way, spot-potted patches. It accepts virtual backpots. It should be sent to CIP-DEV at least the CIP project, dialogue mailing list. The maintainer will review it and accept it once approved. So that is our development process. So our next section is patch review. I will give the floor to the maintainer, Pavel. Thank you. So I will talk about how we review stable patches and patches from our CIP members. Everything we do should be in the public. So we have a repository called LTS commit list. And each time a new stable kernel is released, we add the name, titles of the patches and SAH hashes, and then make notes about the reviews we done. Usually, the good thing would be to acknowledge the patch. But there are other possibilities, like patch may be under review, which is still looking at that. There can be negative acknowledge. Patch may need additional patches to be backported. And the only review patches that are relevant to our members. So for example, Italian architecture is probably not relevant to CIP project. So we will just ignore those patches and met them without review, as long as they don't touch anything they should not. So this is how the review files look like. Nothing much to see there. We do this for each stable kernel 4.4 and 4.19. Sometimes things don't go quite smoothly. And these days, we are trying to help stable kernels a bit too. So we try to do the reviews actually early. Because if the review is done before the stable kernel is reviewed, it's released, community benefits too. So this is one of the results. There was a patch being proposed, but it needed additional patch to fix it up. So it was noted during the review and it was fixed. Or I will talk about that in the next slide. So you may have noticed there are stable kernel rules are described in the kernel documentation. Unfortunately, in practice, we found that there are some differences between what's written in the documentation and what happens in practice. So there's one rule that is enforced and it's that it or an equivalent fix must already exist in Linux history. What the rule doesn't tell you is that very strong preference is given to merging the same, exactly the same patch with exactly the same change log as in the upstream. Other rule says that it must be obviously correct and tested. What is not obvious from the documentation is that the preference is actually given to the first rule. So if it's discovered that the patch was buggy, we will still or not be stable and be inherited. Stable will still merge the patch that is buggy, but will merge the fix too. Next rule says that it must be a real bug that bothers people. We don't find this to be enforced in practice. In particular, tiny memory leaks like once per boot that are pretty common are being merged to the stable tree. Fortunately, these kind of fixes don't usually cause problems, so it's like that. And the next rule makes it clear what kind of errors are acceptable. It must fix a problem that causes a built error and whoops, a hank, data corruption, a real security issue or some old that's not good issue. In short, something critical. That was quote from documentation, but during our review, we found that build time warnings are fair game too, which may or may not be reasonable. Runtime warnings are being purchased regularly, which is probably reasonable, but we also see stuff like confusing print K messages being fixed or adjustments on the long levels and so on, which may be a little bit out of scope for stable, in my opinion. And there's an extra, which says that it cannot contain any trivial fixes in it, but this one, again, is not enforced because preference is given to merging exactly the same patch as is. So these are statistics of our team's contribution to upstream, so for Kernel, we commented on 160 patches. On 44.19, it was 73 patches, and we did 30 backports of patches we thought would be necessary for 44 that were applied to 44.3. Process for patches from our members is slightly different, so basically, we try to enforce similar rules, but we accept features that would not be acceptable for stable. So if CIP member has a hardware, he needs support it, we will actually take the patches. Usually, so we may take hardware support, which would not be acceptable for stable, but the rest of the rule stays, including that the patch must already be upstream, so there should be no bad effects from divergence between our three and main line. So currently, we are occurring more than 600 patches in the 44.3 and more than 400 in the 4.19 tree. The trees are not feature equivalent, so support for some hardware is only being merged to 4.19 because it would be too hard to backport for 44, or maybe it's just not that important. And with that, I pass my mic back. Regarding the SLT real-time support, we use the pre-emptive RT patch, and Pavel is the maintainer of RT kernel in CIP. We will merge it, review it, and validate it. After that, we will release the RT kernel. So for now, CIP has become the Go member of the real-time Linux project, and we will work together with real-time Linux project. So you can get more information via the link. So to sum up, this is a kernel release policy in CIP. We will release the 4.19 kernel trace a month and 4.4 once a month. And regarding the real-time kernel, we will release the 4.19 RT trace a month and 4.4 RT once every two months. Besides, we also release on demand. We support this. It's based on the critical bug or security fix, so this is dynamic. This is the status of the CIP kernel and real-time kernel release in the past years. In 2017, we released our first CIP kernel. So in that year, we released 15 CIP kernels and three RT kernels. In 2018, we released 14 CIP kernel and 17 RT kernels in 4.4 regime. In 2019, we released nine 4.4 kernel and five RT kernel in 4.4. So in this year, we support 4.19 to be a new CIP kernel. So we released 12 4.19 CIP kernel and three RT kernel. For now, we released 38 4.4 kernel, 25 4.4 RT kernel, 12 4.19 kernel and three CIP 4.19 RT kernel. So the estimates in 2019, because they are still two months in 2019. So we just estimate that we will have 42 4.4 kernel, 26 4.4 RT kernel and 17 4.19 kernel, five 4.19 RT kernel in the total year. So next, I will introduce the CIP kernel sec. This project track the status of security issues identified by CV ID in mainline stable and other configured branches. We will track the security issues by using this repository. So this is a QR code for entering this repository. So as you know, the kernel is fairly huge. So we cannot maintain all of this. So we only maintain the scope with the CIP kernel convict. CIP kernel convict is come from the CIP members configuration. So the security issue are determined to be fixed based on kernel configuration provided by CIP members. This is the example. As you can see, we use a YAML format to track the CV issue. For this example, we support it. So we will write down the fixed by information in CIP kernel. And what if the driver or the kernel doesn't support by CIP? We will write down like this. This is an example which will not be supported by CIP. We just write down ignore and no member enable this driver, that's it. So this is the classified file patches. We usually see the file patches email in the stable kernel mailing list. So we create these projects to track the status of your patches and classify the patches into apply or to apply. These are the example for all the results. As you can see, we can track the applied patches and to be applied patches. And we will work on the to be applied patches to solve the conflict issues. And then we will send back to the stable kernel. After that, we will have the testing. Every CIP kernel will have a test and then we will release. For the testing infrastructure, this is our architecture of the testing infrastructure. As you can see, we use the Gileb CI CD to trigger Lava Master. The Lava Master will send a task to the Lava worker and Lava worker will start the testing job in the reference platform. If you want to get more detail, you can attend tomorrow, that is CIP Minisum Meet. We will describe the detailed testing procedure and architecture. So summary, we have routine tasks and occasional tasks. For routine tasks, we will maintain 4.4 kernel and 4.19 kernel, including the real time. We will check down the CVE for security and the applied patches. For occasional tasks, we are going to build up the kernel and RT kernel testing infrastructure and we will define and update the wiki for the CIP kernel maintenance. In every week, we have a regular online meeting in IRC and this is a public meeting. In this meeting, we will align our status and try to discuss online and know each other's status. So if you are free, please join. These are repository from the CIP kernel worker. As I mentioned, we have CIP kernel, real time kernel, CV tracker and fail patch tracker. To get the latest information, you can subscribe the CIP mailing list. That is CIP-DEV list, CIP-project.org. Other resources, you can visit the Twitter website Newsweek and we have several CIP talks at this event. So in today's 3 p.m. 15, we have an introduction to OpenShot project to live a long-end process. And tomorrow, we will have CIP mailing summits. If you have a ticket, please join it. And in Thursday, automated testing summits, we also have a session there. So please feel free to attend this session. The last but not the least, we have the booth at 4 and 4 and 5 sponsors showcase. So if you are half the free time, please visit us there. And CV infrastructure, there are eight companies behind this project. So if you got interested, please join us. Any questions? Thank you very much.