 Thanks for joining us today. This talk is about CRP Kernel Team activities. We'd like to talk how CRP Kernel Team is interacting with Aptreeb and other projects in order to achieve CRP goals. First of all, let us introduce ourselves. I'm Masashi Kudo from Cyber Trust Japan. I'm currently acting as CRP Kernel Team Chair. So, Eze-san, please. Thank you, Kudo-san. My name is Eze-lin. I'm working for Mosa Taiwan Company. Currently, I'm the open-chain project governing board member and the CIP technical steering committee member. I'm also the former CIP Kernel Team Chair. Today, we first explain about CIP. In the CIP, upstream-first is a development principle. So, I'd like to explain what it means for CIP Kernel Team. CIP Kernel Team has two main pillars, that is, upstreaming and kernel-redecing. So, we'd like to elaborate each by each. CIP is working some security standard conformance. We'd like to explain such activities as the CIP Kernel with cybersecurity. Upstreaming and CIP Kernel with cybersecurity are covered by Eze-san, and other sections are covered by Kudo myself. Then, let's get started with what is CIP. CIP stands for Civil Infrastructure Platform. It was founded almost four years ago under the Linux Foundation. When you hear civil infrastructure, you may imagine heavy-weight systems like power plants. That is true, but there are a lot more around us. Even industrial IoT devices can be categorized into civil infrastructure. The way to develop all those systems and devices is getting more and more complicated because mobile and cloud technologies have become commodities. Systems and devices now consist of much more components than before, but the development resources have not been so much changed. Therefore, only the propriety applications are focused as differentiators with limited resources. Now, there are millions or trillions of civil infrastructure devices running worldwide. They share the same industrial requirements, security, sustainability, and industrial greatness. They should keep satisfying those requirements during their life cycles, which are usually very long. Therefore, super long-term maintenance becomes a key. However, there are no common base-building blocks for civil infrastructure platforms. Therefore, similar development and maintenance efforts should be spent each by each separately, even in the same companies. By being motivated to solve those issues, Civil Infrastructure Platform, as known as CIP, aims to develop building blocks to satisfy industrial requirements with open sources. We named such building blocks as open source base layer in short OSBL. OSBL consists of CIP SLTS kernel and CIP co-op packages. SLTS stands for super long-term support, and we aim to maintain SLTS kernels for 10-plus years. CIP co-op packages contain only dozens of packages. They are carefully selected and will likewise be maintained for long term. You will notice that more packages, say hundreds of packages, are needed to develop real systems or devices. While CIP provides OSBL as commonly used building blocks, those additional packages should be added by users or provided by Linux distributors. CIP governance structure is explained in this slide. The governing board is organized with platinum members. It decides CIP directions overall, like CIP budget allocation or investment targets. All the technical issues or directions are discussed at the technical sharing committee, TSC. All the member companies can join TSC meetings. The meetings are usually held once every two weeks with web conference systems. Under the technical sharing committee, the six activities are proceeded as teams of working groups. We formed the CIP kernel team to work on SLTS kernel as well as real-time Linux. The testing team was formed to work on automated testing. In addition, there are three other activities, that is, CIP core team, security working group and software update working group. The CIP core team is working on CIP core packages. The security working group is working on cybersecurity standard conformance with CIP. As this one will explain about it later. The software update working group is initiating a prototype of software update mechanisms for CIP. Currently, there are eight member companies in CIP. They are actively working on those activities. Our annual membership fees are pooled as budget and used to support maintainers and developers in CIP. The budget is also used to invest projects other than CIP. One of the member companies reported that up to 70% effort reduction can be achieved by applying CIP to entire organizations in the company. It is because activities like OSS license clearing, vulnerability monitoring, kernel and package maintenance etc. can be synchronized instead of doing each by each. That is how cost saving was achieved. Up three first is our development principle. I will explain what it means. Here, two development models are pictured. The model on the left hand side is on community model. The project with this model branches its base from upstream and evolves by its own. This model enables the project to ramp up quickly. But in the long run, it will be difficult to incorporate upstream purchasing to it due to conflicts. The model on the right hand side is upstream first model. The project allows patch commits only if those patches are already in the upstream. It may take time to introduce a desired patch. The reason is that if the target patch is not in the upstream yet, it should be accepted by the upstream at first. But this model eliminates the risk of conflicts. At the same time, the project can share its outputs with the upstream. This graph shows a gross trend of commit counts for each stable release. As you can see, a few hundred patches are committed to each stable release per month. This trend makes cherry picking quite difficult. Because CIP is aiming a long term maintenance, upstream first model is a desired approach. As explained so far, the upstream first principle is essential to achieve industrial requirements, especially in terms of long term maintenance. We collaborate with upstream projects. Before using the outputs, we upstream what we have and don't keep them locally. By upstreaming, our code is incorporated into the upstream code. By using the upstream code, we can take advantage of upstream activities with our code. By continuously rotating the cycle, we are moving toward a goal. Upstream for the CIP kernel team is Linus main line unstable releases. In order to explain how CIP kernel team is collaborating with the upstream, let me start with the goal of the team. Primary goal of the CIP kernel team is to provide CIP SFDS kernels for 10 plus years by fixing versions to fulfill the required level of reliability, sustainability and security. Kudo myself is acting as a chair of the kernel team. Yamat-san and Pavel-san are working as CIP kernel maintenance and Chen Yu-san is acting as CIP kernel developer. We are highly motivated to work on the goal and to maintain CIP kernels safe and sound. Because we are following upstream first principle, upstreaming and kernel releasing are two main pillars of our activities. In general, patches are committed to the main line at first, then they are backported to each stable kernel. However, by some reason, such backporting might not be done on some specific stable kernels. It may be because such patches are irrelevant to them or because backporting is not trivial for such stable kernels due to the changes of implementations. The CIP kernel team reviews those patch status. If the team identifies some patches to be backported to some stable kernels, the team contributes to them. We care about security patches as well. We check the status of the security patches by using open source tools and if some patches are missing in stable releases, the team contributes to such stable releases as well. By incorporating necessary patches, we are regularly releasing CIP kernels based on such artifacts. Before the releases, we test kernel. In order to sustain periodical and long-term kernel releases cost effectively, automated testing with continuous integration is crucial. By confirming that everything goes well, the team releases CIP kernels. This is the big picture of the kernel team activities. Now, we are going to elaborate those tasks each by each in the following sections. Offstreaming. In the CIP kernel team, we develop super long-term support kernel. And we follow the offstream first policy and principle. As you can see, the kernel team member will start the development work in the offstream. We will review the offstream patch and review the security patch. And then we contribute in the mainline kernel or stable kernel. After that, we will use the released stable kernel and integrate it to our CIP kernel. As you can see, we review the stable patch in the very beginning phase. And those, we can fix the issue in the offstream. It's a win-win strategy. Currently, we review the patch in the RC stage from the stable kernel. So the kernel maintainer in CIP will review the patch and send the result to the mailing list. So as you can see here, the Pavel, the CIP kernel maintainer, replied the review result to the stable kernel mailing list. And we also record the review status in our GitLab. The next one is a CV check. We gather the security patch information from the mainline stable kernel, Debian, Ubuntu kernel, and then we conduct open source project, which name is CIP kernel sec. This project aims to check the current kernel status regarding the security issue. As you can see here, we can check the status via the web GUI web browser, or you can check the current kernel status via the commonline. Either way, it's okay for the developer. And then the developer will analyze the CVE to determine the necessary reaction. We may put some security patches or we may just fix the issue. All the issues are fixed based on the CIP kernel configuration. The CIP is named as the CIP kernel config. It contains all CIP members' kernel configurations. To the CIP kernel team, we only fix and maintain the issue, which only support by the CIP members. Before the CIP kernel sec, this open source project aims to check the status of security issues. And we identify by CV ID in mainline stable or Debian Ubuntu kernel. Here is the QR code you can easily get to this website by using this QR code. And we are happy to welcome everyone to contribute to this open source project. We want to build a secure and robust Linux kernel for at least 10 years. Another is CIP kernel config. As I mentioned before, we backpull the patches, which is determined to be fixed based on kernel configuration provided by our CV members. As you may know, the kernel is to be more than 2 billion of line code so far. So we have to define our maintenance scope. In the CIP kernel, our maintenance scope is based on CIP kernel config, which provided by CV members. Currently, we contribute lots of patches in the stable kernel. As you can see in LTS 4.4, 4.9, 4.14, 4.19, and 5.4, the CIP kernel team contributes more than 20 or 30, the maximum is 50 patches to the stable kernel. As we talked before, we follow the option first policy and mechanism. We want to achieve the win-win strategy. Here is the detail on numbers regarding the contribution in the long-term stable kernel. As you can see, in the 4.4, 4.9, 4.14, blah, blah, blah to 5.4, the CIP kernel team contributes, such as by report, by sign-off, by debug, by test, by inferiority of way in the contributing of the stable kernel. Currently, we have 1,777 contributions in the off-stream, and we plan to contribute more in the future. I would like to give the floor back to Kudosan. Thanks, Azizan. Then I'd like to talk about another pillar, kernel releasing. Before the kernel releases, we test kernels. Let me explain our test systems. CIP testing team is responsible for CIP testing systems. They aim to develop and operate automated testing systems to sustain CIP release efficiently and effectively. Our setup is split into three sections. The source code is stored on GitLab. Our CI builds are done in a Kubernetes cluster running on-demand AWS EC2 instances. These are brought up when needed and killed when not needed. This is managed by a GitLab Cloud CI tool. We use different sized VMs, depending on the job. As test jobs are mainly just waiting for tests to run on the boards, they only use micro VMs. Our artifact storage runs on AWS S3, and our lava master runs on another EC2 instance. Our four lava walkers run on local machines at various locations. There is currently one at Cybertrust in Japan, one at Denx in Germany, one at Mentor in India, and one at Runesus in the UK. For the CIP kernels, we are currently building 34 different configurations provided by the project members. Currently, CIP has seven reference platforms plus a candidate reference platform. The candidate platform is just waiting for support to be backboarded before it becomes official. The reference platforms cover a range of architectures on V7, on V8, and X86. Some boards are supported in both the SLTS 4.4 and 4.19 kernels, others in just 4.19. Some boards are also tested using CIP's real-time kernel. Currently, CIP is running four different test suites. The first one is a simple boot test. The second is a spectra melt and checker, highlighting which of the speculative executions the system is vulnerable to. The third is a large set of tests from the Linux test project. And the fourth is the real-time testing. For real-time configs, we are running cyclic tests with a background load provided by Hackbench. So here, we are showing the build pipelines as seen in GitLab. On the left, some pipelines are listed as an example. The middle picture is showing a single pipeline with multiple kernel configurations being built and tested. A couple of the builds failed, and the build log from one of them is shown on the right. The developer can now take this information and debug issue. All of this happens automatically, saving a lot of time and efforts for the kernel team. On the testing side, we currently need to switch to Lava to view the results. Lava is able to provide details on the test cases that passed or failed, and it links to the specific test in the logs if you need more details. One downside of Lava is that, at the top level, you can really see if a test run completely successfully on the whole. You cannot easily see if there are any unexpected failures. This is something we hope to improve by integrating with kernel CI. The kernel CI was formed as a Linux Foundation project last year. CIP joined as a premier member from the beginning alongside other founding members. We contribute through code and code reviews as well as help manage the project as part of its steering committee. The next major step is to integrate the kernel CI front end into our build and test pipelines. This should improve our visualization of test results as well as add functionality such as automatic regression detection and automatic git bisecting. Then let me move on to the last but not the least task, CIP kernel release. Again, one of the CIP kernel team's objectives is to maintain CIP kernel's sound. Through stable patch review, the team identifies missing patches and contributes them to stable kernels. Also, CIP members want to backboard their preferred patches. They send patches to CIP Dev mailing list for CIP kernel maintainers' review. By incorporating acknowledged patches, the testing team starts testing. After everything goes well, the maintainer in charge tags it as a release candidate. Another maintainer checks and acknowledges it. Then the CIP kernel is released. The announcement of the release is sent out to CIP Dev mailing list. So by subscribing to CIP Dev, you are notified of the CIP releases. As I mentioned already, CIP SLTS kernels are based on 4.4 and 4.19 stable releases. The first releases of 4.4 and 4.4 RT were done in 2017. We plan to maintain them till 2027 for 10 years. The first releases of 4.19 and 4.19 RT were done in 2019. And likewise, we support them for 10 years until 2029. Currently, 4.4 is released once a month. 4.4 RT is once every two months because commit counts for 4.4 are decreasing. 4.19 is twice a month and 4.19 RT is once every two months, respectively. So far, we have steadily released kernels thanks to our maintainers by following release frequencies I just explained. I also reported the commit counts at ELC North America 2020. Compared with the counts in June, the team made 24 additional releases and the year total so far is 50. Tow the end of this year, even more releases will be added. Here explains how CIP artifacts can be used by CIP users. CIP refers to source or binary packages in Debian. If you'd like to use Debian source packages, you can use YoctwoKey as a build system. CIP core packages contain tens of packages which may not be sufficient for the development of end products. So users can add necessary packages from Debian by writing recipes. Debian provides LTS maintenance and even extended LTS can be provided. So super long-term support, including user-run packages, can take advantage of these maintenance frameworks. Upstreams of current CIP releases are active and we are taking advantage of their outputs while we are contributing to them as we explained so far. However, because we intend to maintain CIP SLTS kernels for 10 years and the upstream life spans are both six years, therefore the gap periods should be maintained by CIP. Because maintenance of 4.4 stable release will be finished in January 2022, CIP will start to maintain CIP 4.4 by ourselves. The CIP kernel team is discussing how to work on this. We have been relying on upstream developers and other stable contributors for their outputs. CIP cannot rely on this after the end of stable release maintenance. So CIP kernel maintenance would review each other's work. The details are still being discussed and I hope we can share with you the plan at some event next year. So, Azizan, please. Okay, let's talk about the CIP kernel with the cyber security. It's a trend to exploit the vulnerabilities of the industrial control system, such as a C4 infrastructure platform which is related to our project. As you can see in the slides, most of the vulnerabilities released in the recent years are related to industrial control system. And those we have to find a way to reduce the cyber risk. Currently, we are using the IEC 62443, which was divided by integrating the standard of the significant industries. 2443 is an international standard series that integrates major industrial security standards for each industry. For operators, building a secure supply chain with reliable equipment is very important. To keep the control systems secure systems and components made in control system need to be implemented with a secure development process. And, of course, its security features should be measured by the latest cyber security. So, as you can see here, we use the 6443 to enhance our security in the CIP platform to reduce the cyber risk. Moreover, the IEC 62443 standard can support the golden triangle from the CIP. The 6443 can support the industrial grid because it's the standard for industrial automation control systems. It can support the sustainability because it has been added with awareness for industry 4.0 and has high interoperability. This is the reason we focus on this standard certification and design guideline. To repeat for those who are listening to our presentation for the first time, the security mission is to provide an open source base layer for developing product components with the IEC 6443 certification security requirements. To achieve industrial grid security, we focus on IEC 6443 certification as an international standard for industrial automation and control systems. More product suppliers who develop using NINACs and package software acting on it will take advantage of our solution and get IEC 6443 certification to secure the industrial and also it can reduce the cyber risk. The CERVICONO team is using the CERVICONO SAC to track the CERVICONO CVE which meets the 6443 dash 4 dash 1 requirements. So by using the CERVICONO SAC we can easily to know the current CERVICONO status regarding the security issues and those we can fix the CERVICONO CVE or just make the check after applying the security patches to make sure our current kernel is secure. This slide shows the flow of how security packages are suggested, implemented and tested. We consider a security package and the security working group suggested the package to the CIP core team. The CIP core is another CIP working group. The CIP core team they are knowledgeable in the package software, those double check the package selection and in the software reference design the implement in combination with the super long-term support kernel which is implemented on the CIP reference hardware and validated on the level lab. The standard of the 6443 dash 4 dash 2 also has the security requirement that it meets to comply with the security level 3. The target of the assessment that we are working on now is software and the compliance of those hardware requirements should be handled by each reference hardware provider. In this way, if the hardware kernel or even the core package that's run in the user space conform to meet security requirement for 6443 standard, this can say that this is the best platform for user product development to complete the certification. Here are some advantages compared to the CIP versus the CIP distributions. In CIP projects we maintain the super long-term kernel up to 10 years and we performed 6443 certification and we monitor the CVE in the CIP kernel. Moreover we also support the Debian extended long-term support packages to secure its security and reduce the risk. We also have the automated testing multiple system chips with the published text results on kernel CI, which is the Nina Foundation project. But not the least it is a strong support from big players of the embedded system industry. And I would like to give the floor back to the Kudo-san. Thanks Sissan, then let's wrap up today's talk. By following upstream first principle, CIP kernel team works on two main pillars upstreaming and kernel releasing. Because one of the current major activities of CIP is for CIP to be certified with IC62443-4 CIP kernel team is also participating in this to achieve and sustain industrial greatness. We are eager to recruit new members. If more people work with us we can expand our activities more and we can contribute to upstream and other projects more so that the whole ecosystem will grow. So please join us if you get interested in CIP. If you'd like to know more, there are links for related information. This page talks about our weekly ILC meetings. It is open to everyone. Come talk to us on CIP channel other meetings. This page talks about repositories on GitLab. Links to open source tools explained in this session are here. And the testing teams links are here. Please check out. And other information is listed here. Thanks very much for listening our talk till the end. We hope you find this useful. We'd like to use the remaining time to respond to questions. So if you have any please feel free to send us messages. Also, even after this session please send any queries to CIP deaf mailing list. Then thanks again.