 Today, I would like to present the topics using open-source software to build an industrial-grade embedded Linux platform from Strat, Scratch, HZ-Lin, for Mosa, a Taiwan company, and also cybersecurity fundamental specialists and deviant developer. You can reach me in this website. For this presentation, I would like to make a survey. How many people is familiar with industrial platform? Could you show your hand? So actually, this topic, this presentation is quite difficult for me because I met over 200 slides in my first video. I tried hard and I caught it into actually seven slides, so I will try to skim and finish this presentation in time. Industrial platform such like page, industrial computing, and network, according to the industrial application is fairly such like a rail, smart grid, and some energy. And those, this type of product is quite wide range. We can have microcontroller or server level, so we might have a lot of choices on this. So before using open-source software, something we should know. That is copyrighting pattern. Copyright is very, very important for the open source and which related to the open-source license. Another is pattern. So for the open-source copyright, luckily, the Nenas Foundation created a project named Open Chain. It's an identifiably recommended process for effective open-source management. We can easily manage our open-source via this project. It teaches us how to manage, how to control our usage with the open-source. Another is Open Invention Network. It's a shared defensive pattern pool that can protect our Nenas. So again, we have to take care of open source, copyright very carefully. We have to use the process to lean and support. We have Open Chain, SPDX, for the large. Both of them are Nenas Foundation projects. We use Open Chain to control, manage the open source. We use SPDX to exchange the software package. We use for the large to scan in the license. Okay, back to the topic. Industrial harsh environment, such as rails with transport, factory automation, or gas. There are some common basic features, such as longevity, stability, and security. And there are also some advanced features, such like performance, real-time, resource limited, and safety. Beware of that, some of the features needs a special power. So we need to evaluate in the very beginning. Okay, this is a life cycle of industrial grad and bad Nenas platform. As you can see, we have a development phase, maintenance phase. In the development phase, we have system preparation and building and testing. In the Nenas phase, we have a long-term test and regular update for the industrial product and the platform. This is very important in this phase. That is because the life cycle, the industrial platform, usually more than 10 years. So we need to take care about the Nenas phase very carefully in the deployment, in the development phase. As you can see, this is the prototype for embedded device. We have hardware, we have architecture-dependent firmware, such like ARM and some BIOS. And those, we have a loader, kernel, kernel, and system code interface inside. You can see the Nenas system and user application there in the roof assist. So we have user space and kernel space. First of all, we have to choose a proper loader. The loader is trying to initiate the hardware and load the loader inside the D-Rain. Then load the second loader to initialize others and load in the Nenas kernel, and then the Nenas kernel will decompress itself. So there are some categories for the loader. Typically, we will use the U-boot in ARM product and the GIUB in X86 product. But for some experts, they will use the core boot or IF find. In kernel, they are two main kernels. The first is Nenas kernel and the second is pre-emptive RT kernel. The Nenas kernel are suitable for the performance and resource-limited target application. Before maybe 10 years ago, in resource-limited applications, we usually use the Micro-C Nenax. But in the recently, Nenas kernel support no MMU architecture and support the tiny kernel. In those, we can use the Nenas kernel in some resources-limited platform. And for real-time kernel, it's suitable for real-time functional safety and resource-limited target application. By the way, I have listed some references for real-time applications, such as SinoMind or RTAI. Okay, once we decided the type of kernel, we have to get the kernel. So there are many sources. The first is SOC boot support package kernel, that is BSP kernel. So this kernel was provided by SOC vendor. If we use that kernel, it causes some issues, like different SOC might use different version of kernel. That means if we have more than 10 SOC, we might maintain 10 versions of kernel in the same time. Most important that the lifetime is unsure. So let's take a look at LOTS long-term support, a long-term stable kernel. As you can see, the Greg and Ben Hatchins maintain 5 kernels here, 5 versions of kernel. So the longest time is about 6 years. It's attained the software uptime for stable kernel, only accepts bug fixes and security fixes. But 6 years is not enough for us, especially in the industrial application. So we have another choice of status, LTSI long-term support initiative. This is a NINUS foundation collaborative project. It's based on LTS and it has an auto test framework. It also adds some new features inside the kernel. The next is CIP, Seville infrastructure platform. This is also the NINUS foundation collaborative project. We also joined this project because we want to support the NINUS kernel more than 10 years. So we have a supportive strategy, we have auto test framework. Our goal is to maintain the NINUS kernel more than 10 years. This is also the project NINUS foundation. So this is the bigger dots you can see, NINUS kernel, LTSI and CIP, the relationship. This is the table version. As you can see, the maintenance period, SOC vendor is unknown, LTSI is 2+, and CIP kernel is 10+. For the features, SOC BSP and LTSI accept bug fixes and security fixes. But in LTSI, it accepts new features, so does in CIP kernel. So the latest version, the LTS kernel is 4.19 and CIP is 4.19, but LTSI is 4.14. For the real-time kernel supported, only CIP kernel support the real-time kernel. So that is the conclusion. Just like I mentioned, there are many applications such as performance real-time. Is that mutual access? Yes, some of you enable the real-time, it may decrease your performance. So what if I want to enable a quadra with different applications to fulfill the multiple scenarios? In on-base, we can use the FIT, front-end image tree. These formats can support the devices to support different kernel and different applications. As you can see, this is an example. You can list the image here, and another image below. So we can have multiple kernel, multiple unified systems in the same product. And those we can configure this product to fulfill the target application. The next is user space. I want to address Elisa, it's a safety critical systems project. This is also the NINUS Foundation cooperative project. This project aims to provide the solution in functional safety application. So it's used to sell to NINUS MP and real-time kernel inside to fulfill the IEC 61508 standard. So if you are focused on functional safety, you can have a look in this project and sell to NINUS MP and real-time kernel. For others, the library, this is very important for the system. There are some open source projects there. For example, G-Lab C, micro-C-Lab C, and G-NEC generation and muscle. Their lessons are quite different. As you can see, they also have some different features. For example, G-Lab C supports stable ABI that was compatible and for simple phrasal, some security feature. And for micro-C-Lab C and G, it's mainly to support the normal new architecture and for tiny size. So if your application is for resources limited, you can consider to use micro-C-Lab C next generation, this tool. And the muscle is also suitable for resource limited application and security application. But please be aware of that the year 2038 problem. This is the issue similar with year 2000. The 32-bit system can be reprinted dates from 1901 and 2038. As you can see, when we get the dates of this time, the integer will overflow in 32-bit architecture. You may mention that there are still 19 years from now. But as I mentioned, the industrial platform usually have a very long life cycle. So we should focus this issue now, otherwise it's hard to fix. So when you choose the C-Lab, you should consider this problem also. Okay, next in this system, in this system, the first application which will be executed by the kernel is managing the application and services. There are some open source tools here. For example, busybox, 6Vnet, system D open RC. Typically in the past, the embedded system will use the busybox because it's very tiny and easy to use. However, the industrial platform on the internet is the trend. So many devices will use the system D or open RC. So you can choose your own internet system depends on your application. I also list the C-Lab library supported. For example, busybox supports Micro-C-Dash next-generation and G-Lab-C muscle. And for system D, only supports G-Lab-C. So if you want to use the Micro-C-Dash, system D is not suitable for you. The next one is to choose the proper RU-Fi system. There are many open source projects to provide the RU-Fi system. For example, busybox, YAKTO, RELOOT, and Debian. You may mention that YAKTO is not a RU-Fi system, RATTO is a tool, yes, exact. So it depends on the meta layer which can provide the RU-Fi system. So busybox contains 300 to 400 applications. So it's not a package, it's just inside the busybox. So it could be a very tiny but functional limited. It's also only supported Micro-C-Lab-C and G-Lab-C. And in YAKTO, the famous project in Nina's Foundation, the lifetime is only supported previous two releases. And the celebrity only supports the G-Lab-C muscle. For BELOOT, the famous RU-Fi system and BELOOT tool is embedded system. The maintenance period is only one year. The number of packages is more than 2,000. It supports G-Lab-C muscle and Micro-C-Lab-C next generation. And the next one is Debian. Debian, the maintenance period of Debian is five years for specific architecture. It supports more than 50,000 packages in Debian. It also supports G-Lab-C muscle. And there is some myths. You may say, Debian is very heavy and very fat, very easy, very hard to customize. That is not true. You can customize a Debian into a tiny profile easily. So we can create a very tiny RU-Fi system for Debian in our embedded Nina's. So we have BRODA, we have kernel, we have RU-Fi system and something like that. So we have to have system development tools like actual BELOOT or some Debian specific tools. As you can see, you can use the actual to build a busy box RU-Fi system. You can use the BELOOT to build a BELOOT RU-Fi system. You can use many tools to build a Debian RU-Fi system. So I also list the tool chain supported here and the license belong. So you can choose what you want. And I want to mention that the left view is a very powerful tool for Debian development. Many directive distributions use the left view to customize the Debian, such like in Kali. Okay, so we use Debian. Why? Because its stability is as a procedure that is stable, testing stable. Scalability from a laptop server to embedded devices. With system security, the Debian has a security tracker. Long-term support for years and multiple architecture and the end is more than 50,000 packages inside. In the build and testing, it's a very important for industrial reform. So if you want to get more information, you can check this in reference in the end. This is our testing and development procedure. As you can see, the developer forked the git and made some changes. After, they will send the patches to send a pull request. After that, this will jump to CI-CD automatic release pipeline. This pipeline will result, yes or no, to our team. And we can pass or not. This is the CI-CD automatic release pipeline. CI means continuous integration. CD means continuous deployment. So we have CI, CD, CD, building, deploying, testing and release. So we use the GitLab and Jenkins to monitor the status of the branch. There are some changes. We will trigger the Jenkins. The select node will do some static program analysis. We use the Jenkins to manage the static testing cases. So, as you know, the nearest kernel is quite large. So we spend lots of time in compiling. We use the distributed compiler to reduce the compile time. This time, here, we use Ice Cream. It's created by Susan. So we use the distributed compiler to reduce our time. And those, we can have more time to do some tests. So as you can see, we have the building form. So in the delivery, we use Lava. This is a tool made by Ninaro. We can use this tool to make some deployment to our testing form. Once we get the binary and the image from the Jenkins, we will send a job file via the XML RPC. And it's sent to Lava master. It will dispatch job via 0 and Q. And then, we will download the image from the Jenkins and send the image to the devices. And next, we will trigger a test framework. And this test framework will send the test cases and start to test. We have several tests, such like dynamic program analysis and platform test, to ensure our quality of the product. So there is an embedded test framework I would like to address. This is Friago. Friago in Spanish means fire. This is a test framework for testing embedded NINAs. So our LTS project, which is a NINA Foundation collaborated project, used this framework to make a test. And there are over 100 package tests already. So after that, we can get the results and send to the developer. OK, the next is maintenance phase. In the maintenance phase, we have a long-term test. Long-term test means longevity testing, reliable testing, robustness testing, and secure scanning. We have to do this test 24 hours, seven days because we have a very, very long cycle for our product. So the test cases are managed by the test framework. First is longevity. It's mainly to test the hardware. So we have endurance and capability test. The next one is robustness. We have a fuzzing test. And I also list the test tool in the end. The next one is the reliability test. We have a power failure test, reboot test, and regression test. You might be curious why we have to reboot to test our product. Because we have several issues before. Sometimes the system will hand during the reboot. And the root cause is that the driver or the application has some deadlock issues in some situations. So it causes devices and the platform to be hand. And the last is a security test. We have the daily test for CVE. So as you can see, we will scan the platform every day to generate security records. As you know, the last week we have SAC, S-A-C-K, security issue in kernel. So we can easily to get the result in each platform that is still in CVE inside. And for stable kernel maintenance, we will have the kernel CI to test our kernel. This is an automatic NINUS kernel testing tool. So it's where the TET, BISEC, report and vector regression upstream kernel trees before release. We can have more confidence before releasing the kernel. So it's a short test on many configurations. That means we can test many boards, many SOC, many devices via the kernel CI to make sure that each changes is OK. The next one is reproducible build. This is a very amazing project. It's also the open source project. It's greater independent, available, verifiable path from source to binary. It ensures the build have identical results. And then we can prove the source code has not been tampered or modified. It's very important for the security. So I also list the open source testing tool below. You can have some look. So the CI CD LT are concepts of software engineering instead of tools of the procedures. Because the tools of procedures could be phase out. So we need to focus on the concepts of software engineering. OK. We come to the next step is a regular update. As I mentioned before, the industrial platform, the product, we have a very, very long lifecycle. So if we want to update the security issue or some bug fixes, we have to update the devices remotely. Because in the industrial application, the platform usually to put in the human-less place. So we need to have the software updates to fulfill the demand, like bug fixes, security fixes, new feature over 10 years. At least some components might be updated. For example, the peripheral devices of firmware, broader devices in Linux kernel, Lufa system, system configuration, and replication. Typically, we update the application more often. And some of the Linux kernel depends on the situation. But I also risk some risk in updating the software. If we fail to update, the devices might become the stone. So at least some requirement here for industrial Linux platform, for example, the hash environment is the middle of nowhere, then we limit it. Multiple versions supported, multiple devices, and then longevity. These are no more requirement for the industrial platform. So the media of the software update, we can use a wire cable, OTA, portable shortage, storage, or unsigned. But typically, we don't use this solution. Too much cost. I also list some software update requirement here. For basic features, we should have a failsafe, rollback, size reduction, signature, and multiple storage support, your system integration, remote access, for additional future, such as online and offline updates, encryption, data updates, successful updates, detection, and proactive updating. And for the industrial embedded devices approach, we can use the image block based. It's a quite large size. But it's much simple. And file based, package based, and database. You can choose what you want, depends on your demand and application. You need to consider the partition architecture here. So you can have some design in your embedded device, and those you can make the best update mechanism. There are some design here, for example, a symmetric and symmetric firmware update. This is a symmetric firmware update design. This is a symmetric firmware update design. For this design, we will have some downtime because we only have the main OS. But once we get dead, we have a recovery OS. So we have failsafe. But for the symmetric firmware update, we have backup OS. So we will keep alive OS. But the drawback is that we have double copy of OS. So you can choose your design based on your application. So typically, in the very low-end devices, we will use this design. But in some high-end application or devices, we will use a symmetric design. It depends on your storage and your mechanism. So I list some open-source projects for software updates as the regard. That means software updates. And I use OS 3. So software updates and I use support failsafe and rollback feature. They also support the delta updates. Signature for security. They support multiple storage. For Yachto and Vrewroot, in my experience, I use software updates because it's very easy to use and suitable for embedded devices. So there are some comparison for others. Software updates and I use support as symmetric and symmetric updates. They support image base and file base. The license is GPL and LGPL. OK, this is my conclusion that we have to plan our design and target application. The industrial application is on the stability and security. We need to collaborate with the community. So we use many open-source. And that's why we cannot the best solution but the suitable solution for our application. That is my presentation. Thank you. So yeah, do you have any questions? I present very quickly because yeah, many slides. Do you have any questions? OK, thank you.