 Okay, welcome to my presentation. Today I will talk about delivering security fixes continuously into your embedded Linux. Main target audience would be embedded product developers but not limited to them. So I hope this talk will be helpful for you. First of all, let me introduce myself. My name is Hirak Toyoka. I live in Yokohama and I'm a senior manager in EM Linux development team at Cybercrush Japan. EM Linux is a Yocto-based Linux environment for embedded or IoT systems. We have been developing and maintaining EM Linux since 2019. I'm also a maintainer of meta EM Linux and MetaDebian extended layer. These are parts of EM Linux. And I'm also contributing to CIP testing and MetaDebian a little. CIP is a collaborative open source project for industrial grade software. Cybercrush Japan is a member of the CIP project. And I have some experiences of contribution to React tunnel. Today's agenda is here. I'd like to start with background of this talk. Next, I'd like to talk about some considerations to get prepared for security updates. After that, I'd like to introduce how EM Linux provides continuous security updates. Okay, let's start. Nowadays, security updates are becoming recognized as essential even in embedded systems. Because lots of cyber attacks on embedded or IoT devices have already been observed. These attacks are against from familiar items such as home appliances and cars to infrastructures such as buildings, factories and plants. In addition, cybersecurity related standards, guidelines, regulations require security update processes. These standards are established by each country or each industry. For example, IoT device in general, automotive, medical or industrial control systems. And procurement requirements may include such standards. So you may be involved even if those standards are not legally enforced yet. On the other hand, how is vulnerability handling in Linux community? For example, in Linux kernel, 150 to 350 vulnerabilities are reported annually. Greg's presentation says average fixed date is minus 100 days and bugs are fixed before you realize it. So this means, certainly Linux community release security fixes quickly in most cases. But what about your embedded Linux? It is often observed that product developers are unable to leverage those fixes for their own products. So you may not able to leverage those fixes for your product when you don't have an appropriate maintenance plan or processes or your embedded Linux is inconsistent with open source upstream ecosystem. From the next part, we will see some points to be checked with how such situations happen and what we should do to change such situations. Next part is considerations to get prepared for security updates. First point to be checked is do you have any maintenance plan or maintenance process? It's too late to think about them after shipment or just before shipment. So if you don't have any plan or process, then you cannot respond quickly to sudden incidents or in addition, your embedded Linux will become difficult to maintain. It is described later. So you'd better to consider the appropriate maintenance plan and processes from the beginning of development. Such plan or process may include maintenance period cycle update model, how to check vulnerabilities, how to prepare the fix, how to verify the update, how to deploy the update and so on. And next point to be checked is does your update process work? Now you have maintenance process prepared five years ago, but is the process itself still maintained after shipment? If the update is assumed as only on demand or once in a decade, then nobody may have experienced the actual update or developer who created the process may no longer be employed or software update feature or infrastructure may be broken. So we need some kind of measures. I think the most reliable way is to practice or check the maintenance process continuously. Periodical practice will prevent these problems. Next point to be checked is where do you get the security fix from? Linux kernel in embedded product is typically based on LTS long-term stable version and customized for your hardware. LTS version has a limited maintenance period. For example, recent Linux kernel LTS trees have been maintained for six years. So once the maintenance period of the upstream LTS version ends, it will be difficult to get security fixes that match your tree. So you should always use the actively maintained version or you need to backport the fixes to your tree by yourself, but it's costly. Next point to be checked is how do you cover the long-term lifecycle of your product? The maintenance period of LTS version is limited. So you will need some kind of update model one possible way is long-term support model that keep using one LTS version as long as possible. If the lifecycle of your product is longer than maintenance period or the maintenance period of the LTS kernel, major version upgrade is necessary. There are pros and cons in long-term support model. Pros is less opportunities to modify product-specific software. Product-specific software means drivers and applications. Cons is a large migration cost at version app and possible delay in security fixes in later phase. Rolling update model always use the latest LTS or stable version. Pros is a quick response to security bugs and new feature utilization. Cons is more opportunities to modify product-specific software. Update model is not limited to these ones. Anyway, it should be considered according to your product. Next point to be checked is how do you take the security fix into your product tree? Cherry picking only the fixed patch from the LTS tree is not recommended for the LTS kernel because LTS trees merge many changes at the first phase. So, prerequisite code is, so prerequisite code to apply the fix may be missing in your tree. If you cherry pick, you need more time to confirm, to confirm if the bug is really fixed. In addition, other non-security bugs will remain in your product. So, you should take all LTS updates into your product tree. And also, you should sync your tree with the latest date of the LTS tree continuously. It is useful to keep your product, it is useful to keep your product ready for the next security updates. Now, I cherry picked some things from other presentations. So, Greg says, small fractional kernel fixes get CVEs and cherry picking CVEs result in insecure system. And Kato-san's slide says, we are using LTS is not enough. It doesn't make sense if you don't keep incrementing x of 4.19 point x. In this way, taking all updates are recommended. Next point to be checked is, how much out of tree code do you have? Out of tree code is the code not merged to main line. Product kernel tree is often based on the BSP or the SOC evaluation board. Most BSPs contain out of tree code or quick evaluation of hardware functions. And additional out of tree code is added when supporting product hardware. But massive amount of out of tree code causes lots of conflicts when you take LTS fixes into your product tree. Then your embedded Linux will become difficult to maintain. On the other hand, some hardware vendors are doing upstream mainlining activities in the Linux kernel community to have their own hardware support code integrated into the mainline tree. You can avoid huge maintenance codes by selecting hardware whose support code is well integrated into the mainline. So software ecosystem is also important when selecting hardware. Next point to be checked is, is your QA system ready for continuous updates? So QA process will be required before the update in the field. So more QA opportunities lead to more cost, especially if it is most, especially if it is mostly done manually. And lead time is also an issue. So test automation, test automation or CI-CD is required for continuous and quick updates at a feasible cost. Next, I will explain how EM Linux provides continuous security updates, as an example of continuous security updates. EM Linux is a Yokuto-based embedded Linux environment. It is developed and maintained by CyberTrust Japan and it is used by various embedded product vendors. Its purpose is to deliver security fixes and bug fixes to users continuously. This is a brief overview of maintenance plan and maintenance processes for EM Linux. Please note that EM Linux is supposed to be customized. So it is not directly deployed in the final product. So maintenance period is 10 years, a major version, standard five years plus rate phase five years. And maintenance updates are provided monthly. Update model is based on LTS model plus major version upgrade will be provided every four years on the current plan. But we may shorten this to two years in the future. For checking vulnerability, Yokuto-CV check feature extended for EM Linux is used. And we have in-house CVE review process. For preparing the fix, we take all upstream fixes from CIP super long-term stable kernel and Debian packages. For verification of the update, we verify changes continuously with in-house EM Linux CI system. For deployment of update, it depends on users. So I don't explain here. So the reason why we choose LTS model is this. LTS model is easier for most users to start the security maintenance process, including CI CD processes. And in LTS model, there are less changes in features or interfaces compared to loading update model. That leads less impacts on product specific code and test cases are reusable for a long time. That said, we will also provide major version updates for strict security requirements. To realize EM Linux maintenance period and cycle, we leveraged super long-term stable kernel maintained for 10 plus years by CIP project. Cyber Trust Japan is a member of CIP. So we are making some contributions to these activities. SLTS kernel has upstream fast policy and all LTS bug fixes are merged. SLTS kernel team is making twice a month release for 4.19 and 5.10 kernel currently. So we use CIP kernel sec to check kernel vulnerabilities. CIP kernel sec is CIP's Linux kernel CVE tracker. This is maintained by Masami Ichikawa from Cyber Trust Japan every few days. This is integrated to EM Linux to have the Yoctos CVE check feature output accurate CVE status for SLTS kernel. This is an example of CVE issue file. So introduced or fixed revisions are included. EM Linux is leveraging also Debian user land packages via MetaDebian layer. MetaDebian is a Yocto project extension for using Debian source packages. It is created by Toshiba people. As a feature of Debian packages, stable version typically accepts only bug fixes. And it has three plus two years maintenance period and extended LTS as a few years. And some packages will be maintained as CIP core packages for 10 years. By using MetaDebian, we can update package version to Debian's latest version. And Debian security trackers data is integrated to Yocto CVE check. In addition, Cyber Trust Japan made MetaDebian extended layer. This includes additional packages for MetaDebian. And EM Linux is continuously verified with in-house EM Linux CI to support monthly updates and daily development. And found bugs and issues are feedback to each upstream projects. EM Linux depends on many open source projects related to test automation. We are making lots of contributions to build a better test automation ecosystem. It is also helpful to stabilize EM Linux. Projects related to test automation include kernel CI, CIP testing, Buildbot, and Lava. Especially in kernel CI, Alice from Cyber Trust Japan is a TSC member and maintainer for some sub-projects. So sub-projects includes Lava Docker. And when we find issues or make bug fixes, we try to give feedback to upstream projects as much as possible. It is also helpful to reduce self-maintenance over the long term. Projects we have provided feedback on include Linux kernel, Linux LTS kernel, Debian Yocto project, OpenVendate, UTL Linux, and so on. Okay, let me recap my talk. I talked about how to deliver in security fixes continuously. To deliver security fixes to your embedded Linux, we need to consider a maintenance plan and processes from the beginning of development. And we need to keep your embedded Linux constant consistent with open source ecosystem. And we need to make whole maintenance system ready for continuous updates. Yeah, so yeah, that's all of my slide. Thank you very much for listening. Do you have any questions? Yeah, sure. Hello, thank you for the good talk. So you said that people shouldn't cherry pick LTS fixes, you should take whole LTSs and integrate them as one thing. I wonder if there's downsides to that as well because we've seen that regressions have been picked into the LTS branches automatically, then they've got merged into people's trees and no one's noticed that actually they've not merged security fixes, they've merged regression by mistake. Is there like, I think just merging LTSs is enough, people need to do some sort of due diligence on what they're merging to make sure they aren't also merging regressions with security fixes. Do you think there's like some process that has to be added on to just not just get merged but some post merge process that has to happen for people to properly use LTSs? Sorry, I couldn't catch the point. Could you just talk one more slowly? Okay, so say there's an LTS and it has a fix you want, you don't just cherry pick the single commit you want, you take the whole, so 4.900 million whatever it is now, you don't take the single fix, you take the whole LTS into your branch and you add that to your branch. But we've seen in the past that because the LTS process is automatic, like it's mostly just like a script, we've seen that inside of patches you want inside the LTSs, there are also regressions for things. And so I'm saying is you can't just get merged everything, I think there has to be some like post process after that. Maybe basically, do you think it's safe to merge LTSs like in one big go? So, I couldn't catch it. So, you mean, so, so, so taking all updates, make some programs you can think. I don't know. I got it, yeah. I got it. Yeah. Yeah, I got it. Thank you very much. Yes, so, yeah, verifying each patch is maybe necessarily depends on, depending on use cases. Yeah, so, yeah, it depends on, so it depends on quality level required to the final product. Yeah, any, so anyone else? Thank you for your presentation. So I have one question. So sometimes it is difficult to define your test case for the security fixes. But when we merge the security fix, we need some regulation test. So how do you do, what test case did you do to ensure that after merging security fix? Yeah, so it depends on cases, but so we, so we can test some interfaces such as CISC or API or other APIs. So, so for checking, degrade, so, yeah, checking if degrade has happened or has happened or not. Yeah, so, and so in, so in, in, in the NACCI system, we test build, boot, package test, small test, network or graphic test. Yeah, but so it depends on use case. So if it is enough or not. Okay, thank you. Anyone else? Thank you for your presentation. I have a question or the one in page 22. You said, yeah, some major version upgrade also will be provided for street security requirements. That doesn't mean you may provide a major version upgrade for specific packages for the, any, many kind of packages at the same time. I just want to know, yeah, some, some impact. Have you ever observed some impact of this change of the, yeah, major version upgrade because you were based on LTS, no, no, LTS style update? Yeah, so this major version upgrade means whole, whole, whole system upgrade. So, yeah, so both kernel and user-land packages, all of user-land packages. But major version upgrade of specific packages is also pass-passable. Okay, yeah, thank you. Anyone else? Okay, so yeah, that's all my presentation. Thank you for attending.