 Good morning, good evening, good afternoon. If you wanted to know more about securing a Yachter-based distribution, that's the right session. My name is Marta Reptrinska and I have a security researcher background and a PhD specialized in network security. You may see a bias during the presentation, maybe not. I have been working in open source for the last 20 years, including the position of Vice President and Treasurer of KDE EV, the organization behind the KDE desktop environment. And a part of that I've been working on hardware related issues like kernel, porting, low level networking, intrusion detections and more. Currently, I am involved in a group called OpenSSF and especially the best practices group. That's the beginning of a talk you would expect from a security researcher, right? Some headings from news sites, millions of IoT devices exposed to attacks due to a cloud platform vulnerability or critical flaws in millions of IoT devices or even more. You are doing IoT wrong. This one is especially funny because it shows a few issues at the same time. From my reading of the research work, the authors show first that some developers do not check error values when it comes from hardware related functions. Of course, everyone in this audience checks error values of every function, right? And the other finding they are showing in the presentation and the research is that sometimes the hardware that is expected to give you random numbers doesn't really give you random numbers. And you should be checking if what it's giving you is actually random. This is probably less frequent that you check for that, don't you think? So a lot of issues, important in the media about embedded devices and IoT devices in particular, you have the issues coming from external suppliers, the cloud, for example. The code you reuse, one of the examples was about network stacks issues, the libraries. You have the software hardware interface right in the random number generator, and of course, you have your own code. That may be scary. And it could be if you try to address everything without a plan. So today, I'm going to cover a few subjects. I'm not going to convince you that you can, in five minutes, secure your IoT device. But I'm going to show you a few ways how you can improve the security of your embedded product in a significant way, especially using Yocto, right? So what we are going to talk about today. We'll start with security basics of Yocto. Then we'll look in my specific use case in particular, because there are a few things that you should know about the work we do. Then we'll look into two items in modters, about the CV checking, if you don't know what CV is, you will learn. And then about the hardening using metasecurity and related layers. I would like to cover also community work done by different people in the community that is currently in progress. And the next step of the work I'm planning to do. Let's get started. A security searcher view of Yocto. There are two main important features. The first one is that you can easily add new software. Easily, it means that you can add it with all of the dependencies. But it also means from the security point of view that you have a risk of including buggy or in the security naming vulnerable components. If it's easy to add, you may, by an error, include something that you should not have in your system. And then, on the other hand, what you have is a large choice of security tools that are actually already available in Yocto. Those are especially in layers like metasecurity, meta-ecilinox or meta-virtualization. Some of those may require a little bit of knowledge to configure and understand the input. But if you are ready for some work, there is no issue with that. And you can benefit from them as you need. Now, from a little further, we do have LTS in Yocto. It means that the project is supporting with the latest fixes. A version of its software for a long time. This is a new process. It has started from the Danfa release, a release that happened in April 2020. So it's one year and a half right now when we are recording this. And an LTS in Yocto is a set of layers. All the layers that are not in the basic set may have their own LTS policies. The support is expected to last for at least two years. And it's possibly none overlapping with the next LTS to come. There is a wiki page of the Yocto project describing all that so you can look into more detail. But what it may mean to you that if you add your own packages or your own personal layers, they are not included into the LTS. So you have to take care of updating them yourself. And then the documentation. If you are trying to secure a distribution using Yocto, what should you be looking at as a priority? This answer is somewhat complicated to find. There is a guideline of the Yocto project about making images more secure with helpful suggestions. And there is more work planned, there is a link, I give you a link to a bugzilla issue about this subject. Now what do you have out of the box in Yocto? And those are the items that are linked in the Yocto guidelines. You heard the compiler facts. You just add one include security flags into your local comfort distro configuration and here you are. Then you should make sure that the debug flags are not there in your release build. That would be unfortunate. And then one big but easily overlooked issue. By default you may get a root login with a simple password. For a product it is way better to add regular users, disable root login and set up passwords, ideally different for each device or require a password change after the first login. Now let's get into the specific case of Austin Rios OS. So the specific case I'm addressing and I'm giving you the lessons learned from. So what is Austin Rios OS? It's a source-based distribution with multi-carnal including Linux, Zephyr and other kernels. And of course it's using Yocto. The distribution is targeting IoT devices with a communication between different classes of devices. That's why you have the Linux and Zephyr. And privacy with security are one of the main goals. I'm not going to spoil more and you can learn more about the distribution from David's presentation that is also presented at ELC. From the security standpoint, we have already done security work on the distribution and some other work is in progress. So what do we have in there? Is there hardening that is added by default in the distribution? That includes Linux kernel options, CTLs, the compiler's options. We've done some image hardening. What means removing services that we do not need? Adjusting the permissions of different executables and services. In tooling, removing unidded services yet again. And we do CVE checks. Two items, CVE checking, Linux kernel options. That's something I'm going to cover a little bit more in the next minutes. So let's start with CVE checking. What you should know about CVEs? First, CVE stands for Common Vulnerabilities and Explorers. It's a base of vulnerabilities. Each has its unique name so you can search easily. And the vulnerabilities have the name in the format CVE year and then some digits. You can find more information in the typical places like Wikipedia. And you can also search the CVE database. What it comes? You have the ID, of course. You have a description. You have the product name, the vulnerable product name, of course. You have the versions of it that have the issue. You will have the score of how serious it is. And you will get some references, links to resources, like the mailing list posts describing the issue. And yet again, you can search them. Now, how it looks like? Yeah, in this example, I'm searching the database for Gileap C. And you can see that we have 166 entries with the CVE numbers, the CVE IDs given. You can click on the CVE ID to see the description. So here, let's do that for the first one. The one we have at the top. And we will get something like that. We have the same description. We have some links to resources. And we have more information like where the record has been created. Now, what a developer should know about CVEs? A CVE is given only when someone requests it. So you can perfectly have a security issue without a CVNAMP. One that nobody decided to apply for a security issue, or even worse. You may have a security issue used in practice called a zero day, without a CVNAMP until it gets discovered. CVE, if there is a number and a description database, it means all the information is back. The database of CVs, it changes. There are some numbers that get reserved from different entities working on the security issues. The content, the description, and the links will be released when the issue is published. And at the beginning, sometimes there are some mismatches like the versions that do have the issue or not may change. There are product names that will change. And you have other databases with more information. And I would like to especially point you to the NVD database with additional information. And then we are going to look for the same vulnerability in the NVD. It looks like that. You will have the same description if you remember the previous one. And then you will have the description of the severity. We have it something in red. You have some hyperlinks that run and even more information. If you zoom at the severity, you do have the base score, seven and a half. It's pretty high. The maximum is 10, so seven and a half is a high one. And then you have a vector. The vector is telling you how the issue can be exploited. So there are the coders, allowing you to understand what this all means if you are interested. And then we also have hyperlinks to different resources like the exploit, like a third party advisory issue tracking, the patch on deeper patches from different sources. And then you even get information about what kind of an issue it is in the weakness and enumeration. You can find out that this is a new pointer, the reference. And at the end, you can see the product definition called CPE in this database. And you can see that this is a G-Lipsy issue. You can take a look at the database to see for other issues. And then Yocto offers you a way to look for the pending CVs, that the issues that are in the packages you have in your distribution, it's based on version numbers. So it doesn't really scan if you actually have the issue or if you have used compiler options that make this issue available, but it's simply checking version numbers. To use this feature, all you need to do is to add inherit CV check to your configuration. And then you build your image as usual. The tool is first download the database and that can last a minute or two, depending on your network connection, of course. And you will see the CV update DB task running. And then the check itself, if you already have the image built and you are just doing the check, it's between one and three minutes in practice, in my case. What you get from it is the log. You get a log for each recipe with the CVs in that recipe. And at the end, you get a common log file with all of the CVs in your image. So let's take a look at the example of the output when you build your image with the CV check. Because you also get some warnings when you are building your image. Yeah, for example, we do have a few issues in the DB package. So you have a list of all the CVs that the tool thinks and fixed in this version. And you have that information for every package. At the end, you get a report, image CV report stored in, with the complete record of your CV scan. And you have all of them for all packages. When we look into that file or in the fragment file for a specific package, we will see two types of issues. One will have CV status unpatched. Those issues are not fixed in that version, or at least the tools doesn't recognize the effects. In this case, we do have one of them that's in the Linux kernel. We have a description. We have a description. We have the scores. We have the severity. And we can see a link. So we can look for more information if we are interested into this one. And then we get also a second type of entries for the patched issues. And here, if you look at the score, you can see something as bad as it could be. 10 is the maximum score, but this one has been fixed. It's patched. As a side note, quite often, very high severity issues will only require you to have a specific configuration. For example, in this case, you do have it in a specific driver. If you don't have this driver, probably you don't have the issue. It depends a little bit on the situation. Now, the CV check allows you to do quite many things. You can configure it to your needs, but I have been looking into results and I asked myself a question, how many packages in the distribution actually have reported CVs? Patched or unpatched doesn't matter, but how many of them do have a CV? And I started asking this question after I run the CV checker on a Zephyr build. I know that Zephyr does have CV vulnerability. So it should be reporting something, and I couldn't find Zephyr in the output file. So I started to investigate how many packages have reported CVs and the result is mostly constant. Half of them. The CV checker is giving them for half of them. For all of the configurations I've adjusted, each Mainline Yocto, On Master, or a Stenorus OS. That is more or less the same about half. Why? You can have two cases. Either the package does not have a CV, does not have any reported vulnerabilities whatsoever. It may be that it is really well written and there are no issues, or it's simple enough and there are no issues. But sometimes there are just horror stories. I have noticed some packages last released in 2009 that are still included. And I started wondering if we really want those packages. But the other reason for not having CVs, like it was the case of my Zephyr build, was the product name mismatch. And I started fixing those issues. They are in progress more to come. To help finding the issues with the product names, I wrote an extension to help the CV check and posted it, I posted the first version of it already on the open embedded mailing list. What it gives you is a similar file with similar entries. And it reports you the package name, of course, with its version, with all the products that can be attached to this recipe. And then it's reporting if for this product it can find the CVs or not. In this case, you have an example of some simple direct media layer, which has two products. And you can clearly see that one of them has CVs. So this one was added to actually correctly report the CVs. And the other coming directly from the library name is not getting as any CVs, because it's the other name that is used in the descriptions. Then let's move to another subject. CVs give you the situation with known issues. But security is not only about known issues. It's also about protecting your system from the issues you do not know about yet. This process is called hardening. You try to make it harder for the attacker to try to find something in your system. It has many different rules and means to the hardening. And some of them are available in the Metasecurity. Now Metasecurity layer is a part of security-related layers in your tool. You have Metasecurity, you have Metas-AC Linux that contains support for AC Linux, basically. And you have Metafirtualization that can be considered as a security-related layer too, because you do have support for virtualized images. In this presentation, I'm only going to concentrate on Metasecurity. Metasecurity on its own is a collection of layers and tools. And it has a number of them with an example from the master version of August 2021, when you have Metahardening, Meta-integrity. With Meta-integrity is giving tools for verification of the system image. You have POSEC for the POSEC specification, security compliance for checking your system, and so on. Now, typically, to add Metasecurity or its suppliers, what you need to do is you add the layer itself to your layers file, of course. And then you add a feature to the distro features. Attention, I'm using the new syntax that has been just merged into the master. So if you are on the order version, use the order syntax. And then you can enable new packages. For example, Blinus at Chexec in this case. There's a special case called Metahardening. This is not a typical layer. It's an example of hardened distribution and it contains the fact of disabling root login. And it also gives you an easy to understand way how to do it. It's changed passwords. It changed some permissions like the UMask. It adds small features like the login timeout, minimum password length, so it's harder to get into your system. And the specific way of using Metahardening is that you add the layer, but then you have to adjust your distro name and put distro equal hardened. And then you build hardened core minimal or maybe a different name, but it has to use the hardened one. And then you get a hardened distribution. In our case, it got a little bit more complicated. First, that there is no Metahardening in Anfo because it was added later. And what happened is we wanted the options of Metahardening by default and not by adding some extra distribution and changing the distro name. So what we did was backporting it directly to the distro layer and added features like more permissions changes. And we plan even more of those. In addition to that, we have a Metasecurity and the parts of Metasecurity that existed in Anfo added to our default layers. So we are defining right now what we are going to risk by default for the user. But we are using those tools to do a system analysis and adjust permissions and adjust our hardening options. On the Linux kernel side, we did kernel hardening using quite many kernel options from the kernel self-protected project. A very useful site when you can find the recommended settings for the Linux kernel. We did some changes, some additions, some modifications, certain options, we do not find that necessary in our use case and we have added some more. What we've done also is that we've added per subject hardening fragments as a hardening something.cfg in our repository and it's all available and you can take a look, reuse, adjust as you see fit. Currently, I see no easy way to integrate that the mainline Yocto. Maybe I'm mistaken and maybe there's a very easy way to do so. Would be interested to know. If you want to know more about the hardening of a Linux distribution based on Yocto, there is another presentation about hardening with Open Embedded Yocto project. That includes more detailed list that I covered today about the packages from Metasecurity Layers because I wanted to concentrate on certain things and you do have the Yocto project security wiki with the links containing interesting information. Unfortunately, some of it is a little bit outdated. Now, what do we have right now in the community or in the Yocto community? First, we have the bill of material generation as well that stands for it that has been just submitted. In short, why is software bill of materials interesting on the security standpoint? Because you have the list of packages, you can verify which issues you have. That's basically the same type of a procedure as you are using for the CV check. And you can also check if you have updated everything. You also have a request to do more security documentation. The title of the issue in BookZilla is Add Security Configuration Documentation. So if you're interested, you can take a look here too. And there are probably more than I'm aware of on my case. In my case, the next steps. There are quite many of them. We are planning to have secure boot and image verification added quite rapidly. That would use metasecurity and likely reuse work from the Ledge Group of Urinaro that is also available as a Yocto layer. You can also have a look. We'll be also looking into the checks for automated package updates to be sure that with the hundreds of packages that the typical distro contains, we are not missing important updates even if it's not reporting CV checks because, for example, there is a name mismatch somewhere. We are also working on integration of findings from our security checks with the IP policy with the licensing information. So to do that, we found out that it would be good to have a more machine-readable output for CV check. Frankly, you have seen it's a text file that is easy to understand. But it's a little harder to parse automatically, to be able to query the information for what you need. And then we have a lot of upstreaming of various fixes and changes. Some has landed already, some like the additional pass in the CV checker and then the upcoming change of the format for the CV check that those are still pending and there will be probably more. So there is still work plan to do. And then what we have learned, of course, or subjective, what we have learned from the exercise. One part of the hardening is removing packages. In Yocto, it's very easy to add them. But a little less easier, and maybe I could even say a little more complicated, is removing them. You need to know which packages do you actually need, what has been pulled directly and what has been pulled as a dependency. Is it really a necessary dependency? Or it's something that can be removed and do you really want that package in your system? In practice, it means analyzing the dependency graph and finding out all the places the dependencies come from. It could take some time to remove a dependency. In our case, we removed the NFS dependency and it took time to understand where it actually came from and that we can remove it actually. Then we are also interested in adding meta hardening, the special case when you have to change the distribution name to add it as a distro feature and not as a specific distro. There are some traces online that was the plan at some point. So maybe at the moment you are watching this presentation, I already know more about this subject. I've told you already that we have been using, we are using Danthal, but to upstream, I'm moving the changes to the master branch. There have been big changes in the security-related branches between the Danthal branch and the master branch. What is good because it means that we get more interesting security-related features. But it also means that applying your patches and maintaining them with the two branches, it's not a trivial task. If you would like to know more about security, there are quite many security-related packages in the octal open embedded. There may be packages doing exactly what you need and they do have documentation, so you can start looking at the packages, what they offer you, look at the documentation, check if they do what you need and then you can dig deeper into the background, into what they do and find out more information. There are hundreds of pages of security-related information available online, so if you know what you are searching for, you can find it. Just look at the dates of the sites to be sure that you do actually look for the current information. And then we have the open source security foundation producing resources directed to open source developers. And there are different working groups working on different subjects of interests of open source developers. So you can take a look at what the group offers. That will be the end of the presentation and it will be the time for your questions and the time for discussion.