 Welcome to the room. Long time ago, Linux security looked like that. I still remember looking at my new router and trying to figure out, okay, it's root, root or it's admin, admin. That was long time ago, and I'm sure nobody of you is working on devices that go to production in such a configuration. No, absolutely not. Because since the beginning of embedded Linux, things have changed a little bit. And the Linux security in 2020s, it looks more or less like an example of this attack, that is a Viasat, a Broadband Satellite attack. If you read through the press release between the lines, you will find out that the attackers went through to a mal-configured VPN and access devices through the management network. And then run malware on devices that wipe them down. Fun. And the rest of the story is shown in a talk from Defcon that inspired me for today, actually. Because the security searchers here, they found a binary that resembles exactly to the malware that was used on that attack. And what we can find out from that malware is that it wrote memories of the device with a pattern like FF01, FF02, and here we go. I really recommend you to watch that talk because they are making fun of embedded Linux security, too. And then the malware was looking for some random devices like DevMTD block, DevMMC block, and just doing that wiping on everything. You can imagine the end result. Yeah. Devices were a little bit unusable after that. And a few serious questions for you. On your device, are your services running with the lowest permissions possible? Are your special devices like a flash from this example protected from random services writing that memory or that device? It doesn't have to be memory. Do you trace vulnerabilities in your software stack? Or can you update your embedded device software stack without breaking everything? If you have doubts, oh, I will ask another question. Raise your hand if you are sure that you have no problems with those questions in your product. And I guess I will need to repeat in a few more languages because nobody knows that. Or you just didn't understand me well. Well, okay. And then there are kittens. We should never forget kittens. And those two nice kittens, they are called, one is called Product Diabetes Directive, PLD. And the second one is called Cyber Resilience Act, CRA. Raise your hand if you have a vague or better idea of what they are. Oh, okay. I should have submitted another proposal then. If you are not aware of those two nice kittens and how they will play with your embedded device in the years to come, you have some reading to do. In a nutshell, those are two legal documents of the European Union currently in the works that are going to regulate the embedded device market. And if your device security resembles to my first slide, you will have work to do. Now, we will go back to your project and embedded security. But first, who is scaring you here? My name is Marta, and I am a security researcher, and I assume that. I have been working in the open source for the last 20 years in organizations and in projects currently working on open source supply chain for different projects and offering trainings and development services for embedded systems. Now, so, how do you design security devices with open embedded and the other project? This will be a non-exclusive set because I have only 40 minutes, but I'm going to share with you the most important things from my experience. But first, let's organize ourselves a little bit. I will divide my techniques into three phases. The phase one is when you are creating your device. You are designing it, and all that you can do at that time. Then you have the development phase, you are configuring your device, the phase two, and then at the end, you have the lifecycle of the device, you have the maintenance after it is out in the field. Let's get started with the creation part. That could be too simple to mention, but I will still mention, follow the best practices from the Yocto project. They are documented in a few documents, like what I wish I'd known about the Yocto project that is in the big manual. The making image is more secure, and my personal trick is to use Yocto check layer quite frequently. This is a script working on the compatibility that is expected to be run when you submit a layer to the compatibility program, but it is verifying quite many good practices and doesn't have to be used only in one context. And then read the docs. If you do not understand certain things in the Yocto project, especially at the beginning, go and ask. There are so many errors that come from the fact that people didn't understand something and were afraid to ask why we do that, and we can avoid such kind of problems. Then most developers of Yocto are open embedded. They start from Pocky, and I tell you don't. What you should be doing is to create your own distribution with your own configuration options and your own decisions. Why you shouldn't use Pocky in a product? Because it has options that are not secured by default. It makes it easier to start your development, but it's not ideal for the product. Examples of such options, there's debug tweaks that is enabled by default. My dream is to remove it from Pocky one day, and there's distro features that has stuff, and we'll get into that a little bit later. And then another good practice that is worth mentioning separately, when you have changes to do, do them in your separate layer, in your BB appends that you put in that layer, and you have them nicely separated. We all know that changes start from one tiny one liner, and then six months later, they are 20 big changes plus a back port of something from somewhere else, and then at the moment when you want to update your layer, that's a headache. You won't be easily able to pull the latest changes from upstream. So, good practice in Yachter project-based product. Your changes go to your own layer, and they stay nicely separated on their own. When we are building Yachter-based products, we take many layers from third parties. They are third-party layers that are of excellent quality. They are third-party layers that are of okay quality, and they are layers that have been, that it would be maybe better not to publish them at all. So, how to find your path among those layers? Yachter check-layer yet again. Run it on the layer you are thinking about including. If it fails, next try. If it passes, that's already a pretty good sign, and you can look a little bit further. So, another subject, is this layer up to date? As with any open source project, there are layers that are abandoned, that people do not have time to maintain, and they are just there with the latest version they have supported from five years ago. First, this is not going to work with your recent distro. If you want to update it, it may take a lot of time, and in case of problems, you are on your own, basically. So, what I do, I look for recent commits. I look for the branches that are supported. In Yachter, we have, for example, the latest eight years. If it exists, if it's actively maintained in that layer, that's a good sign. And then the filling part, look into the code in some random files. Does it look correct? Does it look like a code you would like to work with? If it does, okay, take it. There are some layers in the Yachter project ecosystem that bring you security features. Probably the most well-known is Metasecurity. That includes quite many different tools. We have a choice of my favorite subcategories, like we have scanners in Reciples Scanners. And we have tools for validation of security policies in Reciples Compliance. There's a lot in there. There are also other layers that you may have a look. Have a look at what it provides if you have an immediate need in your project. You know what to choose. If you do not know yet, because there may be quite many tools you do not know yet, the need may come a little bit later. But have a look at the layer. When a need comes, you may find a solution in there. Now welcome to the second phase, the configuration phase. Unneeded features. That's a big subject. We are back to Pocky and Distro features as an example. I've taken the latest Pocky, I think, from yesterday and got the Distro features that is four lines with this font. I've made a small color coding. The red Distro features is one that you should never, ever have in a product. And the green ones are something you should have a look if you need them or not. I do not know if you have Wi-Fi or NFC in your product. If you do not have it, why have that Distro feature? Not only for security reasons, but also for code size. If you do not have a feature, if you have no Wi-Fi, you will never have Wi-Fi. Why have all those packages related to Wi-Fi? All the things I found in production images are compilers, debuggers, and tools like that. Security people have a lot of imaginations. With debuggers, you can do a lot of things in your device. So, debuggers, compilers, all that, they have no place in the final image, except if you are running just-in-time compilation for some reasons. Yeah. Apart from Distro features, there is also a package dependency list. That may be long and complicated, but it's worth looking into and digging into it at least once per product to figure out which packages you are actually including and for which reason. If I do not understand the reason, it usually means that some dependencies are wrong and that results in some dependencies cutting in various places and can remove, like, 100 packages in one go. The fourth of OpenMD DT Octo is that it's very easy to add layers and add packages. But just adding a package or your own recipe does it mean that there is no more work to do? There are many packages, big examples being Docker and Kubernetes, that are pretty hard to configure correctly. Believe me or not, those very popular packages, they are extremely hard to configure correctly and securely because they will work, but the configuration is not going to be secure and they are documents of dozens of pages showing how you should configure them. If you haven't looked into that, that's the moment to review those configuration options and Kubernetes and Docker, they are not the only ones. Every package you include, it will have options. It will have compilation options, it will have runtime options, especially when you are writing your own recipe, find out what are the secure defaults that you should be running this package with. I plan to talk about unique passwords for devices, but there is another talk, I think it's tomorrow, covering such subjects. I'm going to skip just saying that in production you should have unique passwords for every single device, not a single password for all devices. And then hardening. This is a subject we can do a separate presentation on its own. My recommendations are use a separate user for every important service, especially every service you are adding on your own. They are compiler flags that help security, they prevent various frequent attacks. Fortunately, Open Embedded has security flag.ink that you can include in your distra with same defaults. And this one is even included in Pocky. So it means that it's something important. Make sure you have them. If you want to have even more secure defaults, it's possible to override those variables and add a little bit more complicated things. Lowering permission of files, by default, there are quite many files having maximum access permissions. It may not be always a very good idea. There's an example in meta hardening. Doing that, unfortunately, meta hardening is currently a separate distro. We'll cover that in a while too. And then a subject of hardening, hardening of the Linux kernel you are using in your distribution. You may come to a very different kernel configuration in an embedded system that you would in a desktop system because of vendor layers from here and there that enable a lot of subsystems and features. Why some of them are not that great, especially debug features and subsystems that you do not really need and the vendor kernel does enable it because someone may need it one day. I also recommend the implementation of the guidelines from the kernel self-protection project. AppStreamYactor doesn't have it right now. I know that there is implementation that has been done for Oniro. Maybe all the distros have those configuration options too. There is a list of guidelines and configuration options that they recommend. Some of them are very sane defaults. Some of them are less questionable and may lead to some problems, but it's a very good guideline to start from. And then we go to the maintenance part. Okay. For the maintenance, I will be mostly talking about CV checking for now. How it looked in the Yachter project in 2022. You could be checking the whole distro using Inherit CV check and then running your build. It was taking the database from NVD, one of the databases for the security vulnerabilities. You could get JSON or text formats. You could check the image or complete build. Life was kind of good. And what happens in 2023? The NVD decided to put down the old format that the Yachter project was using in, expected in September 2023. And also the CVE launched a new format called Format 50, which has a nice side effect that the whole CV database is in a Git repo. So there was some changes to do. What we have currently in 2023. We have a new feature for the NVD format using the new one. It is currently enabled in Master and Mikudo by default. For Kingston and Duffer, it's not yet ported. We discussed with Steve and decided to wait a little bit still and then port it with all necessary fixes if people using it, using the new feature, report them during that time. There is also a lot of work and discussions happening on the management of kernel CVs. The problem is that the kernel gets a lot of attention of security searches and you get a lot of CVs. You have typically two or three per week. The NVD database is not always up to date. Not always has the exact versions where the fix has been applied. And that generates a lot of false positives for the Yachter project. So there's a lot of discussions about how to handle that and there's also a discussion and a proposal pending to rework the CV check ignore. CV check ignore is a way to tell that CV doesn't apply to this package because, for example, the database is not up to date. To wrap, no. I will come back to my three phases. I showed you the three phases and you will come and tell me, well, you tell us to do things in the first phase, but we are already in the third one with our product. So what to do? For the next product, you will know what to do. And for the current one, you can bring parts of the solution to the existing product. For example, refactor changes you have been applying to layers to a separate layer without changing any functionalities. That you can do. You can remove some options probably while looking if they do not break existing tests. Better if you have good testing coverage, of course, but you have. This is all possible, but it requires work, of course. And on the Yachter project site, I have a pretty long wish list and in progress list. More features, one for the current CVs, rework the current hardening and bring it to the Yachter project itself, bring the post-processing of the vulnerability checker and the SPDAX post-processing and the rework of the meta-hardening that offers certain updates related to file permissions mostly as a distro feature and not a separate distro. And that will be the time for questions. Yeah, everyone is scared, nobody wants to ask questions. We have a question from the back. Try to ask your questions loudly and I will repeat. Okay, thanks. You mentioned, I think Ptest was in red in distro features in your example. Assuming that the package is generated by Ptest on land on the final file system image, is there any reason not to have it besides build time? Besides build time, besides the fact that if you built with Ptest, it adds quite many other things. Tests is not something I would like to, I prefer seeing in images. They are quite often vulnerabilities in scripts, in tests, and people do not really look at those. So if you include tests in a production image, it's quite likely you have vulnerabilities in those tests. And nobody is looking for vulnerabilities in tests because you should never put tests in the image. The only one example when I would keep tests in a production image is the case of self-tests when you are doing self-tests at boot, for example, for certain features. Then keep them, but make sure those tests are well tested and they are not a possible source for yet another attack. There's one question from the front. Wait, wait, wait. You mentioned post-processing of CVEs. What does that mean? We have to justify for each CVEs which is unpatched why it's okay that it's unpatched, right? So do you plan to add a workflow for that or how to integrate it? Okay, the post-processing I'm talking about here is the post-processing of the results of the database. For a build we get a list of CVEs with their status. You get a list per image. What is often practical to have if you are building multiple images to merge those lists and know which that, for example, one vulnerability applies to all of the images you have. Certain vulnerability applies only to certain images that makes it much easier to manage patching of devices and managing of your feet with as pedics you can do similar things because you have the list of all packages and then you can post-process and obtain various metrics from the whole build tree that you have. I have a question from the virtual audience. Can Yocto provide sandboxed own user system deep services configuration? I don't know. This is not my exact domain of expertise but I'm sure that if you ask on the mailing list or on the IRC chart, someone will know. There's one question from the front. Okay, so you mentioned PLD and CRA for Europe. Do you know any other for other continents? There are things happening in the US also that I'm less familiar with. For other countries, I'm not aware of it may happen. Any more questions? Okay. And thank you for coming. And if you have any questions, do you want to say anything else?