 Now in, no, excellent, okay, here we go. Hello everyone. Quite recently, I got in my hands a notebook. A notebook coming probably from some embedded project. It was a diary of someone who was in my opinion and security expert in that project. I looked into the, I told myself it's a great material to talk about best practices in choosing software for your embedded product. I'm pretty sure nobody of you worked on that specific product. So if the stories look familiar, it is not your product. Okay, project starts on day one, right? So let's see what we have on day one. The project started. I got the list of components. The BSP uses the kernel 2.6. Yeah, we sometimes start projects from scratch and choosing the components from Groundsapp. But that is the minority of cases from my experience. Usually you either are building on top of some existing project with some existing hardware support or there is some component to use in that specific project or especially those days, there is something the procuring team can actually buy. So you have some system on chip that is there from the beginning. And what do a security person start from in this case? Asking a lot of questions. Is the socket self support integrated in the Linux mainline? Is it? Does it have correct device trees? And then we'll start looking to the quality of that code itself. There's the socket self, they are peripherals in that sock and the same type of questions. Do they have upstream drivers? Or there's some vendor non-steam drivers somewhere in some tariff file that was delivered one day. Then for which version does it work? And just a short look at the file gives us a hint on the quality. There's the code itself, the SOC support itself, but also the integration. Integration in distributions, all the drip distribution building system. What I like to ask as a question, is there a well-supported Yocto layer for this specific BSP? And the first quality check, does it pass the Yocto check layer? If answers to all those questions are mostly negative, time to go find some other chip and convince everyone that the new one is exactly what they need to have. At day one. Yeah, you have the chip. Of course, you repeat the same thing for all the external chips like the Wi-Fi and everything else. You've got the idea. But also you have the features that the project, the product is going to have. I'm a networking person, so I'm looking at networking first. To have network protocols and do all the network protocols in this project have encryption and authentication. If TLS, if it's TLS 1.2 or TLS 1.3, something else, try to remove it. And then quite many devices nowadays, they do have cloud functions. If you have cloud functions, you will be quite often also in the data privacy regulation space. So go and talk to lawyers. But from the security point of view, first big question is, are you going to transfer user data somewhere? If yes, how are we going to protect it? Big question to ask and not hesitate to ask difficult questions at this stage. Then the project usually has some libraries or APIs that I must have for one reason or another. So verifying what this is in your product, project useful from the beginning. And then the final question that isn't that easy is how long your product is going to be supported. Probably there will be some idea at the beginning, but how long it could change during the lifetime. It could change because of some regulations. So think about in practice, how long you are going to support the whole thing and write the number of years in a well-visible place because we'll need it a lot at different stages. And one small side note here about the vocabulary I use about maintained and supported. Those are synonyms. How I use them through this presentation is, maintained means, we talk about maintained projects. Maintained project is a project where someone is taking care of the project. Someone is accepting user requests, some bugs, poor requests, there's some activity. And then you have a supported version. Supported version is a version that actually gets updates, me mostly interested by security updates, but also bug fixes. And this is usually for a limited period of time only. So what you may have, you may have a well-maintained project that you use, but you use a completely unsupported version. Of course, something to avoid. Or you may be using an unsupported version of a non-maintained project. And that we try to avoid. Okay, let's go back to our diary. Day three. This is the second day discussing the distribution to use. Currently, the most popular option is not to use any at all. What I was thinking is that they probably decided to grab a few packages from the internet, compile them and just drop on the device, right? My answer to that, say no to do it yourself, especially when choosing distributions. Why? Because then you maintain it and you support it, basically. So when choosing a distribution, my first question is, how they support it? So how frequently do you get security updates? That's the question one. Question zero is, do they have security updates? And then how frequently security updates all? Are there known long-standing security issues? It's also an interesting question to ask. And usually at this moment, people start to hide. Then how long is the version supported? And then you go back to day one and see for how long you are going to support your project. And you check if those two ranges actually match each other. And of course, the related question is if the distribution has a long-time support version or not. If it has, normally it means it will be supported for a little bit longer. Taking into account that if your product is going to be supported longer than the distribution supports, this long-time support version, what it means, you are going to maintain it later. Yeah! On the binary or source distribution, it depends. It depends on quite many things. So if you're using source, you have a choice, for example, to use the Yocto project on the related distributions or bid route. On binary, I have seen quite many people using Debian on Ubuntu, Ubuntu especially in the robotics space. So let's get an example of how we analyze of what we see. This is an extract of the Yocto project releases page. So what we can see here. We have two LTS versions, Danfra and Kirkstone. And we have an upcoming new version that will be out. If you are familiar with this Viki page, there are all the releases in between Danfra and Kirkstone. They are not supported anymore. Means no security fixes whatsoever. If you have a product running, one of them do something. For a new product, what would be the choice to take those days? If you have already something in development for running Danfra, that's okay to keep it. At least until 24, that should be okay for the security updates. But if you are starting right now, I would rather recommend you to go with Kirkstone if you want to release rapidly. And if you have courage to follow the Yocto master, not go to the, with the latest version that will be just supported for some time, but just follow the master until the next LTS. Personal recommendations at this stage. Okay, let's come back to another entry. Day 17, today only five new layers appeared. This is better than yesterday. I would assume that they came with Yocto. What is the very good point of the Yocto project that it's easy to add layers? What is less liked by the security people and the maintenance people that it's harder to remove stuff that you do not actually need? So my advice for everyone adding layers or thinking about adding layers is to first ask yourself a question, do I actually need it? And then look at the quality, if it's Yocto project compatible, in this case, less problems on your way. And of course, is it maintained? Now, let's have an example. Someone wanted to include something that included Metasecure call. So the first analysis is on GitHub, but a glitch, it looks like a personal project of someone when you look at the GitHub link, right? So this is not a very good sign because what happens if that person moves away? It means that there's no direct assignment to an organization, it could be problematic. So a warning sign. Then we'll look at the statistics of the project itself. So at the number of commits, the clicker doesn't want to, you have hundreds of commits here. You have quite many folks, you have quite many stars on GitHub. And from all the sources, we know that it's pretty used layer in the area. So this is something acceptable with a warning that if something happens to the maintainer, it may be your problem. Another way of evaluating projects, certain programs, the best practices badge from OpenSSF, it gives you a number of criterias on different areas related to quality and security in general. Projects can get badges, you have spacing and you can get safer on gold level depending on the criteria they pass. The project itself needs to apply for the badge and they need to fill my form, give the information to fulfill the criteria. And let's have a look at some of the criterias. For example, it says that you need to have a bag, you may most have a process for the users to submit bags, quite obvious but necessary. And the project should be tracking individual issues as individual bags, okay. But also the project must acknowledge a majority of bags submitted in the last two to 12 months. Interesting. At least acknowledge bags that they are here, maybe not actually act on it. My personal feedback on using best practices it's a very useful tool because some of the information that is there is not available anywhere else in an easy to grab form. For example, there are questions about the static analysis tools and I do not know any place other than that where you can just look if that specific project uses static analysis and how it uses it. So very rich database of information and more than 5,000 projects those days. Okay, let's come back. We are day 27. I said no to adding one small component, checked. It was bringing eight new libraries. Yeah, unfortunately that is something that happens. One component bringing components that are bringing components. And I have a question. Yeah, this is day of hard questions. Do you know how many components that are in your project? Maybe, maybe you have an idea. If you don't, what can you do? If you are not sure, what you can do? Generate package list from your project and check it from time to time. The package list doubling in size in one day is a bad sign. Then if you are using your project there are few useful tools that you can have especially if you are monitoring from a CA1 to CA1. Bit back minus G with the dependency graph. Security people love looking at dependency graph and getting headache after analyzing what they see. Then the CV check giving you the non-security issues in your image also different when possible and the create as pediatrics for the S-bomb. Three useful tools to have when looking at what you actually have in your project. Let's go to some examples maybe. So I was looking over my morning coffee on the fresh package list and there was Leap Micro HDPD. Hmm, looks like a web server. And where does it come from? And not only this, there was some unexpected crypto libraries after I have been working for some time to remove the crypto libraries that we do not use. So what's going on? I'm not putting the whole dependency chain because it's a bit more complex. The crypto leaps were used by Leap Micro HDPD that was brought in by Debug and 4D that came from Elfutiles. Okay. Okay, asking around, does anyone use it in the project? No? Okay, let's get rid of it. And how easy? It's really easy when you know what you want to remove. In this case, it's one configuration option in Elfutiles that you need to remove and in the recipe, in the yachter, it's all already prepared. You just need to remove one distro feature. Done, removed, all this gone. And if you have analyzed the dependency chain, you manage to remove things. Most of the time I'm spending as a security person is actually on analyzing the dependency chains and finding out why it shows up here and if we can actually remove it. And one more example, a pretty recent one. The Oniroproject works on enabling S-Bomb generation by default all the time. That we can say that it was a little bit of an interesting ride, especially with some less frequent to use layers like Metazephyl, but all fixes are out, let's say a little bit work around it in some places, but still it works. You can watch quite many talks at this conference talking about generating S-Bomb. But what we decided to do is to go one step further actually use the S-Bomb. Yeah, use the S-Bomb. So we gave the S-Bomb to lawyers to the IP compliance check. And as a result, we have a nice discussion, started yesterday on the October project mailing list because they found out that there are a few things missing in the reporting. So they cannot do the composition analysis as they would like to do, but there are a lot of ways apparently to get the missing information. So it's a pretty exciting and nice use case of S-Bombs and actually using them in practice. So I'm pretty interested to see how it ends up and what we are going to add so that this particular use case is satisfied. Okay. Let's go now back to the diary. We are at day 79. Finally, a discussion on updates. I asked from day 11, updates, three months in the project, I would say it should be like that. It should be like that because it's critical. I will be very verbatim here, updates are critical because even the best design project with best people, they will be back. And when the device is deployed, an update is probably the only way you can remove the problem other than destroying the device, of course. So I'm not telling you which other system to use. I'm telling you to look into the scope of the system you are looking into. Does it allow you to update the firmware? And which firmware? If there may be the SOC firmware, there may be firmware for some Wi-Fi or some other elements that you have. Does it allow you to update bootloaders? Because updating bootloaders is typically a pretty tricky thing. And you need to prepare for both update of firmware and update of bootloaders. If you start adding it late in the process, this is going to be complex. And of course, the two other things, updating the kernel applications and of course the file system. Yeah, quite many things that we need to consider when updating as early as possible. And then we come to the, yeah, that is the last entry in the diary. Day 97, two weeks to the release. And there is some name that I couldn't read. Wants to try adding secure boot. To that, I will answer, okay, first, I will answer that we do not do it two weeks before the release. And then I will answer it with a citation. Do or do not, do not try, especially with secure boot. So secure boot is added for a number of reasons. Some of them are very good reasons. Some of them have other solutions. Examples, you can be adding secure boot because you want to protect the device from accidental removal of keys. That something bad happens and the keys just disappear. You may want to encrypt user data because it's some confidential information in there. But you may want to also protect against a little way more sophisticated things like someone resoldering the EMMC or Flash on the cart and putting a malicious software instead. Or you may have an application typically in the T that must not be changed. Protecting against resoldered Flash or for example, it's a very good case to actually do the full complete secure boot. If you want to just protect your for some accidents and not really from malicious users, there are easier things that you can do. Do not need to bring a secure boot that is not that easy to put in place. My advice summarized on one slide. It's very easy to build secure boot broken from the beginning. It's a chain. It's not just another package you add somewhere. Unfortunately, those days, people are working to make it easier but it's still a little bit complex and requires understanding on different elements of the chain and how they interact with each other. It needs to be taken into account early for quite many different things like partitioning the updates. I was at the previous talk about Delta updates and I was thinking, okay, secure boot on with the Delta updates on all the recalculations that you need to do. I thought it is to work cool. And you have the key. You have the key on multiple keys and the big question of user keys on the machine on this key. If you're going to have it and how it's going to work. If there is a reference implementation for your platform and you want to implement secure boot, start from the reference implementation. And at the last point, I'm not joking. That is really what I strongly recommend. If you implement secure boot, get an external update. It's too easy to make it wrong, use the wrong key or just forget to re-inertialize something. Very easy to just miss it and then your whole secure boot chain, it doesn't work. So it doesn't make sense to let that kind of an easy error to pass through. I don't know how this project has ended. It ends, the diary ends at the secure boot. So you understand now why I was thinking that nobody of you was working on that project. Even if you were, keep it to yourself. There are way more stories we can find from this project than another, I think. For the part two, for the sequel, we have a setting up of network protocols, especially do it yourself, network protocols. That is especially cool. And protocols without encryption, even better. We have how to update both support packages, how to we back both security patches and what happens if it goes wrong. And we have the security response team. So a lot of material for part two. But now for the part one, my takeaways. Security depends on most of the decisions you take in the project and the choice of different components you have in a project. Evaluating the software is absolutely the key to everything. And the first question to ask yourself before adding anything, do I need it? And the second one is, does it have security updates? I also recommend you to have a look at the guide on evaluating open source software just released by the Open Source Security Foundation. As a full disclosure, I'm one of the co-authors. And that is not the only reason why I recommend it. It contains a nice list of questions to ask yourself for each single package it contains. And that will be time for questions. We have a mic in the middle if you would like to ask something and hesitate. Or you can ask just like that. We'll try to repeat the question. Yeah, we have the first one. Yeah, somebody has to start. Yeah. If I got you right, then you said that the actor project does a lot of stuff quite well. Please tell us what we're doing not well. Where we should get better? Yeah, I prefer to telling about things that are done well because if I say that enough, then people will keep doing that and won't change their mind. So that's already a very good thing. Then about the things that maybe could have some other outcome. I've talked about the difficulty to find out what you do not need and the analysis of dependencies of having some packages that come from somewhere. And try to figure out where it comes from and why. So this is something that could be worked on. The actor project also quite good on fixing CVs. That is working pretty nicely. More people working on the CVs, that will be good. And also I've noticed when I was working on analyzing the gaps in the CV process of the actor project, I found that there are some gaps like the CPEs that are not set correctly. Found things like mismatch between the CV that was actually fixed and that was marked as fixed. There are things like that, apart from just fixing CVs, working on the general security flow. I know there are many working on that. What I would love to see is see the meta-hardening layer being transformed into a distro feature. That's my project on my to-the-list for a long time. I've never managed to do that. Are the hardening actually hardening options to quite many packages just from the beginning that you can have them configured correctly as it is? Because the default configuration, as you bring the image, it's not ideal from the software point of view if someone just flashes it to a production device. Well, as I understood, the whole presentation is taking care about the product itself. So are there any do's and don'ts and recommendations about the infrastructure that surrounds it? Oh, you mean the CI, the cloud and... Yeah, and when it comes to security specifically, so secret management, all these aspects. I'm between you and the lunch. Do you really want me to answer this question? So, those days I think the big challenge that we have is the use of cloud and embedded because there are a few reasons for that. There's the reason related to the supply chain attacks. If you start putting containers from the internet, it may happen that what you download is not exactly what you were expecting to download. So all the verification of what you are downloading from the internet, we need to... maybe not tools, but to be aware that all those attacks exist. So there may be attacks on the software, you may get just malicious software on your container. And there is the whole thing related to data protection. It's so easy to send stream of user data when your device is a camera, send the stream of the user data somewhere. And that's a big challenge. And I think I've seen the red card already, so I need to stop at this point. We can talk later on. Thank you.