 Hi, thanks for attending this session. I hope you enjoy it. So, in this talk, I'm going to talk a little bit about Embedded Linux security. So, the idea here is to have an introduction on the topic. So, we're going to talk a little bit about what is security? Why should we care about security on an Embedded Linux device? We're going to talk a little bit about thread modeling. So, we don't have much time to go over the topic, but we're going to talk a little bit about it. And then, we're going to talk about a lot of mitigation techniques to improve the security on an Embedded Linux device. So, we have a lot of technical stuff in this presentation. Before we start, so, let me introduce myself. My name is Sergio Prado. I've been working with Embedded Linux development for more than 15 years. Right now, I'm a team leader at Toradex. I also have a company that do some consulting and training services. Usually, or casually, I contribute some of personal software like Build, Route, Yocto, and the Linux kernel. Sometimes, I write some technical stuff on my blog, that's embeddedbit.org. So, let's start with an introduction to security, and then we can talk about the technical stuff. What is security? When you talk about security, we really mean we want to protect something, right? We want to protect something that has some value. In an embedded device, we're talking about, for example, our code, right? We want to, for example, protect the integrity of the code. Or maybe we want to protect the data that's inside the device, right? We want to guarantee the confidentiality of the data in the device. And of course, to protect the system, we're going to have to implement mitigations. We're going to have costs to implement mitigations. So, security really is about managing risks, right? You have to identify the risks on your system and mitigate these risks, right? And of course, we have some costs, and we're going to see a lot of this in this presentation. Some concepts about security. So, we have some, we have owners, right? The owners are those who benefit from the product, could be the manufacturer of the product or the final user. We have assets, right? And that's the real reason we have security, because we want to protect the asset. That could be our data, that could be our code, that could be our reputation, for example. All of these assets are assets. We have threats. Anything that could act against an asset is a threat, right? And we have threat actors that are those that want to manifest a threat in our system. Could be a malicious hacker or, for example, the government, right? So, a threat actor would try to exploit a vulnerability in the system so a vulnerability is a weakness of the system via an attack factor, right? So, for example, we have a device with an internet port. So, the internet port is our, is an attack factor. A threat actor could try to exploit a vulnerability inside the device, for example, in the TCP IP stack to have access to one of our assets inside the device. So, those are some of the basic concepts of security, right? And I really like this diagram because it shows really that security is a kind of cat-and-mouse game, right? We have the assets, that's what we want to protect. We have vulnerabilities, right? The assets, and we will always have vulnerabilities. As we're going to see in this presentation, there is no such thing as a completely safe or secure system, right? The system is always secure enough, but you won't have a 100% secure system. We have threat agents that want to exploit a vulnerability in the system via an attack factor to have access to the assets, right? On the other side, we have the owners, could be us, that are developing the system or the final user. We want to prevent some threat agent or threat actor to exploit a vulnerability, so we will develop countermeasures or you're going to mitigate the risks of the threat actor to have access or exploit a vulnerability to have access to our assets, right? And as I said, this is really a cat-and-mouse game here, right? The threat agents are already trying to exploit the system and we are always trying to implement mitigations to protect the system. And when you think about security, as you want to see in this presentation, and as I already told, we are really trying to manage or minimize the risks of our assets to be compromised. So there is a process, there is a formal process called a threat modeling that could help us to try to identify the major threat to our system so we can have a plan to mitigate it. That's the real objective of a threat modeling process. The result of a threat modeling would be the threat model of your product with all of the risks in it fired and ranked probably because it's important to rank for the most important or the most impactful risk to the less. And then with this threat model at hand, you would try to mitigate the risks and improve the security of our device. I'm going to talk a little bit here about how the threat model works, but it's going to be just three, four slides that we don't have much time in. We really want to talk about the technical parts of the security. So threat modeling basically is represented by this diagram that I created. So we have the assets that we want to protect. We have threats or risks on these assets, right? And we want to mitigate this. So the threat modeling process would help you to identify all of the threats and would help you to find out a way to mitigate the threats. And that's really what threat model is. There are a lot of defined process to help threat modeling the system. Unfortunately, at least I don't know any threat modeling process or methodology that is specifically for an embedded device. So I'm going to talk here a little bit about the Microsoft methodology that's called the Stride and Thread. Just to have an idea of what a threat modeling methodology is. So basically here we have two methodologies. We have the Stride methodology. The idea here of the Stride is to help us identify the threat. So basically we have a category of threats. So we have basically six kinds of threats. We have, for example, spoofing. Spoofing is a kind of threat where they use someone, pretend to be someone who he doesn't use, right? So for example, your device has a web interface, but you don't do any authentication. So anyone could access the device like being someone else. So to mitigate that kind of risk, you would implement, for example, an authentication mechanism on the web page of your device. So we have here basically six kinds of threats. This stride stands for spoofing, tampering, reputation, information disclosure, then our service and elevation of privilege. And then we would try to identify all of the threats using this methodology. And then we have the Dread methodology where we would take all of the threats that we have already identified and rank them. So the idea here is to have a list of the threats organized by the most important threat or the threat that could cause the most impact on your system, right? As we can see here, we have five rating categories, how much damage the threat could cause to your system. We have low, medium or high, how easily it is to reproduce the threat, how exploitable it is, how many users would be affected by the threat, and how easy it is to discover the threat. So you would rank all of your threats from one to three, and then in the end, your threat would be from, as we can see, right? If all of the categories is one, it would be five, or if all of the categories is three, it would be fifty. So you'd have a rank of threats from fifteen, that's the higher point here, and five. Basically, in the end of the threat modeling process, what you would have? You'd have a kind of table with all of the threats that you identified, ranked, and mitigation for that threat, right? Of course, you could extend this table adding, for example, the cost of this mitigation, because that's also important, right? If the cost is too high, like it's higher than the value of the asset that you want to protect, it doesn't make sense to implement the mitigation, right? But that's the basic idea here of the threat modeling, to help you identify all of the threats, rank them, define mitigations, and then plan and develop and implement the mitigations to help improve the security of the system. So if your device has requirements related to security, it's very important to have a good threat modeling process in place, and have a good threat model of your product. So you have a plan on how to improve the security of your device. Well, for now on, we're going to basically talk about mitigations, right? So our focus here is on the technical side of the security. So we're going to talk about a lot of technical mitigations that we could use to improve the security of an embedded device. So let's start with secure boot. So you want to protect your code. Your code is important. You want to protect it. You want to actually protect the integrity of your code, right? You want to make sure that you are running the code that you develop. For that, you need a secure boot process in place. You need to implement secure boot on your device. And as we're going to see in this presentation, every mitigation has some costs. In case of secure boot, of course, you're going to have costs like, you're going to have to think about how you're going to manage the keys of your device. You're going to have to think about the boot time, because of course it is going to impact the boot of your device. Probably it could make your device harder to develop if you have a secure boot, but it could work around that. Anyway, you're going to have costs, of course, implementing. It's always a trade-off, right? The usability, with security, with functionality. Usually we take functionality off of the device to improve the security, for example. How does it work? How secure boots work? Basically, of course, it depends on the implementation, but usually a secure boot is implemented using digital signatures. So every component of the device, when you talk about an embedded Linux system, you have at least three components, right? The bootloader, the Linux kernel, and the root file system. So every component should be signed. So you sign it to one component, and the component before that component is going to check the signature of the device. So the root file system is signed, and the kernel is responsible to check the signature of the root file system. The kernel is signed, and the bootloader is responsible to check the signature of the Linux kernel. So because we have one component of the system checking the other component, we call this the chain of trust, right? So one component trusts in the other component. And we have something like this. So this is a general example, right? We have the root file system that is signed, and the kernel will have a public key. We're talking about a nice symmetric public key algorithm. So the image is signed with a private key, and on the device, we have a public key to check the signature, right? So you don't need here to care about the keys. The keys are public, so there's no problem with keys. That's different when we're going to talk about encryption. You're going to see that it's a little bit different. With encryption, we're going to use symmetric algorithms. So in that case, we need to care about where to store the keys. But we don't have this problem here. So the root file system is signed. The kernel is going to check the signature because the kernel is signed. We have the public key that was used to sign actually. The image is signed with a private key, and the public key is going to check the signature. Of course, we're going to see here that the key is not inside the kernel, but it's going to be in a hard disk that's going to check the root file system. And then we have the bootloader with the public key corresponding to the private key that was used to sign the kernel. And the first component is very important because we have to trust in someone. So the first component we call the root of trust. So in this case, it's the code inside the SOC that we usually call the wrong code. And of course, as we can see, we need support in the hardware to implement a secure boot. If our SOC doesn't have the support, we can't basically implement it because everything starts inside the SOC. If the SOC can't check the authenticity of the code that is going to load to his internal memory, you can't implement a secure boot process. In the next two slides, I'm talking about how it is implemented on IMX6, how we could implement it on the SOC line from an XP called IMX6. Instead of showing these two slides, I'm going to show the diagram that I created. Basically, it's what I already explained in the other diagram, but here's more detail, right? So everything starts inside the SOC. In the case of the IMX6, we have a piece of hardware called Hub High Assurance Boot. And then this is the secure boot implementation from an XP. Then we're going to have to store inside the SOC the public keys. Actually, we don't store inside the SOC the public keys. We just store the hash of the public key. And then when the boot loader inside the SOC, the wrong code load the boot loader to memory, it's going to also load the certificate and the signature. And inside the certificate, we're going to have the public key. So he's going to take the public key, calculate the hash, match with the hash that he has inside the SOC, then to validate the public key. And that public key is going to check the signature of the boot loader. And the boot loader is going to have the public key of the Linux kernel. We usually, when we implement a secure boot on Linux, we usually use an image format called fit image. So the fit image is an image format. Basically, it's a container of images with hashes and signatures. So we can add any kind of image with any kind of signature. And the uboot is capable of opening the fit image and extracting the images inside this container image and checking the signature of each image. So we have inside this fit image, we have the Linux kernel itself. We have device trees, possibly, if we're talking about ARM. And we're going to have our RAM disk, a RAM disk image that's going to boot to check the root file system. So each of these images is going to be signed with private key. And uboot is going to have the public key to check the signatures. And the last step, when the kernel boots the RAM disk, they need from the RAM disk. The RAM disk is going to be responsible to check the signature of the root file system. For that, we have some options. It depends on if you are using a read-only root file system or a read-write root file system. In the case of a read-only root file system, the option that we have to validate the integrity of the root file system is using a device mapper module in the kernel called dm-varit. That's, for example, the way that Android works. And if we are using a read-write root file system, we have two other options at least that I know. That's called, they're called IMA and dm-varit, dm-integrity. Let's give here an example using the dm-varity. So basically the root file system is here. It is read-only. Basically at build time, we're going to use the valid setup tool that's going to hash all of the blocks of the root file system. It's going to create a hash of each block. And then after that, it's going to take every two blocks and create another hash. After that, take another two blocks, create another hash, and it's going to build a hash tree. And of course, in the end, we're going to have one hash that represents all of the root file system. And this hash is signed, and the hdisk is going to check the signature. And during execution, if you change one bit in the root file system, the hash won't match. And then the file system access won't work. So it really guarantees the integrity of the root file system. Of course, we could have and we have a whole presentation on secure boot, so we don't have much time to go deep in this topic. Nothing is 100% secure, of course. So if your root of trust is compromised, all of our systems are compromised, right? And that's what happened almost three years ago when vulnerabilities were found inside the high assurance boot of the IMX devices from an XP. These are the links, if you are interested in this, have a look. Basically, you could craft a specific certificate that would basically crash the home code of the IMX and would bypass the secure boot process. These vulnerabilities were fixed with new silicon, but of course, if you are using silicon manufacturer before these dates, you could really be using vulnerable devices, at least in respect to secure boot. Well, secure boot is good to prevent, to guarantee the integrity of your code, but you would prevent anyone from having access of your code. But if you want, for example, to protect the intellectual property of your code, or that's more common, right? If you want to protect the data of your device, you will need to use encryption. I would say it's not common to encrypt applications on Linux. You could, for example, encrypt your applications, but for example, encrypt the root file system is not very common for some reasons. For example, you are using a lot of open source code, so it doesn't make sense to encrypt open source code, right? Because the cost is higher and it doesn't make much sense. The other problem is GPL V3, right? Because with GPL V3, you have to provide some kind of access to the user to update the software on the device. And if you implement some kind of encryption, you won't make this possible for the user. So I would say that's not very common to encrypt code in Linux, in bad Linux devices, but it could be a requirement related to the data, right? And what we have to work with encryption on Linux? Basically, we could use two approaches. We could do a full disk encryption, or we could use a file-based encryption. With full disk encryption, we basically have a device mapper module called DMCrypt. Basically, this module stands on top of the block device, so every access to the block device would go to the DMCrypt module that we would do at runtime encryption and encryption of the block device. And if you want to do encryption on the file system, on top of the file system, we have two options, FFS-Crypt. That's basically an API provided by some file systems like X24 or UBFS. Another option is E-Crypt-FES. That's a layer that you could add on top of any file system. So it could be other file systems. For example, H3 could be, I don't know, any other flash house system. So it's really a genetic implementation that you could use on top of any kind of file system on Linux. Just to make it clear an example, imagine a system where you have secure boots and with encryption. What would change? In the end of the boot, the root file system would mount. So in this example here, we have a dedicated partition with the data of the device. And the data is encrypted. So an script would run in the initialization to mount this encrypted device. In this example, I'm using the E-Crypt-FES. So it could be on top of any file system. And then we would have a secure boot system guaranteeing the integrates of the application with encryption to guarantee the confidentiality of the data. Here we have one problem, right? Because when you talk about encryption, we talk about a symmetric key. And then the problem is where we would store these keys. This is a very good book. If you are interested in harder hacking, I would suggest you to read this book, Hack in the Xbox. The link is there. It's the book's public. You can download it. It's a very good book written by Andrew Bunny-Hank. He was the first that hacked the first generation of the Xbox. So this is the picture of a small device that he developed. He just connected this device between the CPU and memory buses. So he could just sniff all of the communication between the CPU and the memory. And then he created a special software basically to look for the key inside the data that he collected with this small board. And he could find it. So basically the Xbox had the keys inside the flash memory. And the keys during boot was just read from memory. It was laying there in clear text. And he was able, after he found the key, he was able to encrypt any application, run any application in the Xbox. So the message here is, if you work with encryption, you must protect your key. Your key needs to be protected from the rest of the system. And you're going to need the hardware for that. So you're going to need help from a hardware device to protect the key. Actually, there are some alternatives. If you are developing a device that has some interaction with the user, you could derive the key from a password of the user. That's for example how Android works. So in that case, the key would be encrypted with an algorithm that could be using the password of the user. But usually an embedded device doesn't have this interaction, right? You don't interact with the device. So yeah, you're going to have to think about a good way to restore the key that you're going to use to do encryption. We have some options here. For example, if you are lucky and your SOC has some cryptographic modules, the solution is there. For example, on IMX devices, again from NXP, there is a cryptographic module called CAN. This piece of hardware inside the SOC has a kind of master key that's a unique device. So you could use this master key to cryptograph your key and store your key encrypted in your root file system, for example. So this could be an option, right? But if you are not using an SOC that has this kind of capabilities, what you could do? You could use an external device, for example, a secure element or a TPM device. Both are devices that provide cryptographic functions and secure storage for you. A third option would be a trusted execution environment. I'm going to talk a little bit later about it. So a secure element is basically a secure device, a secure computing device with code running inside of it and providing you secure storage. A good example of a secure element is smart cards. You could use a smart card on a bad link device, but I would say it's not that common compared to a TPM device. Actually, a TPM is not a device. A TPM is a specification, is a nice or standard and specifies cryptographic functions and devices implemented this specification. The TPM device usually has the I2C or SPI connection, so that's why it's more common on an embedded Linux device. Well, none of this matters if your code is buggy, if your code has vulnerability, right? Because, yeah, of course, you could implement secure boots, you could encrypt your device, but if your code is buggy, the threat actor will find the vulnerability and will exploit your device. So it's very important to think about the quality of your code you are developing and, of course, the code that you are using on your device. So code with security in mind is very important if the security is a requirement for you. This is just an example and we have hundreds of examples of vulnerability. So this vulnerability is from the Linux kernel. It was there for several years. If you're running a device that has a kernel from 2.6 to 5.2, you're vulnerable if you are using the rich IO subsystem of the Linux kernel. Yeah, that's why we need to care a lot about security because if our device is vulnerable, it could be exploited, right? So what kind of indications do we have to protect our code? First, use the static analysis tools. So this is kind of tools that could improve the quality of our code finding problems like buffer overflow and things like that. We have very good open source tools like CPP Check and Clank. Clank has a very good static analysis tool inside of it. We have also some commercial tools, but yeah, you're further developing code in CC++ in softwares that are using languages that are not member safe like CC++. You have to use, we have to have some kind of static code analysis in your system. You could also enable runtime protections, right? So this is kind of protections that I run as the name implies, right, at runtime. So for example, Valgrind is a very known tool that is capable of checking memory access and identify, for example, memory leaks. Another protection that is very common, and you have to enable if you have security in mind is ASLR, address space layout generalization. So imagine, for example, that your kernel is crashing. When your kernel crashes, you have the dump, right, the kernel ops message with a lot of memory addresses and things like that. That's what the attacker wants, because then you have the addresses of the functions. And it's very easy for the attacker to develop an exploit using this information. But if you enable the ASRR in the Linux kernel, it would randomize the addresses during boot. So every boot is going to have another address. So it would make very difficult for the attacker to develop an exploit. Of course, it's not impossible, but it would make it more harder for the attacker. Another tool that you could use to improve the security of your software is Fuzzy. Fuzzy is a kind of technique that you could use to basically automate tests, right? So these two escape will generate inputs to your software. Imagine you have a software that collects data from a network interface. So a Fuzzy tool would send data to the network interface and try to crash your application. To find bugs, find that kind of corn case over application that you difficult test. You don't usually test it, right? There are some very good Fuzzy tools. For example, this is a tool that's used to fuzz the Linux kernel. And it has already found hundreds of bugs in the Linux kernel. Well, you could protect your application, but you will always have bugs. Shoppers always have bugs, right? They say basically for every thousand lines of code you have one bug. Yeah, you're going to have bugs, and how do you mitigate that? Even with trying to minimize things you could do, use Linux capabilities. So the idea of Linux capabilities is to basically help you drop the privileges of a root user. So your application would just enable the capabilities and then drop the rest of the capabilities. So if an attacker exploits your software, you would want to have the capabilities that you enable. So for example, there are some implementations of the ping command that has only the natural capability enabled, right? So although it runs as the root user, it doesn't have all root access, right? And that's good in terms of security. But the capabilities is not that flexible. If you want an access control that's more flexible, then you're going to probably have to use mandatory access control in a secure model from the Linux kernel. So we usually, from a unique system, we usually use it to work with that description of the access control. That access control flag is like read, write, execute from user group, the right? But this is not that flexible. For example, you want to prevent an application from access all TCP ports, but port 80. You can't do that with that description of access control, but you can do that with a Mac, a mandatory access control. Basically, in a Mac system, you will have objects and subjects. Object is everything in the system, like a file, a socket, and a subject is basically a process. So you would define what a subject can do with an object. There are some Mac implementations in Linux using the Linux security model. I would say the two most famous are S Linux and API Armor. For example, Android uses S Linux, API Armor is used in Ubuntu. Yeah, and another layer that you could add to your application is sandboxing the application. So the idea here is to run your application isolated from the rest of the system. There are some approaches for that. We could use virtualization, but virtualization is very costly for an embedded device. I would say nowadays we could use on an embedded Linux system containers and T trusted execution environments. So a container is basically a minimal file system with the application and all it needs to run. And a container with some help from the kernel would run isolated from the rest of the system. So if you compromise a software that is running inside a container, only the container is compromised, not the rest of the system. That's the idea right off of containers. It's not that containers is secured by default, right? But you could use this to run your application inside a container to improve the security of your system. Another approach to sandbox your application is using a trusted execution environment. I won't go over much of this topic, but the idea here is that with a container, you can protect your applications, but you can't protect the kernel. So if the kernel is compromised, the system is compromised, right? With a trusted execution environment, if the kernel is compromised, you can still protect the system, protect your assets, because your assets would be protected by the trusted execution environment. That's basically the idea. Let me show you this diagram. So you would basically have another operating system running side by side with your operating system. That would be Linux, right? What you want to trust, you would put the trusted execution environment. Of course, you need harder support for that. So some of you may have heard of Trezone, right? Trezone is a piece of hardware inside the ARM processors that help you implement this kind of segmentation of the system. And it's very common today, right? The team implementations. Like we have teams inside our smartphones, inside TVs, inside set of boxes. Well, just a couple of more slides. All of this is important, right? But you have to have an operating system because bugs exist. You're going to have to fix the bugs. You're going to have to update your device. And then we need an operating system for that. If your device has security in mind, you really need to have an operating system. It could be OTA, of course, that would be good to be OTA over the air because you could update anytime you want. Your third device is connected. It could be offline also, but it's important to update your device because bugs will happen, especially security bugs, and to protect against security bugs you have to update your device. There are some challenges to implement an operating system. And of course, we could have a whole application on that. We have to think about security integrity. If it is atomic or not, the bandwidth that uses speeds, if you have rowback capabilities and things like that. We have some strategies to update the system. What is important here is to have an operating system, a good operating system, an operating system that is basically able to not break the device, right? That's atomic. That's really, really important. And of course, if you are implementing an operating system with some connectivity, you have a network connectivity, you're going to have to think about the security of your network device. And yeah, network is a whole other topic. I won't go much over it, but it's very important to think about it. One really important thing here is that you have to basically decrease the attack service, right? So disable everything you don't need. Disable every protocol, close all the ports that you don't need. If network security is very important, enable an EDS system to detect intrusion, create firewalls to prevent, for example, denial of service attacks. All of this is important to a secure connected device. Yeah, and I would say here that there is no one solution that fits all, right? If we think about security, as we can see here, we have layer on top of layer on top of layer on top of layer, right? We need to protect our code. We need to protect the integrity. Let's use secure boot, but the code could have bugs. Let's use that analysis tools. Let's check the software during runtime. Let's use a mandatory access control. Let's run inside a container. So if an attacker could pass one layer, it's going to block in the other. That's the idea, right? Make it more difficult to the attacker to have access to your assets. And I think that's what this image brings to us. Yeah, so let's close here the presentation. Some general rules about security. So we really need to think about the defense in depth, right? Layer on top of layer of security. Security involves all levels of the system. So if you only think about the security of our device, what about the security of the cloud system that the device is connected on? You have to think about it also. Always use the least privileged principle. Don't use a scoolage or obfuscation. Don't invent our own algorithms. Don't do that. And of course, as I said before, there is no 100% secure code. You will have some kind of visual. So we try just to minimize the risks of our assets to be compromised. So if your device has some kind of security requirement, think about it, design the security in mind, create a thread model to identify the assets, threads, attack vectors, and mitigate risks. Always follow good practice. Don't invent anything. Always use what is there, no techniques, no tools, and have a good update system. If you can't monitor software vulnerabilities and patch the system and update the system. So yeah, this presentation, the idea was to have an introduction on the top. So basically we saw here the main techniques to improve security of an embedded Linux device. I hope you enjoyed this presentation. You have here my contact. So feel free to send me an email. Feel free to get in touch with me on Twitter or LinkedIn. And again, I really hope you enjoyed this presentation. Let me know if you have any feedback. And now we're going to the question and answer session. So thank you. Hello everyone. Thanks for attending. This is me now live here. Thanks for attending this talk. I hope you really enjoyed it. I try to answer most of the questions during the presentation. We don't have much time now. It's less than a minute right now. I'm sorry about the glitches during the presentation. I didn't know that when we record that you didn't get any feedback on the recording. So I would record again, but I didn't know we had some glitches during the recording session. So sorry about that. You can find these slides on sketch.com. I just uploaded today. So if you don't find, feel free to write me. This is my email on Twitter, LinkedIn. Feel free to connect with me. And I'm going to be a while in the Slack channel, in the Bad Links Slack channel. So feel free to go there and ask me questions there. So again, thanks a lot for attending this session and stay safe. Bye-bye.