 Hello everyone, thanks for joining session of the open-store-submit Latin America. So in this talk, I want to talk about how to secure containerized applications with the focus on embedded Linux devices. But I mean, most of the concepts that we're going to study here also apply to let's say deploying containers in the cloud. A little bit about myself. So my name is Sergio Prado. I'm located in São Paulo, Brazil. I've been developing embedded software using open-store software for a while now. I've been working in the embedded field for 25 plus years. Currently, I'm working with my company, BetterLabworks, and I do a lot of embedded development. Most of it focused on BSP development involving the Linux kernel, writing drivers, developing distros with the Octo and Busebrot and so on. I'm also an open-source contributor and I have a blog that sometimes I write about different topics, embedded bits.org. So our agenda here, we're going to have a quick overview of the container technology and how that applies on embedded devices. A little bit of introduction on container security and we're going to really try to focus on how to secure container images. We're going to talk about build time security, so securing container images at build time and also at runtime, of course. A few disclaimers. So I'm just a developer. I'm not a security engineer or a security expert, so I'm just a developer that worries about security. Usually, security requires a kind of holistic approach, so you have to look at, not like in our example, we don't have to, like, we're not going to only look at the container image, but like everything around it, right? So we're going to have to care about how you view those images, where they are stored and everything in the middle. I mean, you have the operating system and everything. So in this talk, we're going to just cover small parts of how to secure applications based on the containers, right? And we're going to use Docker in the examples, but most of the examples could also apply to other container engines like Poderman. So very quick, because I want to focus on the security side, what is a container? A container is just a convenient way to distribute and it's executable, right? So you take an application with the dependencies and then generate a kind of self-contained image that you can run that application. You just need a kernel to run it, right? And that's the only dependency and, of course, a container engine to start the application. You can have only your application there or even a complete operating system inside that image. For that to work, you need several resources provided by the kernel, like name spaces to isolate all of the resources, like networking, file systems, users, and so on. You need control groups to link the resources that a container can have during execution. So with all of the support in a container engine, you can run containers in a device. And why would we use containers on embedded? Well, there are several different, of course, there are always trade-offs, right? So we're gonna have to pay a little bit to use containers. Maybe we're gonna use a little bit of more resources. But we can say that we can improve productivity with containers because you can focus on application development, right? And you can, you have support to run your application isolated from the host machine, the host environment. You have more, you are kind of incorrect to run your applications in different containers. And then you can have a more modular design. You have more control when running containers because you can leverage the container engine to run the containers and then limit the resources that an application inside a container can have. You might have more control over how to update applications because applications are isolated inside the containers and updating container images is easier because you have all of the dependencies there. And if you use it in the right way, you can improve security also. So depending on the implementation, you might use more resources because you are running applications with containers that could bring kind of more dependencies inside the images. And then, for example, you could increase a little bit the usage of storage, for example. But you can get all of these benefits. And I mean, it's not that containers are secured by default, right? Because I would like to do a kind of quick demo here in my terminal, just to show that, like, secure it's not a default option in containers. It really depends on how you design the application. So I have, so I'm a normal user here, right, as Prado, and I'm going to show how I can access this file that only HOOT has access, right? This is a file that I created in my HOOT file system. Only HOOT can read this file. And I'm not HOOT, so that's why I cannot see the content of this file. But because of my user, because my user is inside the Docker group, and because Docker is running with the HOOT user, I can just run a container with Docker and be HOOT and access that file. So I can do something like this. Docker run dash IT, that's dash N. I can just kind of bind them out to my complete root file system inside this container with the dash V option, like let's say HOOT host. And then let's run the Alpine container and get a shell. So now I'm inside the Alpine container, right? I was the normal user. Now I'm inside the applied container with the HOOT user. And I have access because I bind mounted my host file system inside the container. Now I have access to my host file system because I'm the HOOT user, I can catch the secret file and see the content. So that's like we can say that if you are in the Docker group, and if you are running Docker as HOOT, and it's kind of, I mean, you can run Docker in a rootless mode, but currently it's kind of an experimental feature. So most of the deployments still run Docker as HOOT. And if you are running Docker as HOOT and if you are a user in the Docker group, you can be HOOT. So that's really not secure, right? Just as an example on why like running applications inside containers, you are not secure by default. And the idea here is to talk a little bit about how you can secure applications run inside containers. And again, there are a lot of things that you will need to care about in terms of security, right? When you are thinking about improving the security. So you have the build environment that you have to care about. And you have to secure the build environment. All of these are kind of surface attacks. The container registry, that's where you store the container images and also the devices, including the operating system, right? That will run your containers. So this talk is really, really about how to protect the images. We're not going to discuss how to protect the registries and the build environment here. And there are two important concepts here that we're going to try to leverage in the presentation, right? The economy of mechanism, that's a very important security concept. And we should really try to keep the design as simple as possible. Because with a simple design, the attack surface is decreased. And that means the security improvement. The other concept, important concept is the list privilege. So you should really run the applications with the least set of privileges needed for the application to run. So I'm going to have lots of comments here in the presentation. And you could really try to run on your machine. Those comments, if you want, like just copy and paste from the slides that's going to be available in the events website. And I'm going to use this example in the comments, like a very small C application that we'll just open an RTC file. And then it's going to execute an IO control operation to get the current data time. Very, very simple. The idea is to try to containerize this application and run it in the secure way. Let's see how it goes. So I divided this presentation into two parts. The first part is securing the image and the second part securing the application at runtime. So the first part is securing at build time, how to build a more secure container image. And then the second part, how to run that image in a secure way. So that's how we're going to do it. There is a lot of content in this presentation. So I might, like, go fast in some parts because I'm going to try to keep the schedule, the 40-minute schedule. So, of course, later, you can read these slides and get more information from the presentation. So let's talk about securing the container image. I defined in here five kind of mitigation techniques to improve the security of the container image at build time. So first, create a minimal image as minimal as possible to run the application. Second, try to only run images that you trust. So let's talk about this. Third, how to make sure that your application is secure. So running a static analysis tool could be a good choice here. Fourth, and run security scanning tools in the images that are a few tools that can help to improve the security of container images or Docker images. And last, try to make the solution of the container images easily updateable. That's very important because, I mean, in the end, so after we'll have bugs and you're going to need some kind of way to update the images after a while. So let's start with the first mitigation technique, create a minimal container image. And that's very important, right? Because, I mean, if you create a container image with based on a Debian or, I don't know, Ubuntu or any other Linux distribution, you're going to have a very big image with lots of things that you don't need and that might be like, you might have several different kind of attack vectors inside this. Like bugs in software that could be exploited just because you have that software in the image. So the idea here is really to try to come up with a minimal image with only the dependencies that you need to run the application. And for that, there are a few strategies, like try to use minimal images. That is, for example, the Alpine image that's very small. Google has a project called Distroless with really minimal images for specific use cases. Using a multi-stage build also is a good approach, because you don't mix like the build environment with the runtime environment. Maybe link in your application statically or even build in your application with a build system like Yocto, because you can build containers with Yocto or build hood and have really a minimal image to run your application inside a container. Here I have a few comments that I just want to show again. You can just try to run this on your machine and see how it goes and play with it. So here we have a small Docker file. I'm going to go over, like explain what a Docker file is and but for those who don't know, a Docker file is a file that describes how to build a Docker image. And here we can see that we are using Debian, bullseye for the base image. Then we are installing the tool chain and copying the search code of the application, running GCC to compile the application and then configuring the application as the full application to run for this container. And then you can just build this application with the Docker build. You can see the size of the application that's important, 250 megabytes of RAM of size. So a very big container image to run just a small C application. Of course it works, but it's a very, I mean, you don't need all of these, right? You don't need a Debian image to run this application. So what you could do, like in the second approach here, I'm doing two different things. First, I'm based on Alpine for the application. So instead of using Debian, I'm using Alpine. It's a small container image, like it uses buildbox and muzzle as the little C, so it's very small. And the other thing that I'm doing here is I'm doing a multi-stage build. So this is the first stage where I just build the application. And in the second stage, I create a container with that application without the tool chain, the container, right? And the end result, it's a smaller image, like it was 250 megabytes. Now it's five megabytes, much smaller, right? Of course it still works, but we have less things in the image, it's better in terms of security. Let's improve that, like we could just take our image and build from scratch and start to link the application. And then you have really, really small container image with only the application, start to link it with the dependencies. That's another approach, right? And here we can see, like, I'm still building the application with Alpine, but I'm starting to link the application. And then to create the final container image, I'm using this scratch image as a base. That's a basic, an empty image. And in the end, we have a container image with just like 100 kilobytes of size. So very, very small with only the application and its dependencies. All right, so that's one kind of mitigation technique, right? Reduce the size of the image. Another one would be to try to only run applications inside containers using container images that you trust. So if we look at our previous example, like we are using a tag to reference a version of the container, and the problem with using tags is that tags are mutable objects, they can change, right? So let's say an attack, an attacker is able to have access to the Alpine repository and just change the tag to another container image. And you would not know that, right? And you would pull that image without knowing that you are pulling an image that was, I don't know, changed by an attacker. So better than running, basing on containers, using tags, you can use a hash. A hash is an immutable object, right? You can just use a hash instead of a tag and build your containers with that. You can trust a hash digest, because it's an immutable object. Another thing that you could do is to just sign and check the signatures of your images. There are a few tools for that. For example, Docker. Docker has a framework or tool for that called Docker Content Trust that you could use to sign and check the signature of your images. So what has changed here? I'm building the container with a hash. So this is more secure than a tag, because a hash a digest is an immutable object. I'm really sure that I'm really building from the image. So every time I build this container, I'm going to build the same container, because hash, the hash will not change. We'll always point the same container image. Of course, everything works, and nothing's changed in terms of runtime. Only the build time of the container has changed. We could also run our containers with a digest. For example, here I'm not doing that. In this example, I'm running the final container using my tag. I could trust my tag, or if I don't trust my own tag, I could use also a digest, right, run my container. That's another thing that I could do. And the last thing that I could do is to just sign the image. So Docker has, again, we are talking about Docker here. If you are not using Docker, you're going to probably have to search for other solutions for this. So Docker has a common caller trust that you could use to create the keys, just as I'm doing here. Docker trust key generates. It's going to generate a private public key pair that you could use to sign your container images. And I can push the public key to a registry, and then I could just sign my image with that. So here I have the container on my machine. Then I'm removing this container image. Then I'm enabling the Docker content trust. And now when I'm trying to run this container, I cannot run it anymore because I enabled the content trust, and it is not signed. If I build it again and push to a registry, the image will be signed because I enabled content trust. And then I can run the container. So that's another approach, right? To make sure that you are really running your container image because you are tracking the signature and only you write you. At least that's the expectation, right? You have the private key to check the signature, and you can make sure that you're really running the container that you signed it. Another approach and this is really generic, right? It's not specific to containers, but running a static analysis tool is a good approach to improve the security of any application, including applications running inside containers. So the idea here would be as another step when building the container, just run the static analysis tool. So here I'm running CPP check on the search file. And if it fails, you should not build the container image, right? There are several security scanning tools that we could use to improve the security of the container image. So for example, this one, call it 3V. This one is from Aqua Security. A very nice tool. You just give the container image at rest. It doesn't need to be running. And it's going to scan the image and show like the security holes or the vulnerabilities in the image. It has a nice kind of output with a table with all of the software that you have inside that image and the version of the software and the known vulnerabilities. And then you could just like remove something that you feel you should not have in the image. Integrating those image, those tools in a SCI environment could be a good approach. This one I'm using right to scan the images, kind of the same output. Let's see. Try to come up with a solution to make it easier to update the container image. That's very important because in the end, you're going to have problems in software, software have bugs and you have to fix those bugs. So it's important to have some kind of a mechanism to update the container images. There are kind of complete distributions that provide these out of the box, right? We have Alena. We have a Linux micro platform from Foundries.io. We have Verizon Core from 3DEX, CoroS from Fedora and so on. In the end, you're going to have to find a solution if you are designing an operating system and running your applications inside the containers. Second part of this presentation, so now I'm going to talk about how to secure the execution of a container image, right? And there are several different mitigations techniques for that. Let's go over all of this. First one, restrict container privilege. So far in this presentation, I've been running the container with these parameters and this is very bad, right? So here I'm by mounting the complete slash depth directly inside the container. So I have access to all devices inside the container and I'm running in privilege mode. And we're going to talk about what that means running in privilege mode. But I'm basically with those two options, I'm basically root inside the container. I can do everything that I want inside the container. I'm kind of almost the same thing as the root to use in the host to access. So that is not good. Just as an example, right? I'm running the same, I'm running the container with the same parameters. If I start a shell inside this container, I can see that I'm root. I can see the complete slash depth directory with everything that I have in my host operating system. I can just mount my root and fast partition from my host operating system and I have completely access to the root file system of my host to us. So there is no security there, right? We need to improve that. So the first thing, remove this privilege flag. This privilege flag was created so you could run like Docker inside Docker, especially in a CI environment. But we should really avoid this flag when running container emergency in production. Because it enables all capabilities, we're going to talk about that. It enables access to all device files in slash depth. It basically disables ECM or SC Linux if you are using it. So, yeah, you are kind of root inside the container with this privilege flag. So how can we run the container without those two flags? If you think about it, what our application needs is just access to the slash depth slash RTC file. And we don't need anything like everything else inside the slash depth. Only this specific file. By default, we don't have access to those files, right? Because Docker use a kind of feature called C group files to prevent a user from access device files inside the container. And there is a flag in Docker called device that you can use to enable access to an specific device file. With this flag, Docker will buy them on the file inside the container and also create a C group rule so you can have access to this file. So we can remove those two flags and just use this one slash slash device and the device that we want. And that's it. We have access to the device. We can read it, call it IOC control and run the application. So we are really like applying here that concept of least privilege. We only need access to the RTC file. So we're going to only map this file to the container. Another thing that we could do to try to improve the security of the container is to run without foot. Because by default, if Docker, the Docker demo is running as foot, when we start a container, you are hoot inside the container. And to run a container without being the root user, you can build your container with this user instruction or use the user option when running the container. And if you do that, like I'm doing here in this example, so here I'm running as foot without any special parameter. I can see with the ID command that I'm root. I can run the application. Now, if I give like this dash user parameter, like I'm here, I'm trying to run with 2000 user. I can see that inside a container. I'm the user with the ID 2000. I cannot run ping, for example, because I'm not hoot. But I cannot also access the device file because I'm not root. So that doesn't solve our problem. Another approach, but I mean, if you don't need to be root, you should do that, right? Run without a root user. If you need to be root inside the container, another approach that you could try is to use a username space. The idea of a username space is to really separate isolated users from the hosts and from the container. This is a kernel feature that you can use and the Docker engines are able to use that. And the idea here is to just create a range of user and group IDs in the host OS that would map to the container that is running. So here in this example, the user 100,000, here I'm creating a range of users starting from 100,000 and into this number here. And like inside the container, user ID zero would be outside the container, this user ID. Inside the container, user ID one would be outside the container, user ID 100,001. And that's the point here. So you can just configure the Docker with that and run the container. And that's very nice because like when you run the container with the username space enabled, you can see that like you are running as root, right? You can even like do execute root operations like a ping usually requires a root capability. But you can really confirm that you are not a root user because when you try to run the application, you cannot run it because the application requires access to the root user, right? The application needs to access this device file that in the host OS, only the root user has access to it. The root user from the host OS, not the root user inside the container that is mapped to another user inside the container. So that confirms that name space is really working. And I mean, in the end, it doesn't solve our problem. One way to solve this issue here would be to create a group and use Udev in the host OS to change the set a group in this file. So like let's say let's create an RTC group and satisfy with the RTC group. So users inside the RTC group could access this file. That could be one approach here. Let's say you need to run containers with the root user another approach that you could take to improve the securities drop in capabilities. So capabilities are concepts, very old concepts on Linux. In the very long, long time ago, we had a kind of binary decision to define what kind of privileges a process has. So you have everything or have nothing, right? Process running with ID zero have all of the privileges and process run without being zero user ID zero and no privileges at all. And then only on Linux 2.2, I mean, long, long time ago, the capabilities feature were created. And the idea here is really to divide the privileges in the kind of distinct units called capabilities. So let's say you have an application that needs to set up networking, but doesn't need to do credential management. You can do that, right? Dropping capabilities that you don't need when running Asphalt. So that's the main idea here. And usually container engines are able to make it easier for you to configure that, right? So in Docker, you can use this option kept drop to and kept add to drop and add capabilities. So if you take this to our implementation, right? If I run the container without this option, I can see that I have lots of capabilities that really I don't need. If I run the container with without any capabilities, kept drop all, I still can run the application. So my application doesn't need any capabilities, right? It doesn't. You just need to open that file and write to it. Another mitigation technique to improve the security of container images is restricting CIS calls. So the Canyon Linux has kind of 30, 300 plus system calls, you're going to not problem going to use all of those system calls. So if you limit the system calls that you can access, you're going to improve the security of the application. And there is a kind of feature called second that you could use for that, right? Docker kind of uses that by default. And you could improve, like, change the Docker default configuration and, I mean, limit what an application can do, right? So here I have a kind of a second profile that I created. Not everything here because it doesn't fit in the slides. But I created the second profile based on Docker's second profile that is on GitHub. This is the link here. And if I run with the second profile, the application works. And if I go to the second profile and remove the IO control system call, the application doesn't run anymore. So with that we can make sure that kind of second is working, right? Preventing. And then, of course, you could take this profile and remove what you feel don't need in the application. To identify all of the system calls that you need to do, you could do use tools like sTrace for that. Another approach to improve the security of container images, managing the resources of allocated to containers, and that will prevent, that will not prevent those kind of privileged escalation problems, but will prevent the issues related to denial of service ethics. Let's say you have a small device with a container running a web server. An attacker could kind of send thousands and thousands of HTTP requests to this small device and could just break the device, right? Like, the device could run out of memory because the container is allocating lots of memory to run lots of threads to try to answer all of those requests. So if you limit the resources available to container images, you're going to improve the security of the container. And for that, there is a feature in the Linux kernel called countergroups that you can use, and Docker has several parameters that use countergroups to limit access to resources. Like this example here, here I'm running the container as usual without any parameter and we can see that the container has kind of access to all of my memory, right? And if I limit the resources, now the container will only see that. And with this, we improve the security of the solution. Another approach to improve the security of container images is using Linux security models like ST Linux and the API armor. So those security models, it's a kind of extra checks in terms of what an application can do, right? We have the normal checks, like the application can, if the application running as user X can access this file and things like this, and the Linux security models will be on top of this doing more checks on what an application can do in the system. Usually Docker is deployed with API armor enabled, and we can see that like running Docker info. And the idea is that you would create a a profile for a security model. And at least in all my opinion, API armor is simpler than ST Linux. And that's why here I'm showing example with API armor. So here I have an API armor profile just to show like how can you control access to resources using API armor. So here in this profile, I'm configuring the app. This is a binary, my binary, my application. I can execute this app. And I can read this file. And I cannot do anything else. So I'm really limiting here one application can do even an application running as good. You're not able like to do anything else in the system. Only run this application and access this file. And of course, it runs application run with this configuration. Now, if I like, in the second example, I'm removing this line. So now the application cannot access the RTC file. And if I try to run with this profile, I can see that it works. The access to the RTC file is denied. So that's another mitigation technique to improve the security of container image at runtime. I'm finishing the presentation here. So just a few more slides. Security networking is very important if your container, if the applications that you're going to run inside the container, if they have some kind of network connection, right? For example, when you run containers by the full docker, we'll create a full bridge connecting all of the containers that is running in the same host in this bridge. So that means if an attacker is able to exploit a container and this container is connected to the same bridge of another container, this attacker could try to get to the other container. So if you really don't want to have this connection between containers with using the full network bridge, you should really create a specific network for a container that can be done in docker with the docker network create command. Other things that we do like try to not share the network with the host, right? That's important thing. Try to not share the docker's unix domain socket with the container. If we have to provide some kind of connectivity with a container image, try to use TLS, secure connections. So those are kind of some approaches that we could take to improve the security of network communications in containers. In terms of storage, one good approach is to mount the file system really only. So we prevent any kind of writes in the hud file system. That should really be immutable. That's a good approach. Try to develop container images that are immutable. You should really not try to be writing to the file system this flag would prevent that, right? If you really want some kind of temporary file system to store temporary files or things like this, you could mind mount a team FS directory inside the container. And for persistent storage, you could use a volume, a docker volume for example, or you could bind mount a directory from a container. And if you care about what you're storing, right, in terms of privacy or if you have like storing secrets or things like this, you're going to probably need to think about encryption. But that's kind of out of scope here. So yeah, we are closing here. In this presentation, I tried to come up with different mitigation techniques that could be applied to try to write more secure container applications, right? So we started with a very secure way of running a container image. It works, but it's very insecure. You could do lots of stuff with this comment. And here in the comment below, we have a really more secure way to run the container, right? Because we are only limiting the application to access the specific file that it needs. The file system is really only so no access to the file system inside the container will be allowed. You could create a, like in this example, slash run directory to store temporary files to run the application. That's kind of not required for this specific application, but it is just as an example. You can drop all root capabilities, right, especially if you run the applications with and just enable the ones that you need. You can drop all privileges. This will prevent, like if you are running the application with a normal user, this will prevent this user to run in set user UID applications. You can have a custom second profile with only the system calls that you need. You can have a custom API profile limiting what the application can do. You can run in your own network, limiting the resources, if you memory. And the application does work, but it's more secure. So I hope this presentation was useful. Hope you enjoy it. Feel free to reach me on LinkedIn or Twitter. Just connect and let me know if you have any questions. Hope you enjoy this talk and until next time. Bye-bye.