 It's time for next talk from John. So let's welcome John and he's talking about Docker. Okay, hello everyone. So the talk is about Docker security considerations that I have and some thoughts on incident analysis on it. I work as a junior security engineer for Percona for less than a year so bear with me. I'm still doing the transition to the infosec industry, which was kind of my hobby. I do have several years of experience in electronics and I was freelancing as an IT field engineer and consultant. I will be having a problem during this presentation because I wasn't able to do join displays, so I'm just mirroring. I will not be seeing my notes, so some things will be less than expected, but hopefully I will bear with it. So why this talk? This talk is about Docker. I guess everyone around here knows that its use is spreading. It's very easy to use. It's fast to deploy. It offers you isolation, portability, etc. So from a security point of view, two aspects of it arise. One, of course, is a well-known question. It's being asked a lot of times, is it secure? I'll try to touch on the answers for these questions. And the other is how to prevent and handle security-related incidents. I'll try to touch on some concepts on this question as well. So some key facts about Docker. Docker is utilizing the technology of Linux and namespaces and control groups. This is basically what offers it the ability to provide isolated resources on some processes while keeping others hide it. Let me phrase it like that. Another key fact about it is that each demon runs as root and that does have some security implications. Another key fact about it is that containers, by default, come without any control group limitations, which means that a containerized process with Docker can use basically all of the resources you offer it, like CPU and memory. It has a default profile for a system called white listing, and there is also another key fact about it that you can use Linux capabilities. We'll talk about this in the next slide. In order to fine-grained the capabilities that are offered to Docker. So by default, a Docker container comes with these capabilities. There are quite a few, so we will not have time to talk about each one of them. But reading the manual pages from them, you actually see that most of them are not needed when you are in production stage. There might be needed when you are in development stage. You do have to do a lot of things, but in production stage, where you have a container running, I don't know, a server or whatever, you don't really need them. One of them stands out for the purpose of this presentation. I will speak about it in just to make my point on it, but this point stands for basically most of them. So this capability, let me say one more thing about capabilities. Capabilities is a way that Linux offers you to divide the powers of root, so you can escape from the binary root or user, so you may have root permissions, but not with all root capabilities. Many of you may be aware of it. I wasn't aware of it until I used Docker for the first time, and I was motivated to search for more about it. So a capability known as discretionary access control override. Reading the manual pages, it is stated that this capability allows root to bypass file permissions. So this is as odd as it sounds for a process. As noted by a Red Hat Security Standards expert, no application should need this, and if your container needs it, needs this is probably doing something horrible. So the point is that after Dockerfile version, I don't have my notes, but I think it's version three, you can actually drop all capabilities and add them in a need to go basis. So it's something that you should keep in mind when you are developing your application in such an environment. So the motivation, as I said, is very easy to use, very easy to deploy, so I like it, but it has some security implication. You don't have to set up a full blown virtual machine, plenty of images to use and spawn your containers. There is a trap on the above point, not all of them, all of the images that are out there are secure. I'll talk about that in the next slide. So the default configuration, at least from my point of view, is not suitable for production purposes from a security point of view, of course. Docker leaves to the user a lot of decisions to be made and this does offer a lot of flexibility, but as you can assume it also increases risk. There are well-known vulnerabilities difficult to be redesigned. I already mentioned that its demon runs as root, so that is, from a security point of view, a vulnerability itself. As far as I know, there is a rootless demon out there that Docker offers, but it is still in experimental phase, so it can't be really used in production stage. So you can try it yourself if you download the community edition and run this command. You actually have root permission on your system, so you are only one line away from privileged escalation with the default configuration. For the point I tried to make with using images out there. A study was made that was published in 2017 by these guys. It's difficult for me to offer the names, sorry. Its scope was about 350,000 and more images, both official and community images contain more than 180 vulnerabilities on average, so you can understand the impact of blindly using what's out there. Many of them have not been updated for a hundred of days and vulnerabilities commonly propagated from parent to child images. The study itself concluded that these findings demonstrated the strong need to automate and have a better and systematic method of applying security updates to Docker images. So I guess these slides does make another point for why this talk is at least useful. So a few things to take away from this part of the talk is secure your engine. I mean, if you want to use it, use it, I use it. I like it, but there is a lot of stuff to do if you decide to do so. So you have to secure your engine. First of all, you have to secure your host because as the documentation of Docker itself says Docker is only as safe as the host that you install it and I guess that goes for all applications, not only Docker. Drop all capabilities when you start in then only add them in a need to go basis. That's the better approach, at least how I see it. Use trusted images. Use certificates based authentication when communicating with the Docker demon. Protect the Docker demon socket. That's another part that it's not included in the slides, but it is something to be aware of, although I think that now there is a secure API available to talk to the Docker demon because Docker uses a domain, Unix, non-network socket, so you can't really talk to it, but this needs to be updated. It's probably using a secure API by now. Nevertheless, you still have to protect it because it's the mechanism that allow you through a container to talk to other containers and see what resources are available or not. Use APR, more of course, FC Linux, that's all supported by Docker. You should always map the root user in your container to a non-root user on the host because, as previously demonstrated, you're only one line away, so if a vulnerability exists inside the container and the root user of the container is mapped, the root user of the host is, again, just one line away. So rootless demon. That's something I am waiting for it to become available. It is available, but as I said, it's an experimental state, so it's not really there yet to use it. And always try to keep in mind there's Center for Security, for Internet Security benchmark about Docker. It's pretty analytic. It also has a script available on GitHub that actually follows all these guidelines and it does a good job on securing your installation. So you should probably use it. So incident analysis or what to do when things go bad because all these considerations, all these thoughts, all these points I tried to make before, I actually thought about them after finding out that I was only one line away from root user in my system. And that's usually the case. We don't even think about what can go bad before it actually goes bad. So in principle, a foreshadowing analysis on a Docker environment starts like on a regular system. You actually do an HD dump, a memory dump, and you analyze it. However, analysis of the dumps may provide incomplete results because container specifics must be taken into account, mapping to containers, information about weather files, certain files are relevant for the reconstruction of the file system. So you have to think about file recovery and accountability, which file belongs to what image and stuff like that. The known methods that are used is file carving, a file system analysis. That's not something new, but there is some key thoughts that you have to take under consideration when used in a Docker environment. So let's talk about the file system of Docker. Docker uses a layered file system. It's not a classic file system. A layered file system is based on a file system driver which offers the possibility to build a single file system from different layers to present it in a uniform and abstract manner to a process. That's what's illustrated in the picture as well. So that is actually what enables you to save money when you apply that in the cloud or something like that because, okay, you use one image and from that image you spawn several containers. You all know about that. It's nothing new. But there is something you have to think about it when you try to do, when an incident has happened, you try to do some analysis on your findings. As I said, it's not a classical file system, which means the file system is not mapped to block devices as in traditional virtual machines. So when this dump of the host is made, you must answer which image provided any given file, which container used the given file, what's the file deleted at a container level. Just to give a bit more detail here, every from instruction you place on your Docker file is actually adding another image layer on the top and the container itself has all its right accesses on the thin red right layer you see there. So you understand that you have to answer these questions to understand or to explain why the incident has happened, how it has happened, you need to provide accountability of where the file came from, which container used it, was it delivered at the container or an image level. This is my first talk in a conference, so please bear with my English and my anxiety and thank you for them for accepting this talk. I mean, it means a lot for me. So file carving. File carving is a method of linear searching of volume, this key-major file. The file system is not considered with this approach, so can't recover files that are not yet overwritten. It looks for characteristic patterns, what we call magic bytes. The problem with this is that you cannot acquire metadata for any given file, so you cannot actually map the file that you're examining to a specific container. Only possible way using these methods is if the file itself contains some characteristic, maybe a container ID or something like that. File system analysis. File system analysis uses management structures that are stored in the file system. If we talk about mtfs, that would be the master file table. If we talked about extfs, that's I know tables. And the big, the big advantage here is that you actually get metadata, you get the file name, you get the path, you get time stops. That's very useful info that you can use in order to get accountability, to actually explain where your file belongs, where it came from, where it was deleted, and stuff like that. So, these two methods are generally used when the methods, I mean, file system analysis and file carving are the methods you use when you're examining files that come from the container level, the thin red light layer that I was presenting before. If you have a file deleted in the image layer, you have to talk about how it will be recovered. So, docker uses, the file system that docker uses leaves a deletion reference in the thin red light layer, but the file remains in the image layer. So, with this command, you actually see the difference on the directory structures. You can actually see all the differences and then iterate through all the different layers. So, you have to find the file you are using. I had a lot to talk about this slide, but I cannot see my notes now and I'm having a little bit of trouble here, so bear with me again. Linux namespaces. Linux namespaces have various effects with regards to incident analysis because you have to be able to tell which PID is from a container, which PID is from the host, because inside the container, the PID numbers can be the same as the PIDs in the host. So, when you get a raw dump of your disk, this will not be easy to distinct. You have to be able to distinct between process IDs inside the containers and process IDs outside the containers. And that's something, actually, you can do only with runtime information because if you just take the dump and you cannot recreate the runtime environment, you cannot actually do that. For user IDs or group IDs, it's a different case because you can actually do that. There is files like sub-user-ifd and group-user-id under slash-ctc. So, it's a bit easier to do the distinctions and give accountability in that regard. So, that was it in a quick way. I know it goes pretty fast. I don't have my notes in front of me. So, I guess we have time for a bit of more conversation. Help me with your questions. I'm happy to have a conversation with you. So, thank you very much. Before we do that, I just wanted to show you about this event. It's on May 2020. A lot of great minds will be there. So, have an eye on that as well. There is one there, one here. Do you know of any tool that would make the file system analysis or memory analysis easier when you have a container environment? I don't have something in mind. I do know that there are a lot out there that can be used, especially for fire carving. I have some previous experience with that, but not something specific to use in this context. That's what we're asking. There is also another. What's the program of rootless demo? Sorry? What's the program of rootless demo? So, you say rootless demo is not ready for production, right? Yes. So, what's the program of rootless demo? What's the program of rootless... Yeah. Rootless Docker? Yeah. Is this what you're asking? What's the name of the program for rootless Docker? It's not my point of view. It's Docker's point of view. They actually say in their documentation that it is still experimental. So, you can't use something experimentally production because if things go bad, that costs money. That's the main point of it. I mean, you don't use experimental software in production. Yes, there is another question up there. So, VM-like isolation in Docker. You know, like Microsoft have Hyper-V isolated containers? Can you speak louder? Yeah. Sorry. So, how the VM isolation in Docker will help to tackle the problems of security? Similar like Microsoft running Hyper-V mode for the containers. There is it for the Google as well. Each container is actually a tiny VM running. So, can you repeat the question? Yeah. How this will help us to work out the problems of the security? How Docker will help you? Now, the VM isolation. Sorry. Could you speak louder, please? That's my main problem now. Sorry. So, experiments with the VM isolation in Docker when the container runs as a tiny VM. Each container is a tiny VM. Will it solve the problem of this given too much capabilities to the Docker containers? I'm not sure I'm able to answer that. I still haven't understand your question. That's why I really can't answer it. I mean, is your question concerning and comparison with tiny VM? That's... Yes. Oh, if you run Docker inside a VM, that's an approach that it is happening. And it solves you all the problems you have, but all inside that VM. So, all this still applies in the environment of your virtual machine. It doesn't actually solve the problems I describe for Docker itself. I don't know if I touched your question, but... So, a common problem that I see is often containers have the services running as root because they need certain initialization steps. I'm not sure. What would you see as a solution for that? So, what your question is about is when you need services to run as root. No, not exactly. You have a service, and often you give the service environment variables telling it which database to use or what the password for the database is. Okay. And, well, then they do initialization, write their config, but they still have all this information. And initially they have to start as root because, well, file system permissions and whatever are set. And, yeah, that's kind of problematic from my perspective. It is problematic. That's why Docker's daemon is still being run as root. That's the only way that they sold it, so... I'm not sure. So, my question was rather, how could we solve this problem in a different way so that you don't need to run the entire container as root if you have some initialization steps where you usually need root because, well, in my opinion, it would be best if you have three steps for a Docker container. The first is built the image, the second is initialized image, and the third is run the image, or run the containers. I can agree with your approach. I totally can agree, but how does that solve the problem of having your daemon running as root? Yes, that's an approach you can use for sure. I'm sorry, we are out all the time, so let's thank you. Thank you very much.