 So, my name is Sagiha Kaluche. I'm a senior security researcher for aqua security. I did this research along with Michael Czerny, who's the head of research for aqua security, and let's just dive into what's on the menu today and what we're going to talk about. So, we're going to talk about a lot about containers and container development, and the main course of this lecture is going to be how we're going to attack developers. I'm going to break this down to three steps. We'll have a demo for it's stuff. We'll introduce two new kinds of attacks, the most remaining attack, and the shadow container attack. And finally, we'll see like a whole flow and then we'll have hopefully some conclusions. So, containers, what are they? So, this is the risky part. Besides the microphone, I'm going to show you something. Because how many guys have you ever ran a container? Show my hands. If you didn't run a container, I just want to give a feeling of what that is. So, I'm going to drop the mic. Zip Docker is a pretty popular container engine. And you can think of containers as a sort of virtual machine. So, for example, here are my hosts. It's a Windows host. I have two sort of virtual machines. They call images. Images basically, you can think of it as the operating system of the container, more or less. So, I have Nginx and Debian, and now I'm going to run a container. So, I'm running a Debian container, and like I said, I'm running those, and I'm doing LS, and it looks like I'm inside a Linux machine. I can also look at operating system. It shows that it's Debian, and I also can look at kernel. We chose something called Mobile Linux, a little bit weird. We'll discuss about it a little bit later. You can also see that it's 64 bits. So, how would that work? So, if you ever went to a lecture about containers, you've probably seen this slide. It's trying to compare on the left side virtual machine against containers. So, if I'm a developer, like we saw, here I can write applications on top of containers. I don't need this whole heavy lifting virtual machine does. To take care of the operating system itself, and also the hypervisor layer that is needed to run virtual machines. I can simply run a container, which is actually a process, and it's living inside its own virtual little world. So, it sees its own processes, our network stack, and its own file system. And containers basically work everywhere. Obviously, you can write Linux containers and work over Linux, but you can also run, just like we saw here, on Windows and on Mac. And if you want, you can even write Windows containers. So, how big this thing is? I'm a little bit surprised to see so many people know about containers. These are stats from Dr. Khan from the beginning of the year. Again, Dr. the biggest, currently the biggest container engine. And these are from yesterday. They tweeted from Dr. Khan. They documented how much did they grow in the past less than a year. So, we see pretty big numbers. 40 million different hosts connecting to the Dr. Khan and pulling stuff, and now we're up to 21. And these numbers are very big. 1.2 million applications, over 21 billion image pools. So, there's a pretty big community of developers writing and running containers. And developers are pretty favorite target for attackers. Developers are getting paid to write code. They don't get paid to do things securely. So, they have a lot of technical skills. Usually, they're just under the chains. And also, almost by definition, they have access to code, IP, in case of containers, registry. So, even if you infect a single developer and that developer is writing a microservice or an image, you can infect the entire pipeline from yield to testing all the way down to production. So, this was sort of an intro. So, let's dive into the attack. I'm going to give an attack overview. For this attack to work, you basically need two things. We need a developer working, a docker, having his TCP port open. This is not a lot to ask. This was also the default installation once you download a docker for Windows. And we need to get the developer to visit some malicious webpage. Also, not very hard to do. And from there, we're going to dive into Windows 10 attack. We're going to break it down to three, like I said. Like most attacks, we're going to start with remote code execution, and we'll do that by using Docker API. Then we're going to go ahead and do a host rebinding attack, which is a new form of attack we introduced here. This will give us pre-legislation, offer the Docker event, and finally, we'll run a shadow container, which will give us persistency. So, abusing Docker API. The idea of abusing it came when we tried to figure out how this works. Like you saw, I did Docker Run, and the container was running, but what is really happening how a Linux container is running in a millisecond on a Windows machine? So, we tried to understand how that works. And the story is that there is no magic there, of course, which you are doing as you have your command line client that command line client is talking over a restful API with a Docker daemon that is on your host. And that Docker daemon on your host is talking through a virtual socket to a virtual machine running inside your host, whether it's Windows or Mac. And by default, in older versions of Docker, that port was open. So, of course, we thought what will happen if we'll try to abuse it through a Linux's web page. So, it's not that simple. I'm sure the audience here knows a lot of the same origin policies. So, browser security came to the right place, I think. It's a browser's job to ensure that your data doesn't get leaked to another domain. So, at least for example, you go to your favorite news site. You see all your friends that like a certain page, but that doesn't mean that the news site can actually go to Facebook and read the list of all your friends and send messages to them. And one of the security measures that does that is the same origin policy. So, if a request is made across a region, if it's not from the same domain, product, all, or host, then it's not considered to be from the same region, and it's bound through restrictions. So, you can actually send simple messages, and also you can do information. What the HTTP method you can send gets, what you can put in response, you can send less to the body, and so forth. So, we said, okay, that's the playing field. Let's go ahead and list all the Docker API calls that don't file like the same you all see. And immediately we notice the build API call. So, what build does? Build basically tells the Docker engine to construct an image. And it looks at something called the Docker file. It's basically a text file. And it's reading it line by line and does things. Here, for example, I'm pulling from an image called Alpine. I'm adding some shellcode. I'm updating packages. And finally, I'm running some shellcode. So, building is actually executing code because we have the run command. So, we said let's go ahead, let's use this build image API call. Like I said, it doesn't balance emerging policy and it has some very interesting parameters. The first parameter is the tag parameter. So, you can create an image and you can call it whatever you want. The second very interesting parameter is the remote parameter. So, you can build an image from a repository find a GitHub, for example. So, you can use GitHub as a sort of command and control infrastructure. And finally, you can control the network mode over which this building process happens. So, of course, you want to select something that has access to the network of the host. So, now we're going to see a demo and want to demonstrate how from a single API call you can do something malicious. We'll simply build an image and that image is going to open a reverse shell. So, you can see we have here the actual command. It's the post build command. The remote tag points to some repository that we already created over GitHub and we're going to use the network mode of the host. And this is the actual file that we have in GitHub a single file called Docker file. And as you can see, all it does is open a reverse shell. So, let's see how that works. So, the first thing that will happen, the victim, the developer, will visit a malicious web page which will issue a single post request. That request will be forwarded to the Docker demo on the virtual machine. So, now the Docker demo goes ahead and pulls our GitHub which is our electronic control center and then it's running the build container which opens up a reverse shell. So, let's see how that works. Here we have, on the left hand we have victim and on the right we have the attacker. So, first I'm going to list the images and send the containers on the victim. So, we have just one image and there are no containers. On the attacker side, all I have to do is simply wait to host my malicious web page that sends a single post request. I'm doing Python and I'm waiting for incoming connections. That will be the reverse shell. So, once that is happening, as the victim all I have to do is navigate to that malicious web page. I'm going to put on and once I do so you can see the entire build process feedback. This is just for a demo. So, you can hide it in case of an actual attack and once this is done the reverse shell will be open. We can see the actual images that I pulled. I pulled the Alpine image and you can see the container that is running. That container of course opened the reverse shell and now I can go ahead I'll look at the network interfaces from the reverse shell. I'm actually running on the container on the virtual machine and here is the virtual interface. So, of course we noticed we informed Docker that they have this issue. So, their fix was to make the TCP port option and opt-in feature. So, you can still use it but it is off by default. So, that's the first step. So, single command, running code inside the host and now we're going to go to the host rebinding tab. So, of course we asked ourselves like what is the next step of the attack. So, we have a malicious web page. We have a running container but we are limited with what we can do. We can't do anything that we want in front of the Docker email. Of course our API is limited because of the same region policy and we have a limited lifetime of the Docker container. It won't run forever. Another form of that control many of you are familiar with that we thought about is maybe to DNS rebinding. So, a spoiler alert. We don't do DNS rebinding. It's not a new attack. It's pretty old, about 20 years ago. Made famous by a guy called Dan Kaminski and I'm not going to discuss too much about DNS rebinding but just to give a feeling of how it works and then why we chose not to do it. So, for example, if I'm an attacker and I want to attack an internal server used by the victim, I can set up a domain called attacker.com and I get the victim to visit my attacker.com so he issues the DNS request and I answer with two different IPs. One is my actual IP and the second one is the IP that I actually want to attack. So, the browser goes ahead pulls my web content which is some malicious script and all that script does is issue another HTTP request and now I'm resetting the connection so I've effectively bypassed the same original policy. Now, the browser is using the second IP which is the internal IP but he's still thinking that he is over attacker.com so all the same original policy descriptions are no longer working so the attacker script can have full access to the internal server. So, like I said we're not doing the answer binding. It may fail it's something that goes outside of the perimeter and can get caught and we don't want to go through the hassle of building up a domain, maintaining it, making sure it doesn't get blocked by IP reputation and so forth. So, instead we're going to attack another protocol called LLMNR basically name resolution protocol using Windows environments and it's DNS done over the local area network and it's very easy to attack. So, when a Windows machine is trying to look for a host that is not a domain, it's just a host name it leaves local protocols such as LLMNR and will broadcast a request over all interfaces including virtual interfaces including the virtual interface of the virtual machine so you can see where this is going so now we can spoof LLMNR responses from a container sharing the same network as the virtual host and that response is cached in the browser in most browsers it's cached for about a minute and if it's minute so it means it will take us a minute to bypass the same original policy but if it meant it's too much you can skip this cache in Firefox for example you can simply delay the HTTP response in half a second so instead of waiting 60 seconds to bypass the same original policy it will take you half a second so how would that work? So the starting of this part of the attack so we assume we already have a reminder container running inside the virtual machine we got that in the first step and what is going to happen now we're going to redirect from the Windows Web page to HTTP slash POD and what will happen is that the host will query where is POD if we query it over the virtual interface and we can spoof from the reminder and like I said we are POD so now the web server is going to download some malicious script from the reminder and what we are doing in this script is simply continuously trying to query a path that we know will have a good result on HTTP 200 response from Dr. Demin so at the beginning we get for a full from our reminder container and we continue to do so every 5 seconds and finally the caching expires after a minute so another request is issued we spoof it again this time with the IP of localhost and now we know that we bypass the similar to policy we get 200 response and to demonstrate what we can do with it we're going to steal some secrets from Dr. Demin and post the secret back to the reminder container so let's see that happening so here I'm going to my localhost from my own host of course so I'm listing my secrets and I only have one little secret called my secret this is my most secretive secret now when I run my reminder container of course I'm using the network stack of the host and it is beginning to spoof responses and hosting a malicious web page so now I'm going to go over to that malicious web page I'm going to pond the same port number of the Dr. Demin we're going to open the debug part to see the actual requests so you can see once we loaded the malicious web page what is happening is we have a request happening every few seconds and that request doesn't have any response from the web server and this will continue like I said for about a minute so we're going to spin it up a little bit so we won't leave the full minute and here it goes we're spinning it up the caching will expire another request will be issued another spoof will happen and then we'll bypass the storage policy you can see it here I'm reading the secrets and I'm posting them back to the reminder so to recap so far medicine complicated so I'll break it down we've seen two steps of the attack first we've used the Dr. API in the demo I just opened a reverse shell but you can do that to start a reminder container on the second demo and the reminder container's job is to bypass the same region policy so now I have a reminder container running and I have full access to the Dr. Demon and the question is what to do next let me suggest an attack called a shadow container so what is a shadow container and why should we do it so so far we have the ability to run any privilege container that we want on the virtual host which means we have full access to the virtual host full access to mobile Linux full access to this file system and we have access to the internal enterprise network which is still not concealed for example anybody who does Dr. PS will see our container and we don't want that and we're not persistent because if you turn off the Dr. Demon or if you turn off the machine turn us off the Dr. Demon what happens is when the Dr. Machine puts up again it loads from an image so whatever is left on the machine will be deleted and everything except of course the state of Dr. Demon so how do we make this also persistent and concealed so let's assume for one minute that I already have two scripts on the virtual machine one of them is the attack script I'll call it my script and the other one is a shadow script so what happens when the Dr. Demon shuts down it's going to run a shadow script to create a container let's call it a shadow container and that charges the container whole job is to keep the state of the attack so the script my attack script will be saved inside the shadow container and once the Dr. Demon is up again my shadow container will write back the scripts to the virtual machine so I'm back to where I started so now I have this sort of ping pong game whenever the user is actually running Dr. Demon and using Dr. Demon all of my attack is being done and saved on virtual machine and whenever the Dr. Demon is reset a shadow container is created which the user doesn't have access to because Dr. Demon will shut it down and he can't see the attack and when it's back up again I'm back in the game this is the actual shadow script that I'm going to use in this demo it's a pretty simple script all it is loading into a variable my attack script and then this variable is given to the container as an environment variable I'm running a privilege container I will name it shadow and you will also there's a restart policy and the restart policy tells the Dr. Demon to continuously try and execute the container until it succeeds and when it succeeds it fills itself and this is the actual attack script also not too complicated its job is to check if the machine is still in the process of shutting down then it's not going to do anything it goes back to sleep but if it's starting up it's going to rut down the scripts that would be hard is install the shutdown script and finally we can do something bad in my case I could only think about doing one bad thing in writing a file so my bad thing to show that this is actually doing bad stuff is writing a file into writing hacked into the shadow file but of course you can scan the network break into machines, brute force stuff or do whatever you want ok so now let's see a demo of the final part of the attack so we will run the shadow container directly from command line so to show that I have nothing up my sleeves I'm going to list the images and the containers on my host so I have two images two containers and I'm going to run the spectrum container and that container will give me access to the underlying virtual machine and I see I have no shadow file there no big surprise so I did this implement which runs the shadow container and once I do it when I list images and containers we see that nothing has changed I still have two images and two containers but of course when I look inside the virtual machine there is the shadow file so I left nothing behind and I've done my evil stuff which is write the shadow file we can even see the attack script here that we saw in the presentation and the shadow script but to prove that this is actually working we will have to restart the WDMN to see that it's actually we remain persistent so we're going to restart the DMN and then we're going to repeat the whole thing we're going to look at the images look at the containers look inside the virtual machine it's still happening there with me for a few more seconds there it is the DMN is up so now I'm I'm going to use the specter again I'm going to look at the images and containers so we're back where we started we still have two images, we still have two containers and when we look at the virtual machine we still have the shadow file even though the machine was booted up from an image we still remain persistent using the shadow container so we've broken down the attack into three parts but let's see how it works from beginning to the end so to recap first step we're going to visit the images web page which we'll have used the Docker API this will give us a running Rebinder container and the second step the Rebinder container will go for running attack so now we'll have full privileges over the Docker DMN we'll use these privileges to run privilege payload that payload's job is to write down the shutdown speed and when it does so we can delete all of those images from the Docker DMN we don't want to leave anything behind and whenever the Docker restart it will create a shadow container whenever it's back up again and we get the error so let's look at a full demo so it's very similar to what we did in the beginning we're going to look at the initial state we have two images and two containers so when we look inside the virtual machine at the beginning of course we want to see anything no shadow file so we're ready to start so I'm going to go to shadowcontainer.com another website I'm going to click a button and again we're going to see the feedback from the creation of the container something that you can hide of course it's just for a demo below this part which is running the Rebinder container there's another frame and when the Rebinder container finishes I'm going to start spoofing responses so I'll be able to load the bottom frame so it's loaded and now I'm actually inside the post-rebinding attack phase so this will take like I said about a minute and as long as the attack is going on you can actually go back to the Docker DMN and see it going on so if I do I can see the actual Rebinder container running and the images that it pulled and now of course I'll wait for the time to expire we can see the 200 response so we know the time has expired and what we have to do now is run the privileged payload voluntarily so we create a privileged payload we're going to of course start the privileged payload and finally we're going to delete whatever images we downloaded to begin with so once I'm done let's look again at the images of containers so after the attack is done we're back to where we started we have two images and two containers on the host and of course it won't be any surprise that once we look inside the virtual machine we'll find the shadow file good so just again to make sure I'm not fooling you even though this is video we're going to restart the Docker DMN and repeat the whole thing so the Docker DMN is up we're going to do the same stick so we still have the shadow file of course we can still see our attack script and the shadow script inside the virtual machine and of course once I look at the images and containers it won't be any surprise you're back where we started two images and two containers so what's the impact what did we see so we actually see a form of an advanced system threat against developers we use shadow containers for persistency all of the bad stuff that we are doing is concealed inside virtual machine it has a very low forensic footprint because the part of the attack that gets locked on the host is just the initial part when you run the reminder container that will be locked on the host all of the other stuff that you're doing a lot inside the virtual machine and you can delete them because you have full access to the virtual machine and of course we have access to internal network of the enterprise do reconnaissance takeover machines and so forth another danger such attacks like touched on at the beginning is the shadow warp if it's a developer and it's writing images that are used finally in production you can infect the layer you can put some other religious code there and if you do it good enough to propagate throughout the pipeline it won't be discovered what you have to know is that this attack even as we've seen it over Windows 10 you can do it different variations over Mac, Linux and Windows in Mac and Linux of course you won't be able to do the host rebinding attack so you'll have to do DNS rebinding but then if you're over Mac you can still run a shadow container if you have a Linux you don't need a shadow container because you have full access to the host in Windows it's very similar to Linux if you attack a Windows container and you run a print container you have full access to the underlying host so what are the conclusions don't expose your container engine API if you do have to expose it make sure you authenticate to it so not anyone can go ahead and run anything that it wants don't forget to analyze your container engine logs they may tell you when you're being attacked and if you're not using NETOS or LLMNR or DNS just settle them over the host make sure they don't prove in your network and finally you never know how you're going to be attacked so always scan your registry scanning your images and always monitor your containers in runtime so that was me thank you guys very much if you have any questions we can do this online right now or you can ping me for my handle and we'll talk