 I know I'm standing in between whatever Wednesday night beer is or preference dinner. Actually, this would be a very short talk, but moreover, I would like to hear from you. So my name is Arseni. I do not work for Dell, by the way. I do not work for Dell. I'm a solution architect and I've been working on some monetary authority of Singapore technology risk guidelines before like many years ago. They were related to VMware and I co-authored a comparison to help the it is publicly available to help the industry basically leverage the best practices and map. If there is a regulator requirement, how to answer it? If there is a security team within a big financial organization giving you a hard time, how to answer their questions? So I basically want to do the same for containers for dockers and I basically share the digest of open materials that I found around using Google search engine, but I really want to hear from you because I believe there's a very very different crowd today and I really want to hear what you do. What do you do for security in your organizations to start with? I'm not a security expert. So don't worry. It's fine. Not a security expert never had never been it and I don't want to boil the option the ocean is really not something that I believe we'll be able to do in 20 minutes. So forget about it. So two questions. Who has in their production environment today any test driven security pipelines? So anyone does test driven security approaches today? Cool. Who has okay before even going to SAT this acronym from financial markets who works for insurance or for some regulated enterprise that runs docker in production? One, two, okay. Got it. That's good. That's good for me because then I can not screw up that significantly because everything I do is just assuming my experience and whatnot. So I want to just start with a problem statement. Remember the good old days we had very clear demarcation line in between where the developers are and where the operations people are. So the dependencies in here, this is where the gray line is starting to appear with with containerization. So who actually keeps keeps an eye on all the dependencies for containers. So within the container, you may have a particular version of Python within a container running on the same host. You may have like a plethora of other dependencies. So who has to work on that? So in ideal containerized world, it looks like that. You have an application and it has all the prerequisites, all the all the dependencies, installing it and everything is beautiful. But then all of a sudden your guys call you and they say, Hey, app C, we forgot. We actually have a better release. We just migrated it to Java. We want to migrate it to Java. Can you just make sure that the image we are building on the production security image is right now running like on Java, not no longer need all these dependencies. We need it like in five minutes. Well, it's a DevOps world. All you do is what you do, what you do is you install the needed prerequisites, update the image of the container, send it back to the rebuilt pipeline. Everything works fine. Nothing, nothing breaks. But then they wake up and they say like, Oh, by the way, we need a Java eight. We forgot it actually is like optimized for Java eight. Can you do the same but for Java eight and you do that and in the end the risk is with all these images you end up with an enormous amount of configuration drift. So you may have like ports open and something listening on those boards and maybe not even patched to the proper proper levels. And this is how configuration drift could look like for a particular image. It's on quite at IO and they're like engine X image that has been open and downloaded like thousand times. They sent it and it's been used in multiple projects as you can imagine across across the universe across the Internet and there were like 264 medium level vulnerabilities, 126 high level vulnerabilities is just because everyone just patched it a little bit, patched it a little bit there, patched it there, updated the image and the latest became this huge drift from like where it should have been right. So and actually this is very interesting slide that I found. So look at this information security job postings. They kind of kind of tend to go down and DevOps jobs. They kind of tend to go up. So the problem I guess will be a signified in in future. So if we look at the DevOps factory, I'm trying. I'll try to put it in here. This is the usual process. It's a mixed like logical and process state flow diagram of interchange that leads from development of a particular business requirement from business partners and business analysts and all the way into production that is on the operate side and the green side. Do you see anything on this diagram related to security at all? This is if this is docker con this is 2017. It's actually from the keynote of docker con 2017 from MetLife from their presentation to like couple of months ago. Nothing here says about security. It does mention like, okay, we have a CICD tool and magically we do push pull tag and image scanning. No details, right? So I guess this is the area where we need to at least like put a bookmark to ourselves like, well, we need to research it because like not too much of material or best practices are currently in the easy digestible form. How to do that stuff? Yep. So if you look at they're using docker trusted, right? You see if you go to the marketing slides of docker trusted. Oh, yeah. Oh, yeah. I actually included one marketing slide later on, but it again, it doesn't actually give you all the best practices how to how to actually do stuff. So the new reality is in the in the they then there's a new term def sec of develop developers security operations, right? So the new reality is we have a completely decentralized ownership of deployment. So we need to get as early concerned about security as we can and we should narrow it down to like an easy digestible tasks all the way through the pipeline and make sure that there is no configuration drift in the image, for example, and make sure that the injection points are designed together with developers so they understand the concern so they don't come and rush it to you and ask you to give them Java 8 and then provoke this configuration drift that can hence give you a lot of vulnerabilities. So at the one on one level how secure is docker and these are the kind of conversation that we may have with security guys just over coffee in a big firm or like in a Starbucks if you see someone so I went into CVE details and opened up what are the current open docker very vulnerability. So for a vendor called docker they're currently 14 vulnerabilities some still pending from 2016 not yet patched it's not too bad actually. So 14 CVEs just to give you an anchor point what is Kubernetes? Kubernetes actually has three so there are three vulnerabilities currently known to the global to the global population of our planet that are related to Kubernetes and to give you another anchor point our favorite product ESXi from the very old days currently have 31 vulnerabilities so answer is it's quite secure by just by nature so docker and Kubernetes are well built they are not really vulnerable as they are how about running in the cloud so there is a report and again I'll share all this materials and digest of the links with you when you run when you run deployment template for cloud formation that docker provides with docker cloud there are five total issues and like they're two of them are low risk basically they are over provisioning cloud formation AIM roles just with a little bit too much of permissions that needed so that's not that I mean I would say brilliant actually so same for Azure they there was an independent research that gone through recently this research was published there are only three issues that are related to rolling out docker on top of Azure very minor issues again related to more of like preference rather than a problem now what has to happen we used to run apps and again like I apologize it's just a digest of multiple different slides right we used to run apps through the development into the user acceptance testing and then we would have sent them to security acceptance test which are mandatory part of the DevOps in the old legacy world so in the bank for example security acceptance team would take like couple of days and they will call you and they say dude this is not working we're not passing the scans it used to be like a waterfall approach now they looked for everything so security acceptance team would have scrutinized all the code that you push into the production they would have gone and checked every particular source code as well crawled it for customer data crawled it for some stored credentials whatnot they were acceptable when we did not have to do releases five times an hour so it's not really what we can do these days with with with DevOps so the new analysis for vulnerabilities the new analysis for security has to be a bottoms up one so we need to identify specific classes per application like what are the most likely vulnerable areas in this app or in this microservice or in this microservice and we need to only care about just those and we need to push it as as heavily into elimination of of this of this problems early before they actually hit even any particular last minute security test we need to push for that in the development pipeline as much as we can and that would actually support the only way to support this would be the only way to support the velocity of security now just again like some tidbits there are really good security best practices published by Docker themselves so they they talk about the usual things usual things what what I'm saying usual things are the things that regulator the things that your security officer would actually care about so like centralized logging yes it's it's covered in reference architecture yes please do something with cloud watch if you do a lot of cloud deployment in AWS please use the cloud trail for the API traces as well in GCP everything is centralized is quite quite different with Stackdriver do the signing of the content if you use the repository there is the service within Docker that's called notary it allows you to create the hashed kind of like snapshots of the code of a particular revision so that there is no tampering there is no man in the middle substitution so all these best practices are covered there's a special very nice document out there now there is also for the host itself that runs Docker and that is something that could be incorporated into the into the security practices within your firm there is an audit tool that kind of comes as a part of the intrusion prevention scanner so you run a small container on the worker node itself and as you can see it maps into the var runs and into etc and polls the usual Linux sockets and figures out are they configured the hosts underneath running the containers the usual host that you have are they configured as per the best practices against the known latest vulnerabilities so it's again like this is part of Docker there is an audit tool which is pretty cool you run it as as one command line and it produces you a report like like this is your Docker current is the Docker audit files are in place are you blogging into right folders do you do you do like the proper logging level is it like only like exceptional or everything that you do etc etc etc now there's also a separate and again like I seek your input maybe you can chime in with more more tools around it there's a separate project that could do pretty much the same thing but both on-prem and in the cloud and it's called Claire it's developed by CoreOS team what it does it also pulls the vulnerability updates into a small data store and then it also runs a per per layer analysis of your Docker image so if you actually run it on your say version and minus two once it takes like an hour but since you've done it on the layer there was n minus two on the n minus one it would only scan for the incremental changes on that layer so you don't have to redo the entire Docker scan so you actually significantly reduce the time for the vulnerabilities scan unless there is something not changed so take a look at this Claire tool and it's incorporated in several several cloud providers security artifacts this is a pain point but I mean we probably heard about ransomware for MongoDB that recently hit the entire planet people did not at all set passwords but like another problem is when you set passwords you hard code them so the good practice the best practice is to avoid hard coding and to pass the to pass the credential past the security artifacts into the running application through environmental variables and you do that for example by sitting at dash E variable that is then available for your app to pull so you literally pass something into the container from the outer world when you run it and then you take it and use it I mean this is very simple best practice but for some reason people especially develop developing guys they tend to forget when the push when the push apps and there are so many bitcoins that are paid to ransomware guys because someone then commits the code to public github and it's all there and actually I've when I've been looking at a lot of resources there are like some attack vector analysis it takes like 15 minutes to literally like completely blog especially through Jenkins and I have a slide on that so it's it's insane Jenkins allows you to crack it through the console if it's not really password protected well but anyhow so yes please pass them through environmental variables you could also if you're using AWS can easy to container service you can pass them in the cloud formation template you can define them and use the cloud formation script to retrieve it from from some secure vault or from some other from some other source into the cloud formation so even your cloud formation template that then builds a lot of Docker containers on AWS will not have it hard coded you just script around getting it from some third from from some other location and yeah you could you could retrieve it from the metadata of the instance if you're running on EC2 or Google cloud there is an instance you can just curl into the instance metadata from within the app at the bootstrap and then retrieve the tokens that you need to access other services now another tidbit the previous speaker actually headed on one of the slides there is a very interesting startup called Sillium and Sillium does an additional layer of like tapping into the networks at the packet level they insert themselves before the virtualized adapters of Docker and they basically add themselves as a kernel almost like a kernel module and they allow packet filtering to the container plane running on the worker node defined in the Berkeley packet filter format and Berkeley the BPF system is already adopted by Docker for example they allow and deny particular syscoles so from within the Docker container by default there are some syscoles that you cannot run at all they're like 44 of them banned for example you cannot run like mount or you cannot run the process trace you cannot reboot the worker nodes from within the container so they already use this BPF framework in Docker itself but then there is this Sillium company and this is their this proxy architecture that allows you to basically do the packet filtering and even at L7 on the path level do very complex rules to kill the packets to deny access or to deny to deny traffic to a particular application or not so it's running as an agent on a Docker node there is also they have a project that they insert themselves into Kubernetes Kubelet and they have the monitoring they have the CLI special hooks that configure this almost like an IP tables routing sorry IP tables filtering on the packet level and again like it's very low latency it inserts itself as a kernel at the kernel level so very very interesting quite sophisticated startup there's a very good session about it about an hour long so now yet the marketing slice from docker Vincent docker docker promotes and again everyone works for anyone from docker here anyone works there's no one from Dell no one from docker the guy from puppet left okay anyhow no vendors so yeah there is this is the marketing slide that we discussed so it looks like everything is running out of the box again I cannot I don't I'm not a user of enterprise edition of docker but they really made a significant effort to add aspects like role-based access control trusted registry trusted registries your own site completely secured key managed source for the container images that again avoids men in the middle attack substituting your apps code so there is a lot of effort put in place to just run compliance requirements out of the box with docker enterprise edition are back so you can basically label a particular container or label a particular control group and allow some particular users that are LDAP authenticated not just the local user that runs docker but a particular LDAP user to do some actions against control groups of containers or host themselves so it's very definable in enterprise edition of docker nothing like that actually in kubernetes you can do similar things somewhat in in the cloud but again not as granular now the best practices of hardening linux containers reducing attack surfaces very simple right remember the the configuration drift story that we started with like lots of ports that who knows what they're doing there attempt to isolate not only as on the development staging or like user acceptance and production but also on the risk so try to containerize this is the best practices if anyone worked in the bank their business criticalities so you have to go like busy bc-5 the top most one like do or die go fix it or bc-1 like okay let's leave with it if it's down for one week no one would care about that app so like do this on the risk do this on this exposure and then try to scrutinize the security one by one so create like a structured tree of decisions what is my most business critical what are the exposures what are the possible attack vectors apply and enable all security relevant configuration options yes if there is if you already pay for docker enterprise edition or if you already have something enabled by default when you install it do not do not temper with that keep everything up to date like the gentleman from from puppet regularly test security recommendations one to four so everything you do put an outlook calendar and this is what it's absolutely normal to do put an outlook calendar reminder once in two months just go and just sanity check did we change anything did we miss anything just like a reminder and remember complexity breeds insecurity the more complex the more sophisticated set of scripts are the more the more the probability that there will be something hard coded or something that your dev team will inherit from you and it will breed the breed the attack vector exposure now two very cool talks that I've seen one was on enigma and it's been from a guy who works from mazilla and they push as I don't actually use firefox to be honest but they they push a lot of releases like if you look at them at their release at the release pipeline it's pretty much a weekly stable or something like that insane and of course they use the pipeline so what they what they do is they define the baseline for their security so what should be like our central focal point for every release and of course if they do add-ons that's much more effort to control add-ons to firefox because it's a community thing right so you submit developer submits an add-on you have to still like go through the same pipeline because it's part of the dependency release for the core browser so the right tests and they insert them into their continuous integration continuous delivery pipeline so the security baseline for them for for mazilla browser is something like that so they define scores like what would be the build of score for Jenkins to use and like they need to pass some may fail some have to pass they define the baseline their right test using zap and that's called Z attack here we go it's called Z attack oh actually I deciphered it I forgot sorry I think it's Z attack proxy yeah so basically they define them in this in a plethora of tool but one of their focal point is that and zap has an integration plugin for Jenkins it's been released not not too far ago and it's already been yeah it's quite aggressively so there were only 15 stalls generally when they really they actually abandoned it so the they had like a separate plugin for jenkins and then they moved to the new completely new project on Jenkins plugins so it's now currently being used about like 400 installs in the world which is not bad for security the idea behind it is that you are able to define a zap scan I do actually I'm not sure if I have a slide on it yeah I don't have a slide in it but the idea is here like take a look at this one you basically you can define a site you can define what type of like is it going to be like all headers scan like well let's let's try to exploit all possible web server like vulnerabilities out there let's let's crunch it with a lot of header header related like blobs for example and see whether it crashes or not some client sessions you can test your particular app from different paths whatnot it basically orchestrates your HTTP or HTTPS vulnerability scans and it can simulate a total attack on on conventional web servers and after that they just hook it into the pipeline of Jenkins and as you see the scan takes about like 30 seconds so it's not a really long scan if you run it on a beefed up container and it's a container itself it only takes like half a minute in the entire in the entire build process there is also a very big set of different libraries that you can also tie into the Jenkins testing they would test different servers for example Node.js like Rails I guess I've seen here retire GS and like many others so basically all these projects are there for you to test the app on your preferred ID also on your preferred framework tested against the known current vulnerabilities and there's a full list of it so if you Google for awesome DevSecOps you will be able to find this list of testing testing scripts and okay so this is what we've discussed so once you created the criticality the risk of your applications start with the thread modeling so like is data going to be in the wrong hands I've got five minutes I know so is data going to be in the wrong hands or is going to be an accidental compromise or brute force attack and then based on this logic expanded like whether it's going to be a human failure so someone for example lost their laptop and actually losing a laptop has a special attention in here remember these magic tilde dots right do not lose laptops or use the Dell or whatever the encryption of the SSDs right so please like if you lose the laptop there could be a lot of things in your tilde so that's very dangerous especially if you're using AWS now where was it with it with the tree right and then yet focus on code so as early as possible ensure that not MD5 but like SHA 256 is used if you use hashing or something like that ensure that the code repositories are private not public especially if you hard-code the security artifacts into it make sure that the orchestration involves all the testing that we've discussed there are many other cool things that I want to share this is actually what I tried today so in show done it's like as per like 3pm we're gonna actually do that I think if I have yeah anyhow so it was like 3pm today on show done I looked at X Hudson response in the HTTP and basically is the project name for Jenkins and I came up with at least like five Jenkins that are completely unprotected with passwords like totally so and I logged into one of them and it runs on AWS and the guys actually run it in production like there was like Arun Kumar Deliva Kumar I looked him up on LinkedIn he's actually working for Tata Consultancy and he was like I'm like I never actually thought yeah so I never I never thought that it would be that easy but yeah please make sure that all these aspects are like reviewed at least twice a year or as I proposed like every two months return to return to them and sanity check what's the security exposure and yeah so employees guess who loves guess love guess yeah cool make sure when you when you publish the guess make sure it's at least not Google crawlable okay so like so make it the problem of the crawler of the attacker not at least as easy as in Google right and yeah Slack tokens are there AWS creds as we discussed now yeah so and yeah the one one good advice that I picked up for myself is like Jenkins has to be segmented into if you run it on AWS it has to be in a like network ACL and a particular separate security group it has to be segmented to only allow access from the known set of hosts like this is really for me the core takeaway from like what I've seen today with this Jay Kumar example when I just logged into his Jenkins right and then everything should be all across authenticated please please please so yeah yeah that's it so I again like I will probably pile it up in terms of like the links and everything the short link is bitly slash I will update it with with the with the latest format bitly SG Docker security and there is about like I spent about like 15 or 20 hours just watching all this videos and going through this PDFs it's a lot of cool material reach out to me on LinkedIn or ping me on Twitter if you guys have any ideas about security I'll be happy to learn from you that's a big that's a big area we're not experts so let's move on together so just learn from each other yes I think in the death cycle speed Singapore no no at all so there is a death cycle screw in Singapore led by like at least like people and they also did a death cycle conference back in like February or so so it's very interesting to be there on meetup oh yeah I would I would definitely look at them I'm in the wrong room it's just something that I I appreciate it but it's just something that I needed for some like I needed food for thoughts and this is the way like I just structure my thinking right cool guys any questions yes please I would try the other one with the source here so I mentioned once applies that you want your software the vulnerabilities as it like vulnerabilities the vulnerabilities.com wow the website looks brilliant oh nice landing page database okay when you're saying you want to have a security base as a part of your CI security like as a part of your company then what you could do potentially would you could actually scan through they have an API so what you would do what you could do is you scan through them all packages which constitute like which participate in the build process before you actually build it but you mentioned once applies that like have an auto-commander modification which doesn't scale so what instead what you could do is you just scan packages for non-volna unit as part of the build process because it's sure of an API and it's really easy I actually just to correct myself when I was saying about once in two months I was saying reviewing the process like review the process of how we do security for the build not really of course not really checking it once in two months it has to be every build Vulnerable test targets Vuln Hub yeah it's not in here definitely thanks for that take a look at it and the second one you propose is source clear right source clear yeah thanks for these advices alright yeah and I'll suggest an ad to these dudes as well cool any other questions yes please excuse me what's the name of the project help me spill it out B-U-C-K B-U black as in black black not black not black dark there's a genkis flagging for it so we'll have to make a review like dark software alright if you don't mind asking what sort of applications what sort of language do you use it with most of the applications we have are Java in Java alright cool yeah thanks for that that's fun I love it look at that nice website alright thanks for feedback in terms of like container security there's twist lock and aqua security twist lock twist lock yeah twist lock excellent aqua security I think like twist lock is from Israel and aqua security I'm not sure are these companies more likely to be start-ups right so they're not really long term in the industry twist lock and aqua okay wow look at that all these websites are written so landing page you can buy them on theme forest cool thanks for that any other questions or suggestions for the list that we've covered today thank you thank you guys