 Hi, everyone. Welcome to the last talk today, six o'clock. We have dynamically provisioning app secrets during container runtime, or speakers in Renshake. This is the last talk today for the security track, and also a blurb for scale. This is the first time we've had a shared boost between a number of communities, including OWAS and ISSA. If you find some time to visit them on the Expo floor, you know, they're here this year. Thank you. Hello, folks. My name is Imran Sheikh. I am the lead systems engineer at YP. And today I'm going to be talking about dynamically provisioning application secrets during the container runtime. So before we start, I would like to basically bring up some assumptions, like I assume that people will actually know about some of this stuff that I'm going to talk about. First of all, you should know or at least like know about Docker and its ecosystem, maybe like have looked into it or are exporting into it. Or also know about the tools like Mesoos and Kubernetes or Docker, Swarm, etc. to basically to run your container workloads like in production. Probably are looking, if you're not actually running Docker workload, maybe like you are thinking of basically changing your track or like looking into container running your apps in the container workload. And if you are, you are probably aware of some of the limitations like that you are basically faced with. And again, the Docker and its ecosystem is basically evolving by leaps and bounds. So some of the stuff that you might find in the slides might be updated like by a few weeks or about a month. So if that is, and if there's like some new updates in the community, please let me know. I'm going to update my slides or how you can feel. I'm going to mention about the blog as well. So you can actually very well document like a poster comment there. I'm going to update my slides going forward. Okay, so these are some of the assumptions. Now let's see what I actually typically do is that when I actually give some demo or like some talk like in our internal company forum, I love to tell a story. So even today, like I'm going to tell a story, something about like human evolution in the field of computerized world. So let's say like you are this guy, you're coming from the world of VM, you're coming from the world of static partitioning, and you are running bunch of servers for your app. So you are made to convince that you need hundreds of servers to run this particular product. And vendors have sold you beefed up hardware with unnecessary functionality. And you haven't utilized some of the features until the product becomes like the hardware becomes like EOS or EOL, but you still have it. So another thing was that you were sanctioned a budget to use this kind of money if you don't use it or you lose it. So you basically are going through that phase, but I know that you are a nice person. So you don't put fill inside. You say you feel that I mean, I mean, I'm getting paid, but then I should basically also try to save something like same same money for my company as well. Right. So that's why you are basically looking into containerized workload. Like how do you basically run that workload in production? So let's start. So why are we here? One of the main things that you basically are encountered when you are running your container is that how do you deal with application security? Secrets are very important. It's very important to basically secure them in the internalized workload, like in the Docker world. Some of the secrets that your apps or anything that you might use are database credentials, API tokens, SSS keys, GPG keys, certificates or what not. Right. So you need to basically secure them before running them into production. So what do people do? So they actually people who are just like coming into the Docker bandwagon or just like starting using Docker, they actually make the secrets in the image. So that's basically part of your source code. Everything is in there and you feel that somehow if you basically do a change mode 400 own it by root and everything is fine. I think it is the most prevalent anti pattern in security. So what are the issues? I mean, I shouldn't tell what are the issues, but I'm still going to go over it. So when you image, when you publish your image to any registry, anybody can pull down an image with the internal or external. Anybody can basically have the image and can basically have the secrets at the disposal. So none of the GitHub or Docker hub or internal Docker hub or what not is designed to securely store your credentials. Updating a secret is a tedious job. So basically if you basically have a bunch of applications or a bunch of images, you basically update each and every image. If you have a new secret, if it has been compromised, you basically face the task. And consider if you basically have to do that and then if you are tying your image will process using some CI CD pipeline, then you basically go through the entire, I mean, I don't know how you can do that. So you publish this many images for this many versions of the code and it's basically up there. So, and again, like if you have a certificate that expired and how are you going to update all your images, which is running in production, you are pretty much stagnant. And if you basically decommission your machines, and if you have the images there, it can very well leak the secret. Again, what I feel is that updating a secret shouldn't be tied to an image or a code. Rotating shouldn't cause system changes. You should be free to basically change secrets without having to change anything at all. So this is the first challenge that you are basically going to face when you are going to run apps and containers. So now that you have basically jumped into the bandwagon, Docker world, you feel that, okay, now I'm making some progress. Okay, even though I'm not there yet, but actually I'm making some progress. I'm actually, I'm evolving somewhat. So what are the other things you can do, basically, to how can you, how can you stick this in the container? The second thing is that you basically pass it through some environment variables using Docker run, that's E or whatnot. People use it and I'm sorry to say that like more than 90% of the people, those who are running the Dockerized workload, they are basically using this method. I will tell you why. And why do they use that using environment variables because they're part of a 12-hectare app? And they say, as per the guidelines, they say each and every software should be consumed and deployed as a service. And if you're basically exposing the environment variables, you can do that. That's like, basically, people are following that model. And how you do that, simply, I told you like the Docker run and then you have some key value pair and then you basically run your image. So what are the issues? So basically, people like on Docker, they basically captured it very well. I'm just summarizing it over here. So basically, if you use image in the environment variables, if you might be, if you are aware how Docker basically builds an image, it basically creates multiple layers for each and every directive that you see in the Docker file. So when you do a Docker inspect, I can very well see what part of your final image or intermediate image is there. If I have an access to the host, I can see what images or what security you are using. And if you are not running microservice architecture, if you have a bunch of processes running in the container, then all the processes can basically are, can very well read that security because it's a part of environment. And if you are basically are mounting some other link, if you're linking containers, so even those containers basically have access to that. It's very common. We have seen it everywhere that the app got the whole environment, print it out and send it as a part of the page or the email. You basically have all the things that your environment was basically at the particular end time. And consider this thing. If you basically call some third party tools from inside the container, even now they basically have access to that. If you're working any process from your container. Very common for the app store that crashes to store variables into the log file for debugging. I need not extort. So basically now you say, okay, now I'm making some advances. It's good that I'm basically evolving. I'm not devolving back to like my VMDS. At least I'm making some progress. I'm seeing some promise. So what else is out there? There are volume mounts. This is again like very straightforward as a variable. You put your secret in some directory structure on your Docker host and then you actually mount that volume inside the container. That volume can be on your NFS or SAF or your local file system. Doesn't matter. Right. And then you basically do something like Docker run. That's be your mounting something, something like your MNT app secrets. And then your secrets are available in the particular directory inside your image. Guys, I might be rushing. If any time you feel that I'm rushing too fast, if you want some clarity, I can, I can slow down. Okay. All right. So this is basically like the third option, basically baking secrets and then variables. And this is basically like you use something like a volume mount. Right. So what are the issues with this approach? It's a bad, bad design to put all the secrets for all the images on a single machine. And then you basically mount that inside your container. It's a bad design, right? You don't want if the machine is basically compromised, basically all the secrets are basically exposed. Secrets are unencrypted. As in like you're doing the volume mount, everything is very mounted inside the container. So that is unencrypted. So what do people do now? I mean, they, it's okay now. I mean, even like the volume thing is not that great enough. So what, what, what else is out there? I mean, how can I be, how can I convince our bosses that Docker is good and I want to run container as workload? And then how would he convince me to basically make it easy for me so that like I abide by the, all the socks compliance and all of that. So you, so people use something like secret encryption. So, so because, as I said, like some people are paranoid about keeping the secrets in the plain text. So what they do is, and they're even more paranoid to publish that same image to the registry. So what do they do? So they encrypt the secrets using a public key and elliptic curve of cryptography using tools like eJSON from Shopify and others. I'll tell you how it works. The thing is that they encrypt something, their image and then they publish as a part of their image. And it's there in the registry. And you need a private key to basically use that to decrypt that encrypted key while you actually running your workload in the Docker container. Again, to decrypt private keys are hosted on the Docker host in the production. In this case, everything is locked down. You don't want anybody to basically have access to this machine and can read all the private keys, right? So you want your machines to be locked down. At least this way it's good enough that your image is safe from snooping. You feel free to basically put everything as a part of your image, push it to the registry and let people basically get it and they're not able to use it because they don't have the private keys to decrypt the keys that are there. So how does it work? I'm just going to explain very briefly how it works. So this is a tool called eJSON. What do you do here? You do eJSON key gen, similar to SSH key gen. It basically generates a key, a private and a public key pair. And then you want a file. Here is a password. And then you basically put this public key over here. And then you say eJSON encrypt this thing. Encrypt this file for me. So what you get in the end is something like this. This is the encrypted file, right? And when you're actually running the workload now, once you basically have pulled down the image, you do something like decrypt this file because now you have the private key. So now you basically have this in plain text. Simple, right? So coming back to our like, so what are the issues with this approach? So basically to update, to secrete, you need to create new images. Now everything is part of the image. So if you have a new thing, you need to create a new image. The solution is static. It's not dynamic. As I said, like if you basically have some build process going on, I mean, how would you basically update things dynamically in your images? Again, if you have access to the machine, you can still do Docker and then see which things are basically mounted inside your container. So you can see all of that. This is good. At least like, I mean, I'm able to put my image in the public domain with no worries, but then you have all these issues. So coming back, again, then the private keys has to be hosted on the machine, whether you are mounting it somehow or keeping on the local file system similar to the other approaches. So basically now say, okay, now you feel that, okay, now you're making progress. You are basically digging more into it. You're going to make my environment secure. And you feel that now, okay, man, I'm here. How can I be a perfect human now? So what do you do? You use something like a secret store. So there are secrets management and services like from HashiCorp. There's something like Walt from the square guys. There's something like KeyWiz. And there's a sneaker for the AWS. Basically, it basically helps you generate and distribute secrets for services. It's a store, basically, like, I mean. And your secrets are centrally managed. Right? There is auditability with the secret access, who is basically accessing my secret at what time, which contain us basically using all of that. So you can play down all those details, like on the log file. It's an API base. And it's mostly secure. Right? So the way it kind of works is that something like, let's say you have this service called Web. And then on the client, basically use some TLS, like the transfer layer security protocol with the server. They basically authenticate each other with the certificate. And on the server side, what the server does is that basically groups the secret into some particular group. For example, I want dbsecrate into some particular group. And then I map those groups to some machines. And those machines will now have access to all the secrets now. So that's more or less how all kinds of secret stores, they actually kind of work. Walt, KeyWiz, sneaker and other stuff that is out there. So that's how they basically use it. Now, coming back to our chat, you said, now man, man, like now everything is dynamic. I can project it dynamically. And I'm a perfect human. But no. You have gotten yourself into like something like really awkward situation. So I will tell you what it is. Okay, so far, like what I've talked about is like only the runtime secrets. I haven't touched about like the build time secrets. Why are they important? For example, doing your Docker build process, right? You need to hit some private report to pull some dependencies. For example, you're doing some app get or yum update. And then your packages are basically on some repositories. You need some keys to access them, right? To basically build your Docker image. So how are you going to supply that in the Docker file? Again, you need to have maybe like you need to some private keys to SSH somewhere, there should be some things, some packages, and put as a part of your Docker image. And for example, like if you're building your image, you are behind a proxy, for example. Then the run environment, then the deploy environment. Maybe like you need to provide that dynamically. Okay, for I want to build this image and here is my proxy and you can go through that. And it's different from a different environment, for example. So we have like tons of other use cases like that. So how do you deal with that? Again, there isn't a single solution from Docker or from anybody else that basically addresses all these concerns. And similar to the runtime secret as well, there isn't a single solution from anybody. But we'll go in detail about that. What is out there the best alternative? So for some, since they're like not a single solution, what have people done? So for some of the use cases that I mentioned over here, there are some new features available to Docker 1.9, finally. Like they just did like a month or two ago. So you can leverage some of those features to basically provide this build time variables during the build process. And there are some supporting solutions for like basically selling those solutions to the people. And there are some custom hacks. People like us or me, I mean, our team, we cannot basically wait for Docker to basically come out with all these features. So we did our custom hacks. So what is the first thing that you can do during the build process? Like to provide the secret. There is something called like an ENV variable. In the Docker file, you have all the directives. So something there's also a directive called ENV. You can put that as a part of that. And then you can use it similar to like making your Docker file pretty. I don't think there's in particular usefulness with this. There are tons of issues if you basically use this. As I said, all these variables are basically preserved in the intermediate as well as the final layers of the image. When you go to any machine, just do Docker images tree. You will see how many images that particular image has. Like the container basically is built from. And you can very well run like any one of that. And then you basically have all these things available inside your image as a part of the environment. It's not secure. Again, the secure will be exposed if anybody pulls down the image. So Docker file is kind of static with this approach. You cannot override a variable if you're basically behind the proxy. I mean, it's basically part of the image, part of your Docker file, right? And also be mindful that your secrets are basically written to the disk at every intermediate and final layer of the image. So your disk basically has that. Even though basically you use all other issues, it's getting written onto your disk. So finally, Docker came out with, to address the issue, they came out with this build dark variable in the latest build, it's 1.9. So when you're doing a build Docker build, you can provide something like a build arg, and then you can pass a bunch of arguments in there. So this flag basically lets you use the variable with the other director, like run, copy, add in the Docker file. Something like this. So you do Docker build, build arg, and I'm behind this proxy. I want to build my image. So basically go through this, right? So what are the issues? It shouldn't be used when the caching is turned on. And again, like the benefit of this approach is that these variables are basically removed when the final image is getting compiled. So you will not see the variables. So they actually solve the issues when the image was persistent across the intermediate and the final layer. But if you use build arg, it won't be available there. So I think you are safe. But then if the caching is turned on, it's an issue. Docker caching. And again, it basically has the same issue with the environment variables that we talked about in the runtime section, right? Anybody can, you can call third party thing and then they can have access to all of that. So people are basically sick of this. I mean Docker has made our life easier, but in a way, they made it worse as well. So since they are creating this multiple layer for a single image and the values are being persistent across the intermediate layer, so people say, okay, let me just flatten the image. Right? So that way your image size will be drastically reduced. So there are tools like that called Docker Squash. Or if you don't want to use Docker Squash, you can do it like something like you can run an image, you can export and then import. So there won't be any intermediate layer. So there won't be any fear of basically variables residing in the intermediate layer. But the issue with that is that if you do this thing, then you have to add this flattening, hacking logic with your build and deployment process. It's an additional step. I don't think people are basically and developers are actually kind of like that way. They have to flatten everything and whatnot. It will remove the figures but it may still end up in the build cache. And again, basically the same issue. It's returned to the disk at every intermediate and flattened final layer of the image. Right? So that's like one of the other options. So what is the other option? How do you provide SSS keys? So if you look something, here is one sample Docker file. On my left hand side you see like you actually run some Docker container. You basically mount some SSS keys and then you basically expose those keys as part of a service. So in your Docker file when you see something like run on vault, something like that. So at that time it's going to copy those keys from that, run the next command and at the end it's going to remove those keys at the end of the command. So it won't be available in that particular layer. So that's how you provide SSS keys during the build process. So that's like that's what people this is like a custom hack. And if you go along that path so there's also actually let me say that this product is from vault from docketo. So there's something similar as well. If you docker SSS exact, I think that's more better than this. It basically puts those like keys into the memory. So it's never written to the disk. So feel free to check that one out as well. Right? It's called docker SSS exact. So that's like few alternatives. Right? Again, there isn't a single solution from anybody. So there are lots of PRs and lots of full requests submitted in docker. How do you successfully pass the secret container? And this is a big bottleneck for me to run my stuff into the production. People basically recommend it like bunch of solutions unified solution for each and everything that I mentioned in the runtime, build time and whatnot. So these are feel free to go check them out. So these are all the existing full request but nothing has been finalized and still up there. So people have gone their own way they hacked around and they made life easier for them. Let's see how we are doing on time. Okay. So that's that. So until now we only talked about docker. Right? So basically you have an app you do a docker run and then your app is container created into a particular host. Right? Your container is basically running. And what I would like to mention is that you haven't solved this issue about the secrets on this level. How do you successfully do all of these things that we just talked about? Now here comes like another abstraction. For example let's say that you have an app that you want to run for copies of the particular app. And for each instance I want to run with one CPU and one GB. So you would use something like all these tools like Mesos, Kubernetes Dockerform Dockercompose. So you say this is my app now go run four instances of that. Right? And all these tools, Mesos or Kubernetes or what not is going to create containers depending upon your request. Right? So basically on all these many container docker hosts you will do a docker run depending upon how many instances you require you requested. Going to create these many containers So now tell me you haven't solved the issue about dockers or the secrets providing to the docker. How are you going to supply secrets with this approach? With this abstract approach of like using something on top of docker to basically run your containerized workload. Right? So you are basically now saying I am doomed. Like what do I do? So let's look at our previous diagram like where, how we evolved and where, where we and then now suddenly you learned about Mesos and Kubernetes and everything I was like man this is awesome. Let me just use it. But if you are thinking in terms of secrets it's a rod block. What do you do? You will have a tendency at that time man screw that. This is like too much for me. Let me just dewall back to my old good old days. Let me just go run my stuff on and let me just make my life easier. So what have you done? Again the same model. You basically have an app. You want to run four copies of that. I am giving you an example of Mesos because we are a big Meso shop I am from IP anyway. I forgot to mention that. So we are a big Meso shop. So we are using Mesos for deploying our container. So before doing so basically I submit a request to the Mesos that I am doing this copy before this calling my calling a docker run. So we put a layer in between something called like a docker wrapper because we cannot indefinitely wait for docker to come out with this feature. So we basically added so what is the best way to do that is to build something on top of docker. So the way we actually fool it is something like the Mesos engine calls docker. But we just change the path to call a docker wrapper. So docker wrapper is a layer in between. It does some manipulation and then for example this is your workload right? You say okay I want to use this image and I want to run four copies of that and here is my command to run. So Mesos basically will provide you this like the docker wrapper level. So what you do is that based on this we compile something that is a unique stuff that you require this particular secret for example. Let's say like in your docker file sorry like in your payload you say I want to use this particular secret db user. So we compile some shock from this and then we hit the secret store. Secret store basically replies you with the secret. And then our docker wrapper stores the secrets into some shared file system. In our case we are using stuff for example. And then we do we call the docker run using command like docker run and then we mount some stuff inside the container and then we run the command. So we put a layer in between based on your payload we identify a certain stuff hit the secret store, get your secret mount it and call docker run. Right? This is the answer. If you are paying attention to my stuff that I have been mentioning this is a bad idea because you are again storing everything onto your volumes. And you are mounting those volumes inside the container. But at least we have some sort of issue with that. I mean we have solved it something like at least we are able to get it to work with Kubernetes or what not. Even though that is still a challenge. So how do you overcome that? So the way to overcome is we use something like a Fuse file system. Have you ever heard about that? It is a file system in user space. It creates a virtual file system in memory so that it is not written anything to the disk. So there is again a tool like kivis form square. It is already using that file system. So instead of mounting what we need to do is we need to basically put those secrets into the Fuse file system. So virtual file system is never written to the disk. And then since it is a file system you can still use the same mounting but in this case it is going to be a Fuse file system and then you are mounting it inside the container. So your secrets are not written anywhere. So basically that is that. This kivis is actually working with Docker as a Docker plugin but it is not working with this model when we have abstracted. So we just need to put this thing in place and then start using it. I mean in our case we are not using kivis we are in-house solution that is like much robust and secure. I cannot divulge more details into that but we made this thing open source Docker wrapper. So feel free to use that and integrate this thing with the kivis and then mount inside your container and hit in the file system virtual file system memory and you are good. Right? So that is the answer. I mean Docker is not the answer for everything. It basically has some limitations. It is brittle. Any change to the Docker command line will break the wrapper. What we do is that we get something from the mesos in the end we call Docker run. If somehow Docker basically changes the API way we will basically upgrade our Docker wrapper as well to use the right method. There are solutions around that. If you have heard about something like a power strip it is basically it is based on Docker API so you can use some API servers to basically call the underlying Docker. We have not done that that can very well be done easily. Again we are assuming that our tool mesos is going to use the command line for the Docker. They can anytime say in the new release we are not calling Docker run we have been calling Docker API instead. So at that time again we have to basically change our wrapper to use that. So it is not keep on top of basically what Docker is doing what Mesos is doing and then keep on improvising updating your wrapper. So as I said to extend functionality of Docker people have basically written they have been writing extension and now Docker has made it possible to basically very well use those plugins as part of your Docker engine. So the stuff that I basically mentioned about the key is it is actually a wall driver plugin. It doesn't work with the Docker but doesn't work with any abstraction on top of that. Again the same thing that I mentioned so that's that. So what is my, so I'm coming towards the end of my talk. So what is the conclusion? Again Docker wrapper is not basically an answer for everything. So we need a standard and a secure way to pass secrets to the running containers we need a solution. We don't want any more PR. We need Docker to act on it. We need to Docker to provide a unified solution so that we stop hacking around this and start running securely in production. Maybe Docker wrapper is the answer to that. We just need to add the few file system support that I spoke about to that. And again this can be used with any tool of your choice. You can use it with mesos, Kubernetes, Docker swarm. You are just calling Docker wrapper instead and then Docker wrapper calls Docker. So start grabbing our Docker wrapper. It's already modularized. Just add a few if I have time I will actually add that support but since it is working for us but the Docker wrapper can very well do lots of other things. You can very well extend the functionality of Docker at the orchestration level. So feel free to grab it out. It's an open source. It's on GitHub. Go to WIP engineering, Docker wrapper. Pull it and start using that. So that's that. So now, so what are we doing at WIP engineering? We have been running apps on mesos for quite some time now. We figure like Docker there are lots of things that are around mesos that are missing. It's not enterprise level stuff yet. So we have been writing tools aggressively around that. We've been writing software for doing the centralized logging for all the ephemeral containers in our environment. We have been writing tools to ingest secrets like in the container. How we are solving issues around how do you do persistent storage with the container. How would you run a database with a container on a block device. Even we have made that part of the code open source. It's on there on WIP engineering. Feel free to check it out. And we are writing solutions around how do you do monitoring and alerting with these ephemeral containers. And bunch of stuff basically. And all of these things should be there as a part of the mesos or Kubernetes or whatnot is not there. So we are writing all of that. And then we are saying how can we make it like more self-serve so that people can actually you can give mesos as a part to the various teams so they can run their own stuff in self-serve manner. And then how do you guarantee QoS quality of service of that that you basically have this that you get this many resources when you want to run your workload. So again like as I said we are and again before coming to this we sat down with the founder of mesos the other day we mentioned what we have done and then so there is going to be convergence of some of the stuff that we have done with mesos. So we are going to basically help the community drive forward so they are pretty stroke to what we have done. So that's what it is coming down the line. Again, feel free to I post some of the stuff that I write that I talk about on my blog so check it out, Elastic Compute you can follow me on Imran Shake on Twitter and this is my email and we are there on IRC as well. So if you have any question using any of our open source tool we are there hanging around there. Let us know, let us improvise and let us help improve that. And here is my references the stuff that I basically use for my talk some of the links there again like the link is going to be posted online so you don't have to basically you will get all of that anyone like my talk is going to be available So that's that and I will take any Q&A if you have any and the other thing is that before adding that people are doing, people are actively working on the solution. For example, Kubernetes Kubernetes is also doing something about the secret, it wasn't there it just came out like about a few months ago and again that thing is very not mature enough it basically store your secrets like on HCD server and that thing is in plain text and it's not secure enough so people are basically looking into it they are solving on their own terms it's what I'm trying to bring out here is that at least we have some solution that is that works it's sort of secure but again we also need the community to basically push them to come up basically start providing a unified solution if you want to make that thing popular and want everybody to basically start using that okay I will take any I will take any Q&A questions if you have any but then if you are actually running the start command inside the container you cannot pull stuff dynamically you would basically pull you are saying just get the secrets at that time you can very well do that but then again it's going to be you can do that thing as well but there are other issues you will be basically storing those variables like inside the memory like somewhere inside the container you can do a docker inspect you can see all of those if you are using mounts or environment they can see that right it's a certificate they use like TLS like protocol to basically test each other and every time like as I said if you have a new change for example then you will rebuild your image every time if you change the secret somehow then for example if you are running the docker you will basically call it as a part of that but we decided to use it that way so that we can very well extend the functionality of docker itself we can create like more like logging containers we can mount some certain stuff we can add some Kafka queues publish that data up there and then this is still like again what I am trying to say is that this thing is modularized so feel free to you can very well extend the core underlying functionality of docker the file system is basically cached if it loses the connection with the secret store the thing is still there and the best thing about the fuse is that it basically expose everything as a file so you don't have to make too many changes to your existing app they can still read the file they can use those secrets actually we just went with like the queues which is already a secret store and they are using fuse so we went with that but the Kubernetes solution they are using tempfs but again what I am trying to say in the Kubernetes the server side they are running at CD and the passwords are not encrypted it is just like part of the plain text any more questions? alright thank you very much thanks for listening thank you guys