 Hello guys, so first thing is that there will be a lighting talks in this room and the next room after this session the schedule is already you can check on the web and the application and Now our last and but not least two speakers before the lighting talks are Pavel Odvody aka Podvody and Chris Chinheims Hello, hello, yeah That's how it's my better. I don't need to yell anymore okay, so if you want to steal my secrets you actually have to gain full-blown code execution in the given container and From my point of view if someone gets full-blown code Execution somewhere. I think that the secrets are like the least I have to worry about Examples of secrets as a service is Custodia which Christian will tell you more about in a short moment Then we have this walled key ways There is actually a free IPA walled as well. There is like five or six different things called walled By different companies like chef has its own world So yeah, it's pretty wealthy. Okay, so Next up I have some diagrams. So Let me show you how to basically process this and show a few basic steps how to incorporate this into your Infrastructure if you're not doing it like this already. So Please raise your hands who uses encryption at rest for your secrets database Okay, good. So if If you are not doing that this is how you can do it so you have your secrets stored in git and This is actually great because you do not break the mantra of immutable infrastructure and so you build the image and now the built image step is not necessarily like a single thing that you use this one single box for building both application and Your secrets these can be two separate things. So you build the image and Why are the why are the? built-time secrets you actually get your Encryption key to the container now. I don't really care whether whether you use symmetric or asymmetric encryption That's your choice basically. So the thing is that we get the key Into the into the Like into the built runtime of of The image so when we push all the secrets into the database We can simply later on encrypt that image or the database inside the image using some key And we will get this secrets database image with that nifty lock in the corner, which basically means that is encrypted and Yeah, so then later on and When you launch a container from that image you actually use the OpenShift slash Kubernetes secrets to provide the decryption key why are regular secrets in Kubernetes or OpenShift and you unlock your database So it's ready to serve all the secrets now the nice thing about the workflow like this is that you can For example use Like the methodology of principle of least privilege that is like I don't know that that's like the basic Step that you can take when you care about security is to make sure that if that Container is not supposed to talk to some service Then don't give him then don't give the container the token to talk to that service because it doesn't need to So this way you can actually make sure that Your container has only access to what it needs and In the same way now this fully depends on what secret as a service mechanism You choose whether it supports access control list or whatever if you can like further limit the access to the secrets through the API so Yeah Yeah, and the second very important thing is is that the secrets as a service allow you to keep precise audit trail as To what is happening in the system because if you For example, if you rotate your keys every 15 minutes Then it doesn't really make sense to redeploy your pods all the time because if you even if if even if it's fast It usually takes like five or ten seconds to redeploy the given pod but if you do it like every 15 minutes then it means that in 15 minutes time span your container is like ten seconds down and if you extrapolate that number throughout the entire day month year and It's not a single container, but you've got hundreds of them You may realize that your containers spent like Two weeks being redeployed for no reason simply because in the current state of the art The lifetime of the secret is bound to the lifetime of the pod But if you use secret as a service This is not the case because when you need the secret you hit the service you get the token back when you are done with the token you may hit the service again and the key or the token gets revoked and This is also From my point of view. This is very important because this makes the whole the whole Concept of security. It's very it's it's not rigid Whenever you decide, okay, I need to rotate it is key or revoke this one because I Don't know I feel bad and I feel like My infrastructures got breached these things might might have been stolen. So I simply rotate all things and It's very simple Because one of those quotes that I very much like in this regard is that It's this rule of security that security at the expense of usability comes at the expense of security Which basically means that if security is too hard to do it, right? Nobody will water So to make things really simple Please use Secrets as a service and I think that Christian will tell you much more reasons why This is a very good idea and so Yeah, I was very quick Thanks, you have two microphones for two different recordings apparently so While I will switch it over to my slides just press the full screen. Well, yeah, so hi. I'm Christian Heimus I'm senior software engineer for Red Hat security identity Identity department working mostly on free IPA and related projects and one project already started by CIMO and Now revolved by CIMO and me is custodia So slides up perfect. So chapter two custodia So I'm going to a quick intern to custodia. I'll Show you a bit of our design reasonings why we design custodia as it is now Then I go into custodia for containers because custodia was not designed only for containers But for in general as in system to deploy secrets I'll give you a couple of examples how we currently are able to integrate custodia with different systems Stuff we'd like to do in the future and finally questions and hopefully answers too So Secrets are radioactive That's a see how secrets are that's actually not a stranger from fallout Signal vaults, but it's actually a game that was the things sold in the 50s with nuclear material So kids can play with the new power and that's how I see How developers has to deal with secrets right now? They are forced to play with very dangerous things and We like to do we like to make it much easier and much safer for developers So there's a gym running gag. I don't care about nuclear power. I get my power out of my socket from the wall and We like like to do the same for secrets We want developers not to care anymore about secrets, but just use a simple socket and API to get secrets and let the Security engineers with the hazmat suit deal with all this dangerous stuff, so Let's meet custodia custodia Start with the word definition. So it's original Latin word Mean something protecting safeguarding. It's also used in English rather uncommon to Say something about caring responsibility There's also a family relative word called pigs means box is used For storing the old compass on the ship. So if you lose your compass must be protected you won't find home anymore or If you like Christian, hi Christian The container for consecrated brats in the church is also called custodia the large one who picks the smaller one So you see we try to Protect your secrets and be very very careful about them So what's custodia? Let's start from the beginning It's Currently a technology. It's not a ready-to-use product that you can just deploy now and it will solve all your problems It's as Paul said before it's secret as a service. It's an API you can Get secrets somehow explain later how it works It's worked a bit differently than existing technologies that you'd like Docker Kubernetes you actively push From like the cube demon the secret into a Container when it starts up. We do the opposite way. We're going to pull a secret with the API and It's not only Implementation it's also definition how we're going to transport Are we going to route secrets because we just want to have secrets maybe on the infrastructure We have to find a way to get a secret through multiple hops We Have for different reasons and for different use cases different ways to define authorization authentication policies We want to make us pluckable and because to this also current in the name of our reference implementation written in Python I'm a Python developer. I'm actually a Python core developer use the SSL model or a hatchlet model mostly maintained by me Because we get that question rather often what's custodia not just to make it clear We don't want to replace existing solution like at CD or Open-stack Babic and or a hashtag of vault these are of different use cases. So Just not a replacement. It's an additional tool that can help you It's also not just another store for secrets We have internally implementations to store secrets mostly for debugging testing because two years more a transport system for secrets an API and you can use this just to adapt this Existing vault solution. So you could write it back and that talks to Peshacop vault or to free IP a vault or whatever It's out there or even to at CD. So we have met see bindings As I said before it's not just for containers We actually use that currently in free IPA a lot And again not proof of concept. Oh, yeah We use it if you deploy a new free IPA cluster and you credit replicas we use custodia to deploy all these super sync sensitive things into our replication key See I pray private keys for our CA. So we travel thing now a bit deep dive into the design when Seymour started the process and the idea how we designed it, of course, we are security engineers We started with threat analysis. So what Pablo already told you we look at the current problem Yeah, cook look at the current problems and then came to some results. Oh Yo, sorry. Yeah. Yeah, it should be a T threat. Yeah, not like the threading the danger. Oh shit Sorry, sorry. Sorry. Yeah, I'm not a native speaker. So God, yeah, I should have reviewed the slides there So we don't want to have secrets on the fault system because as public said file systems are all bad you can have traversal attacks when you have a web application and a couple of places includes something like tempfs or even the proc file system if you have a Very variable there to start in the proc file system We want to have strong encryption not only at rest but also in transit Which is a bit different than having end-to-end encryption because since we have routing protocol We want to protect both the hops between the transfer between the hops and the enter a chain with two own two different ways and Of course, we want to reduce The amount of access that's required So we basically don't want that the infrastructure that doesn't need to have access to secrets actually get access at all. So Just want to have like the container get access But why should likely communities demon have access to a secret it doesn't need Auditing very important. I want to Paul moose or talk in a case a system get compromised You want to know which of the secrets got compromised? And if you just have to rotate a key or maybe restart your complete complete CA public key infrastructure So without auditing you can't even know what's going wrong He rotation should be simple part of the protocol and That bolts down to if you go to the enterprise PCI DSS compliance for banking and so The very strict rules that you have to obey to use Debal secrets, so we have customers for banks in banks that want to use containers But currently are having problem to use containers because they have to abide PCI DSS compliance rules These and always led to some designs design principles so again, it's much easier to Prove that something works securely by reducing the attack pattern and require the application to Actively pull a secret because if I just push a secret in a container that may or may not need secret Yeah, we have a much bigger attack surface. So we pull a secret on demand We want to use standard protocols because we don't want to reinvent everything on our own And it should be rather easy to write clients and servers and proxies and all the other tools we have I wrote stuff in Python and my first go programs. We have implementations using C in SSSD Again, it should be also simple to deploy and integrate in the current infrastructure because when it's hard to use It's bubble set makes it much more likely that you screw up something and And it's flexible. So we have different demands different customers different use cases. So all the authentication That should be authorization to I am And so it should be flexible and pluggable and Also extensible so we want to maybe support HSM for big enterprise customers We want to maybe generate a secret on demand. So when a service comes up and requires to have a secret So we should say I need that secret doesn't exist Please generate a secret for me according to a policy and all the surface service can then use that secret notification API for having a better way to Restart a service when secret changes So man we came up with the building blocks obviously for transport what we use these days. So what would you use for a transport? TLS No, no pigeon. So, yeah, TLS and HTTP. So simple rest API We want to get something we want to put a secret want to delete the secret Can you just use HTTP over TLS as you said or unix domain sockets and As message format what these kids use today? So I am from area where we use XML, but these days we use Jason with optional Jose for key P encapsulation and so encryption and yeah We have a set earlier we have a storage lay-up section where we have a couple of implementation So it's secret light implementation is just designed for debugging and testing and for demo systems So not something we want to use in production for production Maybe later free IP a vault going to that and even for simple cases when you're back and doesn't support encryption We can do like an overlay we wrap the storage another storage that does just key wrapping and unwrapping The plugable authentication we build a couple of things use the PI cab Ross TLS client certs If you don't have cab Ross you see PA you can use both The other thing I'm going to into a bit later. That's right to a way unique sockets can give you some nice features and Lastly we use up the URL and rebutton of the URL to route secrets so secret gets transported to multiple hops and yeah, give you an example for that later So when I have a simple interface for developers So everybody can do just HTTP and Jason even in C and if you have PHP Python Java Go rust it's all built in The different layers we have give gives us rather good isolation and greater flexibility. We can just stack stuff and With all The parts we put together and the routing we can move the entire policies and the configuration everything. What's Complicated that's not concerned of the developer to the infrastructure the developer doesn't have to care anymore How the secret gets transported how it stored? How it get Secret is processed. It just gets the secret with HTTP get and done And what we're trying to do with a bit of standardization. We hope to establish a very simple basic Rest API that can be used implemented by other vendors We have already other companies or vendors interested in the products or in the technology we defined and Yeah, let's see so Now Custody is not just for containers, but this talk is mostly for container people. So how Does Custody fit into containers? So with containers we have one big issue How do we authenticate an HTTP request from a container? so yeah, we could just Get a secret into a container somehow and then we have like a bootstrapping chicken egg issue because now We need infrastructure to get a secret in the container Just to get a secret from our API and Yeah, but we can use a little trick. So Who's able to spot the part that are the authentication? Is anybody able to spot the part or the admin of the core query that takes care of authentication? No, no, that's just a request. So do a core request and the the answer of the Jason string at the end. So There's no authentication involved in that query itself, but rather the way how the query works internally because it uses a Unix socket and with the Unix socket We can do some nifty tricks and just rely on the operating system to give us security and strong authentication so What we do currently it's a big ugly hack Something it's totally insecure the way it works at the end, but this are from beginning. So I have a custodial Container running that runs in the host p&e namespace So I can't run it in its own namespace for a reason we'll see later And this custodial daemon listens on the Unix socket on this unit socket then is bind mounted into the container and For example this before you're back This bar run custodial sock that's the units units socket device file. It's a file based Unix socket so then we use some of the Features that's called get sock where the option so peer credentials That gives up the PID the effect is user and the effect is group ID of the process that just did the HTTP request That's guaranteed by the kernel that you can't do something back with that. So the kernel ensure that you get correct information the Threat levels here you have to compromise your whole kernel to do something bad But if an attacker is able to compromise the whole kernel the whole OS Yeah, you screwed anyway, so We also can get on as a lens enabled system the as a Linux Process label so the as a news contact the process runs in There's both this information we can then look into the proc file system to the C group file and Use some regular expressions just to extract the container ID from the C group file That's where the bad hack starts and now it gets very scary We also bind mount the docker demon into your custodial socket and then use the docker demon to Fetch additional meter data from the docket That's what you want to remove so did anybody attend to palmers. Yeah Yes, no, no, no come to that in a minute, so I haven't yet diagram next slide up so Did anybody attend to palmers talk about namespace and auditing I Yeah, my boss did and the two other hands so Paul is looking into a way to have like container IDs on the level of the rating system so that each namespace or each container gets an ID and What I like to have this ID instead of using like the PID you are the security credential just use a similar source call to get the namespace container ID from the Unix kernel and then Add some meaning to that so But to get to your question. Yes, we have to run consider on each note But no we don't have to have the secret on each of the systems because again Because to us also routing protocol, so we here have we have application running in container and We bind mount the Unix socket into the container and then have because so they are running for example in a demon set on each of the hosts and that instance running on each of the cublets or VMs or hosts Going to the Kubernetes case that he gets information like the Kubernetes namespace definition from the Tucker demon or rocket and Rewrite the you'll so the container just says get me the secret this demon verify the request Rewrite the you'll and wait said okay namespace from tenant a container or a pot name be tries to request the secret that then gets right rerouted to a Custody instance running on the master using either TLS client certs or diesel API for authentication and Now we can verify Is a container actually running on that machine with that namespace on that pod definition? and is it allowed to request that secret and then we can talk to a key store or Maybe to even another custody instance that then gets the secret can just check daisy chain them And each of the custody instances can enforce a policy and verify each request with additional information With that approach we can even survive a compromise of a cublet so you isolate each customer on a virtual machine or on a bare metal machine and So not to customers run on the same machine Then it's possible to verify that customer a can never request secrets for customer B one of these cases Yeah, that's so HTTPS for the hops and end-to-end with Jose so we can also do Jose Then you have a middle step way Good question. We haven't fully figured out how you do it securely Then we have to get somehow keys into each of the containers, but these keys you can bake in like in your your image theoretically or Yeah, now that's just So in that jargon, we just very just get the meter from the From the delco demon for the pot like the namespace and the yeah So the question was just to repeat how to verify that the container at that point So that's an issue we currently just can't solve with a way We have with it as a Linux policy as the Linux context of the process and the PID if you're going to have the container ID That's suggested by Paul Muir for auditing. We can use the same auditing container ID also to verify the request Yes Next point how to do it better, but then you have to exchange your application to do software HSM So that's the problem. You can't solve with the way currently tokens work or password works Where you like the software HSM so digit of PKT is 11 or how SSH and GPG agent work You just prove that you have Somehow access to a private key without actually having read access to a private key You just prove the the permission to do an operation the private key That could be theoretically or also running out of time directly Handled by having something like a GPG agents or SSH agent similar protocol baked into custodial But then you have to heavily modify your application That's so we want to make slowly progresses slowly. Yeah, so Yes, I agree, but if you make it too hard so then nobody will use it so we have to Yeah, so just because I'm running out of time Couple of quick examples how we can only able to integrate it so make it easy for people edit to your conflict also, so you pass a conflict and The conflict part that resolve the secret for you so that's your to conflict file and for example YAML power so you pass YAML and this magic Dash custodial summation by custodial signals the general power to run a kind of internal extension and that extension just do is like container key. So get me the key for for some string and Deplication only sees the resolved key Our thing you can if you just look at them our repositories I've written a command to you the new docker credential store API and yeah so A couple of years to integrate that Java key store because you'll have an slot our lip secrets so and just and so Sorry, yeah Again, we haven't solved all problems yet I'm right. So the question with end-to-end encryption to get in keys in So the thing on that out so It's not a magic pill. It's External reviews the question was give me an external protocol rules Not yet. So Again, it's just technology and development. We First we try to get some traction and some use cases So we are now on the face where we think it's good enough to let people play with it And it's not ready to use product. It's technology work in progress. So The part where we use it in free IPA again. I'm out of time We can use for save because we have to use the API to get keys, but you can just talk Oh, five minutes for question. Ah more questions. Sorry. Ah So the question with the token to get these secrets from the Unix are good And the so the the in the Kubernetes case each of the Machines has machine credentials. For example, each machine at its own private public key pair to use TLS client sort of authentication or its own Kerberos key tab to do Service-based Authentication so we can then verify that request came from that machine We partly trust the way So we have to trust the cute let demon that it tells up the right Container It's asking for for a machine. We then can verify that that contain actually run on that machine, but we theoretically the system could still lie say I've running two containers on the same machine and Oh, the secret is requested for another container not this. So there's still a trust chain. We have to Trust part of the systems for each of the steps But the other way if you use Kubernetes secrets, you still you trust that you push the secret into correct container when you start a container, so It's not at least the why I understand a modic Kubernetes expert as well as far as I understand how Kubernetes works Our approach is not less secure than the current approach back when at ease any more questions Go ahead it's a lot like four minutes to go for questions Okay, you're describing so the question was So the Question was Sure, that's my the boss of my boss. So That's a slide I skipped a bit so Java he stores just a standard API to get a secret for a running JV Java application or What do you probably know or not know is so KDE? Key ring and no key ring use internal a something called lip secrets. That's just a library that can you compile in has some C API where you use the session debuffs to talk to your like key ring and We like to do something similar just to hook in the lip secret, but for a headless server So using the system debuffs and so these are ideas we have or for Open stack hook into also config. So, yeah Yeah, okay Before you phrase you you in your experience you had issues to convince developers to Use your API because they didn't see a point or was made to how to use That's why we like to standardize on a very simple protocol using like HTTP and JSON that can be easily implemented You can write multiple clients for that or integration points and if we get multiple Companies multiple departments inside redhead outside redhead behind the very simple API then we can work on top of that and extend but first we have to show that Having API is much better than baking your own solutions Yeah So that's oh We have like a replacement thingy because they would be in that slide where you just run instead of the original app just have like a facade in front of that and then we inject for example secrets or drop files that would be a Step to use custodia API without changing your application even change the container again You still have the issues you have to these secrets on the file system We're in N far, but at least you could use the API without modifying buying the image out of time, sorry Only like three minutes over If you have any more questions feel free to contact me