 Hello Then we talk and welcome questions and Comments, so feel free Go ahead So distributing secrets my talk is a little bit about a problem possible solution with distributing secrets in networks So what happened? Historical secrets were used Usually what happens is that This isn't a secret doesn't mean giving passives to users. That was the simple Or all monolithic applications even over networks. You have just that one server that one password That was it Very simple There isn't much to distribute except for giving our passive users But then it wasn't an ethical problem was like interaction Then containers cave But Distribute system cage and distributed system and containers are like distributed system to the maximum. So Let's see what happened. What happened is that now They're not just going to one single place one single password is something now people are using directly or through devices like Biphone's other services that use other services use other services that talk to other things And most of the things to somehow authenticate each other because they run on a network And sometimes maybe the errors within that first you can kind of trust Maybe we need a single machine to have connections that are not authenticated or maybe even not encrypted but in most cases you cannot assume that you have to actually Add credentials Maybe even certificates when you want to do tls connections and all I don't know keys for s3 or whatever other Clusters you have there are a bunch of Things you know secrets everywhere in any distributed system so How do people tend to address this problem in the real mystical system? Like a micro services with a bunch of containers running all over and services they use Provisioning systems Is nice big centralized systems They hold the configurations and on the side all passwords in the clear All the admins that do anything in this mysterious system has access to this data And they just get distributed immediately to all the hostile containers Some people go to the great steps of vacancy So the images and then publishing them and they find out oh all my house and keys are there What happened? Why am I I can't try a thousand dollars? Or maybe in The opposite is Which is also bad because if you remove the password accidentally you get a deeper repository It's still there because it just passed it out, but the previous patch is still in their position. So it's kind of Interesting Thing and if you're doing that, yeah, you're doing something wrong Hold me your wish earlier. Give me four things you do and I'll find five things. You're doing wrong But I secured one of you so We're all the same though The main thing I think people need to really realize especially infrastructure Is that secrets are not Sure, secrets are kind of configured in applications very often But they are not configuration. Most of the case of configuration can almost be public There shouldn't be much in your configuration files That really compromises you critically in that experience except for secrets, which are in your configuration So Maybe there is a better way And I don't know if you'll agree. Maybe you think there won't be a little bit for that So let's try to define a little bit of problem space and if anybody speaks in audio or watch the video Don't worry about it It's not bad So I find things I I kind of Um Find out that are really interesting When they take a problem how you distribute all these secrets you have around they have to manage And on how to provide the secrets update them preserve them protect them and finally out of them So the next few slides that two statements about every any of these problems and one is What happens in traditional provisioning systems is used to distribute this information And what happened if you weren't using API? So provide a secret what does it mean? So you get a secret from application and need to private it So you have a bunch of containers that need to access Remote files system how do I give it credentials? All the classic provisioning system will just have this Usual password or a key In a static file, which is pushed to the machine Uh in the API world What happens is that if you can you make the application actually pull the secret When they or you have a helper at least opposing when you start a good thing. You don't Bang this security in the browser system Update what does it mean? Well Bracers may be compromised and I have to change them. You know I might have to rotate them So in the traditional provisioning Or what you can do is to change the configuration at some point and then push out new configuration and probably wish to have a service You know to make it great this new secret music With an API you can have notifications or have application actually pull it when it's ready And that may be even restarting the application itself just start using the music Preserve So this this is something that affects mostly the container world I think in the container world you have images, but when you're having to do updates What you do is that you just wipe the image and you belong to a new one And you have the problem The data you have to preserve part of this data usually is pass or pass out the service because yes You want that data software, but you want to just keep collecting to that database and so Again, you need to have a way To either preserve these credentials in a way that when the container just started they are still there Or you need to inject them And you know to clear what happens when the region system is not involved it can just start new images already low Again the naked case you just let the application pull it so you don't care How you update a private preserve is there it's in the It's in the service that hands out the secrets and the application just know how to pull it Protect So how do you leave an access to the secrets and one of the big issues I see at least in the Provisioning system is that you have this all this bunch of signal for all these applications And the provisioning system uses a man space that's a group of people That means this group of people have access to all the credentials for all the applications Even though they are not necessarily Managed the application. Maybe they just provide the service to push coffee or a shower Uh in an IP world An IP world is that what you can do is that you can have access control So you need specific secrets maybe database secrets only to people that manage the database or only to the application At that specific data lines. These things are not all Accessible at the same time at the same place placing people. You don't have to have people get it right What if someone sends a brown configuration to a brown server? You don't have to worry that much about their names And find an audience So how do I know The secret of not being compromised in a traditional Case where we just have to you know, something that is used to push configuration you have to audit I don't know everything I've seen I don't nfs I mean, yeah That's been possible actually to retrofit that Auditing in such a system so that you can track who has seen this secret going on In the native care world much easier You will have one service that has a password knows What a knows what application is trying to pull it That's when you can trap that much more easily because you have just two points The place where there's two or in the place that asking for the last you don't have all the other infrastructure in the middle It's not for service secrets But Wait, I've been talking about ipi, right? How do I advocate for the ipi? I don't have secrets Well, here's where it's here. Why I'm talking all the time about applications Right There is a great question, but I'm talking only about applications You kind of have to trust the host But bear with at least Host can be something that you want and I wanted to find but In the cases that I've been looking at simplest one or my dns or containers involved You have to trust the host there's no way the host has access to everything. It doesn't matter What happens the host will have access to all the secrets within the dns all within the containers Just another not the other way around So you have to trust the host What you have to do is that Provision of the host is not sold at home yet, but it's probably much easier to handle that Where you'll assign a credential to the whole host and then it doesn't matter how many applications around it How many he answered me you have Something you can trust there and you can build on that trust the transfer of everything So let's see an example use case that they come up Uh, they made me think of all this problem first and While in conclusion that an api is probably much better than to ensure the vision system So we had the problem of trying to get free api running containers at least just to try out if it could be made to work and Free PA by itself if you don't know what it is it's just an identity management system always, uh Like a centralized authentication system that has a ton of secrets in it And it allows you to educate all your clients and one of the features of api is that It is distributed in the sense that you can have multiple Servers that all serve the same Kind of information and they have to replicate information with each other. So When you have to set up another copy to a lot balance, you know Just for geography to distribute the service We had a pretty manual system where you go on On the existing existing server You know run a command that generates a gpg encrypted file We've all been necessary secrets to bootstrap a new server then you have to transfer this file to this new server Physically be there or Staging there putting a couple of passwords Unpack the gpg file get all these secrets out and bootstrap this new image this new server um It works it is relatively secure Uh, it's it doesn't scale in a dynamic environment. It's it's it's very hard to automate Because you have to have very few access To both the machines the one you have to install and one Existing one and it's a lot of steps. You have to do It can be kind of Plus we were adding features Like this sub-ca feature where we allow you to create multiple CA instances in the IPA where you can do that on the fly later on Once this is the whole system installed distributing, I don't know that 15 20 servers And when you create a new CA You create a new sub-ca, but that's CA when you're private key and you need to distribute that So a number of secrets we have Not only do you be distributed at the time you're creating the icon of the system But at times during normal function where you have to transfer them, you know dynamically on the fly And that doesn't work. You have to man do it So what you know the admin create a new CA and add nodes and blocks and all the servers grab this key Bring it to another server You probably do something wrong at that point So we needed to have a way to fetch Credentials from the server A second method of transfer then And a way to authorize access to these keys. These are the three main issues we have So what do I need to be able to perform these actions? Well, I need an account from the service to the services I cannot do that manually. I really need a service to do that an API that's very important I need to be able to equip this from ratio at rest pending transit I need, well, I don't technically need the rest of the API, but these days everybody does it And I need especially a modular design because what I'm thinking about this thing I can't come across other similar problems in similar application of a cyber API, but Contemporary application have these kind of problems. So I wanted something that's modular enough That it could be easy to extend with you and use it in another situation And well thinking about things like OpenShift, for example Another service that not just even infrastructure, but go all the way The user and have multi-tenancy issues. That was also the issue of being able to proxy around this data and, you know, have dynamic access control. So I wanted something very modern And so I decided to build a prototype and this prototype was the I meant to create a basic Functionality for free API, so we could use it in there, but also explore the problem space I wanted to build it as an HTTP server Mostly because the container Kubernetes Docker DevOption 1 is really involved with HTTP and it's kind of the new IP transport of the new world But I wanted it to be able to use a Linux socket Because I want to be able to use it without exposing the servers that I don't have to Next problem was how do I make sure that when I transfer secrets I can do that securely? This it was specifically important And so I started looking at what they could use to encrypt data over the wire in a way that wasn't Invented by me And I implemented JW Prickler, which is another project that just implemented also Stundam now and got standardized while I was building it And that's just a way to use JSON and a bunch of Kubernetes Do great tokens encrypt or sign or go Any data that you can have any JSON structure And have the Prickler sign based on keys you have and transmit it over the network And also key is extensive So how does it work in principle? It works as a kind of pipeline so The url you pass here is basically the address of the secret or gracious want access It could be local, but it could be hcps for example. So you have this client who wants to ask Please server one or the service one Give me the secret and It can also be It could be server immediately if you're talking to the server that actually has the secrets Or maybe that server is a mailman that simply knows How I'm aware to directly request And it just passes it on to the next server and in moon use the uri will change it and append stuff Or actually prepend stuff, never append stuff And that can go to another server And where that is useful is where you can have making part of the secrets Stored in one system and parts of another For various reasons. That's a base of which you could be asking who could be redirected to different services And you might end up translating that again into database query because both requests and Uh, we'll look at the coming and of course you can go out by five points You can transform that in what you want. And this is just an example to show how It might work So how do we transfer secrets I came up with two apis one simple in fact is all simple where you just go and contact Service and ask for a secret by name And it has it. It gives it back to you As you can see, I'm not Showing any authentication here Authentication can be from the bottom of whatever you want. It will probably be something a header. It's a plus case It's to be basic off And you send a password there Has problems But it could be off, it could be just API protected, it could be anything Or it could be just a local unit something And then you just Check what is the process on the other side? What is the sdl to make what it has on so authentication or and or authorization is kind of major and also Completely important But it's not a hard drive to do any specific authentication application And just or is he gonna say we just put And If you're allowed to do something in the namespace that you choose in this case Just Then we get back to the one okay. You put your secret or an error This is the very simple one the simple one has a single problem. Everything is in the field. So It means you can probably want to have either local connection or national connection This is not useful very much for me This is the more complex one It's still just the Get And you still have to provide the name of the secrets of authorization But Then you have to provide Something else that something else is a jwd. You can add your zone up talking Just basically assign it potentially encrypted, but in in this case i don't know about the sign requests And it's looking pretty far The protocol requires you to basically for encoding because you know You're sending you our eyes to observe us that What these three things is Jason blocks The first one is header header is not perfect It is protected that You have to be careful when you pass it because you will read it before you can verify it And it contains the public key identified Yeah, so it contains an enterprise public key that the other shows need to know And that you use your private code to sign the rest of their lists And it will also include an algorithm that you can choose There are many algorithms that are available in the hose standard You can choose one or of course on the other side you can choose to accept them Then there are the claims And this is completely arbitrary in the jwd There was a standard so this is something i came up with Hopefully it's not too stupid One is This subject has to be correspond to what you are asking So you have to put in to the request the name of the secret this way i can as a sign of name Without having to be on the green your eyes And you have to validate that otherwise so you could send a query by name another one An expression of time you want to do that because you don't want to have someone be able to capture very fast and then we fight later maybe after they were able to They could get keys two years later or something and then Now the value The value is interesting only when you're trying to put the secret When every question is secret And then you get the signature and the signature signs They're probably prepared So you really find a key Make sure the other one is valid Check the signature and then you trust And then You get to the ceiling part then The service need to send you back the secret request at this point the creature is Necessary because i don't know whether this thing is going on here or not You probably won't use dls anyway around this but it won't happen and it is another Use a web token but in this case it's using the web encryption It's more complex as five parts one is the enter signal to be used one Although in addition to the sign algorithm you also have to provide an encryption algorithm Then there is The actual encrypted key The whole blob is not encrypted directly in whatever private public private key you have but you'll create a temporary Symetry key in there Only that symmetry is encrypted with your public key in this case The service will use your public key to represent your your your your private key um initialization vector being needed by The encryption algorithm you choose so if you're wearing stdc, you'll have an id with other accounts you might not be usually doing this um cyber text That things encrypted in the key encrypted in here and an authentication fact to verify that the encryption was actually That's So that's it so That's for the problem. It's it's two basically two tokens types, and that's it now authentication I'm talking to headers usually in their creepy And here is how you Configure them there's a configuration file from stdc service, and you can say authentication header header is the name of the module It has handler, which is a python module Adventure configuration in this case. This is one of the example Um Classes which simply uses freeze the header remote user So not something you want to use Directly on network. It's more interesting where you have a proxy between you And custolia that actually does some authentication and then sticks remote user To the request There are a few of these module available in the project And then there's an example authorization directly instead. I see This is taken from the arcade project In the hasons handle it says it says that Enemy under slash keys to the uris collected by it and there is not any slash keys Just think about apply It tells you that the store can use IPA and it's another Directly that defines the store And tells you where service is basically these is used Assigning encryption tokens and where the server can point its own keys So how does it work within IPA in the end? You have one provision instead And that provision is that is storing in the IPA server Is a has that not a storage The public key that you want to trust For exchange of key material Once the admin sets the public key in LDAP The corresponding private key can make requests and accept and update any of the secrets in the IPA server And that public key has to be stored in a specific place in LDAP where only admin access is provided Then you can start Creating a new server of the replica or simply in that every is already stored. It's just a new Key it will send a request to the sign from the top of the chart before using its own public key What the study will do Is look up if that key is available in LDAP It is available in the fetch also the encryption public key because there is actually a key pair In this case Find out in the secrets of it Rub it up and craft it in the encryption. I'm going to set it back. And that's it How do we do authentication authorization in this case is interesting in the IPA case Any machine that's drawn an IPA already has a key pair So there is a basically double authentication system here One is in order to even talk to the store You must have a key tab And you must use a digital authentication against the Web server the password server where the IPA is stored If you are out of progress there Then the password server proxies your advice to the actual study network, which is recently only on the local server It certainly will check that The service sender request has the Apache uoD And has the SNS context of Apache And it will check that the remote user Patrick set actually I think it's just a slave because it's not out there so again in this case and that after it contains the actual Principal name Of the key tab that we use to authenticate That principal name allows you to go in and out and search for An entry that has that principal name in it and the two public keys So it doesn't matter Really that you send the public key in a sign or type We only allow You to access the public keys that are associated with that principal And then we double check it out correct for us. They will check the signature But if they're on the wrong keys, so if you get access to the key tab, but you don't get access To the signing keys, you still cannot access it from there The service and the bus version so you have to have both the kingdom and increase your assignment keys to get access to this stuff And that's all so You can have two containers that are pretty similar And you have something on the house, which is custodial on the house Do you trust the house? How would you differentiate? Yeah so Can explain a little better For this one So that's a good question So in this case, you can imagine that this one is a it's a local custodial server and this one is actually a container So This one will probably Be related somehow if you're using kubernetes for example docker would be related to that in the sense that When they spin a container They will tell the custodial service or maybe the custodial single query What is the container ID? So when and What is the socket this container can talk to? So when this container will contact locally on a unique socket The local custodial service, they will be now real authentication Because we ask the operating system Who is the uid on the other side? What is the socket that is coming getting on and then we can match that container at that point This one knows who's the container And then you can use for example kill as a ticket to connect with centralized custodial service and ask for give me this secret and it will augment here Either with the name of the container or more problem with the label That is associated with containers that the remote custodial service know about So if you have let's say three applications around kubernetes and you know Bunch of containers that do I don't know WordPress application budget containers that to Run a frigate here and a bunch of containers do something else Then when one of these containers can pack the remote custodial service, they will get a label that is probably a WordPress So when they will ask the next one Then we put WordPress Now this other one will know that This this server can't ask for WordPress secrets because it's one of the hosts that is allowed to ground the WordPress containers and then trust this host That he's asking before we have one of the WordPress containers that will fulfill the request If you have your setup setup your setup in the white where you have only specific host in stuff with uh more press and only specific host Do they start with free again? and the host That you know the WordPress containers pass it with free at the label the central system can say oh well That host is not allowed to ask about this information in your life So basically you have a chain of trust By which you can segment partition your infrastructure and know To reach you can be specific signal now Of course if you're running all the container on the posts Then all your trust just in the host because the host can ask for everything but that's up to you how to set up things So are you thinking of with central control and policy for that too because you have policy Right, so that's something that is not built at all and so kustan is to be flexible. So it's probably 15 months thing to build a single authorization piece And with that you can do whatever you want except for that an external system full policies or whatnot My idea is that if we can Do something like this with something like kubernetes on docker kubernetes already using dcd To do stuff and then you so you would have a policy and then we'll be already pushing the host So at least the host means we'll get that policy as well and How we get these labels in and we'll already know how to label these containers So this part is kind of simpler because all you need to do is to associate the container with some sort of label And then the only needs to know where to go next Uh, the central piece will be more complex. We don't have to have some Better understanding of who's allowed to do work but again Nothing impossible and easy to manage in the sense that they put everything in a few centralized places And you manage all the sectors there or you can say, you know, nobody can update secrets on the admin from another host all sorts of things But that's that's probably a policy system so In the kubernetes in docker kubernetes, you would have to mix that that there will be an instance of custodia serving a tenant or One well Again, that's actually the point of the decision So if you're doing your own infrastructure, you own it Up the bottom, then if you just place a custodia server on the host Everything just all to it I'm sorry, that's a good point. So even in this case if I have different applications running on top of that How I can make sure that my SQL server and my I don't know messaging server, they don't try to ask for a cigarette So as I said, you can have a lot sounds like every native of docker You will have a way to know based on the socket Who's asking for information at the very least you have the user ID of the process that's running And probably in the single label of the sbf case You have to be able to dynamically map them because every time docker runs And your container sbf has an in-landed pcs label But on the local system it's easy to discredit who's who as long as you have that information for either By someone or you have an api to request the information That's that's the only the part we have or you set up your system In a region that you always get You know specific ID you and the specific mcs labels, you know, that's up to the point Uh I think it's more interesting in this case, maybe where you have to actually over the network I decide who to but I would I would say that this case is more dynamic because of the local Random container can be run and this case is more about policy So here you apply the label and here you have a more rigid policy that has to seek a label with a tenant And so if it's all your infrastructure You're only dividing by application probably if it is a more complex system like Open sheet or not where you have actually two parts you have infrastructure people And you have a tenant and the family will have multiple applications In that case, you may have a three layer approach where you have a host system that's allowed to ask something To a container A group of containers that represent an application and each group of containers could have their own We could study a server they could with each of them that would Yeah, you're probably most probably have just one customer service per tenant Even if you have 10 of the patients you just don't want because it's you don't want to have to manage them So you will have access control ways within your customer service to know What they can help so one of the models I have called namespace It's easy Just click the local Database where you have secrets by namespace you have a I don't know what what press namespace database namespace Something else namespace These labels will trigger these namespace a lot of access your configuration file. You said you can have different authorization providers for different paths Right. Do those stack like I have a default provider slash and then Slash will be something. Yeah, they can be stacked. There are some limitations there Especially before ordering Ordering may be an issue if you want to do strange things But the nice thing is that these are patent classes So you could just give if if you cannot resolve something you can always pay the subclass and then do your own thing And I think they do that like a actually so It's kind of flexible Up to you how they want to go to your you usually probably want to I think it's in the In the idea case that you use Two authorization plugins in the same product. One checks the UID and one checks the remote On a header So I check both. That's what I have to flag it Okay, so you can set that So if both pass you're in either fails or out Okay, cool. Yeah Have you written your own? Did you have an opinion on HashiCorp's vault product? Yeah, so many people came to me when I'm writing this thing Oh, I've seen this cool vault thing and I think it's quite interesting as a project The main difference is that vault is not an API. Vault is an actual storage Solution It's true that they even they don't have one of their own database But it's not like you can proxy to a vault or anything Basically, it's the final thing. So I see custodia as something to use both in the back end So maybe you know custodia store all this stuff in vault And basically front-end vault exposing actually a simple API because For most uses you don't actually care about the ad nape and have the API in this case I mean applications just need to store it on to the sticker, but you don't have to do your access controls or anything like that So I see for example anything that mandates access controls, right? You know kind of sort of out of band Maybe not exposed And so yeah, both there is a bolt feature in IPA as well where we can store secrets Unfortunately same name and they're coming at the same time, but they both do the same thing They actually do the final step of store and something somewhere, but they don't have any of the pipeline feature Thank you So you can talk a little bit about Kubernetes, have you looked into integrating it? I haven't worked into it yet because I feel Yeah, I haven't worked in the Kubernetes yet. I've been Finishing the integration of the IPA because we actually need this For a feature that comes coming out in the next version It actually promotes the normal machine to a replica And for the sub-C8 features, I already needed to incorporate that And actually have the demonstrated API actually provides for you So now I almost done and I consider the basic IPA rather solid because it's actually tested by me It works to be the whole thing So then I concentrate on trying to expand the use to random applications and find out what's missing there So yeah, the plan is to look at Uh, probably building a modeling go so they can be Embedded by a Docker Kubernetes or what you have and Yeah Work my way in there Yeah, I don't plan to push, you know, the Docker Kubernetes or what use Python this But uh, that's what we have an IPA The core is simple so translating it to a simple type of thing Should be rather easy Will the plug-in be reusable or you would have to rewrite it? Well, no, no, the plug-in is just Python classes But the plug-ins are really, I think the most complex plug-ins probably that you guys have heard What was important for me is to create a structure that is plug-able But now when the plug-ins are in a circuit box, hopefully Hopefully it works Are there questions? Yeah, thanks