 So I wanted to thank everyone for coming out to CF Summit 2018 here in Boston. I hope you've all had an excellent time so far, a lot of good talks, a lot of good sessions and hands-on labs and whatnot. Now I was told when my talk got accepted that I had to show this slide because the Boston City Fire Department Fire Code mandates it of all speakers at all venues within the city limits. So I haven't seen anyone else do this but I'm going to go ahead and go through what I like to call the fire slide chat. Please note the exit, the location of the surrounding emergency exits and locate the nearest lit exit sign to you. In the event of a fire alarm or other emergency, please calmly exit to the public concourse area, not the private concourse behind the VPN. Emergency exit stairwells leading to the outside of this facility are located along the public concourse for your safety in an emergency. Please follow the directions of the public safety staff. And fire slide. This is my talk, The Vault of Secrets. We're going to talk today about secure credentials handling and how we relate with security credentials, sensitive materials, keys and whatnot in CF applications on top of everyone's favorite paths. So who am I? Why am I up here? Who am I talking and why are you listening to me? My name is James Hunt. I'm on the tweeters at I am James Hunt. I am the author of Safe, which is an alternate vault CLI that attempts to bring a little bit of sanity and ease of use to what is otherwise an excellent software product. I also co-authored Spruce. I wrote The Vault Broker in the Call of Foundry community. I maintain the vault and Safe Bosch releases along with a rag tag group of open source hackers on the public internet. Vis-a-vis Mr. Sarabian and Mr. Daly over there. Go Safe. Secrets are everywhere. We cannot escape secrets in the modern world. We have SSH keys to deal with. We have API keys for all the services we integrate with. We have passwords for accounts. We have RSA keys, client secrets, auth tokens, signing keys, X509 CAs inserts. Where do we store these? We've got some options. We can hardcode them into the applications that we push. Whenever we need a secret, we just pop open the code, import DB, set up a const, use the const, commit, push to GitHub. Everything's good. Long as you trust GitHub. As long as your repo is private, and Dr. Nick doesn't make it public accidentally. We can put them in environment variables. We have the ability in Cloud Foundry to just set n and say this is my database secret. This is my username for my admin account. And then you're only one CFN of a way from your creds being leaked. I do believe this is a problem that the CF Core team is working on. I'm very excited to see if they're going to integrate CredHub into the actual storage of secrets and make sure that spaces and orgs are enforced from a visibility and a maintenance standpoint. But until then, we have that problem. We have a third option. We can put them in the database. All of your apps have some persistent storage usually. Whether that's a MySQL database, a Postgres database, Microsoft SQL, Redis, you've got some persistent storage that your application is hooked up to that you can pull secrets out of. So we make a secrets table. No problem. Code no longer has secrets. Environment variables no longer have secrets outside of the service. All the secrets are safely stored in the database. No one can see them except your DBA. DBA can see your secrets. As long as you trust your DBA, you're fine. You trust the security of the database host. And I guess if no one steals the disks and mounts an offline attack against them, that's not going to work. We can encrypt them in the database. RSA, AES, some sort of symmetric cipher. Our secrets table now looks like this. We no longer have to trust the DBA. We no longer stay awake at night worrying that someone is stealing the disks out from under the database systems. Everything's good to go. We are in the clear. Now we have an encryption key. All right. Where do we store the encryption key? We have some options. We can hard code it. Put it in the app. No, that's not going to work. We can put it in an environment. No. We already ruled that one out. We can put it in the database next to the things that's encrypting. So that's not going to work. But we can encrypt it in the database. We have a dilemma. I like to call it secret zero and it's how do you handle the initial credential to unlock the next round of locks, to unlock the round of locks after that to get to the secrets in an automated fashion that doesn't involve a human operator to go in and set a password on reboot. I have a solution to the problem that secrets are tough. Handling them is difficult. Dealing with them is difficult, but it is a necessary part of our life as application developers, systems architects, and cloud engineers. The solution I'd like to talk about today, unsurprisingly, is Hashicorp's Vault product. It's an open source, secrets, credential storage system. It's been around for several years. It's really nice. I really like the Vault software. It's easy to use. It has some serious thought put into how it does security, both in resisting offline attacks, resisting online attacks, access control. They've got a ton of logical back ends. You can do PKI in vault. You can do straight up key value store. You can do Amazon integration. It's got everything. But what about Cloud Foundry? How do we take this really awesome piece of software and integrate it into our CF app workflows? There's a broker for that. It's out in the Cloud Foundry community. It's called Vault Broker. I want to go through a quick demo. So I have in my possession a Cloud Foundry instance set up just for you fine folks today for this talk. In it, I've got an organ of space, and we are going to take a look at a service broker. So I've got the Vault service broker deployed. It's registered into... Oh, well, that didn't work. That's going to be a lot of fun. Let me see if I can... Is that better? There we go. All right. As I was saying, I have a Cloud Foundry instance in my possession for you fine folks today. It has a service broker. The Vault Broker is currently registered to it, and it's available in the marketplace and providing the Vault shared plan. Now, the way the Vault Broker works is you hook up as an operations team a Vault to the Vault Broker. And you say this Vault Broker fronts this highly available console backed, file backed, database backed, however you want to deploy it, Vault. The broker responds to the open source service broker API requests for provisioning by going out and carving up a chunk of the secret KV store hierarchy. Names it after the instance ID. Binds will create a new token, an initial token that has access to that subtree. So basically, every service will have its own view into the Vault, and they're all completely isolated and segregated. So we're going to create a service. We're going to call it Vault, a demo Vault. It's the Vault service shared plan. There's our service, whoopee. And I have an app. This is just CFN. Nothing terribly exciting here. We're going to bind the service. And at this point, the application now has service credentials to actually connect to the Vault, either via its HA address, singleton address, using the token. And its prefix is included. So here, our route is our path right here, secret C3FF. This root token is basically akin to a user name API key, for lack of a better word, for getting into the Vault and managing it. It's running over here on this 10.0 1629 address. And that's the Vault broker. Go back to my slides. We also have a UI. And this was important to me to have a UI for this demo because what I just showed you isn't terribly exciting. We've all created services. We've all bound services. The interesting bits are in actually using the services. And it's really hard to show plumbing like this without an actual UI. So I wrote a UI. This is my personal GitHub org. I'll probably move it into Cloud Foundry Community after this talk is over. And we have another demo. I should warn you there's actually three demos in this talk. Lots of switching, lots of fun stuff. So back to Cloud Foundry. We have our API. I'm going to go into the app source. I'm going to push it. And I'm going to not start the application because the Vault UI does not like coming up with that Vault for obvious reasons. It doesn't have a whole lot to do. And then it falls over. So once we're done deploying all this, we're actually going to take, create another service, bind that service to the UI, and then we're going to see where the magic is. So here we are creating our service. This one's called Vault of Secrets. Binding. And then we'll go ahead and we'll start that Vault app. So the Vault UI actually approximates safe. I don't know. Has anyone here used safe? Couple people? Have you used safe, Dr. Nick? Couple times, once or twice? So we approximate a lot of the functionality that safe has on the command line, and we bring it to a browser. We've got searching. We've got the ability to generate random types of credentials, different types of keys, SSL certificates, which is one of my favorite parts. And this is what it looks like. So I go up here to plus. I can create arbitrary secrets, which are just objects with keys and values. So username is admin, password is secret. I can generate an SSH key pair. If I need to increase that. Of varying strengths, I can test James SSH. I'll make myself a 2048 bit key. There's my key. I won't use this for anything because it's on the screen. I can also create RSA key pairs in varying strengths. I can create X509 certificate authorities, and then I can use those certificate authorities to issue actual X509 certs and in so doing build a whole tree of CAs for an organization or a set of applications. We're going to skip that though. We're going to go back here and we're going to look at searching. So I can come in here. Take a look at the app or the secret that I generated. I can copy and paste these into various places if I need to. And we also have the not at all ripped off from one password, revelation of secrets that are currently in password fields. Totally original design, not stolen from agile bit software in any form or fashion. So now we have a vault UI. I also want to talk about an example app and a fun little strategy you can use with the Vault Broker plus an application is to bind a service or create a service and have one vault that you share across multiple applications. One of those applications might be the Vault UI and you can go into the Vault UI and you can generate secrets that the other applications can go in and consume. They can pull in keys based on path and you can say use this for your basic auth. I will put it there for you manually. You don't have to generate anything and the operators when they push apps don't have to do it. And then they can kind of live and persist beyond. So I wrote an example application. This one will not be going into Cloud Foundry Community because it is utterly pointless, but it is a bit of fun with API Gen. As promised, here's the other demo. We will go through all the fun bits. Again, we push it without starting because it definitely needs a vault to be deployed. And what API Gen does is very simple. It connects to its vault, which we're binding that same vault of secrets. We didn't create a new service. This will be the exact same one and then we'll start it. So API Gen talks to the vault on the web interface, which we'll see in just a minute. It asks for an email address, which it purports that it will email an API key to you. It will go into the vault and generate that API key, store it in the vault, ostensibly for retrieval later by people like me who forget what email address they use for random things. What we're going to do is we're going to go into API Gen, we're going to generate a key, then we're going to go to the CF Vault UI and we're going to verify that everything's wired up properly. Of course, it looks terrible in this browser. Let's go into the other browser. So API Gen, before we can get started, we will need to get you an API key. This is from my really cool new service. We're going to click give me a key, give it an email address. Boom! It has given us an API key. Don't write this down. This is secret. Please don't disperse or disseminate this information. Zero, two, F3, et cetera, et cetera. Now, if I flip back, close that, if I flip back into my vault and I look up tokens, I have this which I didn't have before. And we can double, triple check that by going and getting another one. And I'll do jhunt, startinway.com, give me a key. This one's 2613. If we flip back over to the Vault UI and rerun the token search, I now have two tokens, both stored securely and secretly in the vault. And if we go in here, 2613, e1017. 2613, e1017. So this is an interesting, to me, illustration of how you can use something like the Vault UI and any other applications that need to communicate secrets, as long as you can bind them in the same organ space to the same service, you can actually securely share credentials without having to put them in environment variables. Great. There are those. So Cloud Foundry gives us the tools. Vault and the Vault Broker give us the rest of the implementation for secrets. Also, Apigen loves you. We will go back to my final slide, which is the end of the talk, running a bit light on content, apparently. So I'm James Hunt. I work at Stark and Wayne. We build lots of things for Cloud. We love Bosch. We love Cloud Foundry. Please come see us at our booth. I think we still have t-shirts to give away. If we don't meet our quota, we won't get to come back next year. So please stop by, have a chat with some of the fine people manning the booths. And are there any questions, thoughts, interests, demands? He's got the mic. Hi. So what would you do if you don't own the Vault and you don't have access to create backends? That's a really good question. I might know the answer. You already know the answer. Yeah. What would you do if you couldn't deploy a Vault and didn't have the ability to create backends? So yeah, we just went through this at Garmin. We are using Vault, and we started with this broker and realized quickly that we didn't want to have Vault dropping backends as new orgs and spaces came up. And so what we did is we enhanced the broker to the point where it now makes API calls back into Cloud Foundry, gets a hold of the org and the space where an app is binding, and then drops or creates a token with a custom role that we have predefined knowing that we know who the orgs and spaces are. So the only back end now that comes up on the fly as the broker runs is the one where it's actually keeping track of the bindings themselves, and everything else we're prewriting all our secrets for the applications as we need them. That's interesting. Yeah, so it's a different paradigm than, hey, my app needs to write secrets, my app needs to use secrets, and I don't have control over where those secrets are or even to create backends in Vault as a Cloud Foundry broker. Maybe you would need them also to make it. And at the end of the day, that's one of the things I absolutely love about Vault. It is not very opinionated about a lot of things, and it's incredibly flexible in what it allows you to do and just compose. It's crypto primitives for however you need to use them. So you had a question? Yeah. How do you actually deal with the chicken egg problem about the passwords whenever you deal with that UI, for example? Like, now we've got another UI that you need to log into because the whole secrets are there. Right, so I will point out that the CF Vault UI, if the keen observers in the audience will notice, we did not have to specify any authentication, glaring security hole in the CF Vault UI, one that I hope to remedy by actually having credentials in either via authentication via UAA and space-scoping things, or allowing the Vault operators to set up passwords in the Vault for the broker to use and to pass back to the UI. So it's an unsolved problem. It's definitely a problem that we're investigating and trying to figure out solutions to. These are very much cutting-edge pieces of software. I believe they were both written within the last four months. It's fair to say it's amazing, but how do you actually, whenever you solve the problem, what you will be doing with those passwords? How do you let the user know? Like, where are you going to store those? Right, now you're back to the secret zero problem. We can't ever actually get rid of secret zero. That's the unfortunate problem that all the smoke and mirrors in the world can't get around. Unless Mr. Sarabian has a talk, I believe, when is your talk? Friday at 2. You want to be there? I believe you're using the cubbyhole backend? Yeah, the cubbyhole backend. Secret minus one. One way people have solved that issue is based on which IPs have access to that. That way, you need to come from that IP to get hold of the initial seed. And only those are authorized to get in. So you protect the front door to your building. So one of the things we've done at Sarkin Wayne with Vault is we've integrated it with our concourse platform automation tools. And one of the ways we've done that is via the, I believe it's actually now deprecated, the app roll app ID back ends. We're super cool at the outset because they allowed you to say, here's two parts of a username or two parts of a password. And they'll be known by different competing systems or different isolated systems so that they can't easily be put back together until deployment time. And it's only valid from this subnet cider range or this specific slash 32 so that the concourse workers can be told via pipeline and some other thing, either pulling stuff out of a vault or a cred hub or any other integration that here's the secret key that you and only you can use to go out here to do this thing to pull creds back. And that's one of the, like I said, Vault really cool pieces of software. I encourage you all to go play with it. Like I said, we do have two Bosch releases out in Cloud Foundry Community, one for experimenting with Vault and one for running Vault in prod. We still have five minutes. Four questions? Thank you. So what are you doing now that cred hub is being released and used in the community? Using it? So are you not using Vault in those cases? So what we do, and I'll probably end up segueing into the Genesis side of one of the things that I work on, big project I work on, we use cred hub for things that we can auto generate and forget, specifically on the platform deployment side. I work mostly with Bosch, by the way, I do some Cloud Foundry stuff, but a lot of my day-to-day is in Bosch and deploying of Cloud Foundry on various clouds. So a lot of the things that are in a typical Cloud Foundry deployment are completely irrelevant to day-to-day operations. The CA certs for mutual TLS, the NATS TLS configuration, very little of that information is used by operators for troubleshooting things. I can't remember the first time I ever had to get the NATS client cert so that I could test the NATS connection to make sure that Bosch was able to see the agents come in. Like, it's not a thing that we have happened. So we segregate our platform cert or secrets into things that operators care about, which go in a centralized Vault that's managed out of band, and things that only the Bosch deployments care about. And those go in Crathub because Crathub has excellent facilities for just saying, I need a cert, and I need it valid for this, and I need it valid for this many years for this name. Go do the thing, and then later I'm going to ask you to rotate it, and I don't want to have to remember any of that state. You do that. The concourse story with Crathub is a bit different because I think they're trying to solve slightly different problems than the Bosch team was, but I think there's a place for both technologies. I think they complement each other quite nicely. Yeah, some people are even taught about putting Vault behind Crathub. We actually had a... I think Tom's still working on that, actually, to do a Vault-based config server implementation just to see where that goes. So, yeah, they're complementary. Cool. Oh, one more question. All right. Hey, what are your thoughts around integrating with other existing services like directory services, LDAP, or internal services? Integrating with, let's say, LDAP, like AD that we already have for credentials and other management and integrating with HashiCarp. Oh, is that an auth back-end for Vault? So, instead of Vault... Vault stores credentials, right? So, you can have credentials. What if we have credentials outside of the Vault? For example, in LDAP, like, you know, AD Active Directory services, we have passwords and such, can it integrate so that we can use Vault for kind of a broker? That sounds like a federated model for this. I haven't put any real thought into it. That's the best I got. It's an interesting space, but integration with credentials is a very sticky situation. It's very, very messy. I think if you go to the booth, they'll give you a free t-shirt, and you can chat about it, and maybe even build a business relationship. We'll talk closer and more intimately. All right. So, Ernest, there is an urgent question. This is it for the first part of extensions. You now can go eat, come back to 35, and either Jeff or myself will be here for the next set of talks. Thank you for coming.