 Hi, I'm Matt. I work at Zipcar. I want to talk to you about a piece of software that I wrote today called Bosch Vault. I'm hiring, we're hiring. If you think this is cool, if you do front-end stuff, infrastructure stuff, Go, Ruby, Java, all that, talk to me, talk to my teammates, we want to work with you too. Now, let's talk about something that really is great about Bosch once my clicker works again. Really? That's frightening. There we go. Manifests, right? They're amazing. You want to move clouds? Solved it, done, easy. You want to move whatever you want to do. You want to deploy something. It's like all the best parts of Terraform and Ansible together in one place, which is great. Who wants to complain? Let's complain. So many credentials inside them, right? Sprawl, we talked about this last year. You've manifest everywhere. You have to have all these different ways to manage them. Even though it's the best part of Terraform and Ansible glued together, you can't commit it to source control, not the real manifest. You could commit the pieces of the manifest. The real manifest is this. You pay somebody to do this for you, you do it for yourself. You're munging stuff together. Where are your credentials? You're going to get them, you're going to generate them in some way. All that logic. All the things that keep us gainfully employed. We love it, but we also kind of hate it. This is IP. This is stuff that's hard to talk about with other people. It's hard to share with other people. Bosch people are so smart though. Couple years ago, a config server came around. Who knows what config server is? Yes. Who's using Vault right now? Yeah, not surprising. Okay. So Bosch came around and they said, wouldn't it be great if Bosch could understand variables? What if Bosch could use an API to speak credentials? So we can't do that. That's what config server is. It's this idea that there's a collection of endpoints that Bosch can call in a predictable way and get things that it needs or generate things that it needs. It's essentially crud, right? I want to create new stuff. I want to update it. I want to delete it. Not really a crazy surface area. And this is amazing. It's a really good idea. And for a long time, config server was also just Kredhub. And that is cool. But I was like, I don't know. Last year I gave a talk about this and I'm kind of still mad. There's some downsides to Kredhub. And some of these things are actively being worked on and that's amazing. But this was my slide from a year ago and I probably should stop because I did used to work at a creative agency. And if anybody watches this talk, like I know there's a drop shadow here and that it's 2019. But it's not lost on me that I'm throwing a little shade. I know that and I'm sorry, but I'm still mad. So I'm coming down here like Paul Revere on Northeast Regional waving vault around and ringing that busted bell. Look at that. That's 21 custom secret backends that exist in vault today. If you use these services, you can generate credentials to vault for them. Vault can handle expiration for you. Vault can handle all that auditing for you. If you don't know what this looks like, here's an example of reaching a database endpoint in vault. Your operations team, your DBA team, whatever is gonna work in vault. Maybe it's your security team and configure a database role that can generate usernames and passwords to databases you want. And your DBA team and your operators and your security team, they already know what the grant should look like. They know what the SQL should look like. They probably have it in a script somewhere that someone has to run or automation has to run. When those things are configured, vault can do all of this stuff for you. Vault does a lot of other really useful features too, like the ability to restrict token usage by IP. And you'll say, but Matt, I can spoof headers. And I'll say, yeah, but not everyone can. And that's security in depth. That's the onion, right? I can leak a token that for most people is not usable, unless you're on the host that's allowed to use that token. Not a silver bullet. It's not a real solution to leaking tokens, but it's better than nothing. And vault has all of those amazing security primitives that I've been screaming about for years. Parent-child token relationships, super awesome. You have a token that's generating all these dynamic creds for you and you revoke it. Cascades, they're all revoked. Vault does it for you. You don't have to worry about it. You have break-class procedures for sealing and unsealing. You have Shemir secret sharing. You have amazing security primitives like wrapping, cubby holes, that kind of stuff. Vault is super audited. And it's compliant in a lot of different ways that you may or may not need to be compliant. The best thing about it, I didn't write it. I didn't have to. They did for me. I could pay somebody to support it. Though you could pay somebody to support just about anything. Still though, if you love those features, this is where you live. This whole process, this bespoke business solution. Man, man. I waited for a while. Last year I saw people talking about config server. I saw people talking about credit hub. And it seemed like the future of Vault was just not gonna include Bosch. And the future of Bosch wasn't gonna include Vault. And that made me really sad. And I started talking to people. And it sounded like that was making other people really sad. They all couldn't show up today, but a few of them did. And we decided to try writing a POC. I didn't know what that was gonna look like. I didn't know how well-documented it was. I didn't know if people were gonna tell me this crazy idea, don't do it. But after last year, I said, you know what, we're gonna try it. We talked to some people, we got some support. And I found that everybody was so into it and so friendly. The amount of people on Slack and in GitHub that were maintaining documentation and answering my crazy questions, like it never ceased to amaze me. I wanna recognize a lot of people for this effort. I'm sure this is not an exhaustive list, but this is a list of people who have been like, just invaluable, I've been building this. I've been able to bounce ideas off them. They've been maintaining documentation that I've been referencing. They've been answering my questions on Slack. And here's what's crazy. That left column is people in the community. People that for the most part I've never met. And people on the right are people that are my colleagues and a couple of friends in tech that have helped me. And what that means to me is that I had twice the support. I had twice the team that I thought I had because I went and tried this out. And that's why I can sit here today, stand here today and say to you that config server is not just Cread Hub, now config server is Vault as well. So let's do this. Projects called Bosch Vault. It's open sourced right now. I would love to have this brought into Cloud Foundry Community in some way, Cloud Foundry Incubator. I don't really know how to do that. I think it starts with me asking you all to care about the idea of config server as an open API. To say that it does make sense to decouple this idea of how Bosch speaks credentials from where those credentials are stored. Because this abstraction of config server is way better than whatever we have in Bosch Vault and whatever we have in Cread Hub because this abstraction allows us to give our director essentially nothing. So Bosch Vault, what is it? At its base level, it's an open source implementation of config server and it happens to be back with Vault. It's a stateless code program which gives you a lot of options when it comes to deploying this. Not everybody's infrastructure is the same. It's also a Bosch release. So we'll talk about some different deployment patterns in a minute. Let's talk about why you might wanna use this aside from just loving Vault like me. If you're already operationalizing Vault and you haven't yet moved to a different secret storage solution or any config server implementation, you haven't written your own, you're not using Cread Hub yet, you should give this a shot, especially if you're already using Vault. If you think that idea that I was talking about before about where I store my credentials should not necessarily be the same server that's responding to questions that like config server should really be this intermediate layer that can plug into any backend. This project might be interesting to you. There's also certain features, dynamic secrets, things that aren't yet done that Vault can do today. That might be a reason that you might wanna try this out. Let's talk about what Vault can do aside from just being a config server. So that means you can send the normal kind of variables block that you would in a manifest and have it generate certificates and passwords and SSH keys and RSA keys and usernames and passwords. All the things that we think about when we think of Bosch speaking variables can do that. It can also do things like allow Bosch to use dynamic secret engines without having to change Bosch at all. And it can do things like proxy credential request if you need to centralize management of things like shared API keys or shared CA certs or shared whatever. Something that you're now manually copying all around your data center. My idea is if you're copying credentials, something is wrong. Get out of there. All the problems I have with manifests, Bosch and config server have solved. All the complaints that I had about the past version of CredHub is solved with Vault. Let's talk about what deploying this looks like. So just like with CredHub or any config server, you're gonna need to tell the director that this thing exists. That config looks exactly the way you would expect. It's the config server URL. It's some UAA credentials. And then what Bosch will do behind the scenes is get a JWT and send it along with every single request it makes to config server. You're gonna need a functional Vault server with a KV2 backend. So that's just normal Bosch key value, but it's the key value store that Vault has that supports version secrets. This is another thing that like, I don't wanna have to maintain that code. You don't wanna have to maintain that code. Vault can maintain that code. How do we version secrets? Mitchell and his team do that. You're also gonna need a token that allows access. This again goes back to that kind of type of token we're talking about. You want a periodic token when that can be regularly refreshed. Bosch Vault's gonna do that automatically. You also probably want it to be scoped to the IP of wherever the request is coming from. Again, not a silver bullet, but it helps. Then you're gonna have to make that decision. Are you gonna deploy this as a binary? Are you gonna deploy this with Bosch? And it really doesn't matter, but you're gonna need all the things that go with that deployment. This is the ideal deployment. So as I mentioned, just what Bosch does when it wants to use variables is it talks to UAA, gets a JWT, so it gets a JSON web token. It sends it as a header across to config server, which in this case is Bosch Vault. Bosch Vault is bound to the Vault server on localhost. Now, if you're generating your own credential certs for Vault, then that's something that you can do. You can add that IP scan. If you're using purchased certs, you're going over localhost, you can choose to skit verification. You'll still get the benefits of TLS, but if something was wrong with that cert, you wouldn't necessarily know. I think the benefits of running this as a sidecar are much more valuable than the benefits of saying not purchasing certs for your Vault server. But it's up to you. Now, some of you might say, wait a minute, hang on, that's a single point of failure. Oh, you're so right. So that's what you should really do. Remember, Bosch Vault is a stateless code program. It has no idea what it's talking to. It's talking to a Vault server. Doesn't know if it's in HA mode or if it's getting redirected, it does not care. So you can just do this. Again, this JWT is getting sent. Bosch Vault is going to verify it using, now there's no connection from Bosch Vault to UAA because every request does not do that. You can tell Bosch Vault how often to get UAA's public key and it'll do offline verifications. You're not hammering your UAA server when you go to deploy things. You can also deploy it completely outside of Bosch as separate deployments or as binaries. Though, again, you're probably gonna want multiples. You can also do this. I personally don't like this because it does write in the configuration file the token for the secrets that the director is getting in the director, which is problematic. You could say, yeah, but if I'm on the director, I could maybe get my UAA stuff and generate a token and do this. To which I say you are probably right and that in a real adversarial scenario, this is probably no different than the other one but I would like to add unwrapped support to Bosch Vault so it could get better. Still, I think the ideal is really this. It's HA, it's easy to manage and we kind of think of this as like the base level of our platform. Like it's still painful. I still need to do a lot of munging to deploy a director and a Vault server. Not too too much, but I do need to do some. But once I have that, everything else can be dynamic. Let's talk about how this is configured. As I mentioned, it's almost secretless but not quite. You do need to give a token in your Vault configuration. There's a really interesting story about using unwrapped primitives at Vault support so that you could give a non-secret value in this configuration. It wouldn't be very difficult to implement but I think there are edges around it that require more input, more people to be using this to talk about what that story should be. Everything else, as I mentioned with UAA, Bosch Vault's not logging into UAA. Bosch Vault's verifying tokens that come out of UAA so it does not need secrets to do that. What port this is bound on is not, I don't consider that a secret. A path to a file, where it should find its certs, that is not a secret. What logging level? Really you should not have any logging or errors only in prod, but it does not log any credentials ever in the clear. Still, not a secret. This is what I really wanna talk to you about with the time that I have left and hopefully demo. The redirects feature. This gives you all the things that you want in Config Server that don't exist today. So the ability to aggregate and audit secret requests from different sources. The ability to use dynamic secrets and get free credential rotation on your Bosch applications. Maybe you haven't yet moved over to Config Server or version secrets in Vault at all and your modification process relies on old version one KV store Vault, redirects can help you migrate to version secrets. All redirects, whatever type you're gonna use here, they're all cached at the expected path in the local Vault and what that means is that you can use redirects for a short while, then remove them and Bosch Vault and Config Server will continue to operate normally because for all it knows, that's where the record came from. When you configure redirects, you have to choose what type. So there's upstream, which is a different Vault server. It could also just be a different path on the same Vault server. You can use YAML pointers for your Vault configuration. Upstream is a versioned Vault somewhere else that you wanna request. Dynamic is the Vault dynamic credentials endpoints and V1 is the non-versioned Vault KV store. And so the thing that's kind of interesting about that too is all of those connections that we're making to Vault in Bosch Vault are made using HashiCorp's provided and exported libraries from the Vault code itself. So I'm not maintaining plumbing code between all these things. They do that at HashiCorp for me. And of course, you're gonna need your redirect rules. So what does that look like? This is what it looks like. It's an array. I'm gonna come find this, define this upstream redirect right here. I'm gonna point it to a global Vault. I'm gonna give it a read-only token. And this is what's kind of nice about this redirect feature is it gives you the ability to have more fine-grained control over things that you want nobody to be able to overwrite. No config server should ever be able to overwrite your APM key. Like you got a data.key, new relic key that changes once every year or two. Nothing should ever overwrite that but an operator. Give a read-only token here and it's not possible. Redirects are only followed on get requests. You cannot post to a redirected Vault. You can't delete from a redirected Vault. You can't update a redirected Vault. It's only for getting. Which means that you can use this type of variable syntax. You want the shared API key. Instead of getting the shared API key, instead get global Vault global shared API key. If that global Vault ever goes down, recall what I said about caching. Before Bosch Vault makes the request to that redirected Vault, it's gonna check the health of it. If it's not healthy, it's gonna get the last known value from its local Vault. Next time you do a get, if that Vault's healthy again, it'll get the new value. This is for a v1, so this is assuming that you have them on the same, so we're using a pointer here to our Vault configuration. We're trying to access some shared key and it's gonna go get it from the non-version secret store. So if you have a lot of non-version secrets today, you can build up an array of version one redirect rules and point all the new manifest paths to the old paths in Vault. Everything will be copied over to KV2. You then delete those rules. You have no more redirects. You redeploy Bosch Vault. And now going forward, you have version secrets that you can use with built in Vault. Dynamic secrets. Look at all these. They return random stuff. This one is for database, so it's a username and password. But you can imagine that, well Oracle is probably the same, but like Google Cloud or AWS is gonna return different types of keys. We don't need to know about any of that because we're just returning JSON into Bosch. And so that means I can just arbitrarily say, I don't need to say what type of variable this is, nothing. I can reference inside my deployment. Now this reference is built up from config server. When you have normal variables, like the ones on that in manifest header there, such as normal Bosch variable syntax that I hope most of you have seen, that's what config server turns that into. It actually requests from config server, what's the name of the director? What's the name of the deployment? What's the name of the variable? And so what I'm saying is whenever you see this deployment, request this dynamic Postgres variable, don't just go and get a static value. Use that configured secrets engine. Read from there instead. And now Vault is gonna do your expiring. Vault is gonna do your auditing. You're gonna get all that for free. And I'm gonna demo this in a couple of minutes. You can also combine all these behaviors at once. This can be abused. V1s are really only meant for migrating, unless you absolutely need to. Don't keep using them. Move to version secret stores. Upstreams are meant for aggregating things that you want other teams to control. And of course, dynamics are meant for frequent rotation. So the last thing I wanna talk about, so my short demo, which I think I'm actually gonna have time for the full demo here. My short demo is just run my test suite. Just not a very good demo. But there's a lot of tests for this. And I would love help thinking through and writing more. Again, just like with our plumbing code to connect to Vault, all of our Vault mocking, Vault core simulating stuff is maintained by HashiCorp. Vault exports it. When I wanna say is this Vault unhealthy, I have several different ways to load up code from HashiCorp that says, here's a Vault that's not initialized. Here's a Vault that's sealed. Here's a Vault that's in this bad state. And I can test against that. When I wanna do password generation, the standard library and load that stuff in, I can make sure that it's getting written and this right ID is getting back. And again, I don't have to maintain all these strange Vault knocks that are kind of an antithesis to what we wanna focus on here, which is that middle layer, that proxying the logic, that decoupling the idea of secret encryption, management, and storage from helping your Bosch director know what a variable is. So let's do a demo. I cracked it because of the Liberty Bell. But then I thought that's kind of frightening. So we'll put it back. And so while I get this demo ready, so you got up here my amazingly secure application that outputs a counter to the screen and a Postgres connection string. It's a great app. I haven't open sourced that. I've just open sourced Bosch Vault. Sorry, you'll have to write that one yourself. So let's just deploy my great application here. And then we'll only take a moment. And so you can see the middle there is Bosch Vault config server. The top is Vault itself. And so that's the Vault audit log that's just coming on standard out. And then the right is where we'll do some Postgres tests to confirm that this is real. So yeah, the Liberty Bell is great because it's like every software bug story ever. Nobody really knows how the bug got there. And they tried to fix it and made it worse until eventually they hauled the thing in the town square and talked about how great it was. And that's like every piece of software, not software we write, but all the rest of the software. All right, so here we go. I have my application. You can see the counters updating. It's great. And now I'm gonna take this user and hopefully it shouldn't come as a surprise to you that I can go down here and I can say like pguser, pgpassword, my mouse. And let's see psql. It's running on localhost and it's running at the very original port name that. And of course I connect to Postgres. There's my amazing test database. Shouldn't be a huge surprise that this does in fact work. Now let's redeploy my application. Okay, so you see that next request come through. And this is not deployed in a heap. So this app is technically down right now but if it was, you know, it's in canary mode. So if I had multiple instances of this deployed, we would still be serving traffic. But the train ride was only so long. I'm sorry. There we go. It's deployed. So now you watch that password and you'll see like, oh, wait a minute, that password changed. Okay, that's cool. I can still connect to my database. That's cool. Now let's just watch what Vault does. And look over here. Password still works. And this is the edge of credential rotation that hurts so much. It's really easy to generate a new password. So easy. What about the old ones? How does it get cleaned up? When will it ever be cleaned up? Vault's gonna clean it up. And I can configure that TTL. And so that top block right there, we can see, we generated that last one at 16.02 and it is now 16.05. So at about 16.07, we're gonna see an expiration come through in that Vault server block. And so you can imagine to operationalize this, you can have concourse deploy your stuff at certain intervals. You don't have to redeploy your stuff and rotate your credentials every five minutes, but you could as long as that timing was comfortable for you. Maybe it would make sense to do like a double thing. Being able to have something else expire your credentials for you and know that it's gonna be expired is a really powerful feature. The other nice thing that Vault is gonna do for you too is even if Vault goes down right now, or even if Postgres goes, oh, look at that, revoke please. Well, that's no problem. I'm sure these credentials still, oh, they don't. They don't. Because Vault did it for me. I could kill that Postgres server. Bring it back up. I could kill the Vault server, bring it back up. And Vault knows that it's generated credentials for it and they need to be expired and it will not stop trying until those credentials are expired. And I know because an auditing company told me it was true. So now, eventually what's gonna happen is these credentials that are up in this application that is currently still working are gonna also stop working. And it shouldn't be a surprise what's gonna happen at that point, right? My application is gonna need to be redeployed. So with any credential rotation, there's sharp edges. But the idea is that these give you sharp edges that you can automate away and that you can audit really easy. And if you're already operationalizing Vault, you might already be able to use this audit log right out of the gate. Vault supports lots of different audit backends, lots of different storage backends. So you have a lot of flexibility. And Bosch Vault is really trying to just get out of your way so that you can use Vault normally. If you wanna change data in Vault that are at paths inside, that config server is gonna expect to reach, that's no problem. You don't have to write special data at all. This can speak generic, any map that Vault returns, it's gonna be able to process. There's that second expiration. There you go. So any of you have any questions? Cool. So I'll leave you this. This is a really interesting abstraction that the Bosch team has come up with. And I hope that if you have one, or you're thinking about this, or you're online watching this later, that you will talk about that. Tell me what a bad idea this is. Tell me what a good idea this is. Let's do this together, because as Ben Franklin said, it is the first responsibility of every citizen to question authority. Thank you.