 All right, good morning, everyone. I think we can get started. Just as an intro, my name is Dan Yoner. I'm a product manager at Pivotal. And today I'm going to be talking about Credhub, which is a product that my team and I created last year. So a quick review of what we're going to be going over in the next 30 minutes. We'll start with the basics of why we created this product, what problems it solves, and how it does that. And then I have a quick mention of where we're going in the future, what we see in our roadmap, and then we can wrap up with any questions if we have time. All right, so let's start with the basic. Why create Credhub? I think the primary motivation, this may be obvious for some of you in the audience, but is the simple fact that leaking a credential can be catastrophic. Just grabbing a few headlines recently from the news we see here, there are varying ways in which this can hit. It can be bad passwords. It can be committing them to GitHub repos, people phishing them and getting access to them. There are all sorts of ways in which people can get access to your credentials. Now tying that back to Cloud Foundry, as I'm sure a lot of the operators in the audience know, Cloud Foundry has a lot of credentials, which introduces a lot of risk along with it. This is a partial dump of the credentials needed for a CF deployment. I cut it off short because I didn't want to make the text any smaller or go into multiple slides. Given the amount of risk involved with having all of those credentials, I think it's pretty plain to see that manually managing those is hard. It's a complex problem. So that leaves us with a really bad choice, which is we can either dedicate the time and the expertise to managing that, or we can maybe cut some corners and say that we don't have capacity or expertise and end up with a not-so-secure platform configuration. Obviously, we don't like that choice. Our focus in building CredHub is to bring automation to credential management so that in Cloud Foundry, it's easy to configure a secure platform. All right, so the next question we get to is, what is CredHub? So one of the defining protocols that we have is we want this to be the central point of control for credential generation, storage, rotation, and access control in Cloud Foundry. That's a really dense sentence, so let me just jump into a few of those points. We think a central point of control is really important because we don't want to duplicate this effort across all of the various components. It also provides good consistency throughout the platform. Pulling this into a single component, we don't have to re-implement that logic. And it also reduces the audit scope if, for example, your CTO comes to you and says, we're doing the security standard. Make sure that all of the encryption algorithms used throughout the platform meet this standard. It's a hard problem when you have 20 implementations of crypto. As far as generation goes, we want to reduce the burden of manually generating these things, just as an ease of automation for operators. We also want to get into the weeds on our side and implement some good defaults. So you don't have to worry about things like secure random number generation algorithms or certificate signing hash algorithms or anything like that. Storage here alludes to the fact that we need to encrypt any data that's sensitive before we store it. As I've said a couple times already, cryptography is a hard problem. We have a staff cryptographer who, at any point we think cryptography is easy, we go talk to him, and he sets us straight. So that's great. And again, reducing the burden of re-implementing this in each component. Rotation obviously is a really core component of managing credentials securely. Rotating all the credentials in Cloud Foundry is a huge pain. It's a painful process. So again, going back to focusing on automation to solve these problems. Our goal at a certain point, and we're not there yet, but our goal is to make this so that we can do this transparently and automatically. Logging is a really basic essential part of a secure environment, especially if you're going for a standard compliance. You need to make sure that, for example, anytime you access or modify data, you have a log of that event. And that can go to the extent of saying, if you can't log it, you shouldn't be able to access it. Sacrificing availability for logging is certainly a thing, depending on how secure your environment is. All right, and of course, access control. So because we're dealing with something like credentials, we need to have sufficiently granular access control so that, for example, credentials for application A cannot be read by application B. All right, so the final point that I want to make is our intention here is to solve these specific concerns specifically for Cloud Foundry. So we don't necessarily have an intention to make this the new generic credential management service. We're really looking at this from the lens of saying, what problems exist in Cloud Foundry and how can we solve those problems? So it's a little bit of a different view on how we're developing these things. All right, so I've covered the why and sort of what it is. So here comes the dangerous part where I try to do some live demos and hope everything works out. To set the stage, I'll go through and deploy a concourse deployment. Throughout that, I'll show you how you set up your director so that you can use Credhub, how that transforms the way that manifests look and how they're processed. And then after that, I'll go through and update the concourse deployment so that it can also pull credentials directly from Credhub. So let me switch over quickly here. All right, can you all see that? Good enough? Great. All right, so let's start from the director itself. So this is already deployed, but I just want to quickly show what you do to your director when you do your Bosch init deploy or your Bosch create nth to allow it to leverage Credhub, so it's really basic configuration. You just include the Credhub release. Once you've got that release, there's some basic configuration for the job, really straightforward stuff like how do you authenticate with Credhub, where do you store the data that's generated, and then last, how do you secure that data? And then once we've included that, there's a simple change to the director configuration, so we just have to enable what they're calling the config server integration, and then point it at Credhub to use that. All right, so I've already deployed my director because that takes forever, so now we're gonna jump forward a little bit and take a look at a current state deployment if you weren't using Credhub. All right, so if I scroll down a little bit, a few things pop out at me, right? We've got all these credentials in plain text, which may not be very surprising to many of you because this is just sort of how it works. If we look here, we've got authentication details for concourse, we've got a TLS private key there, which is great, and then we've got database credentials as well. So to drive home some of these concerns of why this is problematic, the fact that these exist in plain text means this manifests itself sort of elevates its own security level to the same level as the contained credentials. So you can't just grab this manifest, commit it to get and say, hey, colleague, can you update this, redeploy this, change this? Because the fact that it contains credentials means you have to control it. Another problem with that is you sort of lose control of these credentials, right? Like this password could exist in many areas. You're sharing this sort of offline because you can't check it into source control, so who knows who has access to this credential? It's hard to tell. From the certificate standpoint, and a media concern that jumps out to me, if I'm an operator, is I don't like generating certificates, I don't like managing certificates, or CAs, or any of that junk. So from a generation perspective, this is painful, right? And then finally, if I jump out of this and take a look at a really helpful Bosch command here, because I've already deployed this. If I do Bosch manifest, and just to reiterate, any of your operators can do this. Look, there are my passwords again. So the implication of that is Bosch, because it needs to be able to resurrect these VMs, needs the full state of the world. So that means in this case, storing the credentials in their database as well. All right, so let's look at future state, right? So we don't like any of that stuff that I just talked about. What does this look like in a GretHub world? So I can jump back down to my configs, and there's some simple changes here. So instead of actually including the credential, we just include a placeholder, which says essentially, when you need this credential Bosch, contact GretHub and ask for it by this name. We see that that's replaced for each of these. So one of the immediate benefits that jumps out is you can check this into a public GretHub repo. In fact, that's exactly what we're doing with CF deployment, which is this massive CF deployment manifest, and we can collaborate really well on that because there are no credentials. And then the other thing that I'll point out here is these certificates that are referenced are actually not pre-existing. So if we scroll up to this variable section, essentially what this is is a spec that says how you want these credentials to be generated when the deployment happens. So some things are super basic like password. I don't care what the value is, just give me a password and put it in. But then we can also do some more structured and more advanced stuff like setting up sort of like a pseudo PKI for this deployment. So in this case, I'm generating a CA certificate here and then right below it, I'm generating a certificate which is signed by that CA. So it's really, really simple stuff. Also, again, I'll say really powerful if you really can't see a deployment, how they've got dozens of certificates and instead of doing this out of band, you can just do it in your manifest. All right, so let me quickly deploy this and it should be very quick, I promise. All right, I didn't lie. So that looks like a no op but it did actually do something. So if we go back to our favorite command here for stealing credentials from a Bosch deployment and look at the Bosch manifest, we'll see here that we no longer have these credentials in the manifest because Bosch now doesn't need to store them because it knows it has this contract with Credhub that says whenever I need them, I'll ask for them. So I don't need to actually store the credentials. I can just store the reference to them. Yeah, so that's the sort of before and after of how you just do this with any old Bosch deployment. So let me move on to the next one or actually before we do that, let me just jump back to the presentation quickly. All right, so these are the problems that we identified. Manifest must be controlled. We saw how that could change. Generation is much simpler. Also, Bosch is no longer storing these and I didn't really touch on rotation but it's sort of a subtle variation here because the fact that we're storing a reference to a credential instead of the actual credential value itself means you can actually rotate those underlying values without changing the structure of the manifest. So it's pretty simple to rotate something because you just redeploy the exact same manifest, same references, just rotating the underlying value. And then of course we introduced logging and control as well in the Credhub application because it all goes through that, we have that control. All right, so let's move on to the next quick demo here which is a more recent update here in integration that we've done with concourse for pipelines. So I'm gonna go through and quickly deploy this and this will be a real deployment so I'll start it and then switch to another window. Actually, did I do? Yes. All right, great. So if we jump down the one update that we need to do here, you're gonna see a lot of analogs between this and the way that we do this integration with Bosch but the one update that we need to do here with concourse is tell it to use Credhub and tell it where Credhub exists and how to authenticate to it. So I've made that update here in this manifest and then a very quick second that will update and we can actually, let me just move on to the pipelines here. All right, so we're gonna do another before and after this time with concourse pipelines. So if we open up this concourse pipeline manifest, we'll see again, same problems and this should be pretty familiar. We've got credentials that exist, got SSH private keys, some AWS credentials, the worst of all. So obviously you're not going to want to check this into your public at Hub repo and if you do, I can guarantee somebody will spin up a lot of machines in very quick order. All right, so it looks like our deployment succeeded here which is always a good sign. Let me set this pipeline. All right, so I set that, one of the things that jumps out at me of course is the fact that in the diff it's showing all my credentials again which is not a good sign but that's how it exists in the manifest so that's life. So we've got the pipeline here. It looks good but let's look at how we can update this so that we don't have to do the same credential problem with concourse. All right, so I'm opening this, this is after I've updated this to use the credit integration. We'll see a really similar syntax. We implemented this around the same time so it's a pretty similar experience. I can reference here these client credentials and the private keys are gone and the AWS credentials are gone. So all of this is gone. We can check this into GitHub which is the favorite thing to do with manifest. All right, so let's go through and very quickly before we set that let me show you the analog in concourse to the Bosch manifest command that we did in Bosch. So if I look at this because this is set I can get pipeline and here any operator that has access to concourse can see all my credentials again. We don't like that. So let's go through and set this updated manifest. All right, so good sign already. We got rid of these credentials. We're only seeing references. So that's great. Now that we've set this, let me start that to see that it works and let me rerun that get pipeline command. All right, so again here get pipeline will not show the actual credentials because concourse no longer store those in its database which is awesome and it's really straight forward to see there. All right, so the last thing here is hopefully this will go green. But we don't have to wait for that. Well, you can trust me, right? I promise that it works. All right, so let's switch back to one last use case. All right, so quickly before we move on problems are much the same in concourse as they are with Bosch. The one thing that's different here is we don't call out generation as a problem because for the most part the credentials that exist in concourse pipelines are things from external systems. So things like AWS credentials which wouldn't necessarily be generatable in the same way that Bosch credential can. All right, so this leads us to the last use case that I'm gonna cover today. So this jumps up from the infrastructure layer of using Bosch and concourse to the platform itself. So the interaction here is a little bit more complex and it's not necessarily evident if I would do a demo of this. So I'm just gonna go through a few slides of the overall architecture of how this works and some of the interactions and how they've changed. So this focuses on the use case of service binding credentials. So we've received feedback from a number of users that they have concerns about a few different areas of this workflow. The primary one is the way in which credentials are transited between various components of the platform and if we jump to the current state diagram, we can see that here. So operator or actually in this case an app developer initiates the bind request and that goes to the service broker which will go through and provision the service and generate credentials so that the application can access it. Once it does that, it passes it back to CAPI which stores a copy of the credential in its database and then sends that over to Diego which needs to put it into the application's environment. If we look at this with a skeptical eye, we see that the credential isn't strictly required in CAPI or Diego to make this happen, right? It's needed by the service broker and it's needed by the application but it shouldn't necessarily be needed by others. So this is the alternative workflow that we've envisioned for a credit hub assisted delivery of service credentials. It's much the same. The main difference here is with the interactions of how the service broker sends back the credential and then how the application consumes it. So I'll focus in on two different interactions here. The first one here shows the provisioning of the credential so the service broker when it generates that credential instead of sending it back through the platform it sets the credential in credit hub and when it does that it can provision access control for the application for which it's performing the bind. So we get a more strict and explicit access control when we do it that way. And then also once it's done with this instead of passing back the actual credential value to the platform it just returns a reference again to the credential. So it sort of removes some of those concerns about storing the credential in the database because it's just a reference now it's not a big deal to store it in the CCDB database. All right so then the second interaction is how we consume that credential by the application. So we have two different options with this interaction depending on whether or not you're concerned about the idea of having these in the applications environment. We've done some research and it seems sort of split some people care some people don't. So we have the ability to do it either way. So this interaction shows if you don't care about those credentials being in the application environment. So the structure here is that we have this we have this component called the excuse me. We have this component called the launcher which exists in the application's container. And it essentially like when the application starts it creates some context and puts that into the container. So we've updated this so that it will actually reach out to Credhub as the application starts to get those service binding credentials that it needs and then it puts those into the environment. So some of the benefits that we get there are the fact that the launcher uses a mutually authenticated connection to Credhub to request these and it also does it with the identity of the application itself. So we have that strict access control of who can get access to these credentials. And then this is the other option, right? So if you are concerned about these credentials being in the application's environment instead of using the launcher to transparently enable this process you can actually just say we accept the responsibility of updating our applications. So they actually reach out and get the credentials directly. So in that case it's not stored in the application environment. The application just reaches out to Credhub directly to get its credentials. And we've done some work specifically with the Spring Cloud services connector team so that if you're using Spring as an application framework you can actually do this pretty transparently and automatically. All right, so that finishes the use cases. I'm gonna quickly touch on here. It looks like we have a bit of time on the sort of next step wish list of what we're looking at for next use cases. All right, so primary item at the top here of course is rotation. I think that's a concern of most everyone that is concerned about the security of the platform. It's a really high value thing. It's also very complex. So it likely will take a while but we are gonna dedicate time and resources to enabling this in an automated way now that we've got some of the groundwork of integrations rolled out. And then after that we're gonna explore some additional application credential management workflows. So that's gonna be very similar to the service binding credentials except expanded to more generic use cases of anything that an application needs to store securely could be stored in Credub. And then some additional things about pluggable encryption provider interfaces giving more choice around what you use to encrypt your data, encryption as a service and pluggable credential types. All right, so wrapping up here, final ask for everyone in the audience. If these use cases are compelling to you, it would be great if you could get involved. That would mean providing feedback either to me directly or jumping on the Cloud Foundry Slack in the Credhub channel. You can provide some feedback to us. Also check out our repos, submit any issues or pull requests if you're generous. And I'll be around the conference. So let me know if you have any questions. Thanks. All right, I think we have 10 minutes for questions. Does anyone have any questions? Oh, that's a great question. That is certainly the most pressing thing for us to look at. Let's see, I've got two mirror. So much success. All of our releases are compiled. Any other questions? Yes. So we've done an internal audit. I've been told that we can't, we've done, let me rephrase, we've done an audit with an external penetration tester. As these things go, it's sort of sensitive to share results of pen tests, but I can say that there weren't any major issues. It is, yes. I would say it's an open source project, so if you wanna do pen tests, that would be awesome. Yeah, I don't know deeply the politics of, or the legal implications of sharing pen test results, but they exist, at least in the US. Yes. Yeah, so the way that we've structured the application is that we have this interface for encryption providers, and the ones that we support right now are an internal AES-based encryption provider, as well as a client for a Luna HSM module. We plan to make that more generic and pluggable so that if you have any other encryption provider like any of the various IaaS key management systems or a TALIS HSM or something else, you would be able to create an interface for that. Specifically, Vault, I'm not sure how that fits because it's more of a all-in-one solution, but it's something, certainly once we have the interface, we can explore it, seeing if that works. Yes. Yeah, yeah, you can, so let me jump back to this. So there are two different ways that we can do that. So, variables is the command. So the structure of Bosch in how it stores credential references is it has this concept of variables. So it has a list of all the credentials that it's referencing for a deployment, and then it also has a strong tie to the version of the credential that it's using. And this is useful where if, for example, you're rotating a credential, but you haven't initiated a deployment, it doesn't actually leak the change in value into your environment without you actually initiating an action. So this is one way to track it also in Credhub because we have this strict logging requirement. You can also query just based on change date and you can see all the credentials that were changed as well. Right, yeah, yeah, so that would certainly be restricted who would be able to see what has changed. I think I see what you mean. So you're saying because we have strict control on namespacing, you wouldn't be able to rotate a credential. So it sort of depends, it's an operational struggle for sure, but you could, if you granted access to all the credentials for a deployment, have the client that your concourse worker is using, also be able to access and rotate those things. So they exist in sort of separate namespaces, right? There are credentials that exist just for the purpose of concourse pipelines and credentials for deployments. And if you would grant access to the concourse worker to your deployment credentials, you would be able to rotate them through like a sim link or something like that, not yet, but yeah, we haven't explored, like I said, rotation deeply yet, but once we do, I'm sure we're gonna get into a lot of those gotchas and apply fixes for that pain. Any other questions? Yes, yeah, yeah, so as far as production readiness goes, the most mature of the use cases is the Bosch deployment case. We've been using it in production for maybe like six months, it's been quite a while, and we introduced it at Pivotal and our commercial products two versions ago, so that would have been, actually, that would have been six months ago, so we've probably been using it in production for like eight months, yeah. The concourse integration uses similar structure, so we're confident in saying that that will work. They recently released the integration in a major version in 3.5, so I think that might be the latest version that they have. As far as the other things, it's always hard to say with roadmaps when things are going to come, but we'll focus and do as much as we can as fast as we can. Any other questions? All right, that's it, thanks so much for coming. Thank you.