 Peter Blum I work with Pivotal. I'm a platform architect. I'm very excited to be here and coming to Cloud Foundry Summit for a long time I remember Bosch days. They were the best. I guess that's because I'm a wacko and I like Bosch And with me I got Scott Frederick Yeah, I'm on the spring engineering team at Pivotal. I work on a product called Spring Cloud Services for most of my day job we're gonna feature that in the demo here and So I'm a platform architect. I work a lot with customers You know one of the things that always comes up is you know, how do we encrypt credentials? How do we secure things? How do we become compliant and whatever you want to be compliant in and it often ends up with credit hubs So we've been submitting this talk to spring one Cloud Foundry Summit all these different things one again Yeah, spring one again, and so it always seems to come back up on the radar. So actually this time we went through and we This time we went through and we Redid our presentation. So if you want more of the executive pitch go take a look at our previous presentations If you want more of the actual hands-on what you can do with credit hub, that's what you're gonna get today So hold on this version is fewer slides and more demo when you're welcome. Yep. Yep So so be excited hold on you know buckle your seat belt. Let's go By the way, buckle your seat belt in case we have a fire the exits are behind you This was a mandatory slide to put in here Just don't trample each other on the way out. Okay All right, so the story of credit hub You know, I think I think it's pretty easy to understand if you've ever deployed Cloud Foundry How hard it is to generate credentials for Cloud Foundry, right? There's a multitude of CA certs PEM keys all this stuff all over the place and You know, we've all probably use the open SSL CLI I learned from my Windows friends, you know That's not even existent on Windows. So if you're a dot net guy or a Windows guy, it's it's hard to create all those credentials And in understanding really what those credentials are for and how to configure them is hard, right? And so once you get beyond that though, then, you know, maybe one of your operators is such a unique guy He ended up taking a job for double the salary somewhere else and With him he took all your credentials So you lost all your credentials. They leaked. Oh, oh, how do we go back and reconfigure them all, right? So leaking credentials is really easy and then coming back and actually changing them is hard Right, so it's like, hey, we lost that guy And now we have to go retrain somebody to actually figure out how to Recreate and configure them all over again. So with that that's where credit hub came from, right? It's a centralized point. I really like this graph I think it really reiterates sort of, you know, that credit hub is is able to generate encrypt Rotate and all with access to control, right? So all around your credentials. It's able to control that access to them So inside of you know, we're talking about cloud foundry projects and all this You know, where is where is credit hub? Well in Bosch. There's credit hub Inside of Pivotal app Sorry inside of the cloud foundry application runtime. There's credit hub, right? And and there's even integration into concourse And so credit hub really is is becoming a key point to the whole entire ecosystem of cloud foundry the one last piece that we're missing here is the cloud foundry container runtime and That's coming along, you know, I think there's going to be some integration there with the kubernetes secrets So it's coming along stay tuned for that So with credit hub What is credit hub, right? Well, it's a it's a micro service, you know, and it talks to an authentication provider Right. So how do I know who is actually able to access credentials? Well, I have an authentication provider In most cases, that's going to be you a through OAuth 2 and then once I have Authenticated and made sure that, you know, Peter Blum is able to see credentials. Where do we store those credentials, right? We store them in a sequel database and Before we store them in that database, of course, we want to encrypt them So by default we use as 256 to encrypt them You can actually add your own encryption providers if you want on AWS. I've never done it It's like $5,000 for this hardware module. So I don't really have that kind of change throw around They don't pay me that well But you know if you're in Yeah, they should I know always doing these talks so How do we actually interface with credit, right? So we've actually you know, how do we get in there log in create these credentials rotate them and encrypt them? Well, there's actually three different ways to do that And really the first one, you know credit hub was originally designed for platform level credentials, right? And so the first one is is the Bosch config server. I want to emphasize Bosch config server not spring config server difference Different implementation too. Mm-hmm. So so anyhow in inside of there, you know, you might never Understand what the Bosch config server is but when you go in and you create a Bosch manifest and you use credit hub You're actually using an abstraction called the config server there So if you're using Bosch manifest you're using credit hub if you're using the credit hub CLI So we'll have a little demo actually showing the credit hub CLI So there's the credit hub CLI and underneath the covers the credit hub CLI is of course using the API wrapped around credit hub or you can do that with any rest client you want My favorite today is go lang but you know choose your flavor, right? So the credit hub API was kind of alluding to that a little bit You know, it's broken down really into, you know, the operations and authentication, right? So authentication wise you can use OAuth2 with UA or you can use MTLS with a cert and a key Pretty simple We'll have an example of using UA and logging in We didn't really want to get into the weeds with MTLS, but if you really do let me know. I'll show you that As far as the operations, right? They're really broken down into I want to just focus on the first two here But there's really two operations. There's the credential operations and then there's the permission operations So we can get set Delete credentials and then there's an extra one in there that I skipped over called generate credentials And so what that is is you can actually define what you want for credential But you don't have to say what the credential is, right? So I could say I have a password and I needed to have x characters in it And I can do all that and and I can ask credit hub to actually generate that for me And I never even know what the password is You can do that with CA certs You can do that with all these different credential types at the bottom So you can see the credential types are any value any password and user user is consisting of a user and a password Any JSON which we use in spring cloud services? Any certificate any RSA key and any SSH right and SSH I kind of had a question about that and really it's just RSA with the with a fingerprint So and then there's the their permission right so on top of these credentials. There's permission So who actually has access to these credentials? How do we manage that access, right? And you'll see as we move through here There's different ways of managing those permissions and then and then there's this really odd end point called interpolate Okay, a fancy word But essentially it's just for cloud foundry and we'll talk a little bit more about that But essentially, you know, we have this vcap services that has all the keys and we want to actually get values right out of those keys And so we've got is we've got an endpoint that you can just send this whole vcap JSON into and when I get out of it Is all my secrets, right? And so we'll talk a little bit more about that later on And of course I want to point out here. There is a whole API documentation right here Double-check that you ever have questions on the API And so first, you know, I'm going to start off with the with the infrastructure section You know going back to my core with Bosch. So let's let's show you sort of what that looks like And so what it really, you know, it simplifies manifests, right? I think that's I think for me This is one of the one of the best parts of it is is I have manifest that you know CF manifests that went from 5,000 lines to hundred lines, you know and it's all due to It's all due to credit hub right so you can see here I'm use I'm just generating a CA cert and then I'm putting that CA cert inside of my manifest And that's taking up a roughly 18 lines here versus here The way that I'm generating it is is I'm having credit hub generate for me And that's taking seven lines and then when I want to insert it I just put in one line right so it simplifies it down to one line and so with that What do I get well at the end of the day what I get is I get a manifest that I can then share with everybody in My organization whether that's your github However, you want to share it right and then I'll actually be able to relax the access to my Bosch director Right, so I can actually have other people log in run Bosch manifest and they're not going to see my Postgres DB They're not going to see my Oracle DB password, you know my admin password that's in there right and so that's a really a really great You know feature I think with credit hubs. Let me show you some sample manifests that I have out there So this is another example, right? So I deployed concourse one with what I call the legacy deployment Which is where we had all of the all of the secrets in there Pretty good. All right, so so you can see over here that in this one I've got my private keys, you know my public key all this stuff in here, right and To be honest with you. I don't even know where I got this stuff from so That's and that's I think how complicated creating credentials is right You know it's it can be hard to create all these credentials And so then on the credit hub side, you'll see over here all of my variables are defining what I want Right, so I've got this worker key. That's a type SSH. I've got a TSA host key SSH RSA key and then a typical password, right? And so these are these are my these are my credentials, but they're not really credentials, right? They're just references to them and then in my manifest I have them as references, right? And so what you can see here is that at the bottom I've got 282 lines on this other manifest with without credit hub usage and with credit hub I've got 145 lines and that's a small manifest, right? So you can see that you can see the value already with just Bosch manifest and so if I actually If I come over here, and I do Bosch Deployments So you can see here. I've got my secured credit hub secured concourse and my non-secured concourse And so what I can actually do is Bosch dash D concourse Manifest and so inside of here you're going to see all my credentials, right? So here's my here's my actual My postgres password, right? And so this means that anybody that has access to my Bosch director can see all my credentials But now if I take my other one Take my other deployment here. You'll see and inside of here My password is simply postgres underscore password So it's pretty simple. I think you get the idea So that's that's essentially how how the Bosch integration works So let's let's keep moving on so then let's talk about concourse itself, right? So if I have parameters inside a concourse, you know, I want my application to actually reach out to my databases, right? and I want them to To do maybe username credentials or or any kind of credentials inside a concourse Again, I can use credit hub. So there's this integration into credit hub when you deploy credit hub with Bosch You can tell it. Hey, I want to use credit hub to you to store my credentials And so here you can see I've got a very simple Concourse job not going to go into all of how concourse works What you can see here is that I've got a parameter Supersecret value and then inside when I'm using credit hub. It's once again, just a variable, right? And so again, this enables the sharing of pipelines and I don't have to check in those parameter files into vault or some external Configuration system to track them, right? I have all of that built directly right into my manifest or my pipeline and then of course I think that this is sort of an understated value with with concourse is that I Can have access to databases without actually knowing the username or password of that database And the way that is is because I just have the reference to it So you could have your DBAs Administering your database and changing passwords and doing whatever they want But I'm over here doing deployments and I don't ever need to know that password I just need to know whatever reference it is in credit. So I think it's a pretty pretty cool value I'm so I have another pipeline that I could show you but I think I think I want to switch it over to you Okay looking at time Yeah, so Peter's been talking about how credit hub is integrated with Bosch and Concourse for sort of an operator perspective now We're going to switch gears a little bit and talk about some things with Cloud Foundry Cloud Foundry application runtime Which is a little bit more user focused So if you've ever pushed an app to Cloud Foundry and bound that app to a service Then you're familiar with the vcap services environment variable. That's the way that Applications are are exposed to the services that they need to consume and inside of this vcap services block There will be the list of all the services that that app is bound to and inside of each of those services There's this credentials block and that's really the meat of vcap services So if you look at this example on the left This service is exposing a URI that's secured with basic authentication So you need the URI username and a password to be able to talk to this service If it were my SQL database, you'd have a baby a JDBC URL username and password So the fact that these credentials are all in plain text and vcap services Bother some some users who want those things secured a little bit more from the user perspective So now that there is credit hub in Cloud Foundry application runtime Service brokers can be enabled to store this credentials block inside of credit hub And once a service broker is credit have enabled the the vcap services will look like this example on the right Where the only thing in credentials is a single Reference to a secret stored in credit hub and what's inside of that credit hub secret is an entire JSON block So we're going to show a demo of what that looks like now So just to show you what this demo is we've got pushed to applications It's called fortune service and fortune UI are the two applications. I'll show you what those look like So this is a an example of like a back-end database serving application in this case this application Has this one endpoint, which is random and every time you make a call to that API endpoint you make that bigger You get one fortune as if it came from a fortune cookie So that's all this app does it's very simple and then the second app is a user interface application That every time you refresh the page on this application It's making one call to that back-end database service getting a fortune just displaying it So it's a really simple trivial application The services that are available to this app So we have three services here that are provided by spring cloud services, which is a pivotal product that implements the spring cloud Open source projects the one we're going to talk about is the service registry So if I go to the dashboard for that service registry service It's gonna make us log back in again Okay, so this is the dashboard for that service registry service provided by that service broker And we can see that the back-end service app is registered in this registry and the UI app is registered here So the way that the UI app knows how to talk to the back-end app is to look up in this registry What the URL for that app is so that it can talk to it so Better way than like hard coding the route to the back-end app into the front-end app So the only reason we're showing you all this is just to show you that these two applications have successfully bound to this service and They're they're consuming that through the VCAP services But if we go look at look deeper at this application if I do see if env on the fortune service back-end And if we go look at VCAP services We'll first find the MySQL service that's bound to that back-end app because it's fetching all those fortunes from a database We can see inside this credentials block that this service broker is not credit hub enabled yet So we see all these discrete credentials Displayed right here But if we go further down and look at the service registry service, it's just got the one credential It's just this single credit hub reference and kind of by convention the way these credit credentials are named is We give a name for the service broker That's implementing it and this is the name of the type of the service that the service broker is providing And then this is the GUID for the binding just to make sure that every Credential to store it here gets a unique name And then we just put credentials JSON at the end to make it kind of human readable and easy to identify So all three of the services that are provided by spring cloud services are all credit hub enabled So we just see those credit hub references there and I can even SSH into the container for this back-end application And if I look at VCAP services there, and I'm going to pipe that through JQ to pretty print the JSON You can see here that my sequel credentials are raw But even if you're doing CF SSH and looking at VCAP services there You're just seeing the credit hub references. So they're even secured inside the container So these both of these applications are spring boot applications So they implement the spring boot actuator endpoints and one of them Let's you show the environment that is exposed to that application So this is actually what's in memory in the application and we can see there's this VCAP block Where what's in VCAP services JSON is basically flattened into the flat properties and just to make that easier to read I'm gonna curl that here instead I think that'll work right there and I'm gonna pipe that through JQ again and sort it just so we can see all these properties really easily So we can see now in this VCAP services service registry credentials We've got these four discrete fields, which is the URI for the service registry and the OAuth information that the application needs to talk To that registry So all we're showing here is that the application in its memory space has access to the raw credentials While CFENV and CFF SSH and those other things that users might be able to do don't have that access to those raw credentials So how is that actually working? Well turns out Diego is doing a lot for us here So Peter mentioned when we're talking about the credit API that there's this really interesting credit API endpoint That's called interpolate and that endpoint is very aware of this VCAP services block So to that endpoint you can take a VCAP services block with one or more of these Credential sections in it pass that whole thing to this interpolate endpoint to credit hub And it's going to walk that data structure looking for any of those credit hub dash ref keys in there Whenever it finds one it's going to reach inside its own database Resolve that and what you're going to get back as a VCAP services block with all those references replaced with the raw credentials So that's a really powerful endpoint that credit implement it just to make this workflow as easy as possible And in the in this cloud foundry case What's actually happening is when cloud controller tells Diego to stage an application Diego is building that VCAP services environment variable block and Diego knows that it needs to pass that off to credit hub Have credit have interpolate it and when it's actually Starting up the application and building that application environment. It's exposing that raw VCAP services And this is really powerful because it means none of your applications have to change They don't have to be credible where they don't have to explicitly be going to credit by making that call themselves to resolve these credentials It's all done automatically. It's really nice. So spring apps Ruby apps go apps Whatever your app is it automatically gets the VCAP services as the raw credentials just like you're expecting today Yes There are still ways you can get in there So I think the probably the answer to the question if kids you didn't hear the answer the question is sometimes when you want to do CFSSH is because so you want to get like the database credentials so you can test the database connectivity yourself You probably in this sort of environment have to get an operator involved and have them give you some additional permissions to be able to get in there and see the raw credentials There are ways to do it we debated about rather to show some of those ways in this presentation We decided not to turn it into a credit hub hacking session, but Basically, you just have to have the right credentials You have to have the the credentials and we will actually show this in a minute But you have to have the credentials that that service broker used to log in to credit hub to store those credentials And if you have those then you can reach into credit hub like with the credit hub CLI and get them out that way the end the end goal of credit hub right is to obfuscate it right Somebody's able to CFSSH in Then they're able to get into all your containers right and and be able to see any application environment variables So the idea here is that you know Even if you get into the container you can't see any credentials Right your operators. Maybe they're lazy. Maybe they're great. Whatever but they didn't disable it for production And now they can get into that application and they can actually see those production level database credentials That's a problem, right? Even if they disable that then there's other ways to get in there and if there are cups, right? So we'll also talk a little about user provided services and how those work with credit hub So we'll keep moving. Yeah, like I said, ultimately they're in the application memory So this spring boot actuator endpoint knows how to mask things that it thinks look like they're sensitive But ultimately they're in application memory if you're running the application and have the ability to for example Debug the application then you're going to be able to get to them Eventually the application has to get rock credentials Okay, so that's all around managed services and service brokers so We said that your applications don't have to change to know anything about credit hub But service brokers do have to change service brokers have to be modified so that they're aware that credit hub is there and most of them are implementing a Flag in the service broker configuration to tell it Do you want me to store credentials in credit hub or just return them raw? So service brokers are going to have to adapt to this and implement that credit hub integration Okay, so the other user workflow is If you're familiar with user provided services in cloud foundry There are a way to provide credentials for services that are running outside of cloud foundry And the most typical example is you've got a big Oracle database that's been around forever It's run and managed completely outside of cloud foundry But applications running in cloud foundry need to get access to that database How you would typically do that in cloud foundry is to use a user provided service where you create a service instance And you're providing in the CF create user provided service command what these raw credentials are so you really want a way to be able to store credit credentials in credit hub and then expose those in a user provided service way also and The way that you do that is The other service broker that we have installed here is a service broker called secure credentials And this service broker is in the the cloud foundry org on github and it works pretty similarly Similarly to a user provided service so we can show you the command we used to create this so if you see this command we just did a CF create service and Said we want to use the secure credentials broker with that default plan and we wanted to Create a credential that has a password in it and we gave it the raw value for the password here and then in the In the UI app we've bound that service to the UI app So if I do a CF ENV on the UI app We're going to see in there That it's got this one credential. That's the secure credential. So it's working really like the spring cloud services cases it's just providing that for the user provided services use case and we can show an example here So kind of to Nick's question There is this credit hub CLI that Peter talked about and I can use the credit up CLI and I can log into credit hub in this case We've gone into the deployment of that secure services service broker We've pulled out the UA client ID that it's using to Authenticate to credit hub with and we're going to use the credit up CLI to log in with that service brokers credentials Something's changed there That all looks right to me me too Okay, we've been on network on comforts Wi-Fi. Sure. Okay the credit hub get see if we can get the credential out of there Yeah, maybe we're still logged in So once we're logged in we should be able to do this Get and pull the value of that credential out of credit hub. Is that gonna work? And now it's thinking we're not logged in because we tried to log in and fail Okay, I think it's thinking I think we're not logged in now We're targeting. Okay. We're targeting the wrong thing. Yeah, see it's all live guys There we go. Okay So now I guess we gotta go over this other window and So this is you know, we're logging in here to credit hub that's so this is the internally deployed for us credit hub inside of Cloud Foundry. That's already there for you and I don't know what Cloud Foundry version 0.36 I think CF release 3 6 And so what we're doing is we're authenticating with that and then we're actually listing credentials So here you can see all those credentials that have been created. So these have been created Yeah, go ahead Yeah, so we've got These credentials created by the secure credentials broker and then we've got all these credentials that were created by spring cloud services So the ones ending in credentials JSON here are the full JSON payload for vcap services But spring cloud services also uses credit hub to generate some usernames and passwords and client secrets So there's a lot more credentials. You'll see in here but this one from the Secure credentials broker. I have a hard time selecting stuff on Peter's laptop here. Yeah, select that for me There we go, so then we can do credit hub get dash in And get that credential Okay, that's the one the broker stored. Let's get this other one. Oh, yeah, there we go So you could see this for all the credentials, right? So if you're using credit hub for concourse Then you could come into that credit hub and you'd actually see the credit of that all the credentials that you'd be using in concourse, right? So this is sort of your you know, every single Credit hub instance, you know, you can use a CLI to talk to or you can use mutual TLS and talk to that That way did mowing on somebody else's laptop is kind of like trying to run a race and somebody else's shoes It's always fun You're doing great. So there we can see that we fetched that credential from credit hub and we can see what that value is If I were to try to pull one of these other credentials from in here I would basically get one of these created with p spring cloud services The CLI would tell me that I don't have authorization to get those because I'm not using the same credentials with the CLI that spring Cloud services used to log in to credit hub and store them to begin with So that's getting to that permissions level each of these credentials that you see here Has very fine-grained permissions on it that determines who's allowed to read those credentials Who's allowed to write them and who's allowed to change permissions on them? Based on the service broker that wrote them or if you're writing them yourself with the CLI That's all the content that we have here's a little plug for spring one conference That's going to be in Washington DC this year in September The most interesting thing on this slide is this discount code in the lower right-hand corner here If you're interested in going that's a code you can use To get a hundred dollars off for coming to this conference And with that we can take any questions. Are we doing on time? Yes No, that's because so the question is is credit hub a single point of failure is ha taken into account here Yes, it is and the reason is is all the credentials are stored in a mysql database that is ha Okay, that's number one number two is you can deploy right credit hub is made to be deployed with two instances So if you lose an instance, then you're good to go You're still still request will still be going through to get those credentials out As long as you have multiple AZs with your deployment, you should be all set just deploy a credit hub in every AZ Sure No, sorry. Yeah, go ahead It's the typical CF deployment mysql right so it's the same it's all stored in that that general mysql You could make it however you want right so I mean we're at open source Cloud Foundry Summit So, you know, however you want to deploy your CF SQL will work. So the CF deployment Bosch release will do it one way But if you want to go nuts with your own Bosch release and deploy your databases in I see what you're asking. How is the writing and reading done with credit hub? It's a good question. I can definitely table that and have the PM answer it for you Sure, okay here Yeah, when Peter and I first did this talk at spring one last December and we were actually surprised when we see if SSH then didn't see the raw credentials So then we had to start digging into these layers inside containers and garden to figure out how that magic is happening And it gets a little deep into Cloud Foundry internals. It's mostly when you do a CF SSH, right? It's going through the control plane of Cloud Foundry, right? And so they have access to block out credentials if you don't you know and so essentially It's it's all getting done by the magic in you're not quite into that garden container yet When you do CF SSH is kind of the high-level answer You could do that gladly You should still do something stupid You could put this spring cloud actuator endpoint in there and enable that without security and production if you wanted to But that's not credit hub is not a you know, it doesn't solve your stupidity. Sorry All right one more yep, it's totally possible. Yeah You know customers doing that now. Yeah question. There's a little project called spring credit I was like a spring mapping to the credit of API and there are some people who are doing that They're deploying their own credit have separately outside of the one in Bosch and the one in in Cloud Foundry application run time And then they're using it for stuff completely outside of these workflows. So that's really helpful for like operations teams, right? I know a lot of ops teams are using like vault and different things for that kind of and if you're trying to remain sort of Agnostic with with operations and what developers are going to be using with CF I mean you can deploy your own credit hub, right? Just with the open-source Bosch outside, right? And then you can use that for anything right you could use it for concourse You could use it for Bosch itself you could use it for any of your Bosch deployed things You can actually create a client that would right. It's just a rest API You can create a client that pulls all the creds out. There's a CLI for it, right? So there's all this all this ecosystem around credit up so and I mentioned spring credit I should also mention if you saw in the keynote last night in that really cool dot net presentation There's also a dot net client for the credit hub API. So with both Java and net you've got nice ways to write apps that talk directly to credit Yeah, I think we're out of time. We better wrap it up, but we can and we can definitely take more questions offline Scott's awesome