 My name is Summon. I was a core contributor to the Bosch project for over a year and this is Dale. He's been a contributor to the Bosch team for more than six months now. And we are here today to talk to you about how you can make your Bosch deployments more secure through the use of one of our new features which we call variables. So we want to talk to you about the current problems that exist with your deployments, our solution to those problems, our solution to those problems, how you actually use it in your system, integrations that it may have with current systems you have, and the future roadmap and everything that comes with it. So let's get started. What problems are we solving exactly and how are your current deployments insecure? So when you create your deployment manifest, you have SSH keys, certificates, passwords, all in plain text. On its own, this is not necessarily a problem, but what happens is when you want to check it into GitHub or you want to post it on the Bosch Slack channel because you had some problem with your deployment, you have sensitive and secret information exposed in plain text and you just don't want that. Part of making your deployments more secure is making sure that the credentials you use are secure themselves. We've seen a lot of issues or in cases where people have used the default password for jobs and they've used the same certificates across the deployment for different jobs. So weak passwords and reuse certificates is another issue that comes up. Another issue comes when you want to manage your manifest across different environments. You have to, every single time you want to make a change, you're going to have to copy it over and you have to manage different certificates and different keys for every single one and it becomes really frustrating and annoying. So static manifest for staging environments, for different staging environments is another issue. And eventually, you're going to get to a point where you need to rotate your credentials. The default password has been password for more than a year and it needs to go. But swapping it all out is a very manual process which makes it difficult and error prone and you're likely to get more bugs in it. The fundamental issue is that there is no separation between credential and deployment management. The two are always fighting with each other. So for all these reasons, plain text passwords, the lack of ability to put them in GitHub or Slack, weak passwords and reuse certificates, having to manage the same manifest across multiple environments, the difficulty in rotating your credentials, and the fundamental lack of separation between credential and deployment management, we have had Bosch implement an API which we call the config server API and this API communicates with the config server which is our example implementation of a credential storage and generation solution. So this is kind of what we hope to use to solve the problem. So how does it actually work? What is going on to help solve all these problems that we've discussed? And to understand that, you need to understand how variables work. So a variable is a name parameter that has a value. Think like math class x equals three, y equals four, same idea. It can be used for passwords like setting the VM password or properties like setting the database URL on your Postgres job, but more generically it is any value that you do not want in plain text or one that is subject to change. In this same world, a variable can also have a type. Not all variables will have types, just some of them. And some of them, when you require a type, will require extra options to specify on them, like a certificate. So what this means is we're going from a world that currently looks like this to one that looks like this, basically. So we know that a variable is a name parameter that has a value, but how do you assign that value? And this can actually be done in one of two ways. The first is the value can be specified for variables through the CLI defined by the operator. So the operator will specify the variable and its value. Bosch will go through the deployment manifest and the runtime manifest and look for variables and match those values up on deploy and substitute them. The second way is when values are generated by the config server. And in this case, you define the variables in your deployment manifest, on deploy, Bosch will talk to the config server. And config server will either request your variable and config server will return the generated value or the value that's already stored within it. And once it generates a value for a variable, it also stores that within it. So that's great, but how do you actually go about using it? You will require the latest Bosch release and the new Bosch CLI in order to use this feature. You're going to have to set a couple of properties on the director job and you're going to need the config server release or something similar. The config server is going to, is either going to have to be co-located with your Bosch director or a separate instance group in deployment altogether. So the variable that you actually want to substitute goes inside the double parens. In this case, DB user, password, private key and certificate, those are all the variables that you're referencing in your manifest. And when you want to assign values through the CLI, so in this case internal IP and max connection, you do so with the dash v flags. So you're specifying the name and then the value. With the dash v flags, what you want to do, these are variables that are subject to change. They're not, it's not necessarily required that they keep the same value throughout multiple deployments. In the case when you're generating variables via the config server, you're going to have a variables block where you define the name and the type and then you reference it in your deployment. And on Bosch deploy, it will generate that value and substitute it in. So we understand that variables have types and that config server can generate those types. Some types have nested values and they require the use of a dot operator, specifically certificates, SSH and RSA keys. So let's look at an example. We've defined our variables block, we have the SSL variable of type certificate. We specify our options and config server will return a JSON object that looks somewhat similar to this. So when you reference it in your deployment manifest, you do the name of the variable dot the value to get the specific one. But if you have a property that's actually expecting that JSON object, you can just refer to the variable name. You don't actually need to use the dot syntax. Going back to the problem that we talked about earlier with reuse certificates across your deployment manifest, now because of the fact that Bosch can manage all these certificates and generate them for you, we can generate certificates and have certificate chaining. So there you have this example where you have the director SSL key. In variables, you define a certificate and this is actually the root CA that you're defining. And you can use that root CA to sign the certificates you're generating. So you can actually have separate certificates for each of your jobs and Bosch takes care of all of that for you. All you have to do is specify the variables and their options. Which is pretty great. So up until now, we've talked about variables that are deployment specific. All of them belong to your deployment. But what if you had a use case where you wanted to share your variable across multiple deployments? Because, you know, you want to share it. So variables beginning with a slash can be shared across multiple deployments. So for example, if you had the data dog API key and you wanted it across multiple ones, it's just a slash and then the name. Now, say that you're in a situation where you don't actually have a config server. You haven't deployed it. You can still actually use and generate variables and store them using var store, which is your local config server. So you specify a file and that file is going to be your lazy write and read file. You specify variables that you want used in your manifest and variables that are generated are written to that file. So if you have this kind of manifest where these are both variables that will be generated, with var store, the values for them will be written to the creds.yaml file and you can access them later. This comes in handy when you're actually deploying your director for the first time with create end because you have no config server. So you specify var store, you can pass some variables in through the command line and all the credentials will be written to it. So you can actually SSH into the VM later or the director as necessary. If you want to view the variables that are actually being used in your deployment, it's just the variables command, which will return the ID and the name of the variables. The variables that are returned here are only the ones that are generated by config server. So any variables you specify from the command line or through var store will not show up in this list. Also, this namespacing is subject to change in the future. We're probably going to add a GWT or something. So I wouldn't rely too much on that exact format. Now, all of this is great if you're using no credential system already or you don't have any plans to use another one. But what if you do? What happens then? So what we need to remember is that Bosch has implemented an API which allows it to communicate with config server. But really, all this means is that config server has some sort of API interface that Bosch can talk to. You can replace that with anything. You can, as long as it's implemented at API, you're good to go. So Cred Hub is actually our recommended production implementation of the config server API. It does encryption through HSM. And in the future, it's also going to be handling rotation of your credentials. So what this means is you can write your own. The world is your oyster or your Bosch cloud. You can, if you're interested in using Vault, for example, you can just write a config server in front of it and it can still communicate with your Bosch deployment. So you'll have a Vault-enabled Bosch deployment. So really, it's up to you whatever you feel like doing with it. So what's in store for the future and what's coming down the pipeline? In the future, you will actually be able to validate the types of variables you're setting. So in your release spec even, the properties that you set, you'll be able to validate them before the entire deployment goes through so you can get a faster turnaround and feedback on whether the operator-specified values are valid or not. And also, we're going to be making the certificates a little bit smarter. So the certificates that are referenced in each job, their common name or subject alternative names will be automatically figured out by Bosch, depending on the job they're referencing. So all you'll have to do is specify the certificates in your variables section and everything else will be taken care of, which is super useful. So I know all of this is great, but you're probably wondering should you use it? Is it tested? And the answer is we're actually using a config server API enabled Bosch director on pdubs with Credhub right now using variables. So it's working and it's in production. So you should be good to go and implement it on your own. To do so, you can use these links. This gives you a good overview of all the stuff you're going to need, the config server release, the API you want to implement to get it working, how to use variables, the Bosch deployment repo, which should give you config server and Credhub manifests. And you can also use Credhub if you're so inclined. So in conclusion, using variables with your Bosch director leads to more secure Bosch deployments, which is great for all the operators. We're going to have Dale give you a demo of how everything is supposed to work now. Hi there, I'm Dale Wick. And we're just going to switch over to switch over to mirrored mode. I think arrangement isn't showing up. Okay, so if you were at Denny Rupert's talk, or if you're paying close attention to someone's slides, you would know that Bosch dash deployment is a GitHub repo that makes it really easy to create any kind of deployment you want. So when you download it, it looks something like this. So lots of YAML files and a whole bunch of support directories for all the CPIs and so on. And we're going to just take a little tour of two things. We're going to start with Bosch deployment. And we're also going to interact with config server using a command line tool called CFUAC. So let's take a little peek inside Bosch.YAML and see where all the hidden variables are appearing. So there's a whole bunch of variables that have to do with network setup. And so we're going to be passing those on the command line to set up our network infrastructure to work in our favor. And there's lots of generated passwords like the Postgres and NATs password and so on. And there's also some SSL certificates for accessing the director. And all of those are specified what types they are and what their dependencies are down here in the variable section. So lots of passwords. In order to validate certificates, you always need a root certificate. So this specifies the root certificate. Is certificate authority true? And then you can refer to that certificate by name down below. So create it. We will go over here. And so for our environment, we're going to be using Bosch Lite today so that it's local on the machine. And we'll have a jump box user so we can SSH into the director and I'll demonstrate that in a minute. We need UAA in order to use config server. So you could have chosen cred hub, but config server is what we're going to choose. And these are using the dash O option which is ops files. So if you saw Danny and Rupa's talk, they explained exactly how those work. And if you recall, I mentioned that we'll be using the dash V option to pass in the specific network information that we have and our director name. So once we've done this Bosch create M that takes about eight minutes or so to do on my machine, so I've done it ahead of time. We're going to look inside creds.yaml and see what it generated for us. So lots of passwords generated using the algorithm. So again, I did this using the reference implementation to config server. Lots of certificates which are nice and wordy. Lots of exciting stuff that used to have to do entirely by hand. So that's nice to do. All right. So now that we've got our Bosch director set up, the first thing we need to do is start talking to Bosch. So we've looked at passing variables with dash V. We've looked at var store. That's what was generated into creds.yaml. So right now we're doing Bosch interpolate to extract out the credential instead of having to do it by hand and paste them. And we'll use that to talk to Bosch and then finally we'll use that to do a login to the special jump box account. So in order to target our Bosch director, we're going to use the Bosch interpolate command and you pass into it a path. So in this case admin password and it extracts the admin password out of the creds.yaml file and we're going to just set that into the client secret variable. Just like that. And there it is. So that's certainly a lot easier than cutting and pasting it and if it's certificate you have to like reformat it and all those annoying things. So Bosch interpolate, definitely your friend. All right. So now we're going to visit the jump box. So first we'll talk to the Bosch. And so I have two deployments on this director, Ardo and Zookeeper and we'll be working with Ardo today. And in order to get into the director we're going to use creds.yaml to extract out the jump box SSH private key and SSH is very picky about permissions and then we'll just SSH in and that's our local virtual box. So we'll log in there and here we are. If we do a net stat we'll find that there's lots of things running but of course I've turned off my DNS server. Anyway, there's lots of things running. So that's fine. All right. So you can use that to diagnose all of your deep fears of what the director is doing that you don't believe is the correct thing. But it turns out that it is actually doing what it's supposed to do. All right. So now we're going to spend some time interacting with a config server of a UAC. So UAC is a Ruby gem called CF-UAC. So you can just install that. I'm using it with Ruby 231. Doesn't work with Ruby 2.0 because Ruby 2.0 is too old. So I had to show Ruby to do that. So I'm going to extract out the UA certificate. Target my UA server. Get a token for the config server that has the correct permissions. So I'll do that now. That's all good. So now I can look at variables that are in my deployment. So if I do Bosch, Arto, VARs, the variables command, I'll see that I have several different variables. But me being an ops person, I like to sort of poke around and see what those values actually are set to. So we're going to do a get. So the first thing I'm going to try is I'm going to try getting it by name. So UAAC just has a curl command that you can use to work with the HTTP API. And so the data I got was there's one version of a variable by the name guest password. It's possible for them to be multiple versions. It keeps all the history of all the different versions. And this is the current value. So if I use the other syntax, you can get it by ID. You remember the variables command showed all the IDs. So ID 3 was for Web admin password. And so there's its value. So we got variables by IDs. So we used the VARs command to inspect the variables. Now we're going to set an absolute variable. So from someone's presentation, we had a datadog API, which we want to set the API key. It rarely ever changes. But when it does change, we can just change it in config server. And then the next time we do a deploy, all of our deployments will pick that up and reuse it. So we'll put that in and it assigns it to an ID. So it's ID 6. And if we were to reference that in our manifest, it would pick it up with print-paren-slash-d.key-print-paren. All right. So we're almost done. We're just going to try manually generating a variable with post. And we'll remove it with delete and then we'll regenerate it again just for fun. So there's post. And so it sets the log password for your log aggregator to this value. And then we'll use delete. There we go. It's gone. And if I post again, it'll set it to a fresh new PJ RAM value instead of that old value had before. IE 8. So there we go. So that's our presentation. Do you have any questions? So the answer is that it's a bootstrapping file. So it doesn't really contain any credentials the director needs to do any of its work. And it doesn't contain any information that the instances need to do their work. So you could put it in to Creadhub if you want for reference. But there's not really any motivation for the director to do that. Or an easy way. You could write a spanty script. Any other questions? I mean, that file just tells you, like I say, that local file is just for bootstrapping. So you can do whatever you want with that. You can throw it away. But it contains all of the secrets that you use to set up the deployment initially. So all of your active work is done using the Creadhub information. And the Cread information is still rendered into the appropriate config files and so on at deploy time. So it's actually not using Creadhub constantly. It's using it only when you're doing a deploy. So it's better. It's not perfect, but it's definitely better. Any other questions? If there are no questions, that means you're all going to go use config server or Creadhub, right? That's what's happening. Okay. Great. All right. Thank you very much.