 Welcome. Today we're going to talk about Bosch CLI v2. I'm Shathirupa. I'm an engineering director at Pivotal. And before I became overhead, I used to be an engineer on the Bosch team. So last year Maria and I were here at Summit talking about some of the global networking features and how that helped streamline the Bosch experience. And today I'm excited to be back to talk to you about how we continued on that vision and tried to make some changes to make Bosch easier to use. And I'm Danny Berger. I first found Bosch when I was looking for a way to manage my services in the cloud. And then eventually I ended up at Pivotal and I'm now on the Bosch team. So what is Bosch CLI v2? The CLI as a lot of you are already aware is a way to talk to the Bosch director. So with it today, you can target the director, target a deployment and deploy. Once the deployment is up and running, you can manage VMs and instances, you can resurrect VMs, you can cloud check. Basically the CLI gives you a view into the state of the world as seen by the director. And it gives you a way to manage the state of the world and modify the state of the world using the director API. When I talk about the new CLI or the go CLI or CLI v2 today, what I'm referring to is Bosch CLI v2, which you can now get easily from Bosch IO or you can get it using brew. So a lot of you are already used to the Ruby CLI. Why would you switch to a new CLI? That's what we're here to tell you about today. We're here to share with you the story of why we rewrote the CLI and go. We'll share with you as part of that some of the community feedback that we heard about the user experience of the CLI today. So we heard some feedback around the output of the Ruby CLI not being very script friendly. We heard that some of the state that the Ruby CLI maintained was hard to manage and it caused errors where you targeted maybe the wrong director or the wrong deployment and got into a bad state. We heard that having the Bosch init CLI versus the Bosch CLI was confusing. And finally interacting with multiple VMs such as SSHing into them was difficult. And we wanted to address some of these things and provide a better user experience in the new CLI. Additionally, we also saw that SPF was still in use and it was mostly in use because manifest continued to be static. So to change anything, you have to go and edit the manifest manually at deploy time. Manifests continue to be hard to share because there's credentials embedded in manifests. And finally they're not modular. So if you had a really long manifest, building it up was hard and there was no modular way to build it up. And we wanted to address some of those things and provide a way for you to have manifest configuration be simpler. Finally, as you've heard, Cloud Foundry is trying to move towards being secure by default and we didn't want Bosch to be left behind. So today's CLI allows you to do things like skip SSI validation, which isn't necessarily the most secure thing. We wanted to address that by always asking for a CA cert in the new CLI and also providing other ways in which director and deployment are more secure and easier to get up in a secure state. Finally, we'll show you a concrete workflow that brings all these features together in something that you do hopefully on a pretty regular basis, deploying director with a secure configuration. So let's talk about rewriting the CLI in Golang. For this, take a step back and think about the Ruby CLI. When you're using it, some of the things you're often thinking about is commands are really slow. So a lot of commands take multiple seconds to run. There's a lot of inconsistencies in UX. The most clearing example as I shared was having Bosch in it deploy the director versus Bosch deploy other deployments. It was difficult to install, especially so on Windows, but even on other environments, if you had multiple rubies, managing multiple Ruby versions, managing your gem dependencies was hard and that was aggravated by if you had plugins and you had to install even more gems. In the Gol CLI, just by using Gol, we were able to cut down on a lot of the time that commands take to run. We were able to combine the Bosch init CLI and the Bosch CLI and provide a more consistent and discoverable user experience regardless of whether you're deploying a director or other deployments. We now ship it as a binary, so you can go to Bosch IO, you can get the brew tap or you can go to the GitHub repo and get a binary and install it really easily on different platforms. And finally, we provided a Golang director client, which is integrated with UAA, and so you can securely access the director API if you want to add additional functionality on top of the CLI. To get into more details around some of the changes we made, we're going to start on looking at how the commands have changed. It's not backwards compatible because we've renamed a lot of the commands, but the way we've renamed them, it should be fairly easy to understand what the new commands are. For example, we start using upload dash release instead of upload release, and that's pretty consistent among all the other commands. This kind of helps bring it in line with a lot of the other tools that people are familiar with using, both in Cloud Foundry and in the other tools around the world. And so this really helped us bring it in line and easier for people to onboard and understand what to do with Bosch. It also helps make the commands more readable. You no longer have to guess what's an argument versus what's the actual command you're trying to run. And we really think that'll help people bring into the community more. We also heard a lot about how state was tricky in the Ruby CLI. We know there can be situations where you're working in one environment in one tab, but then you're expecting to be talking to the director in another tab, like for development versus production. And we want to avoid the actual dental possibility where you might deploy something to production instead of your development. And to avoid this, we've introduced these new environment and deployment options to the command line. And this way you can be very explicit in terms of which environment and deployment you're using. And this helps make it both clear for readers who are trying to understand your scripts to know what's happening, but also as you go back through history, you can have more confidence in what was executed and where. Kind of related to the state thing is that you no longer need to have your manifest whenever you want to interact with the director for some commands. As you're working to debug things or trying to start, stop, recreate instances, now you can talk directly to the director without needing that manifest locally. It really helps save time and avoid having to keep entire copies of everything around. If it gets a bit tiresome for using these options every command, we also allow you to use them or set them through environment variables. This is particularly helpful for automation scripts or when you have things integrated and your concourse pipelines makes the scripts easier to read in that case. We also heard and experienced ourselves how difficult it was to parse the output of the Ruby CLI, especially when it comes to tables. It was really difficult to get, for example, just a list of deployment names. And we've really tried to improve that with the new CLI. If you are using custom scripts or even things like said or awk, the format now when it's being piped is much simpler. It strips out the headers and the footers and it's all about the data you care about. If you want to be a bit more specific, you can use the column option. And that gives you just one column that you might be interested in, or a couple columns. For example, if you just want the list of deployments, deployment names, you can do dash dash column name. And then you don't have to see the other details about releases or stem cells that they're using. The other option we now have is to format the output as JSON. This is really useful if you want to be doing more complicated transformations with that data. And it lets you pull in a lot more stuff and parse it with other Ruby scripts, Go scripts, build your own tools to build off the output. Along with automation, we heard that SSH commands were not the best for automation scripts. It was difficult to send in a single command to execute it out there when it had to set up all the extra keys. We've really simplified that in terms of now you can pass a dash C and give it a full command to execute on the VMs. For example, you could do uptime to see how long all your VMs have been physically up for. And it executes in parallel, you can target as whole deployment, or you can also do it against a single instance or instance group. Really handy. We also simplified the SCP command to make it match what a lot of users expect. There was always historically a lot of confusion around how that works and what format to give it. These have kind of been the most notable changes that you're probably going to run into as you start switching. Now we're kind of going to get into some of the new features for how you configure and work with your manifests. One of the big things we ran into and heard feedback about was how to make it easier for users, both new to Bosch or Cloud Foundry, but also to teams working with Bosch, how we can make it easier for them to understand what they need to do when they're bringing up a new deployment. People need to skim manifests and figure out under need to figure out which properties they care about for their environment. And when it's can be hundreds of lines long, that can be difficult and intimidating. So we really were trying to figure out how to make it easier. Passwords were another thing where you would probably either leave a default password in there, which is no good, or people may accidentally forget to set the passwords. And we really were trying to avoid that as well. And then our solution for that is more around variables. And this is the example of the terrible things we're trying to avoid in terms of replace me's and update me's everywhere. And variables are a solution for that. The variables help pull out significant configurable values. So it's easy to see that users should care about this IP address, or this password, or this network name. And it makes it easier to onboard because now Bosch knows directly which variables are important for a manifest or set of configurations, and can easily show those to the users. Manifests then can reference the variables wherever they're appropriate, either in cloud config, if they need to deployments, networks, anywhere that's useful. This really helps provide the network, like this is the example of what the networks look like, the variables look like. And in this example, variables are always surrounded by double parentheses. If you need to grab through a manifest, you can easily search for double parentheses and know that's a variable. There's also a nice convenient command in the Bosch to list them out directly. It's kind of similar to SPF, but it's much less dynamic in nature. And that's pretty intentional in terms of we want to make sure users are accurately representing the configuration values they need. And this network is the variable name. It would typically be a string, but variables can also be more complex. They could be entire hashes if we need to. For example, if cloud properties needed to be wholly configurable, that could be a variable in and of itself. There are couple ways to define variables. First way is the dash dash var option. For example, when you're running deploy, you can pass var or uptake cloud config. And this is just the simple key value pair where you could set network equals to a cubo. It's really good for one offs, but you'll all probably quickly find that you need to set a few variables at a time. And that can lead to some long command lines. So we introduced var's file. And this is a good way to import an entire file of variables into your manifest. For example, we've added a zone variable here and now zone can be referenced inside the manifest. So while var's can help make manifest less static and allow some user input, allow you to separate credentials out and specify them as variables at deploy time, there are still some gaps. So think about what if you wanted to add a particular job at deploy time. So you wanted to deploy something with occasionally having a jumpbox user from the user or from the OS conf release and other times you didn't want that there. Or what if you wanted to write a base manifest and share it easily and provide CLI users the option to add these bells and whistles if they wanted to at deploy time. Variable still doesn't solve that. That's where ops files come in. Ops files are YAML files that allow you to specify operations that you can perform on manifests. So in an ops file, you declare the type of operation that you're going to perform, which we currently support replace and remove. And you're going to specify the part in the manifest to perform these operations. So in the first example here, we specify a part to JSON key. You can specify more complex parts and you can identify specific instance groups to apply this to either by name of instance group or by index. A more complex example is with replace where it'll replace service account with a user provided value of service account in that war. The question mark at the end of part is interesting because it tells Bosch that it's an additive replace. If this part didn't exist in a manifest, it'll add this part. And if it did exist, it'll just replace it. Wars can be or ops can be used similar to wars by specifying it with the dash dash ops file as part of your deploy command. You can specify multiple ops files in the deploy command. And so you have this base manifest and you apply more ops files on top of it. They get executed in the order in which they're specified on the command line. So with wars and ops, we showed you a way to assemble a manifest during the deploy time. We showed you a way so that you can share manifest easily without fear of losing creds. And you can provide creds as variables. And this is good. But now that you've separated credentials and manifests, I know you're thinking it would be great if someone just generated this for me. And I wouldn't have to think about this at all if Bosch just did it for me. We agreed with you. So now there's this variable section that you can declare at the top of your manifest. And you can specify a name that you can later reference in your manifest with a double code, just like you would with a regular or double parent, just like you would with a regular variable. And you can specify a type. Currently, the CLI supports generating passwords, certificates, and private keys. So password in this scenario would just be a 20 character randomly generated string. A more complex type is certificate where you can specify options such as what is the CA it is signed with, what are the common names and alternative names that are going to be embedded in the certificate. And when you use the certificate value as a variable, it's a more complex object where you get the certificate, as well as the private key. So these values can be generated by the CLI, or they can be generated by CredHub if you choose to. So what is CredHub? There's a few talks about CredHub at CF Summit and I encourage you to check them out. But CredHub really quickly is an open source credential manager by Cloud Foundry. It stores and generates creds for you. It allows you to rotate creds. The good thing about CredHub and using it to generate some of these values is that then you don't have to be responsible for versioning credentials and you're not responsible for managing them, for storing them on a disk. CredHub takes care of all of that for you. The one caveat though is CredHub is integrated with the Bosch director and not with the CLI. So while you can use it for most of your other deployments to bring up the director itself, you want to use CLI to generate your variables and take care of the variables generated and the credentials generated. Using some of these new features, we were able to evolve the way you can deploy your own director nowadays. We have this new repository called Bosch Deployment and I really encourage you to check it out. This uses the VARs ops and the generated VARs to make sure you're able to deploy a director with a secure configuration and it takes advantage of shared configuration that other users can be using as well. Bosch I.O. has now been updated to refer to this repository as the canonical way to deploy a director and I'm going to go through an example of how you actually end up using the repo. One of our goals with the repo is to reduce the variability and how everyone deploys their director because then it brings around more consistency and we can know better like people are using Bosch in a secure way and we can be more confident in how our users are able to use it. If we go through a quick example of how to use Bosch deployment, the first thing you'll want to do is clone it from the Cloud Foundry org and we keep this repository up to date so it always is referring to a recent version of the Bosch release but also a recent version of the stem cell as well. This helps make sure you up to date with our latest security patches and secure by default but it also pulls in compiled releases which are something we've started using more and we take the time to compile the releases so every deploy you make you don't have to wait for those compilations anymore. Really saves quite a bit of time especially as you're rotating through directors in different environments. We'll start building a command here using the new command line and the Bosch deployment repo. You'll see here we use create and this is the equivalent of the Bosch init deploy you may be familiar with but embedded in the new Bosch CLI. The first or the second line here refers to Bosch YAML and this is the core director manifest configuration includes the director job, blob store and all that other traditional things and we're going to build on top of that. The next line is where we specify a director name. Historically on the manifest from Bosch IO this was hidden in one of the variables way deep in the YAML manifest but now it's really well exposed to users to understand that they should be setting this setting to something that makes sense. It's also the only thing that they need to set for the director itself aside from the environmental configuration. Bar store here is what tells the CLI it should be responsible for generating the dynamic variables specifically like admin password or the NATS and agent passwords and additionally the security certificates. This helps make sure each environment you're working with with the director has its own TLS SSL certificate so you can't accidentally talk to the wrong director with the wrong cert. Also be adding to that command IO specific configuration and this is where we see our first ops file bringing in this IO specific feature set. For example here we're using GCP and this takes care of adding in the references for network or project and you'll be able to specify those values through a vars file. If you ever have any questions about which variables a CPI needs or IS needs they are listed on Bosch IO with the full commands or you can also see the error that Bosch is going to list for you afterwards. We have sample CPI manifests for AWS as your open stack and several others so depending on your environment you may change this and add a few different variables to the command. Once your variables are defined you can then log into your director after it comes up. Here it looks a little different than before. Before you probably would have Bosch targeted it and now target is no longer a thing you can log in directly and we specify where the director is with the IP along with the CA certificate that we would have generated and then our secret password that was generated for us as well. We can then log in and do the regular upload release deploy sort of life cycle. You can also alias this configuration to an environment name something that makes sense like development so then you can just say Bosch-Edev and you don't have to specify CA certs and all those secrets every time. If you do want to make it a little easier we also have an interpolate command which helps pull out credentials like we were working with. Creds.yml was generated with a whole bunch of values in it and we only really care about the admin password and our director certificate in this case. So you could set an environment variable for the client secret and the CA cert and not have to worry about those options at all. Hopefully this gives you a kind of a good idea and gets you excited about some of the ways that you can use this sort of ops file feature set in your own environments and maybe envision a bit more on how I can work. Yeah so we encourage you to try it out. You can get it easily from Bosch IO. One of the things that you can do once you do have it is use some of these more powerful commands with easier output to script against. Be less likely to make mistakes by targeting a wrong director deployment. You can generate and generate your manifest and manage them really easily share them build them in a modular fashion. You can be confident that once you deploy things they'll be deployed with secure defaults and you have to access them with your CA certificate. And finally you can use Bosch deployment to have a director running with Credhub if you wanted to. The great news here is that you don't have to use a new director to use the new CLI. You can continue using your old director and use a new CLI with it and get some of these new features. A easy way to get started as I said is Bosch IO. There's the Bosch help command which will give you more details about what some of the new commands do and we really hope that you are excited to try it out and we're always open to feedback and questions on Slack. Thank you. We have a few minutes for questions if anyone has questions. Yes. So let's say we have an old old director that manages all deployments without cloud config. Is there a way to tell Bosch CLI, Bosch CLI convert all these things to cloud config and the manifest so it will automatically take all the deployment extract the cloud config part and have a nice deployment manifest and nice cloud config so I can... That would be very magical. I can point you to the go CLI client which I'm sure you could come up with the tool to do that but yeah. This is in theory it's possible right? In theory. Okay. If you figure out the way we would love to hear it. Yes. Any other questions? Yes. Yeah. One of the changes in the CLI is that the backup command no longer exists and the hope is that we'll be migrating towards the BBR Bosch backup restore tool you mentioned and there should be... I'm not sure if there are talks about that at this point like at summit but yeah there's definitely an upcoming feature in in the cloud foundry incubator world. Yeah. That'll be the replacement. Yes. In the back. Yep. Like import them from environment variables. There is another vars option which can import a subset like if you prefix a list of if you always had like my deployment underscore stuff you could import all variables which are prefixed with that name. If you do Bosch help on that command it should show that command that option name. I don't remember it specifically if that answers the question. I think they may get into that a bit more but the goal is that Bosch bootloader would be using a lot of these same manifests pulling that in to deploy things and so yeah they play well really. They play well together. Yep. I think that's right Adrian. Thanks. Yeah. Yeah. One of the things we did change is how we take care of the blobs and so several users have noticed this as well. One thing as you're switching between the CLI versions there is a reset release command that you can use and that makes sure it's switching out the blob store caching and all that sort of references. So I think that should help the situation where you run into issues switching as you're working with releases. I think that's it. We're on Slack if you need more. Thanks.