 Hi everybody, good afternoon and welcome to on-demand data services with Cloud Foundry. Quick introductions, my name is Maria and I am a developer for the Cloud Foundry Redis team in Pivotal London. Good evening, I'm Simon Jones. I'm a developer in Pivotal London and I work on the on-demand service broker team. All right, so over the course of the next. Over the next 30 minutes, we are going to have a quick refresher on what Cloud Foundry services do for us and how Bosch fits into the equation. We'll then look at the traditional approach that we've generally followed so far to deploy Cloud Foundry services, move on to the path for on-demand provisioning and finally we're going to look at how we would go about developing a on-demand service broker. Cloud Foundry services are a very important part of what the platform offers to application developers. Just by pushing an application with CF push, I have really no way of storing state or even sharing data and sharing state with other applications. That's where services come in, so we can discover available services using the Cloud Foundry marketplace and there's a wide range of services available. For example, data stores, databases, caches, message queues or even user accounts for external services. Once we've picked our favorite service, we can go ahead and create an instance of it with CF create service. We can then bind our application so that the application can start using it with CF bind. If we ever want to change properties or move our service instance to use a new plan with new configuration, we can do so with CF update service. Finally, when the application no longer needs a service, it can CF unbind itself and then we can go ahead and CF delete the service. An approach that service authors have followed so far to implement the workflow that we just saw has been that of pre-provisioned instances. In that world, Bosch would come in, it would deploy Cloud Foundry, but it would also deploy the service broker VM and a pre-decided and pre-declared by the operator number of service instances. They would form a sort of pull, if you want, of available instances that would just hang around there until a CF user needs one. Then, when an application developer would need a MySQL instance, for example, they would type CF create service, then the Cloud Controller would forward that request to the service broker. The service broker would go away, figure out if it has an available instance in its pull, reserve it, and return successor failure to Cloud Foundry. There are some advantages to that approach. Mainly, the cost is fixed, so we know in advance how many VMs were provisioning, what sort of sizes, and we know kind of what bill will be paying. Also, the provisioning times are quite quick because there's no VMs to spin up at that point and no software to deploy. However, pre-provisioned instances cost money really doing nothing until they're actually claimed. They're not configurable, so they're all exactly the same. It's hard to scale them up because it means that you have to redeploy the whole ecosystem of the broker and the pull. Finally, it's near impossible to scale down safely. We think there might be a better way to achieve these goals. We think this concept we've called on-demand provisioning could help with that. This means that the application developer through the CFCLI creates and delete service actions would actually cause a deployment, a provision to happen, and then that service gets deleted. This really helps with our resource consumption because we know that until the application developer needs to make use of that service, there is nothing being consumed, and as soon as the application developer is finished with the service, again, there is nothing being consumed, so we don't have anything lying around waiting to be used. We can also provide operator freedom using on-demand provisioning. Operators can define plans and quotas so that different types of your service can be created and used by your application developer depending on their needs. We've also got the application developer freedom as they can consume those different plans. They can also, depending on what's been permitted by the service author and operator, take further configuration steps, and these all have to be pre-approved so the operator can be safe in the knowledge of what's happening. What does this look like? Well, the service broker has to ultimately take responsibility for deploying and managing the life cycle of service instances. Your broker now needs to deploy, update, and delete services as the requests from Cloud Foundry come in, and you probably want to use a deployment manager and integrate a deployment manager with your service broker. What did Cloud Foundry have to do to enable this new workflow where we're actually going out and provisioning? Well, Cloud Foundry implements the open service broker API already, and not a lot really needed to change with this. The end points already existed to manage a life cycle of a service, of course, as we saw, but what was missing was asynchronous operations. We previously had a default timeout of 30 seconds for calling the create service command, and that's a little bit too short for a lot of services to go off and actually deploy. It takes a little bit longer. The state changing operations like provision, updates, and deprovision can now be called asynchronously, and we have a last operation endpoint to go and ask about the state of that, so you might see services in the state of creating progress or deleting progress rather than created or delete or not there. That's actually everything that you would need to create an on-demand service broker or an on-demand service. Are we done? Well, we think we can probably provide something a little bit better in the way of tooling. Everyone going out and implementing their own version of deploying software would lead to a lot of duplicated effort. In steps to the on-demand service broker, this is an open source release that Pivotal have developed. This is a web server that implements the open service broker API, so Cloud Foundry can communicate with it. It receives a request from Cloud Foundry and it makes requests to Bosch, which is our deployment manager of choice, and we'll take a closer look at Bosch in a second. The on-demand broker takes all of the generic steps that are involved in making a Bosch deployment, updating that Bosch deployment, binding, unbinding to that. It can't know everything, so we want to be able to use this broker to deploy as many different types of services as possible. We need something with that service-specific knowledge. What we would want service authors to create is this thing called a service adapter, and that ultimately can create manifests for our on-demand broker to deploy the service instances, and it will manage the bind and unbinding as each service has a different concept of what an account might look like. This is available as a Bosch release, and as I mentioned, it's open source, the URL is here, and we'll include that in the resources at the end. Just to take a step back, we've mentioned that we're going to use Bosch as our deployment manager of choice. Bosch is currently used to deploy Cloud Foundry and a lot of the pre-provisioned instance services, so this may tend to pivotal. There should be familiarity, hopefully, within the community, but just to redefine that. It manages a distributed software life cycle, and it covers the packaging, deploying, running, and upgrading of software, and it actually manages the deployments using a declarative manifest, so you define what the desired state that you want your deployment to look like is, for example, how many VMs and what type of VM you want to deploy to, and Bosch will go and converge to that state. We did have to make a few changes, or we had some Bosch features added, which were very useful for the on-demand service life cycle. A lot of this was to do with how deployment manifests used to look, so we would include networking, subnets, and static IPs directly in the deployment manifest. We'd include a lot of infrastructure detail in every single deployment manifest, and this became quite complicated. In order for humans to be deploying Bosch deployments, we came up with tools like Spiff and Spruce to try and help us out. It becomes even more complicated with automation, where we're trying to figure out which IP ranges are going to be available for our new service instance. It becomes very difficult. This logic has been moved to centralised in the Bosch director, so we now provide a cloud config, and we have dynamic IP allocation. The cloud config contains the knowledge of what VMs and networks look like, and each of the deployments can make use of references in the cloud config, but they don't need to know about those things themselves. We also added Bosch teams to Bosch, and the reason for this is that a broker needs to use something with admin-like permissions in order to create and delete service instance deployments, but we only want them to be able to do that for their own deployments, so we don't want rogue service brokers to go deleting your cloud foundry or anything like this. It also allows multiple service brokers to exist harmoniously. The way that an on-demand broker is operated is important to service authors because the input that an operator might give needs to be considered when we go ahead and offer our service adapter, so we're going to have a quick look at that. So, when an operator deploys an on-demand broker, they will write a Bosch manifest just as they use to nothing you hear. In that manifest, they will specify things like, for example, where the Bosch director lives that they want to talk to, where does the cloud controller live that they want to talk to, and so on, but they will also define the catalogue for the broker, which includes how the service offering will show up in the marketplace and also what plans we'll configure and what quotas are applied. Okay, let's have a closer look at plans. So, these are operator defined. They include size, so VM size, disk size and configuration options, and they will be provided as input to the service adapter for the manifest generation, which we will look at shortly. Quotas are also operator defined. They can be applied per plan, and they can also be applied on the service offering globally. They're there to help the operator manage service usage, so they don't end up with way more instances that they were anticipating. As service authors, we don't really need to worry about them. They are enforced by the broker, by the on-demand broker, and the service adapter doesn't really need to do anything about them. Okay, so let's move to how we would go about developing an on-demand broker. There are basically two components that are required to make a service use the on-demand broker. First of all, you'll need your service packaged as a BOS release, or a set of BOS releases, and secondly, you'll need an adapter, so that's an executable, which is also packaged as a BOS release, and we'll look at both of them just now. The service release, first of all, so we saw just before that the broker must be able to instruct the BOS on what it is that it wants deployed in the format of the BOS manifest. That manifest will describe the topology and also the software release in BOS terms that we want deployed on each component of that topology. So the first step of authoring an on-demand service broker is creating that service release. That release will encapsulate your service. It is deployable by BOS, and the main consideration here is what are the configuration options that we want to open up to either the operators or the application developers, and then to ensure that these options that we want to expose make it into the spec file, or the spec files of that release. Okay, so moving on to developing the service adapter. We saw previously that this is an executable that lives right next to the on-demand broker and adds service-specific knowledge to the mix. Let's have a look at how it fits into the larger image of a service, a cloud foundry service instance in life cycle. So when an application developer goes ahead and types CF create service, the cloud controller will receive that request and will send it over to this on-demand service broker. That request includes two pieces of interesting information. The first one is the plan that the application developer chose, and remember that plans are configured by operators, so they contain some configuration there. Secondly, the request will pass on any arbitrary parameters that the application developer may have passed with a dash c-flug. The adapter executable is then expected to use the plan configuration and the arbitrary parameters and combine them to generate a Bosch manifest. The on-demand broker then will give that manifest to Bosch, and Bosch will go ahead and deploy the service instance. We have a very similar workflow with CF update. Again, the on-demand broker will get the current manifest from Bosch, and then we'll give it to the adapter together with the new plan configuration and the new arbitrary parameters. The adapter will then do what it needs to do to combine those into an updated manifest that gets given to Bosch and gets deployed as an updated instance. It's in the adapter's logic, and it's in the adapter's hand to decide the priority of existing versus new configuration. So a lot of the time you might want to apply the new configuration, other times you want to ignore it and just keep what you had in the original instance. And so we're going to actually have a quick overview of how we think some of that logic might look in our service adapters when we are generating the manifest. So first of all, we'll take a look at the arguments that we're going to provide, and these should hopefully contain all of the details that an adapter is going to need to create a manifest in any given situation. So the command is generate manifest. We're passing in the service deployment block, which includes a list of the releases and stem cells along with the versions of those that we're expecting to use in our deployment. We're going to pass in the plan. So this is defined by the operator, but a schema would be defined by the service author. So you know what sort of things you're going to expect to receive in this JSON, but you don't know exactly what the details are. And this will be things like the service plan and might describe what instance types are going to be used in your manifest, how many of them, and also what properties you want to include in your manifest for this particular plan and what values. We're also going to receive the request parameters. These are arbitrary parameters that are defined on the CFCLI by the application developer. Now, again, the schema for this should be defined by the service author. So you should know what sort of parameters you want to expect and what are sensible values for those. And you can, of course, ignore arbitrary parameters that you weren't expecting to receive. We've also got the previous manifest. So the last two arguments are only provided in the case that you are performing an update. So we get the existing configuration in the form of the currently deployed manifest. And that allows us to work out what the deployment currently looks like. And more specifically, if someone has previously configured arbitrary parameters, you may want to keep those in your current deployment. And similarly with the plan, this all plays into this slightly complicated bit of logic you might need to determine whether a parameter was set using an arbitrary parameter command or using the previous plan and whether that needs to be updated to the new plan. And the adapter chooses the priority of that. So it's important for the generate manifest command to always return a consistent response when given the same input. It's not quite a pure function because depending how you implement it, it might have some side effects. But the output does have to be identical. And that's used in comparison when we try to block incidental changes being made when an application developer has asked for an update to be made. And that might happen if the operator has updated the broker and included newer releases. And we basically don't want too many things to be happening at the same time. So we block the developer triggered updates. And it's up to the operator to trigger a platform upgrade to make sure everyone gets the new changes at once. So this is a sort of not quite example, a bit of pseudocode for what some logic might look for generating a manifest. So first up, you're going to create the skeleton. You're then going to interpolate any previous manifest properties, previous plan properties come after that. And then finally, you add the new plan properties, any arbitrary parameters before you print the manifest in plain text to standard out. And hopefully you'll exit zero. And the on-demand broker will know that it's been a successful manifest generation. So looking now at the CF bind service request and how that translates into cost to our service adapter, again, pretty similarly, the on-demand broker will receive that request and forward it on to the service adapter, having also originally first of all requested the existing manifest from Bosch. The adapter method that is called is on the adapter is called create binding and a typical request to create binding will involve connecting, for example, to the provisioned instance and creating a user. Other services simply look at the manifest to get admin credentials in return those. The binding ID is provided by the cloud controller and can be used as a unique identifier for the binding being created. This is quite handy also for the unbind service life cycle because the adapter might have to go ahead and delete that user later. So it's important to keep it as a mapping where required. Bosch VMs is again a list of the VMs that make up the service instance. They are important in the binding creation context in case the adapter needs to go and reach out to them and create users or create topics, for example, in the Kafka on-demand service. The manifest YAML is the existing deployed Bosch manifest. Some adapters use it to extract admin credentials or other information that's unique to the already provisioned instance. And finally, request parameters are arbitrary parameters passed by the application developer using the CFCLI. And again, they could be their use, for example, in the Kafka service instance to create custom topics instead of having a topic named after the binding ID. Many services will behave differently, and it really depends on your specific adapter what subset of these arguments you're going to use. And then we come to unbind. So when the developer has or the application has finished using a service, they're going to call unbind, and it's a pretty similar order of events to bind, and really it should be a mirror image. So whatever you're going to do in bind, you're likely to want to do in unbind but the reverse of it. So you receive the same arguments. That's obviously crucial because whatever you needed to connect to during your bind process, you may also need to connect to during unbinding. The binding ID is very useful, as mentioned, in uniquely identifying accounts, so you should probably, if you are creating a user account, name it after the binding ID so that you can then delete it during an unbind. So that's really it for deleting binding. There is a fourth method that's expected of a service adapter. It's called dashboard URL. It uses a lot of the same themes, but we're not going to cover that. So that's available in the documentation. So again, this is sort of everything you need now. If you implement these four methods, and then collocate on a release with the on-demand service broker, you're going to get a working on-demand service, hopefully. But again, we think we can do a little bit better than this. So your service authors, as service authors, you can choose any language which can respond as an executable to write your service adapter in. But the favoured language of the development team is Golang. And as such, we've made an SDK available at this URL. And that helps out with things like command line passing, which again is quite repetitive. The response serialisation, a bit of error handling there, and provides a really simple interface for service authors, adapter authors, just to fulfil that interface. And that allows you to concentrate on that service specific knowledge and really not care too much about how to manage command line parameters. And this does leave us with one missing part of the lifecycle. And that is a deletion of an on-demand instance. And we didn't mention that because the on-demand broker actually has all of the information required to delete a service instance. So you don't need to implement anything in your service adapter to fulfil this part of the lifecycle. Okay, so to ensure that you get the fastest feedback when developing and also to help improve the reliability of your on-demand service that you've just developed, we do very highly recommend that you test your components. Now there is really no right or wrong way of doing that, also because it depends so much on the specific service that you're building. But we've put together a couple of ideas to consider. So as a minimum, it would be a good idea to cover manifest generation, binding, unbinding, and dashboard URL with tests against the service adapter. These should generally test the executable, call the various subcommands, and make assertions on the exit code and also the contents of standard out and standard error. So any errors that come out and any produced manifests and bindings. It's a good idea to cover updates as well as original manifest generation. After that, tests in the service release are really helpful because they validate the effect that configuration options have on the runtime configuration of the instance. Once you've generated that manifest with your adapter, how are you sure that they impact and the effect that they had in provisioning that instance is what you expected. Moving a level out on the dotted blue line, you would like a bunch of integration tests to test that your adapter communicates well with any external components. So some examples are in on-demand service instances where bindings need to happen in collaboration with a service instance. That's a very good place to test that. Finally, it's good to test your deployed system with a set of lifecycle tests, which is the outer border. Here what we usually look at are end-to-end tests to validate the hubby path of an instance lifecycle. So I create service, I bind, read some data, write some data, read some data, unbind and delete the service instance. Ideally, you'd like the customer to be able to run those smoke tests on their own installation to confirm that they have a successful configuration. Fantastic. Now you really do have an on-demand service available. Just to cover what we've looked at today, we've got the concept of on-demand services as something that is application developer triggered from the CFCLI, but controlled through plans and quotas by the operator and the service author. We've introduced the on-demand service broker, which is a tool packaged as a Bosch release, which will manage the lifecycle of an on-demand service using Bosch as a deployment tool. And then we've identified that your on-demand service, whatever that may be, whatever that may look like, would consist of your service release combined with a service adapter that knows about your service release. So we've got some resources up here, some documentation, some example adapter releases. So I'll leave that up there so we can take some pictures. But the next slide, I assure you, does ask for questions. It uses a command runner to run that as though you would on a terminal or something. Yes. It's definitely been considered. So in terms of the actual lifecycle, it is tested by the on-demand broker team in Pivotal. That would be good to add, I think, but how much we could abstract might be interesting. I think we'd also like to do a kind of configuration confirmation so that rather than testing by results, so if you wanted to test the quotas we're working, you might have to go out and actually deploy too many service instances. It might be nicer for us to say, this is what we think your configuration looks like. Okay. Well, thank you all for joining us so late in the day.