 Excellent, we're good. All right, let's do this. My name is Dr. Nick Williams and this is Justin and Kata and we're from Stark and Wayne and we have some ideas and some experiences some personal sacrifices and Some bold-faced lies. No, we cut those slides out. You have in the area of service brokers, so if you're going to be building service brokers or if you are wondering about what's the future experience of CloudFanity from our perspective, this is I think a really interesting topic. It is based on what we did for a thing called Dingo PostgreSQL, but more importantly, it's its ideas that you might go forward with as you are trying to expose everything you do in your company to your developers, not just deploying apps. So I'm going to tell a little personal story, personified in over here, good, with a couple of little examples and then Justin's going to rescue me from my disaster and then I'll talk about more about backups afterwards. So this scenario played out for me first with Bosch and as a variation, this has happened as I'm about to type it. So with the CloudFanity CLI, you can do this. You can target a staging environment in one terminal to do some staging activities. Meanwhile, in another terminal, you can switch. Why does it do that? Any theories? Why do we have no support people? All right. Your patience and appreciation for waiting for slides of transition is appreciated. Meanwhile, in another tab, you switch to production. To do a productiony thing. You go away for coffee, you come back and you think, you know that staging database? I'd like to delete that now. So what happens next? It's a sad day. That's what that is. That's the day where you have to find the phone number for the guy that might know how CloudFanity, how you did Postgres and obviously it's not just one Postgres, it's probably in a container somewhere on a machine and no one knows how it's done and you're just wishing you could solve all this like just to contain the disaster to your desk and not have to share it with your boss and the IT group and so this is an example of a bigger picture problem opportunity for CloudFanity, which is the idea around exposing first-class love for databases and services. In summary right now, we only create and bind and undo. That's it. There's no other experience that we expose. Because it's in the title of the talk, we'll look at the idea of what we did with undo and the idea of using the experience, what we can do now, what we can jam into CloudFanity as it stands right now with respect to doing an undo without using a CF plug-in. And obviously, you know, we're going to do a little bit of a visualization of how we do backups to support this and how we do layout just so that as you either for your own interest or how you might go about building services. So let's go to the top. So CloudFanity's whole goal is to love on developers. It's not really for admins. It's not really for execs. You know, you come to it because you've got some developer activities to get done. And as the business, you want to expose every possible thing that the developer might need in this experience. How do you get an email account? How do you everything you need for your apps? Because service brokers are part of the experience. But I have been working with CloudFanity for four years and I have watched the services API not change much in those years. So I do beg the question of whether we actually like databases or whether we wish they didn't exist. And just Cloud apps just talk to each other and state was just something on the wire being passed around continuously. And I don't know. It's very easy for me to bitch. So I will try to do this in a very professional and analytical way. So let's do this. There are 361 API endpoints as a bullet points if you go to the documentation. Many of them overlap with each other and 81 of those is related to services. Many of them are duplicates like they get me the list of services, service instances in this org, service instances in this space. There are a lot of duplicates, but nonetheless 81. We have the new V3 API. There are 51 current endpoints, four of which for services. So the new V3 API is not at all focused at making more experiences around services and databases. But you might think, well, we don't really touch those things. So let's worry about the CLI. There are 172 CF commands. And after I printed this slide, I did beg the question of how many of those might be plugins that I had installed because plugins are fun and I may have many. So it might be 150 of which 31 have the word service or services in them. But really, when you get to what can you do with the service in the standard cloud foundry certified way as you can create a service instance without knowing what that really means, binding to it, unbinding and deleting it. What you can't do is there's no commands for listing backups. There's no commands for looking at the database logs. Or there's no health of my database. And as the topic of the talk, there's no way to undo quietly and politely without mentioning it to anyone else from the mistakes you've just made. So another way to look at all this is to quickly look at what in all of the cloud foundry code bases, is there anything that's even useful if you were going to come and write services brokers? The TCU router, which is the sort of the new router that we have was sort of targeted at apps and as it's implemented now, it doesn't really scale horizontally to support lots of service instances. The Diego persistence layer was designed for apps, I think WordPress specifically. You could probably run other WordPress apps, but I don't know if they tested that. Logregator is app centric. When you do the metric on agent, it's all about sending a log for an app. So from a services perspective, you would need to send a log for every different app that might be bound. And plugins are kind of decoupled from brokers. So if you go and provision a service instance from a broker, as it usually you don't even know about brokers, you wouldn't even know if there's a plugin that gives you more features. Let alone how to find it, download it, install it and authenticate. So that's the status right now, perhaps put negatively. And I don't think it has to be this way. I think there's a lot of things we can do that's going to be fun. But for this talk, we're going to talk about Clifness right now. We want to jam in a feature called undo. You want to talk about this? Absolutely. Yes. Thank you. So how do we in our PostgreSQL service broker actually manage to put the undo feature in given these constraints that we have in the API and CLI that we've just had a look at. So this is the command that we've decided upon. Basically, we're passing the dash C with clone from parameter and the name of the service that we want to recreate in. The dash C is used in create service and delete service to pass in extra configuration into the service broker. Typically, we use it for configuration of the database itself that might want to come up or how, what kind of a cluster you want to bring up. But we've decided to use it as an action to perform, as a kind of a hacky way around the existing limitations. Undocumented feature. Undocumented feature. Good idea. All right, there is a problem, however, with doing this. We have a dependency on the name because that's the thing that the end user is interacting with. So clone from production is referencing the name of the database. However, Cloud Controller doesn't remember the name production once that mistake has been made and the database has been deleted. Also, the service broker itself is not aware of the name. On creating the service, the Cloud Controller only passes the UUID to the service broker and not the name of the instance that it's being created. So how did we work to manage to implement this? So the service broker only receives the instance ID. What we do is continue in an asynchronous fashion. So the broker has time to look up the name of the service in Cloud Foundry. And we have to cache that name, a mapping of the name to the ID external from Cloud Foundry so that we are able to look it up again once that recreation command comes in. I'll go a little bit through the interactions a bit more in detail so that you understand what you want to implement this yourselves. So this is basically the asynchronous part of the instance creation. The CLI talks to Cloud Controller and returns with a reference that it's going to continue asynchronously. The user has to continuously pull the Cloud Controller by CF service to find out when it's completed. And eventually, we hope it will complete successfully and the user knows his database is ready to use. So how does the interaction between Cloud Controller and the broker look? The broker receives the request to provision a new instance. And you can see it now. And as we said before, it only receives the instance ID. This is confusing. All right. No, not all right. Okay. I'll lick it. Cool. So it will only receive the instance ID. And we validate the request and return immediately to the Cloud Controller, indicating- I'm sorry, I sat down. Stay where you are. We will solve that, right? I sat down. Indicating that we're going to perform the provisioning asynchronously. At that point, we actually create the cluster that is the service. And we persist an initial backup of the cluster into S3. In the meantime, Cloud Controller, because we indicated that we're doing an asynchronous creation, the Cloud Controller persists its knowledge of the instance into the CCDB. So that once the- you've seen it. So that once the creation of the cluster is completed, we can then look up the name for that instance ID and persist that with the backups. So that we have a reference between the name of the service and the backups that we've created. And that's- oh my God. Come on. It's coming. It's coming. So- It's gone. No. Very bad. All right. So that's how we're able to map. You don't lick other people's devices. I mean, no one really expected that to fix anything. I feel like I've got a TV antenna and I need to- Don't move. Don't move. All right. So let's look at the next interaction. We've deleted the database. It doesn't take sarcasm. That's what it doesn't take. It's not- You're good. All right. I saw that. OK. They saw that. So this- Let's take a screenshot. This is the next interaction. So once we've deleted the database, how do we reconstruct it? We pass the clone from parameter, which gets passed onto the broker. That's what the dash c is for. We again validate the request and we look up in our backup storage whether or not we have a backup with a mapping to that name. So since we have it, we will duplicate that backup into the new instance ID, because this is a new service instance CloudController has created a new ID for us to store it on there. So we have to duplicate the backups. And that creates a independence between the original database should it still be running and the new one, which has new backups. Again, we return asynchronously and the rest of the process is basically the same, including, once again, looking up the name and caching it with the existing backups should we want to go through the same process with this created one. A little detail to have a look at is that we are scoping the references to the space. This is important mostly for security issues. This way we know that when a user wants to recreate a database, it has the rights to do it, because it is currently talking to that space. So by only allowing databases to be recreated in the same space, we don't need to worry about any kind of authentication that would be required when we want to cross over boundaries of spaces. And now I will do my best to keep this thing working. You're a good man. And so this is a very simplistic security authentication or authorization model. And over time, if you have other ideas of how you might map, who's allowed to restore into your champion? Oh, it's on fire. That's a problem. I can see how that's not good. All right, so backups. On here is the definition of the word daily. And I mention it because a lot of us prefix the word backups with this word. And you see it enough. And your DBA tells you this is what you have. And you think, oh, I guess this is as good as it gets. Who cares? I mean, I probably might be working here when I need to use these things. What do I need to worry about? But a daily backup means that at worst, in concept, you can have a 23-hour-old backup. Like a few minutes might be bad. And many, many times more of that. Accurately, as I have a friend, and on Twitter, he has announced that his backups, his daily backups, were taking 24 hours to cut. And I just think that's just beautiful. It sounds like it's the same thing, but it's now 48 hours since your last data was updated. And so whilst this might be what your DBA offers, it can't be good enough. We have time. I'll tell a quick story. So where my passion for backups comes from is my honest thesis. You've got all year to write it. You do not linearly write it throughout the year. You write it all at the end. And so really, I was two weeks into writing it with two weeks to go. And it was 1996, using Microsoft Word 95. Word 95, not very good at large documents. You'd break it up into chapters. And so I was working on chapter three. It's burned into my skull. I remember this very well. And I went to get another chapter. That file wasn't there anymore. I went to another chapter. None of them were there except chapter three. And I went to Windows Explorer. All the chapters were gone, except chapter three. And I thought, I can't believe this. I have no idea, but no one trusts Microsoft Word. So who knew what was going on? Luckily, I was writing to a shared drive that went off to a backup system with the central IT. Great. I can't believe I've lost, like, what, a day's worth of work. I couldn't believe it. So I went to the IT and he pulled out and he said, great. I said, you know, a day's backup, I can't believe it. He said, well, that shared drive is backed up every two days. I said, what? Every two days. I can't believe I've just written as like a seventh of the work I've done. Probably the good stuff, too. And he looks at the backup system. He goes, ah. Oh, that's not good. He looks at it and says, this has failed to successfully backup six times in a row. That's two weeks. And on the upside, it turns out that one of my friends had discovered my password, had logged in as me and moved all my files. So, we are not Facebook friends. No, it was very funny, it was hilarious. And so I have a fascination with backups. So a big believer in continuous archiving. So in Postgres, we have the right ahead logs, which are like, you take a base backup and every little 16 meg of commits send a chunk to the backup system or enter the hot replicas for anyone that apparently couldn't hear me a second ago. With MySQL, we have bin files and of course, Mongo just writes to DevNol so it's not a big issue. With continuously, so. Guaranteed quality. Sometimes it accidentally keeps the data around, but you know, you could just file a bug with them. And they'll fix that for it. I'll send a consultant out. So this feature may be somewhat limited to the system you've got, but it's so important from an end user's perspective, especially when we give back this feature. And Justin mentioned it. What we refer to, I described the problem as, what if you delete everything? Can I undo it? Well, what if you just wanna clone a production database to debug it? Or to use it, you'd sanitize it unless you run a function and everyone's credit cards gets billed a second time, but you'd sanitize it and use it for performance testing, for testing a feature, for testing a migration. We, you know, this is the start of trying to give back features to Cloud Foundry users so they can experiment, so they can do these things. One of the reasons I became PostgreSQLover, for example, was no one, as a previous company, and we didn't know that my SQL locked the entire database, or it might have been the whole table while it was adding an index. And it took an hour and a half to do that. Should they have cloned production and tested that? Yes, sure. But if it's really hard, you're not going to. So if we can make things easier, you make the right things to do easier, though people will do it. And unfortunately, Cloud Foundry doesn't really have an opinion about any of this. So I think it's something of our mandate is to have an opinion of what we wanna give to customers, or users. And to think about that, you might wanna do it with CLI plug-ins and whatever, and it's gonna require to implement these things. But it's important, and it's the future. Let people do this stuff as part of CLI. Let them do it as developers. To help and just to finish off, I wanted to find a way to explain some of these. So if you've missed, if you are not a database person and perhaps you can't quite visualize things, why do I not say what I wanna see? Oh, I know, perhaps I need to do this. There we go, boom. And we'll make this bigger, cause that's always better. And there we go. So I made a visualization of, right, that's right. So this is, whilst our implementation of Dingo PostgreSQL, perhaps give you an idea as you've got about building, you know, Cloud Foundry doesn't tell you how to build a service. That's why half the talks discuss it. It is still left to your imagination. But based on the features that we wanted to give users, like independent continuous archiving, constrained us. We had to give each user a dedicated database so they could have dedicated continuous archives so they could roll them back. We were constrained. And so this is two availability zones. That's the big yellow things. Five servers deployed by Bosch. And there's a white box. So the white hexagons are kind of capacity for some sort of visualization. How many things you could run on it. And we have a cluster running. This is awesome. This is also, I did this in part because I was buying a HoloLens. And it works about as well. It's really hard. You build it, it looks great. And then you put in a HoloLens and you don't realize you've got the sizes all wrong. And now you've got this 16 foot visualization of your database across the room. It's pretty freaky. But, and here we have the router. So I will be an app and I will send data to the leader. And as data comes in, I mentioned the wall log gets cut and gets sent off to the replica so that you have high availability. Different topic, but high availability, good thing. And more importantly is the one going up to the cloud. Yeah, you like that? That's the cloud. And so it goes up. And so because it's continuous at most, a user will have lost, any given user will lost at most in a disaster scenario. Not regularly, high availability. We've got replicas over there. You know, 10 minutes worth of data or 16 meg. And what this means is that we can recreate this thing anywhere across the cluster. We can move it. It gives us a lot of features. It was a really interesting idea. I mean, we will figure out in the future how much of this we can do with other databases, whether we just really benefited from Postgres' advanced features. But we can move things around. And we can also obviously really easily create lots of clusters. And then we get, you know, it's a cloudy day. So hopefully that the idea of, if you can imagine the data flow, you can now go back to what we were talking about, the idea that we can allow people to independently undo because they have their own backups that are really recent. And by pushing experience to the CFCLI and making it, you know, simple, it means that they can use in their CI platform, they can do it as part of development. They don't need, you know, it makes it much easier to do a lot of things. And that's what Cloud Foundry is for. So hopefully when we come back in a year's time, we can talk about some other interesting features to sort of, that we think users should experience. What have we got left? We have... I'm sure there's server food. Oh, I didn't do that one. You're a champion. What do we got? So super important, if we kill something, let's kill this server. Obviously, that's important. I did the failover, it fails over. That's about failover. But the disaster, yeah, the ability for a cluster to recreate and to pull those back archives, it completely internal to the user. And that's the point. We're trying to hide the implementation, but we want to give features. And sometimes you have to rethink how you would implement the whole thing in order to give those features. But to have an opinion, you want to have an opinion of what features your users deserve. Cloud Foundry doesn't have an opinion about that. It's an opportunity. You'd bring that opinion. So user experience. Users, they kind of expect that behind the magic of Cloud Foundry, and your esteemed efforts to run Cloud Foundry, that all their data is recoverable. And yet many of us put databases in production without having quite got around to doing backups and recovery. That's that ticket that just keeps dropping down the backlog. Or, and this is more funny, it's actually two tickets. Someone will say, we want you to do the backup story. And then we want you to do the recovery story. Now, I'm not sure how you test backups without being able to recover them, but I find that very funny. And so user initiated activities. So more and more, we don't want people touching Bosch or going to Bosch machines. There's no authentication, no authorization. We have all of that through Cloud Foundry. So if you're a DBA for some teams or an organization, you should come through Cloud Foundry to do your DBAing, not get an SSH keys and going through the back door. And we have that, we have Cloud Foundry. So bringing more of those experiences to Cloud Foundry is to my mind and it doesn't mind the future. I don't think anyone at Stuck of Wine disagrees, but it's a conversation we have with other people. Well been brainwashed. Good, excellent. Thank you. That's right. Real-time archives are really important because if people do this more often, they will become more annoyed with your terrible backups more regularly. So perhaps think about that in advance. And while we didn't really talk about what we would like out of CLI experience, perhaps a little out of scope, the ability for the CLI to expand to offer more features or even just putting them in. Like, you should be able to ask more questions about my services from the CLI and to be able to do more DBA and systems from there. So we have CLIs for full experience. That's great, but how do we automatically discover them? How do you automatically install them based on the version that's running? If I go and point at Pivotal web services, how do I get plugins? If I go to a Bluemix, if I go to a local on-prem one, how do I get that? Point-in-time recovery, all the things that your users would like. There shouldn't be like these little edge cases. Just give it to them as production features. And you may create a lot of backups and you might want to figure out how to clean them up. Just to finish up, so we're from Stark and Wayne. We're now 30-something people in six countries. We fancy ourselves highly. I mean, I wrote Platform Experts, but we fancy ourselves, yeah. We wear ugly t-shirts confidently. I mean, what else does that say? And so if you have any queries, we are downstairs, but this and many other topics. Clive Foundry is a really exciting future for developers. And sometimes what Clive Foundry does is the answer and sometimes there is opportunity for us to have to go that next step and give our users what they want. So thank you very much. Since we didn't stop talking, we didn't really offer questions. If you have any, that's awesome. Else we can, that's an excellent question and both you could answer. To not answer so much for Dingo, because there's not a talk about Dingo, but I mean, that is really important, right? Think about it, if you've made it just to rephrase the question as I think about it. Clive Foundry makes it easy to deploy 1,000 applications. You better make it easy so they can deploy 1,000 databases because that's often a one-to-one relationship. You better make it easy so you can look after 1,000 databases. You better make it easy to find that some subset have failing backups, or worse, all of them. And so instead of saying how, you know, yes, this is the scope of the problem, and yeah. How do we do it? We can talk about that, but I don't. Yeah, I mean, the whole thing, you should constantly think about bringing those experiences to people and coding for them as not just backlog tickets that, yeah, we should probably think about that one day. It's like, yeah, you are not gonna see this stuff. You have 1,000 databases, 10,000 databases. You don't get notified anymore. You just find out that people are doing this stuff. Back in the day, you probably were the person who got to build the database, and you gave it a name, and it was a little bit better than the last one, and now it's out of your control. DBAs will have the job of being experts. They will go out and help developers become better, instead of just doing grunt work. We will automate their job, in part just to make sure the automation is right. But yeah, notifications and systems operations. Automate all the things. Just should be provided to the users so they know what they can do with the service program for the C-parameter program. I think, I think long-term CLI plugins are a reasonable way to go, but to make that really useful and a good end-user experience, we need the discoverability of the plugins like Dr. Nick was referring to, so that the versions all match up and they have to be documented. And there should be some standard way of knowing when I've deployed my service broker, where am I going to get the plugin from to be able to talk to it? And we would hope that Cloud Foundry would also be able to take care of the discoverability and the authentication when talking to it, so it only allows the correct users. One, just along that line of discoverability, one feature that Cloud Foundry has with brokers is the ability to pass back a dashboard URL. Which sounds like it's for a dashboard, to be honest. Everything but URL and underscore is kind of giving that indication. But if you're of the mentality of how do I get around everything, that's it. So what else can you use it for? You could use it for an API. You could use it for discovery. So perhaps if we expand that idea of what else could the API be in a de facto way, we could start exploring that. We could have a plugin that knew how to look at dashboard URLs to say, hey, do you offer plugins? And I mean, the plugin catalog concept is documented, but we could go down that path. But I think what I, at the moment right now, it's trying to help broker authors think about, want to expose these things, write plugins. And then we can go, right, we've made a mess of that. Let's standardize it. Perhaps a V slash backups, a slash status of backups. This feature we wrote here could be used to test a backup. Did it work? Not just did it get uploaded, but is it corrupt? These are all important things. And again, no developer cares. You just want a system that keeps checking these things out. And also like the dashboard backups, dashboard URL that you can configure for service brokers. It doesn't even get passed back transparently to the end user. You have to look it up with the CF curl command and know the right path to find where is your service broker? What is the ID? Okay, I have the ID. Now I can look for that, Jason. Aha, there's the dashboard URL. So even that is quite verbose to get at it. I think you can't get it to crawl. You think, where am I at? CF target, CF service PG. Oh yeah. Awesome, so there you go. There's another way to get it. And so the dashboard URL typically has been used for doing some sort of authorization. This URL is just the starting point. You'd get redirected to login with OAuth and then you'd be shown your dashboard. And perhaps we can use that. That's how I would hack it in tomorrow. And, but more formally doing it. But if you do things formally, contribute to core, now it's like, well, how long is that gonna take to ship? Like especially if you're also trying to build product for distributions like Pivotal Cloud Foundry where users might be using older versions. And so there is this sort of irkiness where you think, I'm gonna slip it in this way. But we're formally doing it. I would rather people all started wanting to share these different standards of excellence and then we can go, right, we've made a mess. Let's put it into core. There are a lot of ways we can work around this problem. We decided to use this dash C to implement it because it's a fairly simple way from an engineering perspective. We love logs. Oh my lord, I love logs. So currently the solution, there is the Metryon agent and a lot of our brokers do this. Alice still doesn't, yet. So we did spike it. Metryon agent, you can send logs to a, send an event and it will send it to Logregator and Logregator will inject it inside the application logs. So that is a workaround, is to send the service instance logs, the database logs and whatever you'd like events back up through Logregator and they could come back out through apps logs. That is the current workaround to that. I could make up other workarounds. You could have a dedicated fake app for each service instance. That way you could just go to that and that way it would just be its logs. So yeah, I know no bounds to come out with dubious ways to experience things in lieu of putting stuff in core. Excellent question. No, no, that, I mean, as it's currently described there, that that relationship would get lost. So yeah, you would, you'd have to, yeah, we don't, but yes, you would have to keep looking. So as we do it now, let's get the trolley board out and don't rename stuff. Who does that? This is ridiculous. By the way, Crowley of the first thing, by the way, answer question, yeah, a background loop, which should be great. Crowley of the first thing, where I showed those intro slides, where we had the same name for everything. Just in case you missed it with all the flashy around. Go away. Do you really want to delete? There is an idea that sort of made simple by Cloud Foundry. We have spaces and let's give everything the same name as we go through those spaces. Might this be the example to suggest you don't do that? Like until Cloud, until the CF CLI stops taking a global state object as its description of where you're pointing, like the concourse CLI doesn't, the new Bosch CLI is gonna require to always specify what you're targeting. But until we have that, and perhaps even afterwards, unique names everywhere, that's the lesson. Perhaps then you don't have to worry about archives. We're done, hooray! Thank you.