 Hello, everyone. I guess we'll get started here. My name's Steve Wall. I'm from the ECS team, now a part of CGI. My partner here is Steve Greenberg, and he is with Resilience Scale. And we're going to be discussing active multi-cloud today. And it's ephemeral cloud foundry instances. So there's been a lot of talk in the conference about multi-cloud, and that's running cloud foundry on many different clouds. And what we're going to be talking about here is active multi-cloud. So that's running many instances of cloud foundry, and them all appearing as one. So at the heart of multi-cloud is Bosch. Bosch is really the enabler. So Bosch allows you to deploy cloud foundry to a variety of different eye-asses. So with Bosch, you can deploy cloud foundry out to several different eye-asses. And from an end user perspective, they don't know what eye-ass or service provider that cloud foundry is running on. So that's kind of a multi-cloud situation. So you've got cloud foundry running in multi-different clouds. So active multi-cloud is where you have your multiple instances of cloud foundry running on multiple different service providers, but from an end user perspective, it just looks like one. So what are some use cases that enterprises or companies would want to use of active multi-cloud architecture? One of those use cases is resiliency. So if a service provider had a hard outage, so say Amazon went down, it had a regional outage. And your application, in your instances of cloud foundry, went down in that environment. You could simply stand up another instance of cloud foundry in a service provider that was fully healthy. Now, you might see some small degradation in service, but your service wouldn't go away. And then you quickly stand up another instance of cloud foundry in that service provider that was fully healthy. And then you are at full capacity once again. So you have good resiliency when you go to active multi-cloud. Another reason that an enterprise may want to go to active multi-cloud is geography. So say an enterprise had customers distributed globally. They may want to stand up a variety of cloud foundry instances globally. So that they have cloud foundry instances that are located close to their consumers, and they could do something like latency-based routing. So your application is globally distributed, looks like one application, but from an end user perspective, wherever they are in the globe, they get the same type of response from your application. So another reason that an enterprise may want to go to an active multi-cloud architecture is workload distribution. So say, for instance, that some service provider was running a special on the cost of their VMs. You may want to shift your workload over to that service provider for a period of time. Or if a service provider had some high-performance compute you'd want to take advantage of, you may want to run more workload on that service provider than on another service provider, but you still want to have your application distributed on multiple clouds for the other reasons, like resiliency or geographic reasons. So how do you actually achieve an active multi-cloud situation? So in how to describe that implementation, I'm going to turn it over to my partner here, Mr. Steve Greenberg. And he's going to go into some details there. Thanks. All right, so a little reality check. This is not a trivial problem. Certainly not a trivial problem at 5.30 when we're standing between you and getting out of here for the day. But our intention is to give you kind of enough information, talk a little bit about how we've actually solved these problems, and make it easier for you to actually get there if that's what you need to do. All we're really doing, the secret, is applying things like 12-factor app principles, microservice architecture principles. But we're trying to apply them at the Cloud Foundry level. So let's start with a little bit of baseline, and probably one of the most important things when we're trying to actually create an active multi-cloud. The good news is, all the building blocks are there. We've done this for customers. It's actually not that hard. First thing we need to consider, though, is separation of concerns, the power of isolation. We know the power of isolation when we're building applications. We know why that's so important. If we can apply those same principles to our Cloud Foundry instances, we can end up creating ephemeral Cloud Foundry instances, which allows us to create an active multi-cloud pretty easy. So let's start with the really, really simple problem. In this scenario, we've got multiple Cloud Foundries running. We've got users throughout the world. And they want to go to one location. Routing, how do we actually route traffic into these different Cloud Foundry instances? So some user gets on. They go to cloudfoundry.org. It's actually pretty trivially easy using regular internet available technologies to figure out where that user is and route them to the appropriate location, based on latency. There's a lot of solutions for this. Route 53 is a really good one. There's latency-based load balancing solutions. That's the really, really simple use case. So where does this get to be a hard problem? It's all about the data. The data is the hard problem. Our applications can be stateless, but if they don't use data, what are they doing? So let's take a look at what data sources we really have to contend with if we're going to do an active multi-cloud. And then maybe we can reason over some of those data sources. So there's the Cloud Foundry data sources. So things like the Cloud Controller, maybe the Blob Store, probably UAA, where all of our users and roles and orgs and spaces and stuff are. Those are all of those Cloud Foundry type data sources. There's also app-specific data sources. So obviously, our apps have to have data. If they don't have data, they're pretty much not doing anything. But let's actually go back to Cloud Foundry and start talking about some of the things that we've seen. The reality is, if you go into any enterprise customer, we can take a look at our little friend UAA over here. And you're going to be hooking up at least part of that to some enterprise system. And usually, that's authentication. That's going to be LDAP. It's going to be Active Directory. It's going to be a system that already exists. The other mistake that we've seen customers do is they take the UAA that's in Cloud Foundry, and then they use that for tokens for apps. And they apply it to their application level security. So all of their end users are registered in UAA. And all of their tokens are coming from that same Cloud Foundry UAA. It's actually a really poor separation of concerns. And it will lead to very, very significant challenges if you actually want to do this for real. So we recommend separating out those identity services. And then if you're building a microservice architecture, if you were in the last session, and you've probably heard some stuff about things like service discovery and service registries, that's actually part of that app portfolio too. And the reason that we apply Cloud Foundry on one side and we look at the app perspective on the other side is the scope is different. We want to keep our Cloud Foundries isolated from each other, but our apps actually span all of those different Cloud Foundries. So if we look at it from that perspective, then we can start to tackle what this data problem is. Everybody's probably heard of consistency, hopefully an eventual consistency, right? A lot of people think they're kind of diametrically opposed. A lot of people think that a lot of systems have to be eventually, or they have to be consistent. Eventual consistency can't work. The reality is there's far more use cases where you can have eventual consistency and weigh the risk and the reward. So the good example I always use is most of your financial transactions, if they're under a certain amount, say $10,000, are actually eventually consistent because your bank would rather pay the penalty of a difference than pay for the solution to make them consistent in scale. So keep that in mind. In modern systems with eventual consistency, the window of risk is actually really small. It used to be there's large network delays. Used to be that it was very, very hard to send data because maybe the pipe was small. But that's not true anymore. The reality is that in most modern systems, that window of risk is really small. And if we couple those things with other good best practices, things like doing blue-green deploys, things like doing non-breaking schema changes, then eventual consistency allows us to actually gain consistency because that window of risk is so small. So we can gain consistency by using eventual consistency and dramatically simplify the problem at hand. So let's go back to our data picture, cloud foundry and apps, right? If we're trying to apply principles of separation of concerns and isolation, all of these things can actually become eventually consistent. Everything in cloud foundry can be isolated and we can use automation to actually make them consistent. Our app picture, so our service registry also can be eventually consistent. Hopefully you've heard about the container-to-container networking going on within cloud foundry, but we don't necessarily want applications reaching outside of cloud foundry and knowing about other applications because that becomes a harder problem. So we can actually make that eventually consistent for most use cases as well. And that really leaves us with two challenges and they're not always challenges. Our identity service, so think tokens for apps and think about app data. Not every single use case does app data have to be consistent but a lot of them do. Not every single use case does a token have to work in different cloud foundries, but in a lot of cases it does. So let's tackle eventual consistency first. How do we deal with this in isolated cloud foundry instances? Reality check, you don't push apps to anything that's not dev. If you are, you're probably doing it wrong, right? Who does? Concourse does, right? Automation does. Automation should be deploying to everything else other than your dev environment. And we can leverage that to our advantage if we understand that. So how do we do that? We can use automation to provide eventual consistency across different cloud foundries. And maybe that involves writing a YAML file that specifies what orgs and spaces and apps and users and roles exist or should exist across our cloud foundry instances. Maybe we do it in YAML because CF doesn't have enough YAML and we just miss it. But we can feed that into something like concourse and allow concourse to actually keep these things in sync in an eventual consistency fashion. How often are you adding users? How often are you updating roles? You're certainly pushing apps, hopefully a lot. But again, with blue-green deploys and non-breaking schema changes, that window of risk becomes really, really small. The important factor here is that this source of truth, this document that tells us what our world should look like has to come out of a single declarative source. I don't care what that is, it could be GitHub, could be Minio, I don't know, whatever. But if it comes out of that single source of truth, we can use automation to actually make these things look the same. So that just leaves us with the other problem, right? Our identities, our tokens and our application data. Now our application data is gonna be very specific based on what our apps are doing. I can't tell you what your apps are doing. I can't tell you what data sources they should be using. And identity doesn't always have to be consistent, but let's assume that it does. So what most of our customers do initially, when they go to an active multi-cloud, is just declare one as a master, declare one as a failover. And leverage the multi-cloud capabilities at the application level. That's okay, but it's not ideal, right? We obviously have single points of failure, we obviously have failover scenarios, and you pay a pretty significant price, especially if you're dealing with geography, when you're accessing data from very, very far away. But you can introduce caching strategies and you can kind of live with this and you can make things work. What we really want customers to do though, is actually make better life decisions. We want you to think about your applications and what data they're using. We've both had customers that literally just picked a SQL database because their DBAs knew it and it wasn't Oracle. And then they did it for everything. And that's really hard, right? Because there are technologies that actually make this a lot easier. And you should probably go check them out. Hopefully some people went to the Google Cloud Spanner talk. There's other technologies that are there to actually implement distributed resilient data stores. And you should leverage those technologies. Don't just pick MySQL, don't just pick Postgres, don't just pick MongoDB or whatever. Really understand what is it that this piece of data is and then find the best solution for it. Everybody thinks it has to be a relational database because we know them. Yes, there's always relations between data but in microservice architectures there's very rarely lots of joins going on in databases. So do you need a SQL database? No, you really probably don't. You might be able to use something like Redis and go talk to the Redis Labs folks about how to do this at scale. Or use one of the cloud providers like Google or even AWS or RDS that are providing very high level service capabilities for those backing data services. So we've actually done this for some customers. Again, it's not trivial. It's very specific based on the use case of the applications in the portfolio that you're trying to run. It's also very specific because sometimes you're trying to run the same applications on every cloud foundry, sometimes you're not. And those are very different scenarios. But there's a couple of best practices that in lessons learned that we've figured out kind of the hard way and by working with customers. First one is understand why you're doing this. If you're doing active multi-cloud because it's cool you should probably stop. If you don't have a real business use case to try to tackle these problems, you should stop. Because cloud foundry is actually really resilient on its own and most of the cloud providers are really resilient on their own. And so do you really need this? The second thing is weigh that risk reward for that eventual penalty. If your apps can be eventually consistent like in financial services, your job becomes so much easier. If you can introduce caching and they have to be consistent, your job becomes so much easier. It's a really hard problem if you don't. If you're told that it's a requirement for consistency. In most cases it's not a requirement and you need to push back and you need to challenge that requirement. And the last thing is you have to automate everything. If you don't automate everything, and I mean everything, I mean your application life cycles, I mean your deployments of cloud foundry, I mean your updates to cloud foundry all the way from dev through production, this is impossible. So make sure that you automate the life cycle of your environments, make sure you automate the life cycle of your applications, make sure you automate the declarative source of truth that's providing you with that eventual consistency. If you do those things, this actually becomes a pretty easy problem. It's really not that hard. It's just hard to figure out initially what our apps are doing on a case by case basis. And we can go through and we can evaluate what we're actually trying to achieve and we can make good decisions based on those microservice principles. So think about things like separation of concerns. Think about things like 12 factor apps. And apply those to cloud foundry to create ephemeral cloud foundry instances. Hopefully we're all here and we love cloud foundry, but we need cloud foundry to be able to die. We need it to be able to scale up, scale out, fail. Maybe the underlying infrastructure needs to be able to fail. That's the whole purpose of this active multi-cloud. If you keep these things in mind, it's actually not that hard. And I know that was quick, but we wanted to leave some time for questions. Are there any questions? Yeah, sure. Yeah, so the question was which type of data source gives us the most problems? And I mean, it's basically the SQL data sources. I mean, that's all it is. And so if you can avoid using those, that's great. But again, it depends. I mean, if you're building this large, geographically distributed multi-cloud, that's very different than let's say just building a multi-cloud in the US, right? So those are the hardest. If you can pick something else, then do it. Just might be new skill sets. Any other questions? Yeah, so we're not talking about using concourse. The question was concourse replicating data. We're not talking about using concourse, let's say to replicate application data. What we're talking about is making two cloud foundry environments look the same. So the same apps are deployed. The orgs and spaces exist. The users exist. The roles exist so that our automation can work. That should be driven out of concourse or some automation tool. You should really be using concourse. Any other questions? So, sorry? Is Google Spanner available in the marketplace? So the marketplace is different depending on your distribution. So it depends on what distributions you're using. I actually don't know. I don't work for Pivotal anymore, so I'm not sure. Part of what you'll find though with this is that you're not using the marketplace for production services. You might use it to spin up the initial service, but if you're doing active multi-cloud, you're not sharing those brokers necessarily between different cloud providers. And actually, all of your production data sources would be attached through a user-provided service instance. Any other questions? All right. Well, thanks for hanging in there. I know it was the last session of the day. I know everybody's tired, but yeah, we appreciate it. If you have any other questions, feel free to find us. Thanks.