 Welcome to my talk folks. I am Vic Calcar. Today I'll be talking about how to do application development across Cloud Foundries. So just a quick poll here. How many are you using Cloud Foundry today in Dev, staging, prod, just so I have an idea? Okay, that's great. Half of you. So I think hopefully you'll find this talk useful. So what I plan on covering today is what are the challenges you are facing when you're using Cloud Foundry and you have multiple instances of your foundations. And what is the potential solution around solving your data needs? Because you can push your applications very easily to your local Cloud Foundries. But then how do you synchronize the data across various Cloud Foundries that you may have across the geographies? Then I'll quickly introduce Redis as a potential solution and give you a quick overview of what Redis is and how it can address your data needs. And then finally I will present a walkthrough of the solution using Cloud Foundry and specifically I will use the Pivotal Cloud Foundry distribution to show you what that solution looks like from Redis and Redis Enterprise perspective. Finally I'll leave about a couple minutes or last five minutes to open it up for questions and that will give us a good opportunity to ask any follow-on questions you may have. So let me just set the stage for you on what is required or what is the challenge with doing multiple Cloud Foundries on a planet scale with multiple data centers involved. So in your development environments or in your staging environments, production environments, you may have multiple Cloud Foundries. All of them may be within the same region or they may be spanning regions on public clouds. So one of the big challenges if you are a Cloud Foundry user is how do you synchronize data across multiple data centers which might be running in different regions of public cloud. So one of the big challenges of doing that is around data synchronization and how do you make sure that your CF applications are able to leverage the same data set but you're able to push your Cloud Foundry applications across multiple CF targets or multiple CF foundries. So I think that's one of the bigger challenges when you're talking about data services in Cloud Foundry and how do you do data synchronization across regions across Cloud Foundries today. So some of the reasons why you might want to synchronize your data across multiple Cloud Foundries are some of the use cases I'm highlighting here. Some of those are around scorecards and shopping cards that you may be using. You may be using this multiple Cloud Foundries to solve your leaderboard issues or leaderboard solution. It could be related to fraud detection as well as event tracking or you might be using the likes or recommendation system. So for addressing all these use cases you want to figure out a way to synchronize your data, have it be globally available across multiple Cloud Foundries. So there are a couple challenges in the solution to the potential problem statement here. So one of the key things when you're developing CF applications is you want to focus on the same data set across multiple Cloud Foundries. You want to ensure that you have a very good developer experience and a user experience. So you want to give the freedom for your developers to push their Cloud Foundry applications locally, quickly, without having to worry about synchronization of geo-distributed teams. So one of the ways you want to do that is you want to keep very low latencies for your applications by having your applications pushed out to Cloud Foundries that are very close to your user base. And then one final thing as a big organization who has multiple Cloud Foundries across the planet, one thing you want to do is you want to keep the delta of your application in various regions and various continents to be very small. You want to use the same code base and maybe literally change the CF target before you push the application. So you don't want to introduce any additional logic as you're trying to push your geo-distributed applications across multiple Cloud Foundries. Now there are two potential possible solutions that you can leverage today in public Clouds or on-prem, kind of highlighting a couple of them. One of them is around Spanner, Google Spanner, which has a two-phase commit approach. So with this approach, what you are gaining is consistency. But what the trade-off there is in order to achieve that, you're looking at at least 200 milliseconds or higher before your data gets synchronized. Another approach is a quorum-based approach. Cassandra is one of the good examples of how you would do that at a global scale. Cassandra, again, it's a quorum-based system, so it has to have majority of the participating nodes to agree before the commit is written to all the Cassandra nodes. So again, with this approach, the quorum-based approach, you are looking at at least over 100 milliseconds of latency. So in a typical web app, you want very quick response times. You want to have very fast data response rates so that there is no lag for your end users. So if you're already looking at 200 millisecond latency at your data layer, it's definitely going to trickle down to your user application and it will affect your user experience. So one of the things I want to focus on is I don't want the developers, the Cloud Foundry developers who are writing the applications, to focus on implementing the solution themselves in the code. What I mean here is we don't want the developers to look at how to do the retries, how to do the circuit breaker pattern and write all that into your application logic, right? So we want them to focus on delivering value, writing the business use case, and sort of developing the app around that while we let someone else address these concerns about data replication and data synchronization. So let me propose and tell you about a solution that is being used at Redis Labs. And what we are doing at Redis Labs, we have a Bosch-based release which is implementing this solution. This solution is based on CRDTs. CRDTs are conflict-free replicated data types. This implementation of data synchronization is based on a paper that was written by Dr. Carlos in 2011. So Dr. Carlos is actually on our technology board, so he's advising us on how to do this data replication and conflict resolution which is purely based on Redis to solve the data needs of multiple Cloud Foundry applications. One of the key things to highlight in this solution is we are achieving data replication through continuous synchronization and the data becomes strongly eventual consistency. It reaches strong eventual consistency through the replication process. Now what is the advantage of this solution over the previously mentioned options? A couple of the things to highlight here is, again, the data service or the database itself is doing all the heavy lifting of synchronization retry logics so that the developer, the Cloud Foundry application owner doesn't have to worry about those retry logics. And then the Redis is known to be very simple and fast, so it is able to take care of those conflicts that you may have during data replication using the conflict resolution that's built into the CRDT approach. So let me dive a little bit deeper into why Active Active CRDTs. And Active Active, like the name suggests, is around doing the ability to write data to multiple Cloud Foundries at the same time. So it's a multi master approach, Active Active running on multiple Cloud Foundries. What this does for you as an app developer or app owner is it gives you local latency guarantees for your Cloud Foundry applications. So each Cloud Foundry is essentially doing service binding to your own local Cloud Foundry instance, thus guaranteeing you very low latencies for your applications to reach your data service. And then as part of the solution what the Redis CRDT-based technology is doing is it's giving you conflict resolution built in and providing you with strong eventual consistency as you are sort of focusing more on developing your application itself. So another sort of aspects of the CRDT solution is around how you're going to leverage this solution for your Cloud Foundries. So Redis Enterprise is based on CRDT approach, so our CRDBs are delivering lower local latencies for geographically distributed applications. So as the diagram depicts you are able to do high throughput ingest using Redis Enterprise to your local Cloud Foundries while the global synchronization service in the background will take care of conflict resolution for you as well as take care of all the retry logic so that as a developer you don't have to worry about that particular instance. Another sort of quick use case and benefit of Redis Enterprise with Cloud Foundry is it allows you to develop your Cloud Foundry microservice applications a lot quicker and a lot faster. Like I mentioned it makes it possible for the developer to focus on the business logic and not necessarily worry about how to go about doing the data synchronization. All these normal patterns that you use in microservice developments like retry logic and circuit breakers patterns are something that is now offloaded to the data service and the data service will take care of the replication, the synchronization and the conflict resolution. So why CRDTs and what's so unique about them? So CRDTs offer you active-active and I think I want to emphasize this point again. It is a multi-master approach. We call it active-active so it's bi-directional. It's not active-passive. It is master-master replication so you are able to write data to both the Cloud Foundries at the same time and the conflict resolution and replication happens in the background. Another advantage of this approach is the ability to create a full mesh network of all your Cloud Foundry instances that may be spanning regions. Then the couple other advantages to highlight here is the Redis implementation automatically takes care of the counter deletion for you. It takes care of expiry as well. And then the Redis CRDT offers up conflict resolution for these five complex data types. Some of the easier ones may be easy to explain as the last right wins. But as the data structures get complex, the conflict resolution algorithm also gets complex. But the Redis Enterprise solution is already solving that concern for you by implementing various conflict resolution strategies. Before I go any further, let me just briefly tell you guys about Redis. How many of you here are active users of Redis today? Show of hands. We've got a few. This would be good to go over with the whole group here. Let me tell you a little bit about Redis. So Redis is an open source in memory, no SQL database. Redis Labs is home to the Redis open source project. Salvatore, who is the core contributor of Redis, works at Redis Labs. And let me tell you what are the key advantages and differentiators for Redis as a technology. So we are known for three big differentiators. One of them is around performance. The second one is around simplicity. And the third one is around extensibility. And I'll kind of step into these three pillars of our product and product offering in detail in the next few slides. So when we talk about performance for Redis, we are looking at how do we get you the highest throughput with the lowest latency possible for your applications. So the graph on the left is essentially showing you that we are able to achieve very high throughput with very low latencies. And this was a survey done by an independent benchmarking firm to kind of highlight that feature. The second advantage of Redis is around how the footprint looks like for Redis. So the second graph on the right is essentially highlighting the fact that in order to reach a million operations per second, how many Redis instances and Bosch jobs will it take for you to achieve this outcome. So as you can see from the graph, Redis has a very small footprint and is able to deliver that performance that you may be looking for as an app developer on Cloud Foundry. The second aspect is around simplicity and the Redis is no sequel database running in memory and it essentially supports data structures. Like the diagrams on this slide, we have kind of highlighted them as Lego blocks. So what these allows you to do is using these data structures or Lego blocks, you are able to compose your solution and address your use case using these 10 basic data structures that are available in Redis. The third aspect of Redis is around extensibility. What I mean here is Redis gives you those 10 basic data structures. But if you are trying to solve a business need and you have a case where you may need additional capability, Redis allows you to extend the basic functionality of Redis through the concept of modules. So modules allows you to extend the Redis capability and introduce new capabilities that can help you solve your use case. I'm just highlighting a couple of them here. So we have four modules that we want to maintain and highlight and publish from Redis Labs. So Redis Search is a capability that you can introduce in your solution to do basic recommendation system and full text searching. Graph module is about to hit GA in the near future, so that's also a capability that you can introduce in your Redis deployment on a Cloud Foundry. A couple other modules to worth mentioning is around machine learning and JSON objects. So Redis can also do JSON object store on your Cloud Foundry service instances. So let me just quickly tell you what it looks like on a Cloud Foundry. I was going to do a live demo, but I think after I saw what happened at the keynote this morning, not a lot of the demos were working. So I'm going to go with the static screenshots, but I will do my best to explain what's happening here. So I think what you see here is Redis Enterprise deployed on a Cloud Foundry foundation that is essentially running in the US continent. So if you look at the red box, what you will see here is I have a US Cloud Foundry foundation. I have enabled Redis Labs offering of Redis Enterprise in this Cloud Foundry. I have pushed out a web application and bound my web application to the Redis service instance. So the second output here is around when you push that, it's a Python Flask app that one of our architects have written. So I've essentially pushed it out to that US Cloud Foundry, did the service binding so that now you can see that my Redis Web CLI is connected to the US foundation in US. Now I have gone through and done a similar exercise with CF foundation running in the European subcontinent. So now I have two Cloud Foundries, one in the US, one in Europe. Both have deployed this Python Flask app, they're bound to their local service instances, and the data is getting synchronized for you in the background. So I'm going to quickly walk you guys through a couple examples of what that looks like. So let's start with the data structure list in Redis. So what I have done here is kind of taken screenshots of my Python Flask app, and I'm essentially walking you guys through the four steps to confirm that the data is getting synchronized. So if you look at box one, this is in the US data center, I input something into my list. I do the same thing on the European Cloud Foundry in step two. Then in step three and step four, I'm essentially pulling up that same list and showcasing the fact that the data has been synchronized for you and by the Redis Enterprise service in the background without you as a developer having to do any additional steps. So that's a quick example of how we do it in lists, Redis data structure. We have a similar concept of doing that data synchronization using another data structure called Sets. Same concept here. I have a four-step process. In the first one, I'm entering a set in the US Cloud Foundry. I do the same on the second step in the European Cloud Foundry installation. And step three and four are essentially showing you the fact that the sets have been synchronized across both the Cloud Foundries and the developer doesn't have to worry about the retry logic or any of the circuit breaker that you may have to do on a traditional development platform. The third one I want to highlight is Sorted Sets. Again, it's a complex data structure, but same sort of steps. This is a sorted set, so you're seeing the step one. I'm adding some data in Redis on the US side and step two I'm adding more data, but on the European Cloud Foundry. And step three and step four are essentially showing you the fact that without any developer interaction, the Redis Enterprise service instances are being synchronized in the background and you're able to use the same data set for your application needs. One quick screenshot of our Redis Enterprise console. So in this image what I'm trying to highlight here is the fact that you are able to see the synchronization of the Redis service instances across these two Cloud Foundries. So where you see the small green circular icon towards the bottom of the image, what you are seeing there are the two Cloud Foundries that I have connected to each other. So this allows you to essentially see the synchronization happening and you can collect the status of that so that you can focus on developing your application. Now the web interface gives you the clear indication that these two Cloud Foundries are in sync, but we also have a REST API so if you're trying to automate the status collection of the synchronization of data, you can hit the REST API endpoint and you can automate the status update of your data and sort of set up any triggers or alerts for your follow on sequence of Cloud Foundry applications. So in quick summary I just wanted to highlight the fact that using CRDTs we are able to take care of the data synchronization based on these five data structures and you're able to achieve the outcome here of achieving strong eventual consistency for your data, for your Cloud Foundry applications running on different continents or across different regions in the Cloud Foundry ecosystem. This slide is essentially giving you links to our paper on which the technology is based and you're welcome to download it from the Pivotal Network and give it a try and try out our CRDB based solution for Cloud Foundry. So that's pretty much it. I do want to open it up to questions at this point. So I will have my wonderful assistant here running around with a mic so if you guys have any questions I'll be happy to take a few questions at this point. Hi. So you mentioned at the beginning of a presentation that other solutions ensure consistency by two face comments or some quorum approaches. How do you ensure consistency? Because this is probably somehow included in the CRDT concept but for me it wasn't clear how you do ensure the consistency because it seems like it's a more replication based approach to sync the data, right? Yeah, no, that's a good question. So actually the answer lies on the fact that you're talking about geographically distributed systems. So the data has to traverse across the continent so it's an event based system so it's continuously synchronizing but the rate of which it synchronizes depends on your architecture itself. So if they're within the same continent the synchronization will happen a lot faster. In my example it had to go all the way from US West Coast to Ireland so it took a little bit longer. So the answer to your question is it is an event based, it is a continuous replication and synchronization but the actual consistency is achieved based on the geographical distances. So it's a variable number, it is based on vector clocks and we are able to achieve that using that. Alright, one more, two more, alright. I know I'm in a dangerous spot in between lunch and my questions so let's see if we can wrap it up. I had a short one. I had a question with respect to the developers. Are they able to align multiple insert or modification inside a transaction or is it only single transaction which can benefit from CRDT? Yeah, I mean right now we're starting off with a single transaction but Redis itself supports multi-exec blocks but right now we are focusing on getting these basic data structures down for the CRDTs and the eventual goal is to continue that effort that is traditionally supported by Redis. Alright, going once, twice, alright. Thanks everyone. Have a good lunch.