 Are you ready to start? Yeah? All right. Good morning, everyone. So what he said just now is his name is Wen. He's the CTO of Stuck and Wen. He's also a happy fruit. My name is Gao Xiujiao. I'm a cloud engineer of Stuck and Wen. Bless you. Today we will be talking about SHIELD, a backup and restore solution for cloud foundry and its services. We'll fill you in on what that actually means. Before we start to talk about SHIELD, please allow this happy fruit talk about Stuck and Wen quickly. Stuck and Wen is a consultancy that helps you be a superhero to your organization and clients by our customers by succeeding with your cloud foundry and Paz story. We help with everything from the infrastructure level up through automating all of the operations of the Paz itself through the 12 factor applications. We can help you flush out that entire story. We basically come in, we fill in any gap, integrate all the components, maybe you have some legacy, big iron things or whatever. We help you integrate those into your cloud foundry. We apply automation and all of those things. And then we also focus on teaching you how to maintain this thing moving forward yourself. Essentially we come in, we partner with you and your team. We work alongside you guys to carry you past the steep learning curve and greatly increase the speed of your adoption getting you started faster and sooner. That's Stuck and Wen. So back to SHIELD. What exactly is the problem that we are trying to solve with SHIELD? Let's first take a very brief look at the current state of backups, restores for both cloud foundry itself as well as related services. Today there is no overall comprehensive backup and restore solution for cloud foundry and its services. Sure there's a lot of little projects here and there and tools to get you bits and pieces and some tribal knowledge which you have to like sacrifice some kind of animal to and in order to back up successfully or whatever. But really not a good story. PCF itself does have a very well documented manual backup process with a diagram that makes your eyes bleed because of the number of components and steps. True story, we had to rush one of ours to the hospital after it. So basically it's there, it exists and it is very well written. There's also a tile that PCF has that can allow you to like back up certain components and stuff like that. Comprehensive overall systemic, not really, you're kind of out on your own. As for services, oh wonderful services, I love services, I love playing with services, service brokers, ooh, good times. Unfortunately if you are a service implementer or service operator, it's kind of punted. That's yours to figure out how to back up your service, how to provide that. A lot of people wrap custom APIs, they run cloud foundry apps that tie to their services. All kinds of solutions exist to this today. But there still is no real up until now. Great like story for this. Which does bring us to why you guys are all here listening other than the fact that it's the closest thing to the registration desk. Basically why did we start Shield? Why did we go this way? Why did we keep on working on this product at all? So we've been helping GE with this little project called Predix.io and there are some amazing people doing amazing things and we couldn't be happier to be helping out with this project. A lot of stuff was talked about investigating their disaster recovery scenarios. You're in production, you better have a disaster recovery scenario or as Dr. Nick said, all you got is a giant cache of things. So we were helping them with this on the Predix.io. We needed to flush out what the disaster recovery backup restore solutions were for not only cloud foundry core itself. But we also needed to focus on what about the services too? We've got to figure out the solutions to those. We're going to document these on the internal wikis and we go through these ritualistic incantations in order to summon the backup beast and whatever. Or do we have a same way of moving forward with this? Well some very brilliant people that work at Stark and Wayne actually came up with these things and with an approach to this. We already had a project that they were toying with on their own in an open source that basically hadn't had a lot of effort because well frankly you know on your own time that's not a lot of time right? So in order to help GE we figured out that this project would be a perfect fit for this effort. So with GE's help allowing us to work on this a lot we were able to carry this project across the finish line. As of today this project which we're talking about today has been running in production at GE for over six months now. So it's a really encouraging thing. Very very good open source story there. Thanks to GE of course. SHIELD as it stands is essentially a drop in solution to anybody's boss releases and can tie in the services, service brokers, all of these things. So this does address the services story. Essentially it allows you to focus on as a service implementer remember that was on your either the implementer or operators of the services to figure out these things. SHIELD allows you as a service implementer or operator to focus on either implementing your service or operating your service and SHIELD basically focuses on your disaster recovery backup and restore stories as well as migration stories which you will hear about a little bit later. This addresses all of those things. Specifically what kind of services you might ask well today SHIELD backs up and restores services such as PostgreSQL, RabbitMQ, Redis, et cetera, et cetera and several others. I'm not going to go into details on how I'll leave that for later but it can support any services we need. It's for cloud core cloud foundry itself. SHIELD can also back up cloud foundry as well as Bosch. Essentially it allows you to back up any Bosch deployment. Cloud foundry is deployed via Bosch can back up oh hey I get it. So essentially any Bosch deploy system that you want all you need to do is essentially write a plugin to SHIELD to back up whatever the thing is that you are backing up if the plugin doesn't exist already. You essentially deploy SHIELD alongside your existing Bosch deployments using this mechanism within Bosch called co-located Bosch releases and now I'm going to let XJ fill you in on the details of the project. What? Wait I was enjoying your presentation. Okay so now I will talk through the framework architecture of SHIELD and the features of SHIELD. The framework of SHIELD is pretty simple as you can see here. It consists four parts a core diamond, target plugin, storage plugin and user interfaces. One question often asked by our client is why are there both target plugin and storage plugin? So I will answer this question here by explaining how backup and restore works. So when we back up data the target plugin will retrieve data from your data system then send it to storage plugin. The storage plugin here will upload a single data blob to your storage system whatever it is you configured then it will return a key to you which you can use later for your data retrieval. Sometimes accidents happen we lose data then we come to the point we have to restore the data. So when we restore the data how it works the workflow is like this. The storage plugin will use the key to retrieve the data then send over to target plugin. The target plugin will just load the data into your system then you got all your data back simple right? The next piece is a core diamond. Core diamond is a central component of SHIELD. It has many different functionalities here I will cover several important ones. First one core diamond is responsible for metadata management basically it will track what target and the store exists, what jobs you configured, what schedule retention policy you defined, what backup restore jobs have taken place and what are their results. Also it will tell you what tasks are in flight. The core diamond also can configure your backup jobs like you want to decide what you want to backup right? Where you want to put it then you can configure job as you need. A task here basically just a job an instance of job being run. So it can run backup restore jobs according to the schedule we talked about earlier or we can kick off backup restore jobs on demand. Quite nice huh? Scheduling SHIELD car allows you to define the schedule like what time you want to backup maybe 1am that's a golden time or how often you want to backup daily, hourly, how long you want to keep your backup you keep one year or you keep a couple of months you can all configure that. Next one is the core diamond also exposes the metrics statistics of your backup jobs. It allows you to search all your backup archives to see if your backup job are finished successfully. Is your restore finished successfully? All those metrics also can be aggregated. You can send those metrics over to any aggregation system you have. The last piece of the framework is the user interface. We designed a web UI and a CLI. This as you can see this is a screenshot of SHIELD web UI. So using this web UI we are able to view and edit your jobs, view and run your backup and restore on demand. Also you can edit or update your schedule your retention policy. On top of that it also shows us from here you will see it will also show us what's the running task we have and what's the complete task of this log. So it will show you all those. It's simple but it has everything here. SHIELD also have a very well documented extensive CLI. Probably you cannot see clearly here but you can see a basic idea here. So that basically covered the framework of SHIELD. Next I'm going to tell you a couple of stories about the features of SHIELD. SHIELD is distributed as a boss release. Just like all other production grade enterprise software is, right? Like for example maybe Cloud Foundry? Yeah that. What? Yeah I'm glad to make you guys laugh. So in order to use SHIELD to backup data what we need to do here is very simple. You just configure your SHIELD agent on your deployment. For example if you want to backup your Cloud Foundry okay configure SHIELD agent on your CF deployment. You want to backup RabbitMQ, Redis, all that? Just configure SHIELD agent on your deployment. It's pretty easy. Also where you want to store your data? I may want to store my local file system. Okay configure your store on your local file system. You may want to store it on your S3. That's doable too. I'll swift all that. Like it can support like many different of target and store plugins. Yeah another thing I want to say is using SHIELD it becomes very straightforward if you delete your entire deployment then you recover everything after redeploying. That tells us what? This enables a functional disaster recovery plan. Oh I love this story. Yes SHIELD can also backup a restore your boss. So I will share with you a smoking whole story here. Imagine you have SHIELD running. You have all your deployments being backed up by SHIELD. Then someone, some bad guy, burn down your data center. Like bring it right down to the ground. Nothing left. Now zero piece of your hardware is working now. What are you going to do? Probably the first step you want to set up a new hardware right? Then what? Maybe we can spin up a boss. Then we can use boss to deploy SHIELD. After that we can use SHIELD to restore our deployment. Guess what happened now? You just left all your boss database. Your SHIELD deployment is off now. So we designed SHIELD to solve this problem. When you want to recover your boss you don't need SHIELD diamond running. Yeah we just designed that way. If you are interested, check our repo later. So you can just spin up your boss. Use this excellent feature of SHIELD to recover all your deployments. Include SHIELD itself. SHIELD is not only for operation folks. There is also a feature where you can enable service application users to use SHIELD to manage your database backup and restore. I will share you another example here. Assume that you have an application using PostgreSQL database. Somehow over time you got lots of data. You need a larger database. Now how you are going to migrate your data? You're SHIELD. So it's simple. You just use SHIELD to backup your database. Then you create a larger database instance. Also we have RDPG which is a reliable PostgreSQL database service. Okay come back to SHIELD. So then you spin up a larger PostgreSQL database. Then you use SHIELD to restore your data on this new larger database. Guess what? You get all data in the new database. Very easy, right? Writing plugins is just a piece of cake. Yummy one. Cheesecake, strawberry, ice cream. You name it. Okay. Why are you writing plugins just a piece of cake? Because you start with the framework we already provide for you. Basically you don't need to worry about metadata management. You don't need to worry about schedule monitoring retention policy. You can just focus on your solution. How you backup and restore your data. Imagine this. You don't want someone who has access to your SHIELD API. Then just access all your backup and restore information, right? That will be terrible. So we designed the SHIELD in a way we have authentication. It's implemented using OAuth2 and it can be done with basic HTTP authentication, support GitHub authentication. All you can do is through CFUA. So that's all the cool features I want to share with you guys. Now let's help you share with you the future of the SHIELD. So let me take you on a very brief, trippy explanation of what we see or hallucinate as the future of SHIELD. And sort of priority order as we see it. However, that obviously could be influenced by your voices. All right. Primarily, the next one on deck is encryption and decryption. Encryption and decryption at rest. We want this to be fully supported out of the box by default. This is going to be supported at the plug-in level. So basically, if you write into your plug-in, here's how I get the data for my servers and here's how I restore it, you automatically get the encryption and you didn't have to worry about it. That's pretty sweet. Crypto for free. Like that always happens. But in SHIELD it does. Pretty cool. Well, it will. All right. Next up. So we're all familiar with the concept of incremental backups. You take one massive backup that takes God awful forever and then you just kind of take backups at regular small intervals to record the changes over time until you take your next massive one. Hopefully not daily and they don't take more than 24 hours to run. But hey, that's another story. So basically why we do, we want to target this as a supported feature for the plug-ins. Well, two reasons, right? This is about minimizing the storage requirements, making it more a little more streamlined as well as reducing the amount of wall clock, time and CPU and resources that are attributed to the process of keeping backups through time. Next up, we want to take what we have today for monitoring which is very simple and basic and we want to extend this to make it easier for operations folks to get more introspection into SHIELD operations itself such as wouldn't it be great if not just knowing when jobs started, when they complete, did they succeed, did they fail? Well, what was the average runtime of jobs for this particular plug-in or this particular system or this particular thing, right? Average runtimes of execution, average sizes of the backups of all these things through time. Like it helps you to kind of identify, hey, that guy is storing large binaries in his database. What's going on guy? Or whatever else you're actually doing, right? So we want to make it take the metrics to the next level and make it even more useful for keeping track of what is going on and allowing you to get to root cause analysis of why this fill up much faster or preferably even way ahead of time, right? So obviously there's more to that story than that. That's a really brief synopsis of that. Another thing. There is one recurring theme that keeps recurring. Don't worry. His native language is not English. Try this again. There is one recurring theme while we help folks with their production environments and that is handling of sensitive storage of sensitive things like credentials, right? Encryption, decryption keys, passwords, all of these things. This is a very high priority item within organizations and it's usually, unfortunately should be ahead of time maybe, but usually it's a, whoa, we're running a production. Oh, now we got to do that. But still, it's a huge thing to address. Well, we want to make it easy for SHIELD to enable that story for everybody in a straightforward manner so that they don't have to bake up something half-baked and, you know, we want to make a fully baked solution so they can have their cake and eat it too, right? Cake. So basically what we're going to do is we're going to take and add integration points into SHIELD where all the right points of encryption keys and credentials and all these things, we're going to allow them to be stored and retrieved in systems such as Vault. So that's the credentials handling. So, I mean, we have other ideas for the future. We'd love to hear even more ideas, okay? So what I would highly encourage everyone to do is get yourselves involved, right? It's a young project, but it is a production project. We encourage everybody to basically help shape its future. Like, how are you going to do that, right? Well, it's an open source project. How do you do that with open source projects? You deploy it, you play with it, you tweak it, you read the code, you ask questions, right? You send a feedback, pro requests. You write a plugin. Ooh, that's a good one. I didn't think of that. You write plugins. Hey, I've got the most awesome database in the world. Here's a plugin for SHIELD to allow it to be backed up and restored in the context of Cloud Foundry. We all win. Great. So, yeah, so basically you can also, so that's all on the open source front and also, you know, come to us at Starker Wayne and help you succeed in incorporating SHIELD into your own infrastructures. Help us to define your disaster recovery story and train your people on how to actually use it, should disaster befall. Let's hope not, but it happens. So basically, if you do decide to want us to let us help, right, you can email us there, be a hero at Starker Wayne, or we're here this week, right? Come talk to us at our booth over there. I highly encourage it. Yeah. Come for free t-shirt, free business card, free handshake, free hats. Hugs are from me. Handshake from me. Thanks for your attention. Do you have any questions? Keep in mind that we are almost out of time and you can come visit us at the booth in mass for actual questions as well. In the case of services, have you given much thought to how to restore the state of service instances in the CCDB as a part of restoring not just the service itself, but the services state from Cloud Foundry's perspective? So it's the week we've gotten to. Any others? There's no others because they will all come to our booth. See you guys all at the booth in the exhibit hall. Happy to talk to you.