 Hi everyone, my name is Eran Gampell, I'm from Huawei European Research Center. We're going to talk today about cloud disaster recovery. We want to present another approach to doing that. With me I have Juan Hao from Huawei, he's an experience engineer, works in OpenStack for almost two years, contributing Nova and Cinder, and I have Oshrit, she's from IBM, the Research Center, and you will see her work later, a demo of her work later in the presentation. We have a lot on the agenda and we try to squeeze a lot into this session, so we'll cover why we need disaster recovery, what's the the needs for that, the status of replication today, the status of replication in Cinder, show what we did and something we developing Huawei hypervisor based disaster recovery, and then present a new project that we want to introduce and do a short demo. So without any further ado, Juan Hao, please. Thanks, sir. So excited to be here. So at first we need to figure out why we need the disaster recovery. You know, if you go near those walls you may get a lot of answers, but I think there is four points which is important. So first, the end users don't care where the data center is and what happened in it. They just care about where they can upload their photos to the clouded mean. So they want the service in the cloud is available at any time. Second, we didn't have the perfect machines and humans, so the machines may go to fail after a long time running, and the users, no matter any users of clouded mean may make mistakes that will destroy our data in the cloud. So the terrible things is the accident and the nature disaster, you know, the floods, the fare and the earthquake that will destroy the data center in a short time so that we, where we need the disaster recovery plan to protect our data. So, you know, senders have some important features like a warm snapshot backup and replication. You can create a warm from backup and also you can create a warm, create a snapshot for a warm. So that's why we still need the backup as replication. Of course, they are for the disaster. You may, if the disaster happens you may have lost all the storage device and all the snapshots so that you will use the backup and the replication to recover your data to the disaster recovery site. So, as a topic side, we will talk more about around the replication. Okay, we are familiar with the backup, but to the replication, let's see the history in sender. You know, we have talked about it in the Air Horse Summit and we implemented version one in the June release by Ronan Ket and we merged the upstream codes and we make spot to a IBM storage driver and in Liberty release and we got a new version that we call it version two. That is a driver by sender called John Griff and we import and make it readily usable by other backend drivers but there is no driver spot yet. So, I want to use a case to show how the replication works and how to use it. So, you can see there is two data centers. Left is the production site and the right is the disaster recovery site and the OpenStack cloud is deployed across those two data centers. There are storage backend available in both data centers and they are manager by sender. So, storage domain will configure the replication features in sender to spot the replication. So, there is workflow for the older version. You can see the cloud domain will create a warm tab and the users just to use it to use the replication feature and the user well don't know the replication, he just use the warm tab to create a warm and the sender scheduler will to choose a backend that spot the replication feature. Meanwhile, at the backend, the driver will to set up the replication to create a warm and a replicate warm. Next, sender has a project task to update the replication states like coping, error and active. So, we have some API to spot these features like cloud domain can promote replication when disaster happens and re-enable the replication for recover for back. So, the last there is a test API for the users to test the replication feature that can create a column warm of the replicated warm that will to see the data in the replicated warm. Okay, there is a new workflow. We can see there is pretty much same with the older version but there is some different that we should know. The first is the driver could choose multi targets according the sender configuration. So, the next is the replication states that will be reported by driver. Next, there are some different APIs in the new version that we call them for over replication API to make the for over function. And we have the enable or disable API for some use case like maintenance. Of the last, users could query the warm replication states from the new API. So, there is a question that is there only way to get the replication functions? Well, I don't think so. So, Aaron will bring some new ideas for us. Okay, the most what currently supported on sender is array-based replication. And this is very known. So, when you look in array-based replication, you have the storage array dealing with all the things that are needed for replication the one connectivity, the tunneling, the compression, the encryption everything is done by the hardware array. We would like to present software-based replication that we developed in Huawei. We develop it in the hypervisor. Basically, we introduce a new agent into the hypervisor. This agent is on the data path. And I will show what it does. It basically takes the data, write it to the Dix and in the same time duplicate it and send it to the other side. To what you see here in the slide that target the replication agent. So, how does it look from orchestration point of view in our solution? The initial that you have here, the VRG is the virtual replication gateway. The virtual replication gateway is responsible for establishing the one connectivity for compressing the data, for encrypting the data, maybe deduping the future and more feature like that. The DR manager is the one that orchestrate the disaster recovery and OpenStack here is the one that launch the volume, create the we use the OpenStack API to launch the VM to create the volume, to create the volume on the other side. So, the pieces now that I'm going to be focused in the next slides are the IOMirror, the agent that we have in the hypervisor and the right agent is the agent that we have on the other side on the recovery side that we want to write the data there. So, if we go deep dive into how it was implemented you can see that the VM, from the VM perspective it writes to the block device like no changes. Then we capture the IO, we basically tee it, we write it to the disk and in the same time we push it to a queue. Once it's been pushed to the queue, the VRG takes it if it was configured to compress it, it will compress it if it was configured to encrypt it, it will encrypt it send it to the other side, the other side write it to the right agent and we have the orchestration flow to write it to the right volume on the other side. If we look in the state machine how all this is being done so we have two states of replication, CBT based replication and queue data replication. I'm sure many of you know what CBT so I will say it anyway, I will say it very shortly, CBT has changed block tracking it's a bitmap array so each bit represents a block in your block device. So, if in this array you have zero it means no dirty if you have one it means dirty. So, it's very simple and I will explain why we need this state the next state is a queue data replication so queue data replication is a state that each packet that arrived to the queue we are trying to send it to the other side as fast as possible in the future we will add the RPO configuration and more advanced configuration on this setup but currently that's as fast as we can and then it depends on the one connectivity and now if we look in the state machine we have the setup of the connection we are starting to protect the VM when we start to protect the VM we go into the CBT replication we just mark all the bits as dirty the VRG start take the block, the dirty bits turn them off, send it to the VRGW to the other side once we clear all the CBT bits then we can go into queue data replication queue data replication just push it to the other side everything that is being written what's the limitation of CBT data replication you cannot do snapshot in the replica site on the other side because you cannot guarantee that data is consistent so snapshot on the recovery site is done only on queue data replication that's we currently we think we have a way to allow it in the future but currently that's the limitation we have a step of consistency check if the machine crash, if the system crash we want to know that everything is okay on a normal status we just take the data that is in the queue and mark the CBT as block that are dirty but if we crashed and we are not sure if we have a consistency check process between the two VRGs they all take the volumes and the shah of all the blocks and compare it and see if they need to synchronize each other and after that state you can go into the normal state and of course you can stop the replication protection of the VM this is the flow that we currently have to protect the VM so when you define in a flow to protect the VM it's very different from protecting the volume what Cinder has now in the flow this flow for instance doesn't go to Cinder at all so it's a completely different flow you select which VM you want to protect what's the protection policy start the replication create a recovery plan, recovery plan is that how many tests you want to do where you want to recover, how you want to recover fail over, fail back there are some similarity and there is things in the workflow that are very different I would like now to compare the two solutions so when you think about everyone say I provide a base replication you think of course it's way better than an array based replication but that's not the case always there are some scenarios that you will be will prefer an array based replication just to go over it quick for instance the array based replication has no impact on compute node at all from our test it's mostly memory no CPU, mostly memory but it has an effect on the compute node on array based you have transparent deduplication because the storage is there you don't need to cache it in the VRG it can support cross array consistency group, this is with the disclaimer that they need to be in the same array and the hypervisor based replication has many advantages as well it's multi-vendor, hardware agnostic no special admin skills virtualization ready, you can even replicate the ephemeral storage so there's many advantages for both and we have them all in this slide so when you think about it we have multi-use case, multi-pol use case and for protection plan and we want to be able to give the user the tenant administrator to choose the right plan for his deployment, for his workload and so how can we do it so we started thinking about it and we thought that something is missing in the puzzle here we think that we need one API to rule them all and what do we mean we need data protection as a service or application protection as a service that you will be able to define how you want to protect your application and not only protect one workflow and if vendor want to change the workflow they should be able to do that so we start thinking about it and saying okay what we want to protect and the first question that we got so do we want to protect only storage is data only storage and when you think about only protecting storage you will define an API that define backup policy, replication policy verification, file level restore that is very very tied in to storage but we started collaborating with other companies especially with IBM and start thinking about it more and more and we saw that we actually what we need to do is protect an application not just protect storage and we want to protect service we want to protect resources so if we take a typical three tier cloud deployment you have your web tier you have your up tier you have your DB tier in the DB tier you can have a database on a volume or you could have the database as a service and if we drill down into it you can see that it has multiple resources you have the project you have the security group networking element you have images this is glance you have a volume this is cinder you can have database as a service then it's completely other project so you have multiple resources in OpenStack that if you look at it that's your application and that's what you want to move to the other side you don't just want to move the storage so we understood that data is bigger than storage and we need to protect all the resources and what we'd like to introduce now is a new project that we are launching and this project scope will be data protection as a service application protection as a service the name of the project is Maug I don't know if you are familiar with the name or if someone here is familiar with the name it's from Tolkien book The Hobbit the dragon that protects the treasure in the Tolkien book so we thought it's appropriate what we think is the mission statement and it's important to say this is currently in the infancy stage we're just starting with the project so what we currently think we want to formalize how you do application data protection in OpenStack formalize the API, the services, the plugin our vendor introduced new workflow we don't modify OpenStack and we want to be able to protect any resource in OpenStack and we want to have maybe more than one implementation of a protection per resource in OpenStack and allow diversity of vendor solution capability and implementation without compromising usability so when you think about data protection currently it's a bit complicated to administrate to manage the flow to configure if it's array based you need to know special parameter in the array driver we want to make all of this simple and we want to make it standard standard API that you can define it so what's the highlight of the project first of all it's an open architecture vendor can create their plugin and it's pluggable architecture and you can have different implementations from the same protection mechanism user perspective you should be able to protect your application we should be able to show you the dependency if you choose a resource if we have the correct plugin we should show you which resources are dependent on this resource and you can choose we will allow this is configurable how to protect them the admin perspective is that admin would choose which plugin to load of which vendor implementation so there could be multiple vendor implementation for volume protection multiple vendor implementation for VM, for server, for networking so we will allow the administrator to configure that in configuration time so we started going into the detail and thinking what do we need to define in the API layer and we start collaborating with IBM that is already pushing that you could see a demo in Paris and another in I think in Vancouver and they actually have a product that is in some way dealing with this and we start collaborating and got a lot of inspiration from a project that they already have that they would show later in the demo and that's what we have currently and again I encourage everyone to join us and I will advertise later if you want to influence the API layer or the way this project is going to be designed it's a great time to join so we define four let's say segments to the API first one is what you want to protect and what can be protected so it will tell you which plugin are loaded on which resource, which action set they provide you can ask the protection API what is already protected what is in protection status the second one is how to protect we want to define a plan what's your plan, policy it's very similar how you want to protect your data the third one is where do you want to protect and here we introduce a concept called bank bank could be swift any storage could be seen there itself if it has a backup implementation and bank is where you store your data it could be local, global, it's up to you you do whatever you want you can load the plugin with the property implementation of a bank where you store your data the only API that you will need to implement inside the plugin and we'll go into it later is write and read and inside this bank each transaction or each resource will have volts the safe that can be read if you look in the disaster recovery concept from the DR side you can read this vault and recover your application so this is the bank API and I will go into details about that in a second and the fourth one is what was protected so you can look at it in few ways it's log about what happened, when happened made the data about the process if you look about backup what time the snapshot was taken maybe extra data which consistency level did we achieve on this backup or snapshot, etc and it can provide you more information about which bank it was saved and which vault and many more extra information that I will try to go into it in the next slide so on the bottom side we have what we want to allow vendors to plug so what's plugable in this architecture so first of all we want to allow vendor to have resource protection plugin this plugin can protect one or more resource in one plugin or you can have multiple resource protection plugin that protect the same plugin same resource, yes, sorry and this is the administrator decision which plugin to load the second one is which banks do you want to load which banks plugin do you want to have inside and the last part is a plugable plan enforcer service this part we will have as a reference implementation just to give you highlights when you look in the backup plan you have scheduling, you have triggering someone has to manage it, someone has to discover the dependency when you show them a resource you need to discover the dependency tree of that resource very similar to what I saw earlier on the 3-tier application so an application has a security group and it has a virtual router and it has a subnet you need to know all the resource tree that you need to construct so if we deep dive into what we have now so what is protected we have for instance an image, a VM, a volume and each one of them has a volume protection plugin this volume protection plugin currently and maybe we will extend it later needs to implement three action sets one for snapshot meaning point of time labeling second one replication, real time or close to real time that is possible and the third one is backup, take a snapshot take a point in time and copy it some place each of these action set and here it's become a little bit more complicated needs to comply to an interface the interface is very easy it's very loose we allow vendor to add their own implementation in so it's protect, restore verify, option schema and result schema how to protect we have the generic things that we have in a plan name a D trigger it can be manual time on event, which bank to store and option this is the dynamic part so we allow vendor plugin to specify which special parameter they are having their plugin so it can be for instance in consistency you can have a parameter saying consistency level in the schema that says it can be OS crash or application and the result schema provide what's the data that I should put in the ledger to what to protect and where to protect is the bank that needs to support Read&Write API this is a bit complicated but we'll have more documentation about it and more presentation about it to explain it and it's still evolving the concept and you can see under the bank API we currently have Swift S3 you can create your own plugin there so if you like to help us build the the the dragon you see is a little Lego pieces that needs to be assembled please contact us or contact me after that we have a launchpad next week we're going to launch the product start with defining the API and please come to me with any question and now we would like to have a short demo show how it can look from user perspective is from a product the IBM L and I will launch the video and later we will have time for questions and answers we base this work on workload and application centric approach implemented and available in IBM contact manager, cloud manager and PS7 orbit European project with the idea to use it as a base to further extend and develop with others so I want to show you how it looks from the user perspective so we introduced a new disaster recovery service and you can see the disaster recovery panel on the left and it allows the user to log into the cloud protect his workloads and resource all related resources and later recover them on the cloud so first to create a workload policy in this case we are going to protect a single failure with running WordPress so we create the policy we are going to put all the resources and we are going to edit the policy and add the resources we want to protect and define what protection action we want to use at the moment we support for two different resources for servers we support image copy to start it from for stateless servers or snapshot to recover from a point in time and for volume we support snapshot and replication so once we define the resources we want to protect and the reason we put it on a policies we might want to have several different policies with different backup policies we trigger the protect and we see it running we can take a look at the execution and see in the detail the state what it's doing now so what's going now what's happening now is that we actually collect all the resources and all the information needed for the recovery package it and upload it to a safe as I mentioned the safe can be any storage in this case we put it into swift object storage okay so it's working and now it's in protected state and now if there is a disaster so the tenant login to secondary cloud when we use snapshot like in this case the site doesn't have to be up and running during protect everything is stored in the object store so when for a disaster we can set up a secondary cloud login go to the disaster recovery panel again invoke the restore here we can see all the workload we have protected and once we click on the workload we can see all the on the workload policy we can see all the point in time select a specific point in time invoke restore and since we orchestrate this deployment using it you can actually go to the stacks and see your application deploy it again on the secondary cloud okay so you can see the server is up and running on a different cloud thank you if you have a we have a little bit of time I think still so we can have short question and answers for their disaster recovery which one the hypervisor based yeah this okay so the cost I try to cover it you pay memory on the compute nodes you have the virtual replication gateway that you need to deal with it not the the vendor not the array vendor and the performance depends on the one connectivity so currently we try to push it as soon as possible so the limitation in that that you if you write to the same block a lot of time you will send it over the one a lot but if your one connectivity is good you you can get a very good RPO of seconds in this solution yes yes yes and we trying to improve the implementation all the time and there is other approaches you have OS application hypervisor but we just covered the hypervisor and there's many many other solution and we try to improve our solution and we already use it in some of our production yes yes local disk is no problem we we can so the question just if someone didn't hear the question can can we replicate ephemeral storage regular storage that is not volume based so we can no problem that's one of the advantage of hypervisor based replication if they are VM aware virtualization aware they know everything about this VM from the hypervisor perspective usually we recommend to establish a new connection dedicated for that so it depends on your infrastructure basically but you can have a dedicated one connection for disaster recovery of course you need high availability for that or you can reuse your own the normal one but are you talking about east-west inside the cloud inside your DC so it's minimal effect because the only time that currently in the solution we have a huge effect is during consistency check then we load all the volume and we bring it to the VRG but on a regular basis it's just what you write goes so for what we saw it doesn't have a huge effect of course there's some effect but yes but it has some effect we saw it's minimal on the regular use case not in the consistency check state both both so basically when you have this vault and your bank so you have what the two sides need to know is where is this vault located so in this case it was Swift so in Swift they had the idea of Swift and in the other side it just needed to go the data it got all the metadata from the packaging we didn't go into it and then it knew how to orchestrate and to create it on the other side that's the recover part so in the any plugin can implement it this is the recover part of the plugin in which because we present Swift this was the demo but it can be any recovery part I can answer this question in the scenario we showed we used a snapshot method for the server so we got a point in time we uploaded to Swift to object store in this case which we recommend to put it for geography dispersion at a third geography location and then when the recover site is up you can connect it's accessible it can access the same Swift and pull it in the replication case we need to have replication supported storage in both sites and then both of the sites are up and running and data is going for the replication path and what we store in the safe is all the information needed to run it and deploy it we store made data that action is used so we can later dynamically enable them and run the restore API of the volume replication in this case which is going to import the volume so it's going to be defined on the recover site and since the data is already there we just need our template to be aware of the ID on the recover site and then we run the template and so it's it depends on the case but it's recommended if you want each of the components to be we just have another question yes yes we want to support all workflows and for for all the data protection yes okay this is a very good question that we didn't really into it but we currently started by the tenant application the virtual resource you're asking how do I protect my agents so we think that we will follow something similar plugin but we might need to introduce something new it can be using a project that we have here like Kola are you familiar with it that wrap it into a container so we didn't go into that if you like to join us and to define and if you think it's a real use case we would love to add it as the one of the first thing to do yeah we know but we just thought to start with the user perspective meaning the tenant perspective and then go into the admin perspective but it's a very good point that yes so we didn't and you saw we didn't want to say that array based replication is bad just what we say it makes sense for some user and I think you agree with me that the software based replication makes sense as well that's the need for workflow management for data protection and that's why we use the service we provide both by the way because we think they're both viable as we say de-deplication efficiency I didn't go very deep on the hypervisor best advantages as well I just wanted to show that there are both advantages and disadvantages yes this is the link if you want to download it this qr code and I will I will fix it now I'm sorry about that if you have any questions yes more questions yes okay so currently we do it with what we show you as the Huawei replication manager so we manage the mapping we load the VM we know how to how to load and all of that we want all of this logic to be pushed into the resource protection plugin so we will provide a resource protection maybe plugin for ephemeral storage so and it's possible with this architecture okay okay thank you very much thank you thank you Sri thank you