 All right, looks like we're good to go. Welcome, everyone. Thanks for staying with us until the end of the day. This is Haye Storage Engineer. Tell me about backups in OpenStack. Wait, Akshay, is this for storage engineers, the session? No, this session, we are supposed to be the storage engineers. You may or may not be storage engineers in the audience. We'll be providing you information around backups as applied to OpenStack infrastructure. Thanks, so these guys just need to know OpenStack, right? Absolutely, yeah. So my name is Akshay Parthasarathy. I'm a technical marketing engineer at NetApp. You can see my Twitter, GitHub, and LinkedIn details over there. I've been in the technology industry for over seven years now, having previously worked at Dell and Amazon services. I'm joined today by Kapil Aurora. Kapil, would you like to say a few words about yourself? Yeah, my name is Kapil, and I'm a cloud platform architect at NetApp. I've been with NetApp for around six years. And in my current role, I'm helping customers do proof of concepts or implementations around OpenStack and also help them integrate our enterprise storage within OpenStack. Awesome, thanks, Kapil. OK, moving on, we're going to be showing you a couple of use cases. And what we want to mention as you look through the demos is these were done in a proof of concept deployment for the sake of preparing the demos, not necessarily the same things that would be true in production. Our agenda, we'll start with the exciting stuff, which is two use cases and demos. The first use case is going to be with Cinder, Block Storage. And I'll walk you through 10 PMs with Nova and Cinder. And this is going to be it on a VDI environment or otherwise it's going to use Cinder. The second use case Kapil is going to walk you through is using Manila with SAP HANA as the enterprise application. Manila is the file share storage project in OpenStack. After that, we'll take you through what is a backup and why you should be backing up following replication and disaster recovery. And finally, Kapil is going to close off with some application design goals for backup and recovery. So onward and forward, we're going to look at 10 PMs backup with Nova and Cinder. So this is our first use case. Now, this could be in a VDI environment or otherwise. Typically, you might have, as an IT administrator, you might have a lot of tenants in your OpenStack infrastructure taking advantage of that cell service provisioning that OpenStack will be able for you in addition to the cost advantages. So you see here an OpenStack infrastructure on the right-hand side, on your left-hand side, you see some generic compute nodes. Maybe you're using commodity hardware with local disks. We want to suggest an alternative to you, which is you can use an enterprise storage back-end, such as NetApp, for your Cinder. And what this allows you to do, especially in the case of NetApp's solid fire, it will give you that capability to do that live migration of your tenant VM instances. In addition to that, you can use glance images on your enterprise storage back-end. And that gives you that deduplication, compression, thin provisioning, those types of enterprise class features that you can take advantage of. So as an infrastructure as a service provider, you're going to want to offer backups. Nobody wants, none of your tenants wants to come in the next day and find their VDI instance, not have the data it had the previous night when they left. And what we are going to recommend to you is use Cinder for your boot volumes. Use Cinder Snapshot to take a snapshot of your point in time, take a point in time copy of your Cinder volume, and then use Cinder Backup for consistent backups. Now, when you use Cinder Snapshot and Cinder Backup in the way we recommend, you're going to get something that's going to be a consistent backup. So this is something that you can use to bring up a VM from backup that is not crash consistent, that is actually file system consistent. So we will walk you through backup Cinder to Swift, which is a traditional way of doing things. So you may already be familiar with the Cinder volume service. And what that allows you to do is to provision and snapshot your Cinder volumes. And in NetApp's terminology, that's called FlexClip. You can also use Cinder Backup in conjunction to that. And this will allow you to backup to a Swift target. The better option may be for you to consider using an NFS target with the Cloud Gateway. In the case of NetApp, the Cloud Gateway is NetApp's Altavolt. And Altavolt gives you a significant advantage through deduplication compression. And if you do choose to use encryption for security, that's something you can get to. In addition to that, you can also, in Cloud Appliances, such as Altavolt, cache your most recent backups. So when you do want to restore from a backup, it's going to be faster. Now, in the case of Altavolt, you can backup through NetApp's terapetabyte scale object storage known as storage grid web scale. You can backup to AWS S3, Google Cloud Storage, Microsoft Azure. You name it. Most of it is covered as a backup target with Altavolt. There are reasons why you would want to use, in some cases, the public cloud. Reason being that you want that kind of scalability, that kind of elasticity that cloud infrastructures provide. Cinder backup configuration is extremely simple. If you're familiar with Cinderconf, in the default stanza, you will either choose Swift or NFS. If you want to choose Swift, you just provide the backup driver as shown here. If you want to choose NFS, you change your backup driver. And then you need to also provide the backup share location. There are a couple of additional options you need to provide if you do choose NFS. Those are listed over here. Now on to backup commands. The first command and the most important one is backup create. Backup create creates a backup that is not necessarily falters and consistent. And you can do it incrementally as well. Backup delete will delete a backup. Backup list lists all the backups. Backup show will show the details of your backup, including a timestamp of a particular backup. You can restore from a backup using backup restore. You can export and import your backups. Let's say you have an availability zone with OpenStack deployed, and your availability zone goes down. If you have exported your backup backups, then you can bring up. You can re-import the backup metadata into another availability zone. And take advantage. If you have something like an outer wall, you can take advantage of that backup target again. So the VM backup workflow to take a falters and consistent backup will involve a falters and agent. On the VM, the VM is using an enterprise back end. And you do a nova image create. That sends a request to create SIO to the falters and agent. And it creates a sender snapshot. Once you have that temporary sender volume snapshot, you can then take a sender. You can create a sender volume from that snapshot. And once you have that sender volume, you can create a backup from that sender volume using sender backup create. When you use sender backup create, again, you have that option to use a backup gateway. You can then delete your temporary sender volume. You can delete your temporary sender snapshot. And that's how you have a consistent bug. I talked about consistency a bit. And the way we suggest you doing that is using a file system guest agent for consistent snapshots. If you recall from the previous slide, we take a snapshot first. We create a new sender volume. And then we take a backup of that sender volume. So in order to get a consistent snapshot first, let's say you're using KBM or QEMU, the default hypervisor, the most common hypervisor for OpenStack. And you have to have your instance have a QEMU guest agent. For further details on the guest agent, please refer to the reference at the bottom of the slide. The instance, of course, has a file system. The QEMU guest agent gives you a freeze and talk functionality that is going to query SIO accordingly for the snapshot. The NovaImageCreate command sends a request to the hypervisor. The hypervisor, in turn, contacts a guest agent, which freezes the file system. Once the file system is frozen, NovaImageCreate proceeds with creating the snapshot. Once the snapshot is created, the hypervisor then notifies a guest agent to thaw the file system. Once a file system is thawed, you now have a consistent snapshot and IEO operations proceed like normal. I'm now going to show you a demo. This demo uses two NetApp products. It uses a product called NetApp AlterVault, which is your backup K2A. And it uses another petabyte scale object storage called StorageGridWebScale. So going on with the demo. So this is, again, about 10 NPMs in a VDI type of environment. So let's say we have five VDI instances. And for each of those VDI instances, we have them on an enterprise back end. ClusterDataHuntApp, for example, those are your boot volumes. And we notice we have two NFS shares, the first share for the boot volumes and the second share for your backup target. Now let's take a look at Cinderconf and what has been modified over there. All the changes were in the default stanza. And the backup configuration is over there. We're using the NFS driver and we're using that NFS export path. This was the same path you saw earlier in the demo. So these are some scripts that I've made. And I'll make them available later today. This one creates backups. And you can use this in a schedule job to take backups. In order to use this script successfully, you need to enable the QEMO guest agent, take a temporary image that is taken care of it, get the temporary snapshot that it does, get the snapshot ID, create the backup. So it creates a backup for you once the guest agent is in place. Here is the schedule task using Cron. And it takes a backup every 90 minutes. So if you do want to restore from a backup, then let's say you need to use a backup restore command. And we're restoring from the second to last backup in this case. So we do Cinder backup restore. And we enter the backup ID. And if we do a Cinder list, indeed we see that we have a new Cinder volume that is being restored. So we're not going to completely wait until the whole thing completes. I'll save you that pane. So this is a tenant VM. And what I've done here is we are periodically generating random content in order to make this a realistic demo. So we're going back to the OpenStack instance. And we're going to use a script to sort all the backups of one of the tenant VMs. Let's go all the way back up since the first backup. And the first backup was at 2201 UTC, which is 1801 EDT. And the second backup was at 2332 UTC 1132 EDT. So this is NetApps Altavolt, which is a cloud gateway, which is a cloud backup gateway. And let's see what happens with the duplication compression here. So after the first backup, that's our initial point. And the blue represents the normal consumed data that would happen. With Altavolt, we're reducing that to 18.4 from 54 GB. So this is after just one backup of five VDI instances. Let's see what happens just before the fifth backup. In other words, after four backups. So I'm adjusting the window over here. So this is after four backups. After the first backup, we see 2.96x in savings, second 5.86, after the third, 8.73. And after the fourth, we're up to 11.59. So we see a 10x space savings at that point in time. So this got me curious. So let's see at what point in time we get to a 50x savings. So a 50x savings, which means you're reducing one terabyte of data to 20 gigabytes. One terabyte to 20 gigabytes happens after just 18 backups. So that is using NetApp's Altavolt with StorageGridWebScale. And this exercise was done in the lab at NetApp. So you've seen a bit about tenant VMs, VDI instances, and center backups. So Kappa, what are you going to tell us? Yeah, I'm going to show you some enterprise applications and how do you do backups in them, and basically how do you deploy enterprise applications. And you can see the nice logo for Manila. So we based this demo on Manila and an SAP application, which is SAP HANA. So let's see how we deployed our application. And this is an overview of how the deployment looks like. And you can see that on the right-hand side, we have the HANA nodes. So HANA nodes are basically NOVA instances, and we have multiple HANA nodes. This means that we have deployed HANA in a multi-node environment. It's a scale-out architecture, and it's not just one host, but multiple nodes of the same database. And this is running, of course, on NOVA. And on the left-hand side, you can see I have all the storage components. And Cinder and Glance are used to provide boot instances for my NOVA instances, and boot volumes for my NOVA instances. And Manila is used to provide NFS shares for HANA. And all the data that we have related to HANA is sitting on the Manila shares. You can see I have a shared share. I have data and log, which is a very regular way to deploy databases. And everything is sitting on Manila shares, which is related to HANA. Let's move on and see how does the data path look. So basically, the data is connected directly to the NOVA instances using NFS. And you can see the HANA shares are directly connected using NFS. And in case of the NOVA instance boot volumes, we are using NFS, or either iSCSI. In my case, the boot volumes are connected using NFS. And then the hypervisor exposes it as a block device. Now for any enterprise application or any database, there is a very similar way how you take backups. And that is also the case with HANA. So you would first query a secure database or put it in a read-only mode. And that's exactly what I'm going to do with HANA as well. And I'm going to put it in a read-only mode, which is called HANA Snapshot. Once I do that, the database is actually ready to take backup, ready to take a storage-based snapshot. Once we do that, I'm going to take a snapshot of my Manila shares. And in this example, I'm creating a consistency group out of all my Manila shares. So you can see I have three shares. Those would be data shares. And I'm creating a consistency group. Although HANA doesn't require me to have a consistency group, but in this example, I'm just trying to showcase the functionality. And the advantage that I got out of that was all my data shares, say, there were two. So I did not have to take two snapshots. I just needed to take one snapshot. And imagine I have a huge cluster database, and there are multiple data shares involved, then it makes my job a little easier. And as a last step, I'm going to close the HANA snapshot. So closing HANA snapshot means telling the database, OK, my storage-based backup was completed. Now you can go ahead and start regular operation from your side. So this is the backup workflow. And I'm going to show you in a demo how we do that, and also show you the recovery. So you can see these are my two HANA instances, HANA OSK1 and 2. And these are the Manila shares. And now this is the Manila CLI. I'm just listing the shares. And what I'm trying to show in the next couple of seconds is that I've created a consistency group using Manila. And I have two data shares, and both these shares are part of the same consistency group. So I have the consistency group called HANA data CG. And this is my first share. I'm just matching the IDs so that you can see that these two shares are part of the consistency group. So this is my second share, and it also has the same ID for the consistency group ID. So now that we have made that clear, this is the HANA node, the NOVA instance. And I'm just showing that we have mounted the Manila shares here. So these are all the shares, and they are mounted on my database node, both of them. And now we will log into our database. So our database is deployed, running, and we are creating some sample data. So for my tests for recovery, I'm creating three tables. One is called region. The other is called sales. And there's one more called, I forgot the name. Yeah, so there are three tables, just some dummy data. Now that we have some data, let's look at the HANA UI. So this is basically an administration console, and the HANA database is running. All the services are running. We can see that there are two nodes, and all the services are green. And that's where you can actually start a backup. So I'm telling HANA, OK, I'm going to take a storage-based snapshot. You go ahead and prepare for that. So this is like a HANA snapshot or preparing HANA database. And we see that the HANA database is prepared. Now the next step is to actually take a storage-based snapshot, and that would be the consistency group snapshot. So I'm just taking the consistency group snapshot. It's just one single command, and my database is actually backed up. And the last step that we saw in the diagram, we just need to make sure that the HANA database is aware that we took the backup. And we are just closing the snapshot and telling it that the storage snapshot was successful, and it can register it. And as soon as we do that, you can see that that is the HANA backup catalog, and I have an available backup. And you can see the time that we took to take the backup was 55 seconds. And that is the advantage of a storage-based snapshot. Now let's try recovery. And what I'm going to do is I just deleted the sales table from my database. You can see there are just two tables now. Maybe that was my mistake, and now I want to go ahead and recover it. So I start recovery within the HANA studio itself, and I have to provide my username password, of course, and the HANA database is going to go down and start recovery process. So it's just collecting some data at the moment. And now it starts a wizard for me, and I select, OK, start my snapshot-based recovery. And I provide my log location. And now I can see, OK, these are the snapshot-based backups that are available. So what I need to do now is actually restore from the storage site. So I'm going to do a recovery of my consistency group snapshot. You can see that's the command to do that. Use the snapshot that we created. And then I have two new Manila shares, which need to be mounted on the HANA nodes. So I do that. As soon as my data, which was backed up, is available, it starts showing green in my wizard. And now I can go ahead and start the recovery. So HANA is aware. Now the backup is on disk, and I can start recovery for this particular. And we start the recovery process. It takes some time, but here it is a little fast-forwarded. Once that is completed, we can see that the database is back up, all the services are running. And now let's see if my sales table is back. You can see that the sales table is there, and also the content of the sales table exists. So our recovery was successful. So that was a simple backup recovery workflow of a database. And that is not the only use case that somebody needs in an enterprise storage, enterprise class application. We saw the use case where we use snapshot. And that can help us actually to do a HANA dev and test or backup system creation. And that applies to all enterprise applications, including all SAP systems. And you use Manila snapshot for that. And the Manila replication technology, which we didn't showcase, but if we use the right tools from SAP, and we use Manila replication technology, we can solve the disaster recovery solutions, which we need in an enterprise class application. And the onboarding of the traditional applications are enabled by features like replication and consistency groups, which are important for enterprise applications. And that was my enterprise application demo. And now, actually, can you tell us more about why really do we need backups? Exactly. That's a great point. So you've seen a couple of use cases, VDI, tenant VM instances with Cinder, and SAP HANA using file shares with Manila. But maybe you don't have these two applications. So does that mean you don't want to be taking backups? No, that's not the case. So you could have human errors, data corruption, application bugs, you name it. You want to be creating backups of your data. And the only variable in consideration is how frequent do you want to keep these backups? What needs to be backed up? Pretty much any data you cannot afford to lose, and your tenants cannot afford to lose. VM instances, databases, documents, files, logs, of essential servers, things of that nature. So a couple alluded to this earlier. Replication and disaster recovery. Now, this is a topic that is very related to backups. And in the case of Manila, we have introduced share replication in the Metacle release. Share replication allows you to replicate a share and aface ourselves from one availability zone to another. Or if you have a single availability zone, a single data center, you can do that too. You can replicate between multiple clusters in a single availability zone. But if you have only one cluster and you want to replicate your shares within that one cluster, that is another option. If you want to do that, not particularly something you would do in the event of a disaster, but you can do that. Why do you want to use share replication? Non-disruptive operations. So with multiple availability zones, you can provide for disaster recovery, and you can keep your clients running and using their file shares. So with share replication in Manila, they make use of the API, Scheduler, and share services. So shown here is your first availability zone. And this is, let's say, in a replication domain. So in Manila Contacts, replication works using this replication domain parameter in your configuration file. Let's say now you have a second availability zone sharing a common Keystone infrastructure. Then that availability zone has another storage back end, and it has a share back end for that particular storage. When you issue a share replica create command, that goes to the API service. Within your first availability zone, the API service then identifies a share back end that can fulfill that request to create a replica. Once the share back end receives that replica, it contacts the storage in that availability zone. The share is then replicated using the vendor's replication technology. In the case of NetApp, it's known as Snack Mirror. Once the replication is complete, then the replica status is updated to available in your OpenStack database. So that's what happens when you do a share replica create. In the case of Cinder, the story is evolving. So Cinder replication is available in SolidFire and some other vendors. If you do choose to make use of SolidFire for your Cinder application, then you also gain some other advantages that SolidFire is going to provide you. De-duplication in particular, compression, and you have that high-performance solid state desk that you can make use of. In addition to that, SolidFire provides you that minimum and maximum IOPS that you can set. So you can have a guaranteed quality of service for your talent's block storage. For a Cinder replication, we're going to have it upcoming in Newton for cluster data on DAB and E-series. And E-series. In the case of SolidFire, you do replication by configuring a replication device, as shown here. The first part of it with the back end ID is generic across back ends. And everything after that is specific to your storage. So that's how your configuration file needs to be modified. Kapil mentioned this previously. Like in the case of disasters, how do you ensure that you minimize downtime and you keep your enterprise running? Let's say you have a number of compute nodes and you have VMs, tenant VMs. And each of those tenant VMs has boot volumes. And these boot volumes are resident on a storage back end, such as NetApp. You make use of storage side replication. In the case of NetApp's cluster data on DAB, it's known as SnapMirror. It replicates to a storage device in your second availability zone. And in the second availability zone, you have these services running in hot stand by Nova and Cinder. Let's say a disaster does occur. Then what you do is you break that replication relationship, and then you bring up your VMs in your second availability zone. So you can also make use of storage-based replication for your glance image repositories. So this is similar to Cinder. You might have customized administrator images that you want to preserve. So you cannot afford to lose these images. You can have hundreds of images created by tenants. And they would not be happy to find that their images are gone. You use storage side replication again. In the case of NetApp, it will be SnapMirror. And there are significant efficiency gains that you can have. These technologies may use compression. So that's going to save you a lot of network bandwidth. In addition to that, periodic snapshots will also be replicated over in the case of cluster data on DAB from NetApp. So we have touched on two applications. We have touched on SAP HANA as an application, and we have touched on BDI as another use case. That just got me thinking, what about general application design codes, including those for cloud-native applications? Yeah, so as Akshay mentioned, we just talked about one application, which is enterprise applications. But there are lots of other applications which get deployed on an OpenStack Cloud. So I just want to leave you guys with some thoughts around, how would you deploy these applications? What should you think about before considering backup for these applications? So the first and foremost point is to know, OK, there are three important storage projects in OpenStack. And whenever you are considering persistence of your storage, you should think about these three. Cinder for block storage, Manila for file share services, and Swift for object. And now let's look at some of the design goals. What should you actually do when you are designing your applications? First and foremost, I would say that making your servers stateless is very important. What I mean by that is that all your ephemeral and reproducible data should be as part of Yanova instances. And these instances should be stateless. And how you create these instances should be automated. So these are some of the DevOps principles as well. So basically, you are decoupling your software from your hardware, as well as your persistent storage from your non-persistent storage. And when you automate your servers, you should also maintain them in an SEM or GitHub so that you can document it really well. And when we talk about application backup, nobody else knows how to take a backup unless it's a backup tool or application itself. So you cannot just use Cinder backup or Nova backup, Nova snapshot to take a backup of an application. You need something which can talk to the application. So it has to be a backup tool or it has to be the developer who needs to provide an API which can put the application in a consistent mode or in a read-only mode, so to say. So it's important to consider backup tools when you are taking backups. And another DevOps principle I would like to mention is monitoring. It is important to take backups. And you would, of course, schedule all these backups. In my demo, I showed one backup and one restore, but you would, of course, schedule them and create a script and create a cron job like Akshay did. But it is also important to monitor them and see if you have recoverable backups available all the time. And lastly, once you make your server stateless and once you automate the process and also separate your persistence storage, what it enables you is to do automated recovery as well, because your storage is separated and you can also replicate it. For example, and on the other side, you can also use the same scripts to create your new systems. So it enables automated recovery for you. So now let's look at how does the picture look when we talk about all kinds of data and what goes inside Nova. So I talk about ephemeral data. So what is that? Basically reproducible data, your images, your binaries, which are static data that goes into your ephemeral storage. And of course, Nova, sorry, Cinder and Manila and Swift, these are the persistent storage projects. And what do you store in them? Of course, application data, configuration files, your logs, and in case of Swift, your object storage, right? And why are we doing this decoupling? And how can it enable us? Basically, this makes your application mobile. It makes it platform-independent, so to say. So once you have segregated your persistent and non-persistent storage, you can move your application from one platform to another or one technology to another. And how you do that, it's very simple. Because your data is segregated, you can write new automation scripts to provision your servers or your compute, for example, using new technologies, like containerized technologies, Kubernetes or Magnum. So if we use such principles, and in the new age where you have multiple clouds, multiple new technologies coming up, it makes a lot of sense to be able to segregate these data and provide mobility to your applications. So that's what I wanted to leave you with. Thank you so much. Akshay, you want to say something before we leave? Sure. So on behalf of NetApp, a couple of myself, thank you for staying here till the end of the day. If you do want to meet with some of our executives, please schedule a VIP meeting with us. Follow us on Twitter, and please visit netapp.github.io to get more information. That's where our open stack details are posted. Thanks again. You guys have integration with Horizon that you can probably do one click kind of backups of a VM or particular volume. I know it's not very difficult, at the same time have you done it? So I personally didn't use Horizon for the backups, but what I can do for you is using the CLI. My experience has been that we don't get a time ordered series of backups, not yet. So I will be making a script, a few scripts available later today, and that's going to give you a time ordered series of backups. But regarding Horizon, that's something. Okay, you want to talk? So usually you would not click backup and say back it up. Usually they are scheduled, but of course all the backup tools provide that. That was something I wanted to mention. So it is not a very critical function usually, because backup functionality is usually provided by the administrators, but it could also be a click in backup, which is a good point, I would say. But we don't play around with possibility of that. Right. Yeah, you're right, yeah. It looks cool, I would say. It's nicer to do a demo with that instead of the CLI. So it definitely does look cool. The thing with doing it via Horizon is that we wanted to showcase a case where you have a very consistent backup that is not necessarily going to be crash consistent, that is going to be something that can bring up your center volumes in a state that can be used immediately. So that's why we went this way. And in order to do that, we had to actually take a snapshot, and then we had to create a temporary center volume, then create a backup room for that volume. So that's why we went through this route. One more, I did. I know you mentioned that you had to do snap mirror-based replication across sites. Is the replication is supported in Manila? It has only backend device capability that it's relegated to the storage array? Or is it something that Manila itself can do, or both? Right, so right now Manila replication is available in NetApp, and other storage vendors will be adding that capability. So what happens is, and we do have a developer here, what happens is the Manila replication command does use a vendor, an enterprise vendor command, that will do replication, right? So in the case of NetApp, it's snap mirror, exactly. It's got to be a backend provider's capability. It's not native to Manila, but it is a capability that's providing and it's using that backends. Thank you. There are no any other questions, and thank you so much.