 Hello. Can you hear me now? Hello, there are people there. I think I see a few of you out there. Welcome to Developing a Backup and Recovery Solution for Tenancy. My name is Mark West. This is Preston Bannister, and we are with EMC Corporation. This session will cover what it was like to build a product for OpenStack outside of the Big Tent and some of the challenges that we faced in that matter. Go ahead. All right. About two years ago, we were asked by customers, our existing Avamar customers, to build a backup solution for OpenStack. And we learned a few things along the way. And part of the objective here is to sort of give you some context. So when you're looking at backup solutions, including ours, you know what to look for. I think that's about what I want to say. The other thing I should mention is that prior, we built backup for a VMRV cloud. So building backup for a cloud is not a new thing for us. So that is Preston's background. My background is more on the professional services side and doing services delivery, building private cloud for the enterprise for about 15 years. So my contributions come into building software pipeline and working with our OpenStack partners, Marantys, Red Hat, and Ubuntu, where we test and certify all of our products in our labs. Ready? Go ahead. First, let's clarify what sort of backup we're talking about. If you look for backup and OpenStack using Google, you generally get back three different kinds of results. There's backup of your OpenStack infrastructure. That's not our focus. There's backup of the bulk data in the cloud. That also is not our focus. Then there's backup of instances for tenants within the cloud. And that's where we're focused on today. We built a backup service for the cloud. We learned from the exercise. We hope we can give you some context from doing that. Once you have a backup service, you need to integrate with operations. That's more Mark's side. And he'll speak to that later. In the end, we hope you will have gained enough to ask the right set of questions when you're evaluating a backup service. Thank you. For a moment, I'd just like to take a quick show of hands. How many people have data protection for their OpenStack clouds? Oh, that's scary. But with that, you might be realizing or come to be realizing that having this enterprise backup or backing up your tenancy data is probably even more important than backing up your OpenStack infrastructure controller nodes, et cetera. Oh, sorry. Thanks. Need that. We're offering a different backup on an instance volume basis, also potentially self-service. Whether you're an engineer in ISV, our Enterprise Data Protection Solution allows you to have full continuity for your backup lifecycle. This provides an appliance-like backup functionality for OpenStack, allowing your tenants to be self-service within their tenancy, or allowing you to apply policy to global level. We'll cover more on that later. All right, let's visit the pets versus cattle analogy. You've probably heard this before. The original notion was that if you built a cloud, if you build a cloud like Google builds a cloud, what you would put in your cloud would be cattle. Instances that had no data to be backed up, all the data is actually in the cloud and object storage or something like that, and replicated external to the instance. And so cattle, they don't need backup. They don't need traditional sorts of backup. What we call pets, on the other hand, are the more traditional sorts of applications. These instances do contain data that we need to protect and preserve, and thus we need backup. If clouds contain only cattle, then traditional sorts of backup, what we're selling, is not needed. But our customers have told us very clearly that they're moving pets to the cloud. They're valued applications in very large numbers. Also, from my point of view as a developer, if I'm going to deploy an application, I like the idea of deploying it to the cloud. I would rather deploy it there than anyone else, so pets make a lot of sense to me. When you're building a backup product for enterprise use, there are four things that we need to mainly consider. Consistency, efficiency, automation, and control. First, we want to collect everything possible to have a high-quality backup. This means capturing everything of use and in the most consistent state possible. Second, we want to collect backups as efficiently as possible. Backups involve huge amounts of data, and we want to minimize the amount of impact on your infrastructure. And last, you need control over usage. This means providing policies and schedules at both the site and tenant level. Speaking to design considerations, backup puts load on your hypervisor hosts. It puts load on your storage. It puts load on your networks. We need to think about all of those aspects because we're moving huge amounts of data. We need to give your site control over where usage and load occurs and when. We need scheduled and on-demand backups, not just on-demand backups, and one-step restores. And generally, we need to be a good neighbor within your infrastructure. There are other challenges. OpenStack development to date has not been especially focused on backup. This means that the OpenStack APIs do not offer everything we would like to have when building a backup service. There are a couple of backup APIs in Nova and Cinder, but if you're trying to use them in your operations at scale, they're really not what you want. In terms of implementing backup, getting a backup out of OpenStack is kind of like getting a backup out of Windows in the 90s. If you build a backup application now for Windows, they have absolutely excellent APIs for backup. But we're not yet there with OpenStack. Main things we found that we're missing, at least for our purpose, we need to be able to capture time and space-efficient readable backups. We need to get change block tracking at the source. That vastly reduces the amount of data we have to use, have to move. And we would like to be able to create clone snapshots of volumes that don't count against the tenant quota, but account against the backup service quota, which means cloning across tenants, which is not possible in OpenStack today. And we would like application-consistent backups. There is very nearly support in Nova, nearly landed in Mataka, but didn't quite make it. Things you should look for, kind of reiterating things here. Make sure that the backup service you're looking at is minimizing the impact on your host and your network and your storage. If we're moving around a lot of data, you don't want to move it around any more often than you absolutely have to. You want to be sure that your running applications are minimally disrupted by backup, or not at all. And you want to know that the backups taken are complete and consistent. We need to capture the entire backup state, all the instances, all the volumes, all the metadata, everything associated with the instance, so that we can do a restore at a later time. And given that we've done a complete capture, we can do a one-step restore. I'm going to mention our implementation a little, and that's about it. We have an internal use service that lands on the controller nodes. We did this so we can get HA, along with the rest of the open stack services. That's critical for doing backups. We land proxies in each availability zone. The reason is we need locality of storage. Storage is not necessarily accessible across availability zones. We offer a REST API for management. This is what you would be programming to or using. And we keep a tenant within the cloud. As much as possible, we've implemented our backup service as a service within the cloud. And so our tenant owns the management and proxy instances, and owns quotas, so you have control over usage there. And our back-end storage is with EMC Avamar, which means you've got a mature target with all the other features that it offers, which we are not going to enumerate here. Yours. Thank you, Preston. So our EMC data protection service is designed to work with Avamar and data domain at the back end. So if you have an existing Avamar data domain in your enterprise environment, this is an obvious solution for you. There's no additional licensing. There are no additional fees. It is an extension of the Avamar that you've known and loved for years. We work with any block storage. It does not matter whether you have EMC storage or not. However, there are some reasons why you might want to have EMC storage. As for policy-driven management API, one of the key things that was important in the development of our product was giving administrators this ability to implement a backup policy to all of their tenancy. So whether you want to instantiate backup and be self-service or force a backup against all of your tenancy, that becomes an option through this policy-driven management API. One of the things I also wanted to discuss during our development of this product and continuing development of this product are the tools that we used in our development environments. One of the things that I mentioned earlier were our OpenStack partners, Mirantis, Red Hat, and Ubuntu. And so as we're building our Avamar data protection service, or our EMC OpenStack data protection service, we're noticing that we have to test on Mirantis. We have to use fuel to deploy our Mirantis nodes. We have to deploy on Red Hat and use Platform Director and deploy on Red Hat in that fashion. And then we use Juju and Mass to deploy Ubuntu. And so in our labs, we run three different architectures of OpenStack or three different flavors of clouds at all time. And this allows us to be more agile and allow us to do rolling upgrades much more quickly. The tools that came in handy in doing this were primarily things like Ansible, including Ansible Tower. The team had a really short learning curve with implementing that particular configuration management tool and orchestration framework versus competing products. We also leaned really heavy on the form inside, which is a free open-source software tool, if you've seen that within community, to handle our bare metal. And so when we weren't using provisioning tools from OpenStack partners, we were spinning up Ubuntu and CentOS or RELL images using our form and product. And then, of course, VMware. We really relied heavily on our VMware environments to create pseudo labs or spin up all-in-one instances. So whether you're Dev or QA, you have access to a fully functional environment that concludes our API, our running product, our latest version of our RPMs, QCAU2s, et cetera. We currently target Kilo and Liberty with support from Ataka coming shortly thereafter. One of the important things to remember and that differentiates our product from other products is that we are a purpose-built backup appliance. I've been at the booth this week. If you are on the floor, please come see us at the booth. And one of the questions I had was, well, the other solution, it provides cloud for me. Well, that's fair enough. But what happens when your cloud goes down and your controller node infrastructure is no longer in place? How do you restore that data? Where does that data come from? So this creates a point of recovery that lives outside of your OpenStack cloud, something that differentiates our product within the marketplace. All right. Sort of to sum up what we learned in the process of implementing this backup service, we have to work with what we have, what's present in OpenStack, in the versions we're shipping to customers. We're using the existing OpenStack APIs. We can't yet. Nobody can. Offer entirely optimal performance. Change block tracking is probably the biggest issue here. We ran into some limitations in OpenStack, trying to do restores on instances that are currently active. You can't attach the boot volume. There's a bug in Nova about that that will probably get fixed. We need volume clone across tenants so we can do it in fewer operations, less risk of failure. And we need efficient volume clone from sender drivers. Something else I kind of wanted to mention, because I really haven't seen it mentioned in OpenStack in context very much, is that in doing our performance testing, we found that the Linux iSCSI QMU stack made a great deal of difference in terms of performance that what has happened between earlier versions of the kernel, what's in Red Hat and CentOS, and what's in Ubuntu trustee in theory is a very large difference in performance. And we care about that a lot because we're doing a lot of volume IO. In future, what we're looking for is the change block tracking coming from QMU. I learned a little bit extra about that today, by the way. We're looking for proper application queues in Nova. So again, it almost landed in Mataka so it should land in the next release. And hopefully we will be able to address the remaining limitations with the OpenStack community. Yours. Moreover, we want to continue to contribute to that community. And so have our OpenStack management API for backup and recovery be built upon whether that orchestration is through heat or CloudForms or Juju or others. We want to make sure that all of that orchestration templating is available within communities so our customers can build upon that. Are you going to attempt to show the demo here? No, we are not going to show the demo today, unfortunately. And you can check out the demo in our booth. It is available at BoothB7 and that's what we have. Any questions? Well, we were supposed to mention there's a raffle for an Amazon Echo. We both forgot. The Amazon Echo is coming up at the end. We have Q&A. There's a microphone right there so if they want to ask us questions, we ask that you please come to the microphone. Yes. I guess we were incredibly clear or something. Thank you for attending.