 Alright, I think we are getting queued to start here, so we'll jump in here and make good use of your time. I know we're all running around between as many sessions as we can pack in. Real quick introductions, I'm Chris Nelson from Swiftstack. This is Kenny Grip from Percona. We'll share time a little bit so you know what the next 30 minutes is going to look like. Obviously, the topic as you have in the abstract or in the title and on the screen right now is using Percona's software to back up my SQL databases to Swift. That's the specific topic we're going to talk about. To kind of set the stage, I'm going to take the first seven, eight, maybe ten minutes and talk a little bit about Swift, OpenSack Swift and what Swiftstack kind of does in that space. I know Kenny's going to talk through for his five or ten minutes around Percona and the software and kind of why back up. And then in the last ten minutes of the first 30 or so, the intent is to do a demonstration. So we'll actually show using Percona software to back up OpenStack databases into Swift. We'll see that live up here as we go. I should save a little time for Q&A, but don't hold us to it if it runs a little bit long. So with that, I'll jump right in and kind of move along quickly. Real quick, if you don't know Swift or Swiftstack, that's what I want to hit for the next couple of minutes. Swiftstack as a company, we're in the hall, obviously, come on by and see us if you want. Swiftstack is a couple of things. One we are the lead contributor to OpenStack Swift, a project that many of you know and hopefully love. It's been a part of OpenStack since its inception. We are based in San Francisco. John Dickinson, who many of you know, project technical lead works at our company, so do several of the core developers and a whole bunch of other people who are in the business of making Swift, OpenStack Swift, the world's most scalable object storage, most widely deployed object storage, easy to use in kind of an enterprise scenario. So we've got OpenStack Swift. If you're not familiar with that, it's the project's real quick rundowns of a couple things here. One it's in use everywhere. There's no more widely deployed open source based or really any object storage in the world, HP's public cloud, IBM, Rackspace, Oracle, any of the Swiftstack deployments and many of these are based on Swift or are running Swift. So it's a very well proven technology. Tons of developers are scattered around here who have contributed in major ways and some of these company names that you see here have been a big part along with Swiftstack and kind of moving this forward. If you missed it today, just a quick plug for Joe Arnold, who I think is back here somewhere. The book that Joe and a bunch of the team at Swiftstack wrote on Swift, we've been given away. He did a book signing today. I think we've got another one tomorrow. I'll swing by the booth if you want more details. If you want to feel for where Swift has been deployed in the real world, not just in, you know, somebody's OpenStack POC in their lab, real world deployments both in the public space and private space, I mean, it's everywhere, right? No question about how well proven it is. Not every one of these is a Swiftstack customer. Some of these are clearly using OpenStack Swift, the distinction we can talk about another time. But real quick, what does Swiftstack do in addition to or beyond what you can go and get from GitHub with OpenStack Swift, of course, which is an open source project in the OpenStack community? You know, one of the marketing taglines we talk about is making Swift enterprise ready. The things down here in the bottom, we'll get to a little bit of detail here in the bottom, the OpenStack Swift gray and black kind of section, that's what you get with core open source, OpenStack Swift, right? You get a RESTful API. You get the scalability across geographies and continents that's not a byproduct of it's eventually consistent nature. You get the fact that it can be deployed in anything that you can run OpenStack on, right? Commodity or industry standard hardware, any X86 hardware out there. And more, you get concurrency, the performance capabilities, and all the rest. That's part of OpenStack Swift, whether you talk to Swiftstack or not. What we do is add pieces on top of that that are more or less required in the enterprise space. So obviously we provide support. That's a big part of it. But also, our Swiftstack controller is a management tool that sits in front of this and provides simplified deployment, management, monitoring, integration with things like LDAP, authentication, Nagio, Zabix, provide a file system gateway. So if you have applications that don't yet talk to a RESTful API, the Swift API, but they know NFS or SIFs, you can go through a gateway to get your data into a Swift cluster, et cetera, et cetera, et cetera. That's the value add that Swiftstack brings on. I don't want this to be a sales pitch for Swiftstack. So I'll keep it brief. With that, we'll stop Swiftstack. I want to jump back into a bit about how Swift actually works, because this is going to be relevant to the part Kenny's going to talk through about actually using Swift as the target for backups, and specifically my SQL backups. So hopefully this is not new to anybody in the room here. But in case it is, real quick, an API, RESTful API, every object that is placed inside of Swift object storage. Imagine an object being a picture and it's associated metadata. You walk out there, you take a picture of an airplane, a seaplane taking off, it was taken in Vancouver, it has a name, it was taken on your iPhone 6, or whatever the case may be. You apply a little bit more metadata. It was my OpenStack Vancouver trip. All of that, the picture plus its metadata, makes up an object. Every object in Swift is referenced by a URL. And it includes pieces, like you see on the screen here, the Swift cluster that it is in, plus a couple of additional pieces here that can help sort out where it's going, a container, a count container, an object. You'll have time to get into lots of details there. But we can talk about that more a bit later. So everything is referenced via an API. When you're putting data in, when you're writing data into Swift, you are writing via HTTP puts, the operation that you would expect. And when you are reading data back, you're using HTTP GETs to this. And there are some additional API calls as well. So what does the data path look like? I make a call to the cloud, or my Swift stack object storage, at that URL. Now what actually happens? What are the pieces that are in play? I'm going to get to a hardware slide in a minute so people can get their heads wrapped around this. Number one, the client, which in this case is going to be Percona's extra backup tool. The client is going to communicate with, if it's a large enough cluster, probably a load balancer that's going to handle the traffic coming from a variety of clients and point them to a tier, if you will, or a proxy service. The proxy service is this traffic cop, if you will, that says, OK, client request comes in. I need to go put a piece of data. And I've got potentially thousands of disks, potentially hundreds or thousands of nodes scattered around the world. I know where that data should go. Or on the flip side, I know where that data should be. I'll go and retrieve it. So you can scale in a Swift cluster independently this proxy tier, this kind of traffic manager, if you will, tier from the actual, you can scale independently from the storage nodes that are really holding that data. So we spread, we use an algorithm, if you will, inside of Swift and Swift Stack when we set all this stuff up, we will take the smallest deployment, you give us three drives, and we'll spread the data as uniquely as we can across those to give you the, we emphasize availability over almost anything else in that sense. You give us a small cluster of a handful of nodes, a large cluster, or even a multi-data center cluster, multi-region or multi-continent cluster. And we'll take that data that you give us and make sure that you have availability, even if you lost a continent or a network partition or anything like that along the way. Real quick, one other feature that's relevant, especially in the backup space, if you heard kind of buzz from Swift, the OpenStack Swift project last summit when we were in Paris, we were talking about that and kind of the previous one. We were talking a lot about storage policies. Storage policies is new features that a lot of companies collaborated in to put into Swift that allow us to say, I'm going to segregate devices that are participating, again, thousands of devices from different ages and different characteristics, SSDs, flash drives, spinning drives, big and small, I can put them all into the same cluster and I can segregate parts of those and say, this is kind of a high-performance chunk of stuff, this is a high-density chunk of stuff, or this is my European Union physical hardware, so all the EU data is going to be stored there, or this is a, the news that Swift has made, this summit is, erasure coding is now available in Swift, so maybe I'm going to store my backup data in an erasure-coded container, but there's other stuff that's serving kind of a more tier one web application. I want that to be in a replica-based container as well, storage policies is what allows us to do that. So, busy slide, don't worry about the details, I know all the stuff will be posted later, but real quickly, to give you a feel for, Swift can be deployed on a variety of industry-standard hardware, common reference architecture, these are flexible, this is not a hard and fast rule, common set of reference architecture, you can kind of see on the left-hand side what it looks like to piece together some memory and RAM and CPU and networking, and then you can see that this can apply across a whole range of vendors that are out there providing the hardware that's in your data centers today. Last piece for me, before I get Kenny up here, common ways that Swift is used, obviously today's conversation specifically around backups and backups of my SQL in particular, but Swift is used in a wide variety of ways, our Swift and SwiftStack customers range from the backup and archive space which we're talking about down to kind of your enterprise Dropbox, right, Think File, Sync and Share, box.com, backup or storage, obviously anybody that's using or has known OpenStack for a while knows that Swift is a natural thing to use for your glance images or your sender backups, and then we spend a lot of time with customers in the M&E and the genomics kind of bioinformatics spaces and beyond. So that's a real fast rundown, I think there's a link on the last slide, so the last thing I will say, and then I'll hand it over to Kenny, is that if you want more details on Swift specifically, we've got a workshop running, I think it's about six hours on Friday free to any OpenStack attendee this week, you're welcome to come and we'll go into all of that stuff in a lot more detail and give you chances to get some hands-on with software. So with that, to Precona and Precona's backup. Thank you, Chris. OK, first of all, a little bit about Precona. Who doesn't know about Precona? I kind of want to have a question. I kind of want to know who has heard about Precona. Great majority, I like it. OK, good. So it's a company founded in 2006 by former MySQL employees. So we are privately funded. We have a lot of customers worldwide. I'm one of the Belgian employees. We have a lot of employees all over the world. I think it's about 30 countries at this time. So we focus around MySQL services and beyond. So here's a list of some of our customers. I think, yeah, you can see it in the slides later on. But here is, the thing is the most important thing, is what software we provide. We do a lot of open source software. So a lot of the things we create, if not almost everything, we make open source. So we have our own MySQL version called Precona Server. So it's a patched Oracle MySQL version with added diagnostics, added performance benefits. It's more suited towards the cloud environments, things like that. We also have an extra DB cluster, which might be known by some who's using Precona X3D cluster in OpenStack. Nobody? Galera cluster, maybe? Yeah, some of them. OK, great. So this is actually bringing synchronous replication to the MySQL world, so where you can have better high availability consistency in a MySQL cluster. So another thing we have is Precona Extra Backup. And that's where I'll focus on a little bit today. This is an online hot backup for InnoDB storage engine, which you're most likely to use anyway. So this is where we're going to talk about today, how we can take backups. So it has the name Precona Extra Backup, but it does also support different MySQL flavors. And I'll show you that later on. We also have Precona Toolkit, which is like a set of open source scripts that help to manage your day-to-day life as a developer or operational person to fix replication, fix inconsistencies automatically, do get some certain metrics out of it, and everything. We also now have something completely different. So we're not only MySQL anymore. We have Precona Toku MX, which is a MongoDB distribution, which brings multi-version concurrency control transactions to MongoDB, and it has a very good compression ratio, and it has improved write performance. So maybe it's good to check it out if you're using MongoDB. So we do more. We also do OpenStack services, Linux, Hadoop, Vertica. We kind of tailored towards the general cloud and everything. So we basically try to focus on what our customers want us to do. Like the features we make in Precona Server are features that is really necessary, that our customers really need, that we see that are kind of problematic. We try to optimize it, bring it together, bring it open source, and release it for everybody. And this is one of the things that I'm going to talk about today. So here's an image about OpenStack core services. It's a little bit outdated. But what you can see is we've got all these kind of services like Glance, Nova, Cinder, Neutron, Keystone, maybe. They all have a database. And in the majority of the cases, this database is MySQL. Who is not using MySQL as some of these databases? Who isn't using MySQL? Only a few? OK, and S-Core service. OK, so here's one of the OpenStack user committee surveys from 2014. And you can see that the database usage, if we look at it, all the ones with the arrows are actually MySQL-related products. So we're talking about 68% is using MySQL. So the majority of the people use MySQL from Oracle. Then we've got MariaDB, which is a fork of MySQL. It's still compatible. It uses the same MySQL protocol. It has different functionality, different features. But it's basically kind of MySQL still, right? So we have MySQL with Galera, Procona XRDB cluster, and MariaDB Galera cluster. They're all built upon that synchronous replication technique built by a company named Codership. So this is different flavors or different distributions of that software. We also have MySQL with DRBD, which is still used a little bit. And then we have Procona server, which is about 3% of the installation. So if we think about it, 70% of everybody using OpenStack is using MySQL. So backups. Anyone know who is not doing backups at the moment? Oh, one person. OK, one person is not doing backups. So this is kind of an important thing. We need to do backups, because if we don't backup that data, of course, you will have maybe a problem bringing your OpenStack back again. Maybe all the important data is there. So if I go back, we're talking about Nova, Cinder, Neutron, maybe Keystone. All this information is stored in MySQL. So if we think about a decent backup architecture, what do we do? So we have this is kind of general backup recommendations, I would say. So what are your business requirements? What is the recovery time objective? How fast do you need to be able to restore? This is an important thing when you kind of create backups. Many people just take a backup to some other file system, so they do it manually. Extra backup, maybe stream it to another host or to another file system or something like that. Well, how long does it take to bring that backup and bring back the service? So what is the business requirement there? Another important thing is the recovery point objective. How much data can we lose? Can we actually take a full backup every day and then whenever we need to restore, we can lose up to 24 hours of data. Is that something that is acceptable or not? So this is something you have to wonder. Another very important thing is off-site backups. You'd probably want to be able to recover in case of a data center failure. So you could use Swift maybe. If we use backups and stream that to Swift, then we can use Swift to make sure that the backups are located somewhere else and not only in the same data center. Another thing is the backup retention plan. So how long do we need to keep full backups, incremental backups? How do we need to do daily incrementals or hourly incrementals? Another thing is in order to reach to the recovery point objective is to do like point-in-time recovery can be done with MySQL binary logs. You need to backup them as well to be able to go back to any point in time in case that somebody deleted some wrong data, for example. Maybe you need to skip that and then make sure we recover everything except that delete statement, for example. Another important thing is to test backups and your restore procedures. Who tests their backups regularly and automatically? Not many, none. Okay, one person, good. So this is something very important and something that actually should be automated as well. And actually with this way that we take backups now with backups to Swift, which I'll really demo very, very soon, it makes it a lot easier to be able to test the backups, see if the backups are working correctly or not. So what type of backup solutions exist? So we've got LVM snapshots. So if you're using LVM, you can take an LVM snapshot, mount that snapshot, copy the data over. But the problem is that there's a right performance impact. There's a big right performance impact and I think it's a ballpark figure 10 times slower in terms of rights. So this is not exaggerated, this is true. So if you have a right heavy environment, LVM snapshots will not do it for you. If you think about MySQL dump, which is a logical backup. So we convert all the data to SQL statements. So imagine having to load that data. If we're talking about two gigabytes, it's gonna be fast. If we're talking about a terabyte, it might take a long time, even a week to restore that single threaded, doing SQL statement by SQL statement. So this is something that is very, very slow to restore on larger data sets. We can back up a node, if we're talking about a Galera cluster. So we can say this node will take a backup from there. You can actually shut it down, copy the data files, start it up again. Or you can use a replica. So using master slave replication or master, master depending on what you use. So this does require extra hardware. You need to take that replica out of production and things like that. Another thing is disk subsystem snapshots. So some hardware like sans supports snapshots. The thing I can say there, it's block level. You cannot have fine-grained control on which part you wanna backup, which part you wanna restore. You kinda have to restore everything, make sure that that works, and then you can maybe extract some data out of it. So it's a lot more difficult. And also file system snapshot, yes, if you're using InnoDB, it is durable. So you can expect that the database will recover and it will have no data loss, depending on some settings of course. But keep in mind that if you want point-in-time recovery, some settings need to be correctly configured or you might have a corrupted database anyway, if you're using such snapshots. So that's kind of what there is, but what I wanna talk about is Procona Extra Backup. So this is an open source, free, completely free MySQL hot backup solution for non-blocking InnoDB backup. So we're talking about InnoDB here. ExtraDB is also mentioned. So ExtraDB is Procona ExtraDB. It's actually a patched version of InnoDB with some extra features, diagnostics, performance, improvements, and so on. It supports Procona Server, Oracle MySQL, MariaDB. So it supports any flavor of MySQL. So it's not limited to Procona Server. So what features does it have? So again, online, you can do multi-threaded backups. So file per file, you can compress and encrypt in the same command and you can do it multi-threaded as well, each of them. You can stream backups. We have this for a long time already. We can stream backups from one node to the other node. We don't have to copy it to some local disk first and then copy it to the backup server. We can do it over the network. Using Netcat, SSH, different things can be done. And that's where the backup to Swift also comes into play. We can do incremental backups, delta backups however you want it. So it is a physical backup that is being taken of the raw data files, the raw table spaces of InnoDB. You can do partial backup, partial restore. I can say I only want to restore these files, only these directories, that is no problem that works. So what we added in extra backup version 2.3, the latest version at this moment is 2.2, is support for streaming to Swift. So we've added a tool called xbcloud. It also has S3, so support, but I shouldn't say that too loud here, I heard. But okay, it's currently in development. We are expected to release a beta very soon. But soon I mean maybe this week, next week, somewhere like that. Also, I just heard that it is planned to be in Trove for database as a service in Liberty. So this is going to be great to have a feature for that for Trove as well. And now I'm going to kind of do a demo. I have screenshots as well for if the demo doesn't work because demos usually don't work, but I'll try. So let's have a look. Sorry, I will also copy paste things, just to be sure that I don't make any mistakes. So first of all, I've got like, is it readable for everybody? Who cannot read it? Somebody, okay, let me make it a bit bigger. Is it okay now? Perfect, okay. So I have a command here, Swift stat. Just checking, can we authenticate using a user backup and the key, sorry, not to save backup. So we are doing a Swift stack VIP. I have a Swift stack setup and I have a Dev stack setup, so very simple, very lightweight. I'm running this on my computer, just in case internet didn't work. So we can see we can authenticate, authentication works. So what I want to do is I kind of want to list the files I have currently for that user. So I have a container here, so I do Swift. I can do it here. Here we go. So we can see, do the same thing, we do Swift list. And we can see I have a container called core backup. I created that, that's where we will take and put our backup in. I also want to show you the GUI. So we kind of have the Swift stack interface. So this is one of the interfaces here. So I have a container called core backup. There's no files in it yet. And I also have Cyberduck, which has connected to the Swift API. So everything is empty in this core backup. So let us take a backup right now. So I'll just go over it quickly because it's kind of a command that we need to run. So what do we have? We are compressing the backup right here, highlighted. So we're compressing it. We're encrypting it using AES 256 with a very, very, very, very insecure key, which we mentioned here. You can also include a key file if necessary. We're streaming the backup. We're not making a copy to some file. We're streaming it using XB stream, which is extra backup stream. And we're taking a backup. This is just the command to say backup. We're piping this through XB cloud, which is the new tool that we've developed. And we are doing it in multi-threaded. We'll send multiple requests through the Swift REST API using Swift. So we have the Swift URL here with the authentication again and the container we wanna put this in. So we have full backup here. So this is just the directory name I'm using. So maybe you should use a date, time, maybe you should make a full directory. So there's still some work that you should do, depending on your requirements, on how to put it in there, what retention policy you have, and things like that. So if I press Enter, a backup will be taken. So you can see it's basically taking the inodb files, copying them, doing the online backup, piping that through XB cloud and immediately putting it into Swift. So if I go to the interface here, this is the directory is created. Sorry if this is a little bit small. And you can see that all the directories are in there. I can do the same using the Swift GUI. I have core backup, all my files are there. So what we see is we have different things. Let me just refresh while the backup is being taken. Let's go into Neutron. So we have different files here. So it's using multiple files, split up in multiple files. And we have the IBD files, which is inodbtablespaces. And it has the .qp extension, which means Q-Press. So Q-Press encryption is being used, sorry. XBcrypt is the encryption that we use. So it's a compressed and encrypted file. So each of them is compressed and encrypted. So you can see we have a full backup of all our tables at this moment. So very easy, just one command and it streams it. No spooling necessary, no intermediate spooling necessary to be able to put it in a backup. So depending on your configuration of Swift, you will have multiple copies offside, might be in multiple data centers. Whatever you configure in Swift, Swift Stack, the data will be arranged over there. So we have our full backup. And I kind of wanna also do a restore, of course. And I'm just going to remove everything here. So I'm going to do restore the data to slash restore. So this is just XBcloud get that I'm actually doing. Just one moment, I don't want it to execute immediately. Yep, so what do we do? It's the same instead of XBcloud put, we do XBcloud get. And then we say the container that we want. So we have core backup here. And we want to take it from the directory full backup. And then we stream that through XBstream again. And we put it into slash restore. So if I enter that, there's a little typo there, but this is still a very alpha release. I'm not using the latest one. This is the one that worked a couple of weeks ago and I left it like that. So I didn't wanna upgrade this morning. So you can see it's copying all the files, taking everything from Swift and putting it into slash restore. So the database here, Dev Stack, nothing much happened. So yeah, everything is done. So if I go to slash restore, you can see we have the same amount of files here. We're not ready yet. So we're not ready to just use it for MySQL because it's still compressed and encrypted. At this moment, it's not supported to automatically decompress and decrypt it. But we've been talking about adding support for that. Not sure when that will be though. So the next thing we need to do is do that, decrypt and decompression. So we can do that. We do that here. It will do, currently it's doing some cut pipes that through the different commands to be able to do that. So it's doing this, should be finished shortly. And then we have an extracted backup. Now a backup by itself, an extra backup, if you use extra backup already, is not ready to be used. You cannot just run MySQL on it. You need to prepare it. So this is the second phase you always need to do. So basically, you need to do dash dash prepare and say I want to prepare the data in slash restore. That is it. Now keep in mind, if you do incrementals, this would be the full backup. We can then say, we take from xbcloud, so from Swift, we take an incremental, we apply the incremental on the full and then we'll prepare it. So these are things that we can do as well. So this would be a multi-step process if you use incremental backups. So we'll do the prepare. So what actually basically happens is an INODV recovery. That's the implementation of how Prokona Extra backup works and that's it. It prepared it, everything's ready and we should be able to just start MySQL on this and this is just a regular MySQL data directory. You just replace the directory, start MySQL and we're good to go and we have all the data recovered. Now, because we are using INODV file per table by default, we can do it like a partial restore. We can for example say, oh, we want vipservices.ibd. So you can say only recover that file or that table. So we can say, okay, we only want a subset of the data. So that's basically it on how to do it. So it's a couple of steps. So in the slides, I have exactly the same so you could see how we do it, the process that we do it. So that is something that if you wanna use it. So currently this is a 2.3 alpha. You need to take the trunk from GitHub to be and compile it yourself at this moment. So if you just hold on for a small amount of time, you'll be able to download packages and just install it on whatever distribution you're using. So this is basically it. Do you have anything to add, Chris? Yes. I'll add one comment just to clarify. One of the pieces is, as Kenny was saying, directory and you saw that full backup directory. So think if you're a Swift guy, that's a container, right? So where we were backing up all of the files from Percona XRDB, that was he created and dumped all that stuff into a container. The UI that you're looking at, if you're not familiar with Swift stack, the UI you're looking at here is a simple, extremely simple browser-based kind of console into the containers, accounts and containers that are inside of this cluster. We have, this is a very common way to kind of go interact with and put a little bit of data into a Swift stack cluster when you get this up and running, kind of in a POC type of environment. Kenny used CyberDuck as well. It's another way to view what's in there. Commonly, you would be interacting with the Swift stack cluster or the Swift cluster with some application, exactly like Kenny did today with Percona, or imagine your other apps and things that are out there. The other UI you didn't see from Swift stack, which you certainly can, and either in the workshop or just by going to our website and signing up for a free trial, you can see the management UI that kind of controls getting all this stuff set up, which we did offline and didn't have the time to mess with here. So I think that was it on that last slide. I know we'll switch back to the slide here in a second. We'll do that. We've got a couple of links, obviously, to the Percona site, to a Swift stack site, to our workshop. If you want to sign up for the Swift workshop, that's the middle one there. If you can't read that link, just come find me and we can get you some details on that. Any questions? I think the way to do this is either to go to the microphone or shout it out. We'll try and repeat back for the recording. Any questions about what we covered here today? Oh, yes, please. Questions? Always nice. So I'm just curious as granted, this is with extra backup, but is the InnoBackupX wrapper script? Will that be updated to allow this as well? So good question. So you're used to Percona Extra Backup. So a new feature in Extra Backup 2.3 as well is to get rid of InnoBackupX completely. And you're smiling, so yeah. I agree. So yeah, all the functionality is now implemented in Extra Backup. So there's only one command, no wrapper around it anymore. Any other questions? All right, very good. Thank you, Kenny. Okay, thanks, Chris. Thank you. Thank you.