 from you guys. So thanks for coming. Let's get started. My name is Matt Griffin. I'm a director of product management for Precona. If you are not familiar with Precona, we're a database services and software company focused on MySQL MongoDB. We drive a lot of our services innovation and learning into our open source software like Precona server, which is a MySQL alternative, and our clustered version of Precona server, Precona extra DB cluster. So we also participate in the Trove project and help that team along as well as the docs team and the HA guide is what we're currently working on. And we'll be in the demo theater today, so come visit us over by, or in the Explore area, so come visit us by the demo theater. So I'm presenting with Amrith Kumar from Tessora, and he's this guy, and he'll tell you a little bit about himself sometime in this presentation. If you want to tweet about this presentation, please use these Twitter accounts, and this is the agenda we're going to cover today. We're going to cover some of the introduction to the Trove architecture, where we are with replication in the Trove project, leading up to Kilo, and then Amrith's going to talk you through a demo of how it all works, step by step in replication, and then we'll talk about the future in Liberty and Beyond. And if we have time, hopefully some Q&A. So let's first begin to understand where do we stand right now with Trove. This is the user survey, and the bottom left are stats from the November 2014 results showing where Trove is, in terms of the top bar is production, then dev test, and then POC. So 9% production, 9% dev test, and 16% of reporting clouds are using Trove, and 16% in POC. This is great because when you look at the current results that were just released a few days ago, just above the six months ago area, you can see that there's definite growth in production clouds running Trove, but it's 12% if you can't read it. Then 13% in dev test, which is actually a really, really component or part of the cloud experience in using clouds. Amherst colleague, Ken Rugg, had an awesome article in SuperUser about it a few months ago about the importance of dev test. And then I think what's really, really promising is the rise in POCs, and so this is at 19%, I believe, which is really, really great, and I'm anxious to see what the results of this survey are at the next time. So I think that's here, yeah. So these are the stats. Next, we will talk about why do you want to replicate? Why use replication with basic replication? So these are just a few use cases of when you'd use replication. First, you want to have a scale-out strategy. And so you might use a master-slave topology or you might use a master-master topology. There's trade-offs with each, like master-slave, you have to deal with the slave lag and promoting slaves to masters, and then master-master, it can be really complex. We won't go into all that because it's not really why you're here. High availability might be another way where you can use MySQL replication to support a high availability strategy and an offline copy. There are other use cases around how to do upgrades and how to set up really good backup strategies using MySQL replication, but these are just a few of the other reasons why you might use MySQL replication. So next up, Amrith is going to talk about the architecture of Trove. Thank you, sir. Don't kick my beer over. Thanks, Matt. I'm Amrith, I'm with Tesora, we're a company which is focused on Trove. We're also in the exhibition center or the exhibit hall. Come over if you're interested in learning more about Trove. So what does Trove do? What is the Trove architecture? How many of you here are familiar with Trove? Show of hands. How many of you have used it? Neat. How many of you have used Trove with MySQL? How many of you have used it with a NoSQL database only? DB, Cassandra, anybody? No. All right. So you're probably all familiar with some of the names on this picture. I'm going to talk to this picture very quickly as you're probably familiar with everything on the right of this picture. That's your traditional open stack. You've got Keystone, you've got Neutron. Keystone does authentication and identity management. Neutron does networking. Starting from the top, you've got NovaCynder, Swift and Glance. Nova being compute. Cynder and Swift being storage. Glance being image service. Okay. The stuff you're here to talk about is on the left of the picture and that's Trove. Trove has primarily three components like most of their open stack services. There's Trove API. It exposes the Trove RESTful API. The Task Manager, which does most of the heavy lifting. And Conductor, which we'll talk about a little bit, does some very important offline out of banned things which are useful for databases. The objective of Trove is to give you a database as a service. You need to be able to provision and use a database without having to bother about all the minutiae which goes on in provisioning, managing, replication or clustering or all of those things. The way in which all of this works is requests go to the Trove API. I can't quite point, oh I can't. Request go to the Trove API. You get your responses from the Trove API. Behind the scenes all the work gets done by the Task Manager. Some of the work by the Trove API as well. And they communicate internally over a message bus. By the way, if you have questions at any point, raise your hand, stop, ask. Very important thing to remember about Trove is that Trove provides you frameworks with which to build a database as a service. Trove is not a database. With Trove you get databases you already know and hate. MySQL, Oracle, Mongo, whatever. Trove is not that database. Trove helps you orchestrate databases. So you can provision an instance with Trove. Trove takes care of the provisioning for you. At the end of the day what you get is a MySQL database for example. If you want replication with MySQL for all the reasons Matt mentioned, Trove will help you provision a replicated pair of databases. Trove does not itself do the replication. MySQL does replication just fine. Thank you very much. We just give you a replicated pair of databases which you can use. And the last thing is when you need to do a failover, when you need to manage your replicated pair, Trove gives you an API bill which you can do this. On the other side, what does the database do? At the end of the day your data is stored by MySQL. Queries are processed by MySQL. You get answers from MySQL. The actual movement of bytes for replication is done by MySQL. So there's a very clear contract here. What does Trove do? What does the database do? All right. So Trove provides a framework. The first version of this framework came out in Juneau. For those of you who are familiar with other OpenStack projects, we use the standard strategies mechanism. In Juneau we supported asynchronous replication with what MySQL talks about as binlog replication. Quick show of hands. How many of people here know what binlog replication is as opposed to gtids? Great. Okay. This is going to be a short presentation, guys. So in Juneau, version one of replication was built for MySQL 5.5 and was built around binlog replication. Okay. For those of you who are really interested in going back and looking this up in the sorcery, Trove guest agent strategy is replication. MySQL binlog is the actual implementation of we1ofreplication. In Kilo, we added a new capability here. I say we. I didn't write any of this code. There's other people sprinkled all over this room. All right. Quick show of hands. How many of you here are Trove contributors? Just raise your hands, please. All right. There's a whole bunch of people here. Blame them on one guy over there if there's any bugs. All right. In Kilo, we came up with the gtid-based replication. We extended the exact same framework to support gtids. All right. We're going to talk a little bit about gtids and MySQL binlog replication and so on. The focus is going to be on Trove and the framework which it provides. Okay. So let's get into the real stuff here. The first thing, how do you provision your master instance? Trove create. The underlying restful call is just the create API call. Trove create m1, 2, dash, dash, size 10. Okay. That basically says create me a single instance. Call it m1. The 2 there is the flavor. Flavor 2 is small with a 10 gig database. So what that command is going to do is it's going to create a singleton nova instance. It's going to hang a 10 gig volume off of that. That instance is going to be of type small. And it will have on it, and let's assume here for the purpose of this illustration that we're talking a Kilo-based system, the default you will have a MySQL 5.6 instance booted up on it. Okay. All right. Then you go and do your usual stuff. You load some data and you create some tables and all this fun stuff. Here's the interesting part with replication. All you do is this one command here. Trove create, give it a name. Flavor 2, size 10. And all you specify is make that new one a replica of m1. You just created an instance called m1 up there, and you just added this little piece up here. And what? Yes, go ahead. You have to do what? Nope. You'd literally do exactly these commands. It will all be taken care of for you. Now, we'll see why you don't have to do that. And you can do this as many times as you want. At the end of the day, you get a picture which looks like this. This would be m1, and this would be, you know, all the replicas. And for those who are going to be watching this video later, the question was do you have to put the master in read-only mode before you do the replicas? The answer is no. Okay. So what exactly does Trove do? This is really the guts of replication. You gave it this simple sounding command. So what Trove does, and this is GTID-based replication, MySQL 5.6, first it's going to create a snapshot of the master. MySQL knows how to create a transactionally consistent snapshot. I don't need to put the database in read-only mode for that. Database runs along just fine. But we also store MySQL stores the master log position in GTID form. Then Trove goes and creates a new instance, and it launches it. It hangs a flavor 2, 10 giga storage off of it. It loads that backup, that snapshot which it created. And then it identifies m1 as a replication source, and then starts up replication. Now, these are exactly the same thing. Okay, so let me ask, let me say this the other way. How many of you here have set up replication manually? How many of you have done the exact same things? Well, at least a couple of you. Okay, that's good. You had a question, sir? No, it does not. Oh, the question is, does it pause the VM to take the snapshot? Nope, it's a MySQL snapshot. It's going to send it straight to Swift. Okay, so it's effectively going to take a snapshot from MySQL's perspective. The VM is going to transition for a short while into, quote, backup state. Let me look to the guy who wrote this. It goes for a second to the backup state, correct? Okay, and then it'll come back out. But no, it doesn't... It uses Percona X or DB backup. Gentlemen of the red shirt, Nikhil Ptl. Good point. While it's in backup state, other Trove commands will fail because it'll say in backup state you can't execute other commands, but your data plane access still goes on fine. You do exactly these same steps if you set up replication manually. So in Juneau, again, in Juneau, we did this with traditional bin log replication. Only master slave topologies, any number of slaves you wanted. None of the other fans here topologies possible. And all Trove would do was orchestrate the creation of the slave, load the backup and all of that stuff. Okay? All right. If you actually go and look at that implementation, which I told you, you will find these exact lines of code. And the lines of code there in red are exactly what you would do if you were setting MySQL bin log base replication. Nothing fancy. Trove orchestrates this for you, so you execute one command and this is done for you. That's the value that Trove brings. So create a snapshot, change master to master host, master port, so on. Master log file, master log position. Those are the two commands which identify this as being bin log replication based. Okay. In Kilo, we do just one slightly different thing. Master auto position is one. So for those of you who are familiar with bin log replication as opposed to G2 replication, that's fundamentally the difference from an administrator perspective of what you have to do different. Okay? So for the entire duration of time that this whole operation is going on, you're not sitting and waiting. You came along, you issued up here top left Trove create replica of something or the other. The Trove API service is going to give you a response immediately. The gentleman who's taking photograph, there's seven pictures which are going to look very similar, so get ready. The next thing which happens, the API service does just the light stuff. Oh, you want to create a new instance which is a replica of something else? Sure. Here's the idea of the instance. Have a nice day. Come back later and check. The API service tells the task manager, somebody wants a new instance which is replica, go do the heavy lifting. So the message shows up here. The task manager then goes and says, okay, I'm one, go take a snapshot of yourself. Snapshot lands on Swift. Okay. Task manager then says, snapshot is done. Nova, can I have a new instance, please? I have a new instance. It talks to the guest agent on the new instance and says, there's a backup there. Go load it. When it's done with all of that, it says, okay, establish replication. Now you have yourself a replicated pair of databases. Under the cover, Stroves is doing all of this for you. One command word. Okay. We talked a little bit about bin log and gtid replication. We're not really going to talk in a great lot of detail about it. If you really want to know more about it, go to Percona Live. No, just kidding. Matt, you're up. By the way, you should really go to Percona Live. It's a good show. Yeah. Join us in Amsterdam coming up in October, I think. Yeah. Anyway, I don't know. So I'll be brief. Just going to give a quick overview of, here we go. Bin log replication. So bin log base replication, it can sometimes introduce problems. And that's kind of the key reason why we're moving to, or the move was made to gtids. And it's really essentially because the slave knows very little about what's happening with replication. And so in master master setups, there can be problems with where should the slave, what's the most recent version of the data, unless you really know what you're doing. But even in master slave scenarios, whenever you want to recover that slave, there's typically a lot of manual effort involved to make sure that you are recovering all of the data available. In the case of gtids, it solves a lot of those problems. So gtids are a global transaction identifier. It's a unique identifier created and associated with every single transaction committed on the master, essentially. So now the slave has a lot of rich information on what is the most recent piece of data. And you can do things like automatic failover from the master and bring up a slave, or do more advanced things with Trove. These were introduced in MySQL 5.6. So I just want to give you all a brief thing. If you want to know more, yeah, go to Pokémon Live. So I mentioned failover, and I'm going to talk about failover. Thanks, Matt. All right. So if you were using Trove with Juno, and you can take my word for it if you didn't, everything was built with 5.5 in mind. And 5.5 did not have gtids. Therefore, if you attempted to do a failover, it was possible. Again, let me stress, it was possible in some circumstances that you could lose data. Therefore, there were some restrictions on the kinds of things you could do with Juno and 5.5. Again, now in Kilo, we have support for failover. So we'll talk about that. So now that if you have a replicated pair, let's talk about some of the commands you can use to manage these. When you have a replicated pair of databases, you have a master. Again, 5.5 or 5.6, Juno or Kilo, one master, many slaves. All right? This is not multi-master. This is not Galera cluster. This is just master-slave replication. Okay. There are three things you can do if you have a replicated pair of databases or a replicated group of databases. The one is to actually do a failover. And a failover is an operation where the node which was the master is no longer the master. Okay? And the other thing which you can do is you can take a slave and say you have no master anymore. Those are effectively the three things you could do if you have a replicated set of databases. So let's talk about failover first. I have a master. I have some number of slaves. The master is reachable. I don't want that master to be the master anymore. Okay? That instance is, you know, I want to retire the physical machine on which that instance is. I need a new master. Okay? The other situation is your master is failed. In a situation where you have a failed master, you need to elect a new master. So that's the second case where you need to do a failover. And the third we talked about is take a slave and make it not associated with the master. I've given you the commands here or at least the name of the command. If you have a existing replica and you want to make it the master, you should promote to replica command... promote to master... promote to replica source command. If you have a failed master, you eject the master. And the last one is you detach a replica. Everybody clear what these three operations are? When you would use each one? Sounded like, yes. Okay. Go ahead. If you detach, how far can you... Okay, so I'm going to go out on a limb here and we'll find out if I'm right, okay? I'll detach it again. Am I correct? Yeah, I win a prize. All right. Okay. You'll see that actually. There's a slide later which shows that. You rip it off and then it's totally done. Sorry, somebody here had a question? No. All right. So I'm going to start with this illustration here. This was that instance M1. It has three replicas. A master, three replicas. It's running 5.6. This is, by the way, annotated output from Trove Show. Trove Show is actually a whole lot more information which I just ripped out from here. So this is what it looks like as a picture. I have a master. It has three slaves, 12, 13, 14. So the first thing which I'm going to do is I want to take 14 and make it the master. So what I want to do is go from this picture here to this picture here. All right. So question, anybody can answer this. Raise your hand. What's the command you'd execute? What are you going to target? Anyone, not an ATC on Trove. OK. And which one would you target? M14. OK. Promote to replica source M14. That's exactly what you do. And when you're done with that, so I issue the command Trove API, says yes, you're done. Task manager goes, does all this stuff. At the end of this, if you run Trove Show M1, it now says it is a replica. M1 is now the replica of M14 because if you say Trove Show M14, it now has three replicas. So if you're a MySQL administrator and you had to do this manually, there's a certain set of things you would have to do. That set of things is long, interesting, sometimes error prone, all done for you one command worth or one API call if you choose to use the RESTful API. OK, so far? All right. So now I'm going to detach a replica. So I start with, now M14 is the master. M1, M12, and M13 are the replicas. And I want to go to this situation and this answers your, this is part of the answer to your question. I just want to take M1 and detach it. And the command you'd use, of course, is detach replica M1. And when you're done with it, notice there's no mention of replica of anything. It's now a free-running instance all by itself. Go take a snapshot of it, destroy it. But at this point, it is no longer related to the other three in any way. And you cannot ever reattach them, not through Trove at least. OK. And the last command, which, oh, and by the way, if you go look at M14, which is the master, it only has two replicas left. The third one's gone. OK. All right. And the last example we have here is a simple failover. Now I'm left with M14 with two replicas and I want to, and I want to eject this, this one as being the master. OK. M14's the master. For some reason, it's non-responsive. I want to kill it. I want Trove to pick a new master for me. OK. So a couple of things to keep in mind. If a instance is a master and it is not really failed, you can't eject it. OK. So Trove maintains a heartbeat. And I mentioned earlier that there's a conductor, a conductor service. The purpose of a conductor service is every so many seconds is to pull the database and say, is the instance really up and running? If the instance is running, you can't kill it as the master. So a little safeguard here. The second, if I try and eject M13, recall, M13 is not the failed master. M13 is a replica. If you try and eject something which is not a master, it says, I'm sorry, it's not a replica source. You cannot eject it. OK. You finally pick the right one and you eject it. It's a failed instance. And when you're done with this, sorry, go ahead. What type of a heartbeat is the heartbeat? OK. So give me a second. Let me go back a couple of slides. OK. So when you create a Trove instance, a Trove instance master replica, whatever you have, it's one of these things up here. It's a nova instance which has MySQL and a guest agent. Guest agent is a piece of software which exposes a Trove-specific API and an implementation which is specific, in this case, to MySQL. So if you had replication for some other database, for example, the mechanism for doing replication would be a difference out of commands. This guest agent generates a heartbeat every, I think it's every minute. And that message is sent by it over the message bus to the conductor. The conductor stores that in a persistent database here. So it's basically doing a MySQL ping under the covers. If you require a different heartbeat mechanism where a different command is executed, you can definitely modify the guest agent to do that. But it does some best effort check for, is this MySQL database here good or dead? That's what it does. And there's a measure for how long it should have not had a heartbeat before it'll be considered to be dead. I think that number is one minute. So we're here. And I want to eject this guy. When I'm done with it, if I looked at M12, it would say M12 has exactly one replica. Remember, you just shot M14. M12 and 13 are now the two databases left. One is a master, one's a replica. Yes? Yes. No, it comes back up properly. And here's one of the reasons why. Sorry, the question was if you had a network partition and the guest was not able to talk to Trove for a while and then you shot it and then something happens and the network comes back up, will everything be happy or not? Paraphrase, that's your question. The guys at MySQL have spent a long time figuring out how to do replications and masters and slaves. All we're doing is we're telling one guy he's the master. If you tell a person he's a master and another one comes up and says he's a master, there's a quorum election process which says, I'm sorry, your GT is not as new as mine. Go away and sit in the corner. So Trove does not get in that process at all. Trove just leaves the database to do what the database can do just fine. Oops, I lost my mouse. Okay, so we're now down to two instances. One's a master, one's a replica. And looking to the... Yeah. Okay, so the question was is there any automation built into Trove where if a instance fails, Trove automatically does fail-overs? At this point there is not, but this slide says, coming soon there will be. So yeah, this is... So think of the progression. We started with, in Ice House, we started with single instance MySQL. In Gino, we had replication with Binlog. In Kilo, now we actually have the ability to do fail-overs. Next step, build a mechanism to do automated fail-overs. Now part of automated fail-overs is a policy which says, how long do you wait? You might want to wait a minute. Somebody else might want to wait 30 seconds. So coming up with those, how do you actually do that is what we're going to be talking about for the next couple of days. And that is definitely something on the roadmap. Other questions? Yes, sir. How does guest agent talk to rabbit... Oh, okay. So I keep losing this mouse. The question is how does the guest agent talk to rabbit MQ? So this instance, this compute instance, which is actually your trove instance, you will have two interfaces. One interface will be the customer interface, the customer network. You come in on that to do your queries. The other is a private management network. The private management network is used to talk to a, typically a trove-specific rabbit MQ server. Therefore, that is something which is entirely off the customer's network. Is there any security issues? So the question is, are there any security issues with the guest agent talking over the message bus? The only thing which it can communicate to is to the conductor. The only thing which it sends messages to at this point is to the conductor. And the only messages which it sends are the heartbeat message and in some cases it sends other information. When you create an instance, there's some specific message which goes over that. So there are things which you have to do in order to harden it. We're going to be talking about several of those. There's been discussions on the mailing list about some of these over the next couple of days. We're going to be talking about some of those things. So one of the things which we are looking for over the next couple of days is input. And the question was, we talked about automated failovers. Is there any talk about introducing load balancing or load balancing as a service? Integration is there. I have not seen a blueprint for that at this point in time. We've had some conversations about how far we need to go in this release. Over time, I assume people are going to want all of these capabilities. There is, of course, the question of does that fall within the purview of databases of service or is there something else which you would want to do? So I'm not saying the answer is no. I'm also not saying the answer is yes. The answer is it's a good question. So we'll start at the back, gentlemen in the back. So the question is, is there a way to find the replication status of the slave through the Trove API? The answer is no. You connect directly to the slave and find out its lag. The gentleman right in front of you. The question is, have you integrated Trove with Heat as yet? The answer is this question came up this morning. It depends on whether you want Trove to orchestrate instances with Heat or Heat to orchestrate Trove instances. Heat to orchestrate Trove instances. I don't know the answer to that. Sorry, go ahead. The answer is yes, it does. The gentleman behind you said so. Somebody here had a question. Yes, sir. Can you use replication without Swift? So the reason we use Swift is as a place to put backups. If you don't have Swift and you have some other storage mechanism which supports the Swift API, I believe the answer is yes. And I'm going to look over in that direction and say, was I right or wrong? You need Swift. OK, I was wrong. Happens once in a while. OK, what is the question? The question here is, what exactly do you need from Swift? Is it keystone authentication? What more do you need from Swift? Or is this a question which we should hold for the next two days? OK, other questions? Sorry, it does work with who? That doesn't sound like a question. You're saying it does work with the Rados gateway, but we're not just backing up a volume. OK, other questions? There was one other hand there. Thank you. Trope didn't get into that business. Databases figure that out just fine. Trope, no, if you issue the command which says, I want to do a failover, we'll try as immediately as possible to get that command down to whichever database instant. If the database decides to say, oh, there's a long-line transaction for some reason I can't do the failover, the task manager is going to have to wait for a while to get a response. But Trope itself does not know anything about what's going on on the database under the covers, what transactions are going on under the covers. Yes, sir? So I will give you the same. So the question was, something happened, and somebody figured out that replication was the problem, and the DBA went in and did something and fixed it. Will Trope be smart enough to figure everything out? To recon... OK. So there's a little warranty on the back of my phone which says, if you open it, all bets are off. If you SSH in into an instance which is running Trope, all bets are off. Same story. Because at that point, I've given you... Choose your metaphor. I've given you a gun with a whole lot of bullets and I've said, do whatever you want with it. I have no idea at the end of that what's going to happen. You can do a whole bunch of things which Trope just has no way of ever discovering. So all bets are off. I missed the first part of that. Where in the roadmap do you... So the question is, where in the roadmap do you have integration with the Oracle database? OK, so with Trope, currently you can get Oracle with Trope from my company to Sora. I'm a Oracle 12C guest agent. You can use it. We're doing 11G guest agent. We're also talking to the community about doing Oracle pretty soon. I would give you a more definite answer in four days. We are very, very interested in... We are very interested in doing Galera cluster and we'll be talking about it with you and with the rest of the community in the next couple of days. There is a very significant interest in demand for replication. Not everybody is comfortable with just having a synchronous replication. So yes, there's a demand for Galera cluster. And we're going to do something with it. Yes, TBD at this point. Do you have ideas? Well, no. So there's different ways in which you can do it. So the question is, if you're doing replication, if you're doing clustering with Postgres, you have a number of options. I've done clustering with Postgres many years ago. I don't know what the current state of the art is. But if you do, we're going to be talking about it over the next couple of days. So come on over. You're not allowed to ask questions. I'm sorry. Oh, you're saying I'm out of time. Oh, okay. One last question. Sorry. I'd suggest we talk about that outside of this presentation. It doesn't have really much to do with replication. It has to do with whether we want to use config drive or something else. How about you combine, and there's a bunch of us who are interested in that question for other reasons. Other questions? If not, Matt and I are available for the next several days. Come on over. Ask us any questions you want. They're going to kick us out of the room. So we should probably get out of here. I'll just leave this slide up here if anyone wants to grab information. Thanks for coming.