 Hi everybody, David Christensen from Endpoint Corporation as was just introduced. My talk today is going to be on choosing a logical replication system, some of the issues that, and considerations that go into that. A lot of this is going to go into the design of the specifics here, and we're going to be talking a lot about the kind of the historical stuff rather than kind of the upcoming things, although I will touch on that briefly. I just want to say also that my day started out winning $100 bottle of scotch, so it's downhill from here. I will be raffling off a red bowl though. If you guys want to pay attention. So first of all, let's talk about just replication in general. Is there one right choice for a replication system? And the answer to this is no. If you were expecting anything different to any silver bullet, still time to go another talk, sorry to disappoint. It's just simply a matter of understanding the constraints and applying engineering decisions. Man. So replication is of course used for many reasons. It's used for disaster recovery. It's used for high availability and failover, read and write scaling, backups. Those are a lot of common reasons for doing this. I think probably nobody at this conference has to be told why you would replicate data. But of course there are two choices here. There's logical and physical replication. And the question is why would we need logical replication systems when we've got all this great Postgres built in native replication using the physical wall streaming? Well, logical replication supports differing hardware and Postgres versions. So as you probably know, the physical replication requires the exact same version of Postgres because the wall records need to be the same format. The system catalogs need to be the same. So there's an inherent tie there between the systems that can be involved. With physical replication, you have to replicate the entire database system. You cannot pick and choose certain tables that you'd like to do. Depending on your application, this may be something that you care about strongly. There may be a few tables that you need to purpose-driven replicate to other systems and you don't worry so much about all the other tables. So particularly in these heterogeneous systems, that's a consideration. And writable clusters on the slave. Of course with physical replication, the slaves are all... You cannot write on those. So basically you have one master that you write to. That's the only thing that can be done to that data. And so there's kind of a single point of contention there. And of course, the reality is that not everyone is running the latest and greatest systems. So even as we move forward, we still have clients that are running 8.2 or 8.3 different systems where we can't rely on things being available, all the functionality and features being available. And so logical replication will give us an opportunity to kind of have some of the benefits of replication without having to... We'll still try to get them upgraded, but that's ultimately their decision. So we'll talk a little bit about physical replication. Of course this started out with warm standby back in the early eights. This went to hot standby in nine. The physical, of course, comes from duplicating the actual disk structure. So if you have a replica on one system, it literally will have, once it's caught up to a certain point in the wall replay, it'll have the exact same on-disk structure for all the data. So Postgres core applications built upon its recovery system and basically will make all of the same changes. It's kind of, I have to assume, one of the great design advances that Postgres made just being able to use this, and I really appreciate it. So yeah, it started out warm standby. It was better than nothing. Management's very primitive. You had to manually create backups using some of the PGSTART X-Logs functions, which is, of course, pain. You have your 16 meg wall files that were generated by default. So this data ended up, basically, your recovery choices are you can set archive time-outs and have potentially non-filled data segments which would speed up the recovery process or ensure that if you did lose data, it was minimal. But those were very fairly primitive. Of course, in Postgres 9, we got hot standby streaming replication, which really reduced a lot of the latency things there. We inquired our standby servers. We're never too far behind if we're using streaming. And later advances with cascading replication. 9.4 brought big changes with the logical decoding. We'll cover this some more, which is kind of the overlap between physical and logical replication. Of course, the benefits of physical replication. It's easy to set up and use now. Just use PG-based backup. Do something like that. Create a replica really easily. There's a lot of other tools that are available in the community where people have had things to really sweeten the deal as far as working with physical replicas. This also supports synchronous replication, I think, as of 9.1. It's ideal for high availability and re-scaling. Any DDL, DML changes, everything is automatically propagated because all the rights to system catalogs are propagated as well. So those are definitely benefits. There are limitations, of course, to the physical replication. It requires, as I said before, the same Postgres versions, hardware architecture, standby servers, read-only, no local changes at all. And every change in the entire cluster is replicated. So you cannot replicate a subset of tables or databases. That can be somewhat limiting, depending on your use cases. So let's talk a little bit about logical replication. Features that were introduced in 9.4, I imagine we'll be talking about some of these in this upcoming talk. I'm looking forward to the BDR talk, so a little pre-pump-up there. There were some big changes introduced for the logical replication system in Postgres 9.4. We introduced logical log decoding, which works via parsing the wall stream and extracting all the information about modified tuples. So if you change some data, you insert a record, you delete a record. Those tuples, that information is all stored in the wall stream, and the new infrastructure in Postgres 9.4 enables you to write or to utilize programs to extract that data out into a format that you can use. So of course, there's new postgres.comp settings. There's the wall level logical, which is kind of the highest wall level now, but that's required. You also have to set your max replication slots in order to do anything useful. Replication slots themselves, basically in order to have these wall stream changes, we need to make sure that there's sort of a semi-permanent way to keep track of what clients are available when, and that's kind of what replication slots end up doing for us. It's basically a way of keeping track of a wall location. You have a specific name, sort of a token that is assigned, and that will be created, and even preserve over system restarts, things like that. It ensures there's consistency. It ensures wall files don't get recycled, and so basically you get to set a spot and read events. It's kind of a built-in queue that uses the wall stream for its source. So slots are created and managed via the replication protocol. You can also use PG receive logical program in the binary directory to do that, and of course there's some SQL functions. If you're wanting to create replication slots, there's an invocation here. You can see that you specify the slot with the slot argument, the database, and then the action create slot to drop slots. It's the same except for the drop slot option, and then you just indicate start, and that program will start up. So there's also SQL function equivalents. So you can call this from within Postgres itself rather than kind of externally, which I think could open itself up to some interesting applications. The one difference here is you have the option to peek the changes. So basically you can kind of see things that are upcoming. Any of the changes that are pending can then not be consumed, but maybe use that information to decide who's actually gonna consume that data. It's one thing that comes up to mind. So in order to use this, you have what are known as output plugins, and basically the wall has to be decoded. So this is the job of an output plugin because no one output's gonna be sensible enough for every tool. So we want to basically allow people to have programs that will parse this data out and generate something useful. You can provide that in the PG receive logical option with the plugin option, which is essentially a piece of code that's written and will be loaded as an output module. So that's of course great for things looking forward. There's a lot of infrastructure here. There's a lot of things that will be interesting and good and useful for developing new logical replication systems, and that's really great for people who are on 9.4 and future versions. Clients still are paying the bills. They need things done yesterday. They don't want to can't upgrade. And even if they were on 9.4 today, that's not an entire solution. There are tools that need to be written, things to be adapted. So it's a great infrastructure. It's a great way to start, but we're still looking for real realities of engineering. So let's talk about logical replication for now. Basically kind of reviewing some of the stuff that we talked about before with logical replication that's fundamentally just capturing the logical changes that you make to the database as opposed to the physical changes. So it's the table that you make a change to, it's the columns that you change, and it's the type of action that occurs. You have the insert, update, delete, those are all things that will change. And essentially most logical replication systems will propagate these changes. In general, they work with triggers on the tables to record any changes and some sort of daemon to propagate any of the changes to the intended nodes. It's inherently asynchronous in nature, which in general I would also say is fairly expected. Most people will not need synchronous, and you're probably using the wrong system if you're trying to do this with these tools with that requirement. So yeah, traditionally this has been done with trigger-based systems such as Slowney, Bacardo, and Laundiced. So I'm going to go over Slowney and Bacardo in a little depth here through the rest of this talk. How am I doing on time? Okay, so some of the benefits of logical replication. It runs on different architectures and Postgres versions. You can replicate a subset of database changes, specific tables, sequences, et cetera. The database clusters themselves are still writable, so for instance you could have a separate database that is not part of this replication set, and it would itself be writable. You could still use it for other things, which particularly if you're working with an existing installation can be very good. And of course there are also some limitations when it comes to logical replication. The systems that are out there, you need to kind of specify replication explicitly. What gets replicated, how it gets replicated, if new tables are created, they're not automatically added. It requires more planning. Again, oh man. Also there's special handling for DDL or structural changes, so most systems out there are not able to just automatically take changes to the table structures and automatically detect that and apply that. So there's at the very least some coordination there that has to be done. Then it also requires a daemon or an external process to monitor and manage, which of course is another moving part. It's another thing that could break. Let's talk some specifics though. So we'll look here. We're going to look at Sloney and Picardo and briefly at BDR. So there's a little matrix I kind of put together. We have kind of the master-slave type of replication, which all the systems support. We have multi-master option, which Picardo and BDR support. Custom conflict resolution. Again, not relevant for Sloney because there's only one source of changes. The other two systems have that. Custom data transforms is a feature that Picardo itself has. I don't remember if I get into this or not. Multiple replication groups, non-postgres target databases, et cetera. And then cascaded and custom topologies. So Sloney is a system that was designed explicitly for single master multiple-slave replication, so that is its intent and that's what it does and it does a very good job at that. We have kind of our base anatomy here of a Sloney system where you can see that we have a number of nodes where each node is essentially a database. Within those nodes you have sets and within the sets are tables, where basically the tables are the things that are replicated, of course, sequences too, but I didn't include those. Each node has its own Sloan daemon. That's the little green item there. And then there's a Slonic program. I'm not sure if I'm pronouncing that right, but that effectively is what communicates with the Sloan daemons in order to manipulate the cluster, make changes to things. So in this example cluster that I've got here, we've got two nodes. We've got two sets and two tables in each set. So as this example here shows, you can see the replication proceeding via the set. So all the tables in a set are replicated together, effectively. You can have a different origin node for each set. So you can see that on the first node we've got the red to set and the second node we've got the blue set. And effectively, even though they live in the same database, there are two separate groups of replicated tables. So that's kind of a brief overview. I'll go into some more detail on each of these. Covered a lot of it nomenclature just now. We've got a path, of course, which is basically the DSN to connect from one node to another. Now, depending on how complex your Sloney cluster is, this basically will be a method for each one to get IP address, house name, database name. That's that information. At its core, Sloney is a distributed serialized event system. Its internals are stored inside a Sloney schema in each node's database and it contains all the support functions, triggers, et cetera, needed for the database level part of Sloney. Sloan daemon runs for each node, listening for and processing events. Each node has its own event queue, so it's not a coordinated queue across each node. So each node is able to send and receive and independently confirm its own events with all the other nodes. In general, the Sloan communication is done via Slonic, which has its own domain-specific language for creating all these events and distributing them across the cluster. This one here appears to more or less be a duplicate of that, but, you know, effectively, let's see, with Slon daemon, you generally will run that on the same machine as the database, but that's not a requirement. Any machine that can connect via the database, you know, latency-wise and whatnot, and just availability-wise, it's wise to run that on the same machine, but by no means do you have to. A little more about the nodes themselves. It corresponds to a specific logical database instance. It's basically just a mapping from a node ID to a name, a conceptual name, and that's purely for kind of organizational purposes. You basically set everything up with the node IDs in order to describe the relation in the topology and the source of the events and whatnot. There are several tables that are irrelevant here. You have the SL node table, which defines all the nodes in the cluster, so every node has its own copy of this. You have an event table, which basically queues up any of the events that are created for this particular node, and then I don't mention it here, but then you have a confirmed table that basically writes who's seen the event in question once it's done. So any changes to the actual node configuration and whatnot are done via the events and that has its own event type. Talked about paths already. That's basically the connection information. I think with current versions of Sloney, the listens are created automatically, but they used to be important. So Sloney sets are the next thing. We saw conceptually how those are kind of the groups of the tables. They're things that are replicated together. Each set has an origin node, which is the node ID master for that set. That's the only node in the cluster that you will be able to write to that set of tables with. It's replicated together. That's stored at the SL set table in the Sloney schema. While the tables themselves are stored in the SL table, those have a one-to-one node mapping as far as what set they belong to. So the tracked table is in only one set, and I provide some of the options with Slonic for creating this. Then you have subscriptions, and this is where you kind of describe, actually, what is done with the sets. So subscriptions are basically a mapping from sets to nodes. If a node is subscribed to the set, it will receive any data changes related to it. You have to be a listener to be a subscriber, but basically the subscriptions themselves are what's replicated where. So if you have node one and node two, and in a three-node cluster, if node two is subscribed to set one from the first node, it will get all the events. It will propagate that data. Sloney also allows cascaded subscriptions, which basically are when you use a non-direct route to get to the node, let's say. If you have a master and you have several slave nodes, then they can either all connect directly to the master, or one can kind of daisy-chain to the next, and that can come in handy. In one of my case studies, I kind of go over how we were able to utilize that for a good purpose. So anyway, there's some extra special syntax when setting that up. You have to just basically say that you want to forward the events, because normally when events are propagated, they just by default will not be sent and stored on that table on the other node. So this kind of serves to have this work as like a dispatcher. A little bit more about the triggers. So when the table is added to a replication set, Sloney installs what are known as the log, or the deny triggers to the table. So when that's subscribed, it'll basically depend on what the role is. So if you have an origin node, it gets the log triggers. If you are part of a non-origin node, if you're a listener, you end up with the deny triggers. And that's what prevents you from modifying the table data itself on the other side. This is also done when you subscribe. Sloney has checks in place to make sure that you have the exact same table data in place on both sides of the system, so it can guarantee that all the systems are set up the same way and have the same data. It does that with a truncate and a copy to initially do this. So this is a little bit about how Sloney tracks changes. It takes ownership of the tracks tables, and on the origin node it adds the triggers, which will fire on insert, update, or delete. Probably truncate, too. I'm not 100% sure. It can. It does truncate. Makes sense. Anyway, these log triggers end up storing information about what row was affected. Essentially, the key in question, it contains this tuple data and the current snapshot of the transactions that were in place at the time of running, because that comes in useful when you're replaying that. On a subscriber node, you have the deny access triggers, and that basically just does nothing, so it effectively aborts whatever action you were trying to do in the next case, and that way you can guarantee and know that your data is the same, and Sloney can take that for granted without having to do anything else. Applying changes. The node Sloan daemon listens for sync event notifications. Those are types of the SL event types. Effectively, that'll go and look at the provider node, gather the data from the log table since the last applied sync event, and pull any of the logged information across. Once it applies that data, it'll confirm it on the remote node, and so this ensures that we have serial changes, so all of the changes that were done in the same transaction are applied at the same time on the slide. So let's talk a little bit about Bacardo now. It's a standalone replication system. You can push changes from Postgres to Postgres or other databases. It's also trigger-based. It's asynchronous. It supports both master-slave and multi-master types of replication. Here's a brief anatomy of the Bacardo system. At the top, you have the master control process, the MCP. You have control processes, the CTL. You have kids, because Bacardo can't get enough goat puns. And those handle the actual dirty work of the replication itself. So this is, of course, a brief overview. You can see an example here. For the red sync, we've got... that's a multi-master kind of sync, so the data is flowing both ways. For the blue sync, we've got the one-way data transfer. So the kid handles the actual propagation of the data. The controller makes sure the kid's around forks and when there's changes needed, etc. And the master control process just runs everything and makes sure everything's great. And we've got the Bacardo command-line process there, which is your primary point of interface with the Bacardo system. So a little history about Bacardo. It was developed at Endpoint. It was started at Backcountry.com in 2002 using Postgres 7.2. It was released publicly in 2007, so I think that's what, eight years old at this point. We're up to Bacardo 5, which just has true multi-master. Previous version had had just dual master. There's lots and lots of improvements over the previous versions. Some of the strengths of Bacardo include low requirements. You've got Postgres 8.3 is the minimum, PL.Perlew, DBDPG. No changes needed for Postgres or its configuration. There's one daemon process which connects to all the databases in question. It can be run anywhere as long as it can connect everywhere. It's fast. It handles poor connectivity well, which is very handy a lot of times. So you can have situations where, if a connection goes down for a while, once it comes back, it'll automatically reconnect and start the replication process again. It has good command-line monitoring, all of the configuration and monitoring for the most part is done with the Bacardo program itself. There's an easy install setup process. More strengths, depending on how you look at it. Targets, slaves are not locked, which means that you can write data to those things. Now, that is going to come in handy. Also, the configuration is in the database. We also allow custom code, such as conflict handlers, data transforms, et cetera. So you can replicate different data as actually changed. You can do custom aggregates to get pushed out to another server if you have reporting things that need to be done, so that can come in handy. There's multiple target types, Oracle, MySQL, Mongo. This is an interesting difference here. In terms of slowny, if you make, let's say, 100 changes to the same row between sync runs, it's going to queue up 100 events, which, based on its design, it should do. In the case of Bacardo's design, if there's multiple changes to that same row, we only mark that once. We store the primary key only, and we don't care about the interim states. We basically say the final state of the row is what's interesting. I think I touch on this more later. But Bacardo itself is good for heterogeneous environments, so multiple database systems. If you have a legacy system that's running Oracle, and you're in the middle of a Postgres migration, you can push things across. So there's lots of benefits there. Of course, it comes with its own set of limitations. There's no automatic handling of DDL. There's no built-in failover. Bacardo considers that outside of its domain. So in the sense of multi-master, if you have one master, it's up to you to kind of deal with how that's handled. Are you writing to both databases, or are you writing to just one of them? So it leaves that convention up to you. Of course, it requires primary keys for most of the syncs, which I honestly should not be a limitation, but it's just good design philosophy. Again, the target slaves are not locked, which could be strength, could be a weakness. So if you want to ensure that the data has not been changed, Bacardo does not give you that guarantee, because that considers it out of its purview. And again, it replicates only the final state of affected rows. So for instance, if you had some triggers that were supposed to log on every change, such as audit triggers, slowny would be a better choice in that situation, because it's going to make every change, run the triggers that are needed on that side, whereas Bacardo would just run it once with the final state of the row, if it was not keeping up today. So a little bit about Bacardo's fairly peculiar nomenclature. We call tables or sequences goats. You have the named group of goats. This is kind of the slowny set. This is a herd, because what else would you call a group of goats? And then you have sync, which basically specifies how the kind of data is replicated and how. And this is similar to the subscription in slowny. As I touched on in the diagram, we have the Bacardo architecture. We have the master control process, which runs and monitors everything. We have the controller process. It's spawned by the master control process. It's responsible for essentially listening for any changes in replication state and handling monitoring kid processes. And then the kids are themselves workers that do the actual replication. I just can't get enough goat pens, like I said with the kid. So the primary point of interface with the Bacardo system is through the command line program. This used to be called Bacardo control, which I think makes it a lot less confusing in this talk, because we say Bacardo, kind of referring to the entire system or the daemon itself or the command line program. So I apologize for that. That lets you do such things as install and set up the database, the Bacardo system, add any of the objects, teach it about databases, teach it about the herds, teach it about any of the things that you want to do, query for status, kick off a specific sync. And it has lots of good built-in help. Probably one of these days I'll do a demo, live demo with things, because I feel lucky. The Bacardo configuration itself is stored in a special database. It's Bacardo, it just keeps, instead of trying to keep anything in a config file, it just stores it all in the database, which has lots of validations and triggers and whatnot, so it ensures that everything is set up properly. Quite a bit of, but the Bacardo system lives in its database. So it keeps track of the databases, tables, the herds, sync settings in its own database, which has to be owned by a Bacardo super user. This is all set up for you automatically via Bacardo install, you pass it to DSN via the command line when you do that. And it basically is able to store state about the Bacardo daemon itself, all the syncs, it's essentially its own metadata database. So how Bacardo tracks changes through its system is, again, with triggers. Whenever an insert, update, or delete is done, the primary key of the table is logged in what's known as the delta table, which lives in a Bacardo schema on the target database. So it doesn't matter if it's an insert, update, or delete, if it's a delete, the only thing that's logged, essentially, along with timestamp and whatnot, it's not logging the actual table data, it's only logging the primary key. This can, depending on your sync settings, trigger notification that data's changed, and then Bacardo will go ahead and start processing and applying the changes. When Bacardo runs syncs, either automatically or manually kicked, it checks the delta tables for all affected PKs, and then deletes them on the target, and then it copies the current row in case of any deletes. So this handles cases like, if you delete it, then it's not gonna exist in the target table, so when it tries to do an insert, it's not gonna have a source row for that, and it basically handles things all itself. But it does that in batch, kind of as I had described with any changes that were done. Bacardo is, of course, multi-master. That was one of its original development purposes. Bacardo 5 has true multi-master using round-robin syncing approach. Every master in the cluster logs change rows by PK into a custom table. Of course, when you have multi-master, you have the potential that multiple people have changed the same row at the same time in non-compatible ways. So as part of the application process, Bacardo checks to see if there's any conflicts. There are different handlers that you can define for a specific sync that will tell Bacardo how it should handle these things. Basically, any conflict is a bad situation. We have to figure out what's the right thing to do. So Bacardo has some standard conflict resolution methods. There's the source, the target, the random, or the latest, which basically... the source means that the originating database, the current master basically wins no matter what. The target is the opposite, where the remote master wins. You could randomly do that, which I'm not sure that that's a good idea, but apparently it exists. And there's the latest where the timestamp, one that was changed last, is the up-to-date version. And also, we allow you to write your own custom conflict resolution. So let's talk a little bit about BDR. It's the new kid on the block. Wish I had pictured Donnie up here, but couldn't figure it out. It's the bidirectional replication. Works via the logical streaming features of Postgres. I'm not really covering it extensively because I know there's a talk following me that seems pretty cool, but this looks like a very nice feature for Postgres to have, especially in core. Some of the strengths here is that it doesn't use triggers, so it's able to avoid right magnification. It handles DEL modifications automatically for many cases, such as table structure changes and whatnot. New tables are automatically replicated by default. They're added as they get put in. The current limitations of BDR is, of course, it requires a custom patch Postgres install. Their goal is to get it into core, which I would expect probably 9.6, realistically, but we'll see. Some of the other limitations are that certain DEL features are currently restricted in the version that's out there now. And, of course, it only works on Postgres greater than or equal to 9.4. You also have some limited control on cluster apologies, so it's not as nice as, say, with Sloney or Picardo in that regard. I'm probably going to skim through this particular section. I didn't realize it was going to go longer. I felt I was going to go short, but we've got the side-by-side overview here of how you'd accomplish similar tasks with Sloney and Picardo. So constructing the cluster with Sloney, you basically can use the Slonic scripts to have a turn your config file into the cluster. It sends out a new cluster using the node configuration and the IDs that you've done. It creates all of the paths and listens and basically connects to all the databases and initializes everything. For the purposes of Picardo, you would use the Picardo tool to run the install process. Then you add the database names with various options. I think this slide is actually inaccurate for Picardo 5, because I think we did away with DB groups, but basically you would define the herds, you would define the sets of tables that should be replicated where and when. Here's the syntax for specifying the replication groups. Within Sloney, you basically create a set with the defined tables here. You add them together. This is the syntax here with the Picardo control program to do the same. You can modify the existing replication sets within Sloney. If you have to, let's say, add a new table and you want it to be in the same replication set, you have to create a set, a new set with its own ID, subscribe that set to the same providers as the original set, and then merge the set together, but that's able to do that atomically. You've got Picardo where you basically just use Picardo itself to modify the sync definition, or you can. Also, I believe, modify it in the Picardo database directly if you want to. A little bit about the logging differences. I guess I've already covered this. Sloney itself logs the full row data, because it needs to do that in order to replay those changes. Whereas Picardo just logs the primary key and is only concerned with the final state. If you need to apply DDL changes to the Sloney cluster, you create a SQL script to make those changes for you, and then there's a Slonic function, or command called execute script, which basically will coordinate the running of that across all the nodes in the cluster just to set everything up and make sure that it runs at the proper time. Anything generated before or after will have the right log data structure. For instance, if you're logging changes to a table and then the table changes, you can't replay the structure change until you've redone all the events that require the old table structure. With DDL, with Picardo it's a little messier. You can stop the replication, make DDL changes manually, restart the replication. It's also smart enough, though, if you want to do something like a simple column addition or drop, you just do that in a specific order, because it effectively does a select star from this table in question. If you add the column to the slave first, then it'll still be replicating the old data correctly, and then you can add that column in the master. But in general, it's better just to start and stop the replication yourself if you can afford that. Failover, Sloney has formal failover, but moveset is their preferred method. That switches the origin node from one particular node to another. It does this in such a way that the entire cluster is sane. So, for instance, if you have a missing node that goes down, it ensures that those events are in the proper order and properly acknowledged by everyone. If you have to, if there's a failover command, that'll forcibly remove the node from the cluster by contacting every other node except for the target one, and that's basically something you only want to do if there's, like, no hope at all. So, Bacardo itself, of course, has no formal failover, but it does have the deltas that continue collecting. If you have one of your masters go down, the other masters will still be queuing up their changes, and then when it comes back, it'll notice that, restart the sync and make sure that it runs, you know, whatever conflict handling it needs to. Some monitoring with Sloney. There's the SL status view, which returns just information about every node in the cluster from the perspective of the node that you run that on. So, it'll show you the number of events you lagged. It'll show you the time that's been applied. This can actually be out of sync within the cluster, so one node could be lagging behind relative to the other ones, but everyone else sees everything fine. So, that's essential for debugging and monitoring things. Bacardo provides similar information through the Bacardo status, which, you know, will show all the syncs and any lag and errors that are, it's countered. Sloney, of course, you know, wants to try to stay up to speed and replicate things as quickly as possible, which is the general case for most things. Bacardo lets you kind of change how that approach is done. You can have more or less immediate syncs, where basically when there's a change that's made, it will kick off the sync for you. You can also have it be a timed process or a manual process where, let's say you have table data that you don't care. It needs to only be, like, every 15 minutes or something. You have some business use case, so you can set up the sync for that specific table, so that only runs. All of these tables can, all of the syncs, excuse me, can have their own policies with that. So you could have some getting kicked immediately. You could have some, you know, every five minutes, some every 10 minutes. You can kind of customize the necessary behavior there. All right. I feel we're not going to have a lot of time to get into the case studies, but the short answer here, the talk's called choosing a logical replication system. And so, of course, there's no right answer. You have to kind of pick this scenario, depending on the strengths of what you need, and pick the right tool for the job. We have an example here, which I think dates this specific one, about a minimal downtime upgrade. We had Postgres 8.2 with Sloney. I can't remember, but it was 1.0 something in a four-node cluster with one master, three slaves. And it wanted to upgrade to the shiny new Postgres 9.0 hot standby. So what we ended up doing, because, unfortunately, the version of Sloney we had didn't support Postgres 9. We couldn't modify the cluster. We couldn't update anything. And we had all those pesky app servers that they wanted to keep around for some reason. This was a client where there was no downtime maintenance opportunity. So what we actually ended up doing was using Picardo to replicate the Sloney cluster to a hot standby cluster that was set up on the other data center. So that was actually three types of replication being used the same way. You can actually run Picardo and Sloney on the same system, and that works fine. So basically, we had very simple schema, but very large table, lots of changes going on. We had to make sure that this all happened. But using Picardo, we were able to kind of have the application downtime itself be measured in minutes regardless of the size of the database. So this was a large database. We ended up creating the new Postgres cluster with the hot standby streaming replication just on its own. Kind of an empty replication cluster. We tested everything on that side. Created the schema on the other side, and then basically installed Picardo on the new system. So we made sure that we were collecting the deltas in the Picardo tables. So at that point, we could then do Postgres dump, PG dump, everything still getting captured in this time. So we was able to load that. Then we were able to start Picardo up, have the replication run, be able to copy any tables or records that had changed in this table over. Essentially, so we were able to kind of take this approach using Picardo, get things going, updated, and working on the Postgres 9. And then once we were done and tested everything, we just removed Picardo from the equation and they had their new setup. I don't have enough time, I think, to go into this other data center migration, but effectively what it was was a sloney cluster that we had to go across a very expensive and error-prone VPN with a huge database that took 14 hours to subscribe. So we were able to minimize that by using the cascaded subscription to set that up, do the replication once over the expensive link, and then do the other subscribes over to the other nodes on that side from there. So I guess at this point, that's about all I have, so I don't know if anyone has any questions. We've got a couple minutes maybe, but thank you very much. Anybody? All right, who wants the red ball? Okay, thank you.