 Hi, everybody. So yes, I was already introduced as a Postgres consultant. My main expertise area is high availability for databases, and especially Postgres. So we do a lot of consultancy for a lot of local customers over here. And we build a system for them for high availability using Postgres application. So my talk also today goes through the new feature that came in Postgres 9.0. So just to get an idea, how many of you guys are using Postgres in production? Very good. And how many of you tried Postgres 9.0? Great. So you did. So there were very nice new features that came with Postgres 9.0, and one of them was streaming replication. And that was one of the most wanted features for quite some time, especially when people compare Postgres to MySQL. They obviously said, oh, MySQL has got the application out of box, and there's nothing like that in Postgres. And that was a very valid complaint, because there were application solutions available for Postgres, but they are very, very hard to implement, like Sloanee, Bucado. Has anybody tried using Sloanee over here? And did you find it hard to maintain and to monitor the Sloanee processes? Because it gets a bit annoying, because it's all trigger-based. So it was always needed in the Postgres world to have a solution that's based not on triggers, but on the transition log files. So in Postgres 9.0, they came up with the proper solution. So we'll be covering those areas today. Agenda. We'll be covering a little bit of history on Postgres application solutions, and a little bit about the streaming replication internals, and what the limitations we got with the current application setup, and how do you quickly set up a house standby, which is pretty easy with the latest version, and how do you monitor lags with the streaming application, and a little bit about conflict resolution. Because with every application solution, you get conflicts. How do you resolve those conflicts with the streaming application in 9.0? And a little bit about the future of streaming application for Postgres. Starting with the history, we all know that Postgres doesn't have anything out of box for application. So there used to be a lot of solutions, and they're still out there. Like Sloanee, Becato, Alondes, by Skype, and they've been there for quite some time. They're very stable, and a lot of people are using them, and they're very happy with it. So with this new replication architecture, like using streaming replication, those solutions are not going to die, because they've got their own, like they basically covered their own areas. Like with Sloanee, you can replicate a set of tables. You don't have to replicate the full database. You don't have to replicate the full cluster. So they have their own advantages and their own disadvantages. But now at least Postgres has something that is out of box, and that's very easy to set up. And none of them are using transaction log files. They're all trigger-based. So you have that added complexity to your database that every table has to have a trigger on it. And another thing that used to be a problem with Sloanee was that you need to have a primary key on each and every table. That becomes a bit of a problem when you've got a big table. Then you have to maintain a big index as well on that table. So some of the databases, you can't really have a primary key on all tables. So yeah, there used to be a problem for Sloanee and other replication solutions. Prior to version 9.0, Postgres used to support something that was using VAL logs. VAL log is a transaction log file in Postgres. So I'll be using that term a lot of times in the presentation. But a VAL is like a log file, and that's like a transaction log file for a Postgres database. So before 9.0, there used to be something that's called warm standby server setup in Postgres. But there was file-based shipping. There wasn't record-based shipping. So there used to be a little bit of a lag between the primary server and the standby server. And another thing with those warm standby server setups, you couldn't query the standby server. You can just have it as a failover setup. Like whenever you want to do a failover, you can just take your standby server out of recovery mode and make it a primary server. But you couldn't do a record-based log shipping. Because it would only ship a log file once it reaches that size, that 16 megabyte. But with 9.0, you can do record-based log shipping. So warm standby server in Postgres, before 9.0, used to be very complex. They used, it's still possible in 9.0 as well. So if you have a system that's using the old method of using warm standby server, it will still work in Postgres. If you're using PGNOSco standby to do that, it will still work. But you just have to be careful with a few settings. So I will cover those settings later on in the presentation. But yeah, everything that was working with prior versions will still work with 9.0 as well. Streaming replication, what it is, it's similar to how you had before for warm standby server. But it's just that now you can query that standby server. You can run your read-only queries on the standby server. And it's record-based. So it's not file-based. So you won't have those big lags between the primary server and the standby server. And then it's asynchronous. It's still not synchronous. The original patch that came out for streaming replication had synchronous replication in it. But because of the complexity, and there were so many areas that were not properly covered, they didn't release synchronous replication. But it is coming in 9.1 that's going to be released around the end of this year. And then it supports multiple concurrent slaves. So you can have one primary server and multiple slaves with that set up. But you can't do cascading replication, which you can do in slowning. So by cascading, I mean that a slave cannot send data to another slave. But only a master can send data to all the slaves. Again, it's very simple to set up. It's not like the way you used to set up warm standby server using PG standby. That was very complex. But now, because everything is out of box, so you don't have to worry about setting up external tools to cater for everything for replication. Something about the internals. With streaming replication, there were two new processes that were introduced in PostgreSQL 9.0. One is the VAL center, and the other one is the VAL receiver. So you can see those processes as you do a PS and look for the PostgreSQL process. You will see those two new processes when you have the streaming replication set up. So what these processes do, the first one is the VAL receiver. What a VAL receiver does is that it uses LiPQ, and it sends a call to the master server, and it tells the master that I need to start streaming replication. And once the master gets a request for streaming replication, it starts the VAL center process. And the VAL center process is then meant to send all those replication logs from the tonic and log files to the slave servers. So what Hausten by does is that, as I mentioned before, prior to version 9.0, we used to have one standby servers that were able to do the replication through VAL log files, but you were never able to queue with those servers. They were always in recovery mode. So if you had to fare over, you'd put up a trigger file, and that would basically promote it as a primary server, and then you can just use that primary server. But with the latest version of PostgreSQL, you can set up your standby server in a way that you can queue with those servers as well. So they work in a Hausten by more. They always have the latest data. But again, there will be a little bit of a lag because it's asynchronous, so you can't basically say that it's 100% exactly a copy of the master. It's more like eventual consistency. So eventually, it will be consistent, but it won't be consistent all the time. And Hausten by lets you fail over to the slave server without dropping the connection. So if you have a connection to the standby server and that is open, so if you do a failover and you promote it as a primary server, that connection won't drop. It would just change the transaction more. Because in a Hausten by server, on a standby server, whenever you create a new connection, it always creates a connection with a read-only transaction more. So what it does is that as soon as you do a failover, it would change the transaction more so that you can do read writes on that as well. Now, we can look at quickly setting up Hausten by. It's very simple with version 9.0. So there are a few new settings that were introduced in version 9.0. They are well-level, max-well, senders, and well-keep segments. We will go through the details of those settings now. Well-level, there are three options that you got for this setting. One is minimal, archive, and Hausten by. Minimal is exactly similar to what we used to have before 9.0. So if you have a setup that's using PG Standby to do your worm standby server, if you got your own scripts that are managing the worm standby server, and you want to use the same setup in version 9.0 as well, just keep it minimal. If you're not using the latest functionality, you can use your old systems with the same setup in version 9.0 as well. But if you want to set up a worm standby server, then you just set it to archive. And if you want to set up a Hausten by server to which you can send the queries, send the read-only queries, you need to set it up as Hausten by. There's a little bit of a performance hit if you're using archive or Hausten by, because the verlog will be storing a little bit of more information to cater for setting a worm standby server or setting up Hausten by. But it's very minimal. So you can do a benchmark on that. There are a lot of people who are doing that right now. But because it's still in development phase, there is a version that's available. That's supposed Hausten by, but still there's a lot of work that is going on on making it much better. So again, there's a little bit of a performance hit, but it's very minimal. It's very, very minimal. The next setting is Maxwell senders. As I mentioned before, there is a new process that was introduced in version 9.0 for Postgres that basically caters for Hausten by. So it takes requests from standby servers. And once it receives a request from the standby server, it starts sending those VAL records to the standby servers. So this setting will basically enable that process. So you need to have a positive value for this setting. So it should be more than zero. And it depends on how many slave servers you've got. If you have, say, six slave servers, you just set it to six. So you need that many number of senders to send VAL log files to the standby servers. It starts with a default of zero, but that wouldn't let you do any Hausten by setup. You need to set it to the number of standby servers you've got in the Hausten by setup. OK. So next is Maxwell segments. So that's, again, very important. If you're setting up Hausten by, that means that every time a new VAL log file is generated, it ships it across to the standby server. And it wouldn't keep any copy of that VAL log file until you tell the server to do that. So in most of the times, setting it to zero will basically do the job as well. But at times, you can end up in a situation where something went wrong with one of the standby server, like it went offline or something bad happened to that box. And once you bring it back online, you want to take it back to that state where the primary server is right now. So you need those VAL log files to do that partial bit of recovery. So setting it to zero is not really a good option. Just set it to a value that is according to your disk space, because each VAL log file would take 16 megabytes. So I generally go with setting it to around 100 VAL log files. But you can put it to any number you want. It depends on the amount of disk space you've got. Yeah, it's just a few recommendations of that. So set it higher if you're using HallStandBy, but make sure your disk space is available. And in case you're doing a VAL standby server, you're not setting up it as a HallStandBy server. You don't really need any VAL log file for that case, because as soon as a new file gets available, it gets shipped to the standby server, and it's always replaying that VAL log file. But if you're setting up a HallStandBy server, in that case, you really need those VAL log files in case of a crash on the standby. A quick start on HallStandBy. So have anybody over here has done PITR in Postgres? Point of time recovery? Nobody? Okay, so when you're setting up a HallStandBy server, it's basically you need to take the exact same data folder dump, like a file system dump of the whole database to the standby server, because the ORDs cannot change, because what streaming application does is that it takes the full VAL log file and just replays that onto the standby server. So the both of the environment should be exactly the same. So what we're doing is we're taking a full file system level dump of the whole Postgres installation and putting it onto the standby server. That's how you start the setup. So before you do that, you use a few Postgres functions to take that dump. The first one is PgStartBackup. What that does is that it asks Postgres to do a checkpoint. A checkpoint in Postgres is where you basically write all the data and dirty buffers to the disk so that you don't have anything sitting in the memory right now. So you just want to make a consistent dump of the whole database. So you run that command and then you take a full file system dump. You can use rsync or you can use, you can start up a full database dump and move it to the standby server. And then once that is done, you do a PgStartBackup. That way you have an exact same replica of the primary server on the standby server. And once that is done, we will go through the next steps. And for that initial step, there are a few tools that are still in development. And that would basically make the job easy. Because right now you have to do it all manually. You run that query, take the dump, and then do a PgStartBackup. But there's a new module that's called PgInnerScoreBaseBackup that's coming in version 9.1 which will help you do that automatically. So you just do PgBaseBackup, give it the folders of the primary server and the secondary server. It automatically takes the full backup of the primary, ships it across through the standby server using rsync or the command that you give it. And that will basically make the job a lot easier. Because I've seen a lot of people making mistakes while doing these dumps. Like they would do PgStartBackup, but they would forget to do PgStopBackup. Or they wouldn't have proper checks to get the return codes of these functions. Like they would do PgStartBackup, and they wouldn't check the return value of that function call. Because at times, it might just give you error that I can't do a check when something else is going wrong over there. And they will still take a file dump. They wouldn't do it properly. So that new tool will take care of all of those problems. Next thing is, once you've taken a dump and you've shipped it across through the standby server, you need to set up a few things on the PostgreSQL or Confine in the master server. And they are setting up the VAL level, as we discussed before, that you need to set it to horse standby for a horse standby application. And VAL keep segment should be set to 100. That's just a starting point. It depends on your own scenario. If you are setting up your server as a worm standby server or a horse standby server, it totally depends on that. You can go with a zero or a number that you are comfortable with. And then Max VAL senders. I've set it to one because I'm just setting up one slave server. If you're setting up multiple slave servers, like you're setting up six slave servers, you set it to six. So you need one VAL sender for each standby server. And then Archive mode, it should be enabled. And then Archive command as well. That basically tells the Postgre server that whenever you got the new Archive file available, the VAL log file available, ship it to my Archive folder. And then the next thing is, you need to do a few things on the standby server as well. The first thing is setting up the pg.sqlhp or conf entry. pg.sqlhp or conf file is basically has all the authentication rules for Postgre server. So you need to enable access to the replication database. That's an internal database. You can't really see the database on the primary server. It's invisible to the normal user. But it's used for doing all the management work for the host standby server setup or for streaming application. So you need to enable access to that database from the standby server. And it has to be trusted. Usually people don't like putting trust in there. But yeah, you can't really go with empty file where it just doesn't work. So it's going to be improved in the next versions. But for now, it's just trusted. The next thing is setting up the host standby setting in the pg.sqlhp or conf file of the standby server setup. So you set it to on. If you want to set up your server as a standby server and then setting up a recovery.conf file, it's exactly the same as you used to do with the vom standby servers in the past prior to version 9.0. But the only change is now a new setting has been introduced that's called standby underscore mode. So you set up a new recovery.conf file and set up a restore command that's similar to the archive command that you set up on the master server. So once that is set up, you start your server. And as soon as the server sees a recovery.conf file in the data folder, it automatically goes into a recovery mode. But the difference between a vom standby server and a host standby server is that when it sees a recovery file in there, and it basically can see a value for standby mode enabled, it automatically tells the poster server that I can accept connections as well. Because in a vom standby server, it can't accept connections. It can only replay the valid log files, and it's always in recovery mode. But in a host standby server, the recovery mode means that it can even take connections and it can run read only queries on top of it. Yeah. So we'd have to do the same thing as far as, like, R-syncing the files over or whatever, because, I mean, you have it set that it's a shared directory right now. Yeah. But we'd still have to go through those steps of R-syncing the files. Yeah, R-syncing is for different purpose. It's basically, R-syncing is used for taking the full file system down of the data directory. No, no, I meant your restore command and your. Oh, you can use R-syncing over here as well. It depends on. Right, so because your examples show that shared folders, I just want to. Oh, yeah, you can use any command. You can use your own script as well over here with proper exit codes that you can check if the file is properly copied or not if there's anything. Because I would prefer having your own script over here instead of using CP command. So just have your proper checks in there that if the file has been properly copied or not. Because I've seen problems where the file is not copied properly. And it can take the server into a state where it just doesn't want any more connections. So you need to have a proper script that will make sure that the file is copied across to the other side. So yeah. Some limitations with the standby server. So you can't run DDL statements on the standby server because they use transaction IDs. And the standby server doesn't support any transaction IDs. It wouldn't generate any transaction ID on the standby server. It's always taking everything from the primary server. Next thing, you can't exit. Oh, sorry, it was DML. So a DML is like insert, update, or delete because they use transaction IDs. And then the next thing is DDL because a DDL will change the system catalog. As soon as you do alter table or you create a new index, it will create new entries in the system catalogs. And we can't do that on a standby server because it's taking everything from the primary server. And then queries like select, for share, for update, they wouldn't work as well because they do a few things with the underlying data files. And we can't touch those files on the standby server. We're taking everything from the, yeah. Oh, yeah, yeah, yeah, you can do everything on the master server. It's just that you can't do it on the standby because you want to keep everything consistent. So if you do anything on the standby, it won't go back to the master. That's multi-master if we can support that in the next row, it's it. OK, so some more limitations. Two-phase commits, they won't be possible as well because even if you're doing a select with a two-phase commit, it still creates a bare log entry. It still creates knowledge and log entry. So you can't do a two-phase commit even if it's for a select statement. So yeah, two-phase commits won't work on standby servers. You can do them on master, but you can't do them on the standby server. Sequence updates, they wouldn't work as well. You can't use temporary tables on standby servers. So they're going to be supported in the next version. But right now, you can't do any temporary tables. That's a bit of a problem because if you're using a standby server for reporting, you want to use temporary tables over there. But right now, that's a kind of a limitation. So does that mean you can't find the value of a sequence from the slave since curval won't work until you call it? Your current value will work, but you can't increment the value of the sequence. In current versions of Postgres, if you're trying to call a current value before the next value is called, it fails. Yeah, it does it. If you call a current value before the next value, oh yeah. Yeah, because the sequence hasn't been defined yet. Yeah, I haven't tried that with host and buyer, the sequences, so I can check that and I can let you know. But you can't do a next value or a set value with that because that basically is changing the sequence. So it won't be possible for the standby server. So if it is giving an error with version 9.0, that's a problem, but it shouldn't. You can try that with 9.0. Yeah, it shouldn't give you a problem with 9.0. I'm pretty sure. No temporary tables. That's a bit of a limitation because you really want to use temporary tables with reporting. But you can't do that right now. And if you're using a hashing dex, because a hashing dex right now doesn't generate a backlog entry, so that wouldn't be replicated. If you're doing anything with a hashing dex on the primary server, it wouldn't go to the other side. But the normal index on Postgres is a B3 index. They're not normally hashing dexes, but if you are using hashing dexes, there's a bit of a limitation. In every replication solution, there are all these conflicts, so there are some conflicts that are possible with the host and buyer setup. So if you're dropping a table space on the primary server, it can cause a problem on the standby server because there can be a few objects in the table space that are being used in queries on the standby server. So as soon as the standby server finds a drop table space command, it will try to drop the table space. But because the tables that are in the table space are being used by the standby server, it will go into a wait scenario, where we just keep on waiting and waiting and waiting. So that can cause a lot of issues because your queries are just waiting and your whole standby server starts lagging because it can't replay that backlog entry. So that can be a problem scenario for a host and buyer. But there are ways to resolve that. I'll cover that later on as well. And then if you're trying to drop a database on the primary server, that will make the standby server drop all the connections because it has to do that. If you are dropping a database, you can't really make any connections or database on the standby server, so it will just drop all the connections and drop the database. And if you want to get a report on all those conflicts and how they were resolved, you have a view in Postgres that's called PgStat Database Conflicts. So it will list all the conflicts that have happened and how did it resolve it. If it canceled the query or what exactly it did to resolve that conflict. The two very important settings in the Postgres log-con file on the master server that can help you resolve those conflicts. One of them is Max Standby Streaming Delay. And the other one is Vacuum Defer Cleanup H. So the first one is Max Standby Streaming Delay. So what it does is that it puts up a timeout on the query, like say you were dropping a tablespace on the primary server. And one of the objects in that tablespace was being used on the standby server. So that means that it will try to drop it and it will go into a wait scenario and it will keep on waiting forever until that query has ended for the select. But if you set up a value, say, of 30 seconds on Max Standby Streaming Delay, that means that after 30 seconds it will automatically cancel that query and it will execute that valid log record. So that way, you cannot have a more than 30 second delay on your standby server. The next one is Vacuum Defer Cleanup H. Most of the problems with whole standby server are with Vacuum because on a primary server, in order to do the maintenance, there's an auto vacuum process which keeps on vacuuming tables every now and then. And it basically looks at the number of inserts, updates, or deletes that are happening on the table. So what happens is that, say you have a table which is just getting hit all the time. It's getting inserts, updates, deletes all the time. And on the standby server, you're trying to get records out of the table. So as soon as the primary server starts a vacuum, it basically starts changing the tuples on that primary server for that table. And those tuples have different rows, different records on the standby server. So both of them start having conflicts because the standby server is showing the old records. But after a vacuum on a primary server is showing the new records. So in order to take care of that kind of a scenario and to get rid of that conflict, they introduce a new setting called defer cleanup age. So what that does is that it will defer a vacuum process by a specific number of transaction IDs. So if you set up this value to 1,000, it won't vacuum on the primary server until you've done 1,000 transactions on the table when the vacuum was actually going to happen. So it will put up a delay on the vacuuming. And it can introduce a little bit of a bloat on the tables as well. But it will help you resolve those conflicts where you're trying to run a vacuum on a table that's already in a transaction or in a select query on the standby server. So that certainly helps with the vacuuming issues. Because vacuuming is causing a lot of problems for horse stand server right now. It's just a fix for now. They're going to improve it in the future because even this setting is causing a lot of problems. Because as soon as you put a defer cleanup age, that creates a lot of bloat in the tables. So if a table was supposed to be vacuumed right now, it won't be. It would wait for another, say, 1,000 transactions to get to a point where it can be vacuumed. But that will solve a scenario on the standby server side that it can be used for doing read-only queries. I'll tell you a bit about administration of horse standby. So there are a few parameters that you can't really change on the standby server. So if you want to change those parameters, you need to change them on the primary server as well. And there are max connections, max prepared transactions, and max log per transaction. So if you are going to change them on the standby server, make sure that you change them on the primary server before you do it on the standby server. And they always have to be greater than or equal to the value on the primary server. So you can't have them less than the primary server. Anything that uses DB link in Postgres, that won't be affected, because as I mentioned before, that everything on the standby server is read-only. But if you're using DB link to connect to a remote database and do an insert or an update, that will still work the same way. So it wouldn't affect that. So a transition is read-only only for that database. Anything that is outside of that database server is not relevant to the horse standby server. You can still do writes, but on to remote databases. And you cannot create additional indexes that are only on the standby server. If you want to create an index, create it on the primary, and that would get it replicated to the secondary server. You can't have just indexes on the standby server, because that's not possible. And as I mentioned before, if you do a drop database or alter database, if you change anything on the database, anything that's connected to that database will connect to the database on the standby server, it automatically quit. And you have to reconnect all those clients, because primary server won't allow you to do that on the standby server. And because it has to copy the same valve segment on the standby server, and it can't do that if something is connected to the database. And the statistic collector on the standby server will work the same way as it worked on the primary server. It will still keep on collecting all the stats. And you can even analyze on the database and it will work the same way as it used to work before. A little bit on monitoring. In order to check the lags between the primary and the secondary server, there are a few functions that are available. You can run the PG Last X log receive location function on the standby server. It gives you the value of location. And then you can run PG Current X log location on the master server, and get the two values, compare them, and how far are they from each other. So that will give you a lag between the two servers. A little bit about tuning how standby. So as I mentioned before, there are a few settings that you can use to resolve conflicts for how standby. But you have to be really careful when setting these values. So if you set up your vacuum deference to very high, that can create a lot of blood in the database. So if you are playing around with those settings, keep on monitoring the database server to see any bloods in the database. Because if you get a bloated database, that wouldn't really help on the standby server side. And then the max time by delay as well. When you're playing around with that setting, again, you have to be very careful because if you set up a high delay, that can create a lot of lags. And if you're using it as a failover, no, that can make it a little bit of a problem. Because if you have a big lag from the primary server, that is not really a useful standby server anymore. There's some people that have already started working on writing some very useful tools for how standby. And one of them is being written by second quadrant. Second quadrant is, again, they do PostgreSQL consulting and PostgreSQL support. And they've written a very nice tool that's called Rep Manager. And that does the whole thing that we just discovered automatically. And it has got two components. One is a command line tool to run all those commands for setting up a host standby. And the other one is a management console within the tool, which basically monitors all the lags and does all the administration work for host standby setup. So it would be a good idea if you guys can go through the tool and go through the documentation in order to get rid of all those manual commands. And it will just do the whole job automatically. You don't have to do anything of that. So future roadmap, what they've got planned for, the next features in streaming application. The next thing that is going to come out very soon is PGBS backup. I saw a few days back that it was already committed in the 9.1 branch. So it's already done. And it's already available as a dev release. If you guys have access to the CVS of postage, you can just get the latest code and try it out. It works with version 9.0 as well. I've tried it and it works sweet. That's going to be released as a stable product in version 9.1. And the next one is they're going to improve monitoring. Because right now when you execute those functions to get the lags, they give you a number that's not easy to understand. So if you're writing scripts around those values, it's really hard to do that right now. So they're coming up with proper views in the Postgres database that will tell you how much is the lag between the primary and the secondary server. And the next thing that's very important is a synchronous application. The code is already there. But it's just going through a testing phase right now to get committed in the version 9.1. And that is going to be released out at the end of this year. That's about it. So if you guys have got any questions, have we do take them? Two questions. Any plans for clustering at all? And also, any plans to integrate something like PgPool so that your read queries would automatically go to one of the sliders or whatever? Yeah, clustering. What do you mean by clustering? It is doing the whole thing right now. So what exactly do you need? Just distribute the loads on writes and things like that. I guess multi-masters. That's a very debatable topic. And right now, there's only one tool that can do the job that's PgPool. And PgPool, again, does give you a lot of trouble as well. It's not an easy to use tool. It is easy to set up, but it gives you a lot of trouble. You need to do a lot of monitoring on that tool to make it useful. But I haven't seen anything in plans yet to do anything like that in Postgres. What I usually do is I do the load balancing on the application level. Create my right object and I have a read object. And for the read objects, I can use any of my slaves. And for the right object, I only use the master server. So do that on the application level. Because PgPool gives you a lot of trouble. Like, I'm not a fan of PgPool myself. So basically, you could use something like LVS as a front end to your slave. So you could send all read queries to that virtual IP. LVS? I haven't used that. And then it's virtual server. Can it understand the data? It's basically load silencer, sorry. Can it understand what an insert is and what a select is? No, it's real simple. It's just a TCP base. Just re-badges the Ethernet header with, well, for direct routing. Re-badges the Ethernet header with the MAC address of the real server, which is behind the load balance. My question is going to be, is that going to work with Postgres in the way that we use that kind of system with MySQL at the moment? Because as you mentioned before, PgPool is more of a connection pool, which can understand the database commands. But if you're using load balancer, like you mentioned, it can't understand what an insert is and what a select is. It doesn't know where to send the insert. I'm talking about read-only. Oh, read-only? Oh, yeah, you can do that then. That's not a problem. That's not a problem at all. If you're just doing read-only, that just works. I think we'll be moving to using Postgres. Oh, that would be great. You mentioned this replication manager thing, where basically you've got this whole resource manager written around it. Why not use Pacemaker? Why not plug into something that is already there that you can just use? Why reinvent the way? Which one are you mentioning? Pacemaker, the Linux cluster stack resource manager, which happily manages replication from MySQL, by the way. And it already has Postgres integration. So how does it manage its replication? Does it use the transaction? No, it currently doesn't. It's something that I've loved to add or seen added. And I was kind of wondering what the motivation was for basically instead. The only motivation was basically people complaining that MySQL has everything out of box, and Postgres doesn't have proper application, and whatever. No, no, no, I'm not talking about implementing the replication in the database server, which is perfectly fine. Yeah, you're talking about too. But yeah, but having the cluster resource management in a separate tool rather than plugging into an existing infrastructure. That's just a tool that somebody did. It's not an official Postgres tool. It's just a tool that somebody could use. It was done by the original writer of streaming application. So he did that tool just as an add-on. So it's not a Postgres tool specific tool. Anybody can write that. Even myself, I'm writing a tool right now which is going to give you a front end, basically a GUI framework from which you can easily see your clusters and you can run all those commands. So everybody is writing tools right now to make it easy to use. So that's about it. You can write your own tool as well. So yeah. OK. Is there anything you need to do to your database when you set it up? Or can you have it running as a normal standalone thing and then at a later stage enable this? Yeah, you don't have to do anything on the database. It's not like slowning your other replication solutions, because you don't have to run those scripts to create all those triggers. And you don't have to do anything. It's just very, very simple to set up. It takes five minutes to set up the standby cluster. It's very, very simple. That was the main motivation behind having a native replication solution. Because everybody, whenever somebody looked at slowning, they're like, oh, OK, we don't want to touch it. It's very complex. We don't want to have primary keys on every table. And slowning doesn't do DDL. So whenever you want to do a DDL, it's terrible. It's just terrible. It will lock the whole thing. And then it will do a DDL. But in this case, it's just plain simple. You don't have to do that. We better make this the last question. You talk about getting the data to a standby. But what do you do when the master fails and you want to use the standby? Yeah, exactly. Very good question. So when you set up your recovery.con file, I basically didn't have that in here. But you give it a trigger file. When you set up your standby server, in the recovery file, you have a trigger underscore file setting, and you give it a file name over there. So as your standby server is in recovery mode all the time, as soon as it gets that file in that folder, it will go back into the primary mode. And it will become a primary server. But if you want to bring your failed master back into the cluster, you have to do it all over again. So that's a bit of an annoying process. And that's the same thing with slowning as well. If you do a failover in slowning, you have to rebuild the whole thing all over again. Yeah, that's a bit of an annoying thing. But I saw a few mails on the hackers list in Postgres, where they are thinking about doing something for that specific case as well, where the master can again catch up with the standby, and then you can promote it back as a master. But the tool that I mentioned, RepMGR, that can make your life easy. It can do those things automatically for you. Like once you bring that node back in, it can take those backups, and it can help you do those manual jobs automatically. Yeah. Let's thank Shob for presenting for us. Thank you, guys.