 Hello. So I'm going to talk about where I see our replication, the Postgres replication scenario being today and where I think it should go and where we know it's going and just how those play together. So just to understand where that we start from a common point of view is what is actually replication. Replication is moving the content from one system to another in the most generic sense. Having similar data or the same data on several systems and then the interesting part here is what we have right now built in in Postgres is the so-called physical replication and that works on a block level basis pretty much. It works by saying okay we have this data on disk pretty much and then we move that on the other side. That is very good, it's very efficient, has some disadvantages though. So there's a reason why other people build logical replication solutions and those just work on a content level. So if you insert the data there what it creates is usually say okay an insert happened and that is the content and so and I think the other answer here you probably know what Postgres is by now. So Postgres started when it started it didn't have any replication solution built in and then Jan, I don't know if he's in here amongst others created Sloney and that's a logical replication solution. It works by creating triggers so every time you modify a row it creates a trigger, creates a row in another queue table and that queue table then gets read by another database and then the change gets reapplied. That works great well and I think without having Sloney Postgres would not be here where we are today. I think nobody of us would be here because we couldn't afford having a conference in the Soto. So it's also the case that both Sloney and Lindy still allow you to do things that none of the solutions allow you to do. So they are still important. They also show that they have some problems which is why Postgres in version 9.0 added Streaming Replication and I think again in the case if we didn't have Streaming Replication none of us would be here because that was I think one of the linchpin moments of Postgres development and Postgres popularity and Streaming Replication is much much simpler to set up and maintain than Sloney and Lindy still it's also much more efficient but it has much much less flexibility. You can't ever create a temporary table on the standby. You can't ever have different indexes on the standby. None of that is possible ever. You always have to replicate all databases in a cluster so it definitely has some limitations. So what you also got in the last couple of years because I and others worked on it is we have in Postgres the infrastructure to build a better replication solution or a more modern replication solution. What we introduced in 9.4 is logical decoding. What that allows is to get all the changes that happen in a database in a configurable format and do something with that. One of these applications is doing replications or there is feeding it into different systems, doing cache and validation. There's many use cases but that's what we edited. Some other products call that change data capture. So and we also built a lot of other infrastructure. One of them which I think was very important step even if it wasn't actually that complicated is so-called background workers. We now have the ability inside one database to start additional tasks that do something. For example we receive changes by another system. That's why we edited it but there's many other use cases. So why do we think we need anything but what we have today? The trigger-based solutions for which Sloney and Lindista are an example have their fair share of problems. Most importantly they have a very high overhead. Both in run time every time you do an insert it calls a trigger that trigger then insert that row into a queue table and that has a certain CPU overhead. It also prevents some optimizations. For example normally if you insert data with copy it can use bulk modes. All that doesn't work anymore if you ever have triggers. So trigger they have a run time overhead. They also have a space overhead because every insert that goes into the actual table also has to be inserted into the queue table. So it then can be replicated. That means we have a right amplification of about two in many cases even worse. So that has quite the high price. The other problem is that due to issues which about ordering and so on you can never do synchronous replications with the established trigger-based replication solutions because they figure out in which order to replicate data by doing the replication like in bigger blocks than one transaction and that prohibits you from doing synchronous replication on a one transaction basis. It's also the problem that the current solutions can't replicate DDL because there's no way to just get notified about oh the schema is changing. And in my experience that's one of the most frequent issues for systems breaking down because suddenly somebody added a unique key on one side but forgot it on the other side then they fail over suddenly they have data corruption. Stuff like that I've seen that so many times now. They are also in not very easy to administer. And that's one of the bigger problems. I've seen so many down times caused by somebody not actually understanding how slowly Lundista work. It's also very hard to do multi-master because if you insert into one system it will then insert the row into the queue table it will be replicated on the other side and the other side will then again try to insert the row into the queue table and then we'll go back and forth. Then you build loops. There's a way to prevent that by having columns in there that say oh this is a row that I've been replicating but that gets quick, ugly quickly and you suddenly every user sees additional columns so it's not that easy. Another problem is that it's not built in. And I think many customers or many users there say go to the postcode website and say oh I want to do replication and they see the built in physical replication say that's okay I can live with that and then they discover oh they can't do all these nice things. They have to replicate everything and then they get unhappy and then they go to some fancy new NoSQL or whatever thing. The other problem with physical replication, the big problem is that you can't ever do selective replication. You can't ever say I only want to replicate my configuration data but all the other data is sharded and I'm only needing it on two systems for availability but everything else please don't replicate everywhere. It's not possible to do that with physical replication. Also you can't ever create a temporary table. You can't have non-replicated tables on stand-byes and that's quite the big problem. So physical replication is great for HA but after that it starts to be less useful. So what we worked on based on these feature that I mentioned earlier, logical decoding, background workers and some smaller ones is that we built the logical replication solutions using those tools. And yesterday we released the version 0.9. Unfortunately we haven't yet managed to send out the release announcement. It's a bit complicated but yeah. So and that works with stock 9.4. You can use this extension and replicate data from one system to another in one direction. That was unidirectional means, UDR, unidirectional replication. So you can replace things like stream replications, lonely, longista with that. And it will have much less overhead. How much less I'll come to in a second. And what it can do is do initialization in parallel already or what I think is already a great feature it can initialize from a physical base backup. You can safely say I'm doing a PG base backup and then catch up all the changes that happen since you made the base backup and not miss a single transaction. And that is very good if you ever want to upgrade from say 9.4 to 9.5 doing it via a logical dump of the database and then catching up all the data since that takes a long, long while because on a large database running PG dump is not exactly fast. So I think initialization of logical replication by a base backup is a very good facility. What we also have is partial replication. You can say I don't want to replicate data that's done to some table to that table because it's the lock table and I only want that locally because it's too high volume. What we don't do unfortunately in 0 of 9 is even if you say I don't replicate changes to those tables it will still initially when you create the clone it will still copy the tables. So we will have to add some more work there to not do that. It's also what it also doesn't handle. You can change which data is replicated but it will not if you say initially I'm not replicating the lock tail but then discover oh logging is really nice and I want to also replicate it because suddenly my lock analysis tool wants to look at it or whatever then we don't have a facility to automatically say reclone all the old data. You can start replicating it but not more and as I said it allows to upgrade relatively seamlessly from one version to another with like a couple like a second or three of downtime where you redirect the rights to the standby and that's I think how we are going to do much more seamless upgrades to new major versions. Obviously that doesn't help you very very much right now because this only works with 9.4 and there's no 9.5 yet so upgrading from 9.4 to anything is not that interesting yet. It's also now fully configurable via SQL and I think that's a big step. We used the first version of both BDR and UDR were configurable via postgresql.conf and it confused very many people it required restarts to reconfigure it and it was obviously not good enough. It was also much simpler but yeah. So how can you configure it now? You need to change a couple of things in postgresql.conf they are pretty much independent of which logical replication solution you use but you have to change those once. Unfortunately those all require a restart but once you've changed those you could have 10 replicas for this database and you need to allow the replica access via in pghba.conf but that's pretty normal because yeah. And after that you only need to create extension BDR and on both nodes and then you say select bdr.bdr.subscribe tell it how is my node called and to which node am I connecting to and that's also important how am I connecting to myself because unfortunately postgres doesn't have the ability to grant connection authorization tokens or something that allows you I'm allowing you to connect to me with my authentication that would be very nice but we don't have it today and that will automatically clone all the data and do that in the background and if you want to do it in the foreground you can say okay I'm waiting till it's done. And you'll notice that it says here BDR and that's because we only wanted to create one extension not one to BDR and UDR but if you say BDR.subscribe it only subscribes changes that will only work in one direction so it will pull all the changes from the other side but it will not push back any other changes. If you want to do it in both directions I'll come to that later. So and why did we actually do this as a very brief introduction so with using hot standby and we can with pgbench on a moderately good EC2 instance we can push 14,000 tps write tps per second. With UDR it's very very very slightly slower but there's barely any difference because the right overhead for pgbench type workloads is minimal there's barely any overhead. With BDR it's pretty much the same I think the difference here is mostly runtime variation. And then we also compared that with londist and you can see the overhead here already is pretty big. If you go to more systems that have more complex writes that write to more tables in one transactions that difference gets even bigger we have had a comparison where there was a factor of three or so difference. Slowny is very slightly slower here but I think that's not really measurable so that's just what you have to pay for using a trigger based replication solution. So what we next did we have a simulation where we didn't just use replication but we also wanted to measure how expensive is it to then pull out these changes and apply them on the other side. It's very nice if you can produce changes as fast as you can but the replication can't catch up. A replica that's three days old isn't very interesting. So what we did we said every time there's a lack of more than two seconds we throttled the workload to see how long can we keep up. And with hot stand by we could in that slightly different workload we had to write a custom tooling because pgbench can't do that. We could do 25 something TPS. With UDR it was slightly less. With BDR slightly more than UDR there's actually a technical reason for that. I'll come to that in a second. And if you do see the really crazy comparisons then with long distance Slowny. The overhead of pulling the data out of the log and then applying them in transforming the format and applying them on the other side is quite tremendous. And that shows here as well. What we measured here is for the pgbench workload how much right ahead log are we producing. So for 10,000 transactions we how much many megabytes of right ahead log are we producing. This is a bit badly labeled. The primary here is with hot stand by off and the stand by is with hot stand by on. We should probably label that differently but whatever. So that's about how much right ahead log we are creating. With UDR on the primary we are pretty much the same as hot stand by. There are workloads where UDR will have a noticeably higher overhead than or bigger than hot stand by but pgbench doesn't trigger those. But you notice that the stand by has produces quite a bit more right ahead log. And that's because we have we are missing one feature with it that we are proposing for a postgres 9.5 that allows you to safely say we have replicated up to here and I'm not going to lose any transactions. And that's required because UDR and BDR work on a per transaction basis to support things like synchronous commit as synchronous replication. So you can't batch transactions sanely and that's the overhead of doing that saving the replication progress in a table. So in BDR we don't have that problem because we have the feature there and that's there's somewhat more overhead on the stand by on the other side but it's pretty good if you compare it with the other side. And then you go to the trigger based replication solutions and you see they slightly write slightly slightly more. We also had buccardo here at some point in somewhere here or so. So I think that is evidence why from an efficiency point of view we you we really want something else than trigger based replication solutions and why hot standby is very very and string replication is very good for some cases. They do it in a much more coarse fashion they don't apply individual transactions and if you didn't do it on a particular transaction basis the overhead is obviously lower but then you can do syncrap pretty much fundamentally. So that's the reasoning here. So that's what we can do with 9.5 4. With what we released now you can pull it from git.postgreSQL.org I'll link the later. There's now official release docs we don't rely on the wiki anymore. It uses the postgres documentation format so it will look familiarly old style. Yeah so and but also the announcement for this talk said we were talking about multi master because what we that's what where we how we got started how this helped down this whole path. What we have now is with BDR we have a working asynchronous multi master solution and I'll mention that it's in production today there are still sometimes problems but it generally seems to work reasonably well. It basically uses all the code that is used for UDR that works with 9.4 and then adds additional capabilities because with unidirectional replication obviously that's not multi master so we added to add more feature and one of the features that I think is the most coolest is if you execute on one side ddl it will automatically replicate to the other side. There's some caveats that I will mention in a second but yeah it's we add customers that said please can you somehow magically transport this to 9.4 because we really really want this and I think it's too much work so we said no. What you also did besides just the asynchronous replication and multi master replication is to allow sequences to work across multiple systems because it's there's various ways to assign unique IDs across systems you can use UIDs you can say you can assign every odd values on this system and every even values on this system but it gets relatively hairy and it's not very user friendly so what we added is when you say create sequence you can now say create sequence using bdr and then it uses a voting approach that says this server gets the values from 0 to 1000 this way server gets the values from 1000 to 1 to 2000 and so on and there's some caching to avoid short down times to cause problems and similar things. What you also added is conflict resolution mechanisms because if you do asynchronous multi master across that is you can have conflicts because two systems can do the same change so what we have by default is last update wins but you can add different conflict resolution strategies by saying calling a function set that registers a conflict handler and that conflict handler then gets both sides of that change and can decide what we want what we want to do with that so and why did we actually go to all that trouble this is several person years of work here it's very hard to do geographically distributed setups where without replicating data to the other sides because latencies of 150 or so milliseconds are really really painful to work with it everything calls slows down to a call so and those are becoming more and more common so I think that's the prime reason to do asynchronous multi master because with synchronous multi master you have latency problems without without replication you have also latency problems because you're all every white right has to go to the other side so they're pretty much if you want to have shared state there's relatively little choice to then to go there it's also very useful to scale read mostly applications without saying I'm separating read and write if your application doesn't write that much data and you can somehow ensure that the rights aren't conflicting very often that can give you quite easy scalability boost even if you're not to graphically distribute if you're in the same data center it's also very useful to do very easy failover failure bug I don't know who of you has implemented procedures for failing over of a hot standby node then bring up the old master as a new standby that's not all that simple so it can be very much easier to say I'm just redirecting all the rights to the other side waiting for catch up and then say okay and fall have finished falling over what we also have customers wanting very much is that they want to have some shared state between clusters of but where they like the configuration data has been there the you the authentication in for is there but all the rest of the data is sharded so they want to not have the chat messages or whatever on all nodes but they want to have to know kind of send a message to that user just that user exists they want that information almost and that's in my opinion quite large use case for a wider range of applications so what can these setups look like what you can have is have say a one data center or two data centers in Europe they are fully interconnected between those two and you can say my data is important I want to do synchronous replication between these European data centers they are in the same city just state different data centers to not avoid to avoid losing data I want to have my important transactions to be replicated synchronously but you can't do that across the pond because that's just too slow so you have two another database servers data centers in of America for example and they do the same spiel they replicate synchronously between them and but other to the other side of the pond they do a synchronous replication as you can see here to set up BDR currently you have to every node has to connect with each other and that's because otherwise it's much more complicated if one node goes down you have to reroute traffic around that it's just simply to implement that way there's no fundamental assumption that we are always are only going to do this it's just what we have chosen as the first step so to go back to what was saying about transparency the other application normally you can just say I'm now executing DL I type my create table whatever and it creates the table replicates it to the other side and executes the DL again that works great for create but there's some commands we can't replicate at all because commands like create database alter user or create user and similar in Postgres they're shared between different databases in one cluster and since BDR and generally logical replication usually works on per database level it's unclear what does it mean that in a different database you execute a create user you would have to know that in some other database in that server there's a logical replication configured and you would somehow need to replicate it then so that's why these commands are not replicated there's also some commands that we currently prohibit from actually being executed you will just get an error which is not so nice but yeah better than corrupt data you can't at the moment change the type of columns when you specify using a clause or if you have generally if the alter table requires a table rewrite we don't allow it there's also a good reason why you don't ever want to execute these in your actual application because those will cause a table rewrite and that takes long long time the all the commands that just add a column without any rewrites that those are completely fine or dropping a column that all works just everything that slow is not allowed if you grant and revoke that's actually allowed as long as it's doing it to a local table so you can say grant select on table and that works what you're not allowed to do or what's not replicate is grant connect on database because again global state so when is asynchronous multi master and specifically bdr a good solution if you only have if you have data where for example every user is only allowed from to connect from their home country or is automatically redirected then you can't ever have conflicts and then the it's pretty easy to cope with the possibility that you have data that's not completely in sync the other thing is that you just write your application and you're aware that there can be inconsistencies for a while other workloads where you just can't have conflicts if you have only inserts and you don't have any unique constraints that will fail then it's just very easy to implement this just generally if you're geographically distributed especially over long distances this asynchronous multi master might be a good fit what you when you don't want to use it is if you have something where consistency is really really important it can never ever be violated because you're dealing with whatever medical data for example and you don't want to lose a patient record because there was a conflict and somebody said oh this patient is ill and somebody else otherwise accidentally overwrote that with this patient is actually a healthy please send him home that would not be good so last update wins might not be the best idea for to resolve conflicts by two doctors making through the same patient it's also much harder to validate that all your applications are doing something correct if you have 10 applications written by 10 development teams three of them don't even exist two of them have died since then it's very hard to guarantee that you can actually cope with possibility of conflicts it's also very very important to note that asynchronous multi-master does not guarantee unlimited right scalability to forever you can have some benefits by doing you gain can gain some right scalability but it's usually not super drastic and it very very quickly tapers off so as I said for conflict handling we have the built in last update wins we use the transaction timestamp when it committed as to resolve that by default and you can add newer conflict handlers it's not that interesting I think unless you really started to look into this so it's a bit complicated how all these fit together at the moment we have Postgres and that has built in logical decoding it has built in background workers and in 905 you already added some more features and then on top of that there's the UDR project that's open source you can download it there's Debian packages there's RPM packages for Fedora Red Hat Centos I think and that you builds logical replication but only unidirectional it allows you to upgrade in in the near future to new versions with a very short downtime and it supports 9.4 which is important then on top of that we have a slightly modified version of Postgres and we modified that because we needed the additional capabilities we have a couple features in there where we had to modify things like for example the DDUL replication required us to get the DDUL statement back in some form that we can replicate we have the ability to create these new types of sequences where we can say this sequence is a distributed sequence we have the commit timestamps with which since have been committed to 905 but if you want to use 904 because you don't want the most bleeding edge in production version in production we have that as a modification and then we have this replication origin progress tracking which is why you saw in UDR earlier that it wrote more than the primary then BDR did and using these modifications we have a version of the extension that's the same source code as the UDR that uses these additional capabilities to implement multi master to implement these distributed sequences to implement DDUL replications and I will to emphasize that all of these features with the except this one have been submitted to the PostgreSQL mailing list we've worked on several of them through many iterations trying to make them committable and so it's not like these modifications are something secret we are that we are keeping it's just we're trying to develop all this and it takes a long long time and PostgreSQL takes a long it has a year-long release cycle but if you develop something new it takes a while a year-long release cycle is not very attractive because the feedback you want to get it out to your users you want feedback from users you want to have product tests where you can actually use this in a realistic scenario because then you know you screwed up and what your user interface didn't work out that great for example we had this configuration format where it was in PostgreSQL.conf and that really didn't work out great in the customers who already tried it it was confusing so we had to try something else which we now don't and I think for these iteration for these relatively early development where it's very useful to have something that develops faster than Postgres it's it's very good that Postgres doesn't release every week I think you all would be not very happy about this so there's good reasons for Postgres developing the way it does but it also who has some problems because it takes a long while to get things in yeah that was what I was saying so and these what we what Simon came up with as a phrase we're not forking Postgres with the BDR special sauce but that really really follows Postgres pretty closely we are submitting everything upstream we hope that this difference really really vanishes soon are you alive so down there it's also very important that we are every time Postgres is going to release a point release we are also going to have a point release and we are pretty much merging the changes that are happening upstream pretty frequently so where are we now with regards to core Postgres we have lots of infrastructure we have logical decoding in core we have background workers in core we have now commits timestamps in core we have a relatively except some conflicts here with Robert and Hakey have a relatively sane version of the replication progress patch it even has docs we have sequence AM and Hakey was actually somewhat happy with the last version which is nice we have the ability to develop these event triggers that do give you this sane version of the DDL that you then can replicate and this is not specific to Postgres at all you can also integrate that into other replication solutions you can integrate it into your development workload you can automatically generate the evolve script for whatever database schema tracking you want and for auditing and similar those have all been submitted we hope to get them committed relatively soon the only one we didn't we didn't manage to get very far is the valve based queuing solution that's just because it depends on the replication identifiers so it was there was not that much to do before that was in but but but the big problem is we have don't have a built in logical replication solution it's very hard to sell to people that you have to go to some website you have to install this you have to install a different repository then you have to download it then you have to read different documentation it's very complicated and that if you have already a big Postgres shop you're probably probably willing to do this but if you are new to this and are choosing which database you are going to use it's not going to inspire that very much confidence that you have to go to some random other website and download another project so that's really not enough we need to have something that integrates much much more seamlessly but you might think why on earth did you develop something separately if you want to commit it to Postgres and the reason for that is really that it's very hard to develop stuff in Postgres Postgres has very high quality standards it has a very debate enthusiastic community and let's go that's good most of the time but it makes it very hard to prototype a solution where you know I'm going to make compromise because I can't develop the best solution on earth within a couple months so I think it's very important to allow for some of that stuff to happen outside and you might ask yeah sure you can do it outside you can just work on your patch for three years but a users don't really like not to have this capability they're happier to have something outside of core that works also we actually need to get paid for developing this and our users that pay us for this are not very happy to wait several years or they actually have worked a couple years but even more years it's also very hard to have something that actually is nice if you don't have users testing the development version so where can we where do I think we go for the incorological replication how can we make this happen I think we have to accept that we are not going to integrate BDR as is into Postgres for A there's lots of compromise in there core Postgres might not accept it's also just honking a lot of complexity that you can just absorb immediately so we're going to try to add the unidirectional replication first and but design it in a way that we based on our experience can then extend to multi master and how long that takes we'll see I think it's also important to note that we realize that it won't exactly you look like UDR for UDR we had very different constraints than core Postgres has we couldn't just say oh we want this new statement in Postgres we didn't want we couldn't say we have a new create replica command or create primary server whatever command because we can't we didn't want to modify Postgres so that might be different for the integrated version but I think we can reuse some of the code and very very much some of the experience and some of the infrastructure we built to get there for example the logical decoding API hopefully doesn't need any changes the commit timestamps when won't any any changes the background worker API won't need any changes and so on and while we are working on integrating this we also working on have will continue to provide UDR and BDR because I think Postgres can't really survive we have customers that need it and I think those aren't the only ones so I think that's just saying and we'll try to maintain those but at some point for example we are not going to need UDR anymore so we are going to retire it but we'll still need BDR for a while but at some point I hope we come to the point we say okay now it's so so major the this stuff that is in core Postgres we don't need an external version anymore those features aren't that urgent anymore that you need a pre-cooked version for immediate use but until then we are going to maintain those and we are also going to work on new features in BDR and UDR just to extend Postgres later with them and to get experience with them and also to just make a life for users of Postgres easier and here are the features that we are at the moment thinking about working on or are already working on one of the big ones is I think partial clones that you can clone a database and it will only have the tables that you want the replication the want to replicate changes to we need to make a failover to the other side a bit easier there's quite a bit of manual work involved there most importantly you need to have one manual action afterwards to bring up the sequences to the highest value that's not so nice so I think we need to make that easier probably not too hard the the way you configure replication sets which is our solution to saying only these changes are needed on the other node I think we need some more improvement if you configure if the for example right now if you say I want this node replicates all the changes from the other node and you change that configuration we are going to automatically need to pull down all the missing tables delete all the other tables and also you don't shouldn't need to kill have to kill some node or restart to do that yeah that's actually already there for also what we want to do as soon as possible is to move the ability to DDL replication from BDR into UDR so because there's no nothing multi-master specific about doing DDL replication right now if you want to do DDL it will tell you you can't do that please use use this function and then you have a function that does the DDL and replicates it that or you can say I know what I'm doing let me do this for BDR we want to lift some of these restrictions I was saying earlier that you can change the type of a column if that requires a table to rewrite we want to allow that right now you can't do create table as whatever we want to allow that stuff like that what also we want to do and I think that's I personally find that a pretty cool feature we want to have transactions where you say this is a really really important transactions I can't allow any conflicts here and then we can guarantee by using 2PC internally this transaction is going to be replicated correctly to all nodes and I'm only going to return a successful commit if it actually worked and I think that allows you to design systems where the relatively low amount of actually important rights always succeeds but the rest is fast because obviously using these conflict free transactions will be far from free they will have to contact all the other nodes synchronously so you surely don't want to use that for everything we also want to make distributed sequences a bit easier to use right now you can configure how many of them are pre-allocated and things like that also if you will get an error instead of blocking if there are no new values at the moment available and just some usability enhancements so a rather important question is where can you get this stuff it's on bdrproject.org the docs are under bdrproject.org slash docs and there you can see the different versions and Bux please report them to GitHub but please also try to give a sane report of this we have had screenshots sent to us and yeah you know there's a reason sometimes it's better to develop outside of law if you have questions about it to us there's a list at second quarter to come where it can reach the people that are familiar with bdr but if you have other questions you can just send an email to general and hopefully someone will pick up there and it will be archived and other people can chime in and it'll help other people to see what you're doing and not doing we already had people helping other people so so that's good yeah if you want to help you can read the docs and tell us whether you understand most of us are so neck deep in this that they really don't understand the problem anymore it's obvious using it would be nice too reporting problems and so on it's also another thing is to please help reviewing patches I think we're why are you grinning this is lots and lots of work several people here in the room have helped this and I'm very grateful for that but I think we all need more review and you don't really don't need to be the most biggest Postgres expert to help here every pretty much everyone can start doing review and you learn quite a bit of Postgres and the knowledge you have will help you at some point in your daily operating life it's not something that is only applicable to development the knowledge there helps in all aspects of Postgres it's also quite possible to add features yourself it's Postgres licensed so it's open source we'll try to integrate it if it's good if it's not not yeah so any questions we have on my laptop and my workstation I've tested that's what we have on the website 48 nodes and that that and since then I've tested something with 60 something at that point you started getting into problems because you have a lot of lot of processes running I think I was had to configure my machine because it's like 10,000 processes at that point if you do something with it you won't probably should test that with more machines than just one workstation there's actually an option in the configuration file where it artificially introduce latency we can say br.applyDelay equals whatever I think and then it will just delay committing which is very useful during development I think we tested with a couple seconds there I don't know the exact number yes I have I'm not I think there's a good argument to be made to say we are going to trying to continue using functions I think it would not be with the worst idea I think some other people are not so much on board with that I think we don't definitely not wanting to have it in a BDR schema but yeah on the other hand it's not very it would not be absurd to try to reuse the create server and similar command and then subscribe to server because we have this nice infrastructure for the foreign data wrappers my problem with that is that it gets slightly murkier if you want are doing multi master because you if you have if you want to build the mesh connections then you create servers dynamically on all the other nodes I'm not sure I think there's very different strategies I know that's one of the reasons why it would be a good idea to say we reuse some at least the table I don't know how the table is called right now to have the common knowledge which what is replicated where that might not require using the same syntax to create them because there's no need for that necessarily except especially for the multi master case where it's probably most useful to have be able to query all the other nodes I think there's a lot of discussions we can have I think I want to get these dependent patches in and I think at that point we need to have some long emails yes vigorous discussions vigorous agreements yeah anything else it definitely is possible so the logical decoding in combination with that replication identifier and progress tracking has all the information to make that possible where I'm not really I don't have really many doubts about how to implement the technical part what I find very complicated is about how to figure out how to configure this how to make sure that you don't have nodes that can see any some changes that we'll get is not terrible and that's the part so the user interface of this is what worries me and where I have not yet found something that oh yeah this is obviously right so but it's not that hard from the technical point of view you can basically do it right now if you want by just not using the interface functions and saying I manipulate the internal configuration directly you can build these kind of setups already it's just not something I would suggest doing you'll definitely get to keep the pieces yeah I think we I think it depends on which shared object you talk about actually I think we're not going to want to do stuff like create table states replicating I think that's pretty boring and it doesn't really make sense because it assumes all the systems are the same and I think the biggest problems is probably users from a practical point of view because to create database why would you want to replicate it around databases one thing that is pretty interesting is to replicate alter database set so and so because that is actually something that's pretty clearly what you know understandable what you mean and we could just say we refuse alter database set when BDR is configured for any database but the one you're have configured it from or that would not be the worst rule out there but yeah roles are pretty much harder I've not yet figured out a good way to do that I think the capturing the command is not that hard we just would have a shared queue for those but what if you want to replicate from different systems and merge and then it gets very murky the other opportunity would be to say we introduce database local roles I'm not a big fan of that but yeah but I'm out of time so I won't sell for the problem today