 So, welcome Gülçin Dilerman for the next talk in the Post-Cross-Cross session for today, and it's about the evolution of fault tolerance in Post-Cross-Querl. Hello, welcome. Firstly, I want to say sorry, because I'm a bit sick and my throat is a bit hurting. Hopefully you will hear me. Do you hear me now? Yes. Okay. Thank you. And as Christoph mentioned, I will talk about evolution of fault tolerance in Post-Cross-Querl. I want to picture how we evolved in terms of dependability and fault tolerance by describing the features, the mechanisms and all the features that is supporting and closely tightly connected with fault tolerance. So, who am I? I am working for Second Quadrant on Post-Cross and Automation, and I also help for organizing Post-Cross events and do some talks. And also, I'm doing my masters. Hopefully, I will start writing my thesis soon. And that's it. You can find me from Github and Twitter, and you can add me from Skype if you want. Agenda that is, first, we will talk about Post-Cross-Cross database in general, but just a few, I mean. And then, main mechanism for fault tolerance is VOL. Then we will try to describe what is transaction, what is commit, and what is checkpoint, what do we have for replication in Post-Cross-Querl, and what do we expect in Post-Cross-Tan, as well, and how the replication evolved from physical to synchronous replication and then logical. And in the meantime, we will also cover timeline issues by failover, switchover, and PG1 too. So, first things are, first, Post-Cross is robust, and why it's all nature? It is already having asset compliance, and we literally follow SQL compliance for SQL 2011, I guess. We try to keep the standards as standards. We don't want to change them so much. And also, transactions are the basic for all relational databases, but this is important for Post-Cross. So even if you don't run in transactions, Post-Cross-Cross still treats every statement as transactions. The thing that is transaction log, which we will talk about a lot in the talk, is wall. We save all the statements in the record in wall files, and then use them for application, for archiving, for all kind of reasons. And we have for detecting hardware defaults, you can use data block checksums, and there are a variety of diagnostic tools and lots of backup solutions. So you can be sure that if you configure Post-Cross-Querl properly and use all the environmental tools, you won't be facing with disastrous things in Post-Cross-Querl. And this is very important. The last point, it has greater than five nights, which means in fault tolerance is almost not achievable in almost all systems. So this is quite a good number for a synchronous replication, which we will talk if you configure and tune properly. There are a variety of levels of synchronization as well. We will talk in the talk. So what is wall? The wall consists of series of binary log files, and these are all under data directory of Post-Cross-Querl. It's in pgxlog subdirectory. So all the changes that you do, you run some statements like insert, whatever, some command. They are all recorded in this data directory, data file, sorry. And just not only these ones also indexes or when you run a vacuum, so all your changes are recorded in this transaction log, which is very important because when you want to recover from a crash or want to do point-in-time recovery, you will need these logs to know what happened, what changed. So this is your kind of diary of Post-Cross. And the thing that you should know when a transaction is committed, it is not directly to return to disk. So you should know that you can lose some data because there are some time between the commit and the disk commit goes to the disk. So we will talk about this as well. So when we say transaction, what we understand? There is this begin and commit that we all know. It is bundled. There are intermediate steps that you run your queries. And when you commit it, it is like we call this transaction. So as I said before, in Post-Cross everything is considered as transaction. Even if you don't write transaction as begin and commit, it is still is. There are isolations levels, four of them from SQL standard is coming. The important thing is here that we should know that Post-Cross doesn't support dirt to read. What is dirt to read? When I said that there are these begin and commit and there are intermediate steps of that we don't in Post-Cross, the other concurrent transaction doesn't see these steps between. So let's say I updated something in the between, but other transactions doing something with that table, they are not affected by these changes. Because if there is some failure, it won't affect, I mean, we will roll back and these changes not affect the other concurrent transactions. But there are some systems that dirt to reads that reading this intermediate steps and which is not very good for you and for your applications. So Post-Cross doesn't support dirt to read. I don't know how to point this. Yeah, dirt to read. And the most heavy one is serializable. It is very hard to achieve, actually. So these are the transaction isolation levels in Post-Cross. Okay. So what is checkpoint? We said that when there's a crash, we recover by replaying the walls that the logs that we wrote before. So how do we understand at which point we should replay these logs? There should be a point, that safe point, that we know that before that point, everything is written to disk. Because as I said, when you commit a transaction, it is not immediately flushed to disk. There is a time. So for checkpoint, in terms of checkpoint, that is the point that we can be sure. Let's say you had an issue or have a failure and fail. Then there is one checkpoint. You can be sure that before that point, everything is flushed to the disk. So checkpoints are very important for that reason. You should tune your checkpoint settings. There are lots of variables in Post-Cross Qualcomm. It shouldn't be too frequent, or the time is important because if you have a failure and you put a checkpoint some very long time ago, then you need to replay all the changes between the written into walls. So it should be, I mean, depending on your system, your load, your changes, and so that you should be sure that it will be like, I mean, ever it's the normal. It can be an immediate or scheduled checkpoint. Immediate means that you can write your command, for example, like checkpoint to command, and then you can immediately run this checkpoint. Or it can be scheduled, which means that you can also tune your Post-Cross Qualcomm, and it is scheduled in some time, or it's like, how do you configure? And it is decided by Post-Cross Qualcomm. So we talk about wall. That was the files that we write in binary form. The changes are written to the files. In replication, what we do is, there is one, like in classic replication, there is one master and one standby, and there are changes captured on master, and then sent to standby over to our network. So here, what are we doing? There's master capturing the changes and writing to wall files, and then sending to the standby, and it is replayed, and then now we have a maintainable copy of our system. That is basically replication, how it works. A copy of your system, basically your master system, and you have a copy on a remote server. That is generally Post-Cross Qualcomm. Post-Cross Qualcomm replication, we will go in details, but replication is this. And also in modern systems, reliability is important, and for redundancy reasons, we need to have replicas, right? So I don't want to go in details more here. For an overview, we need to understand how we started. Like things are not like this all the time, right? There was a history and loss of work behind it, loss of commuters and loss of people contributing to this. But in the very beginning, I mean around Post-Cross Qualcomm 7, 2000 and 2005, there was a philosophy in the in Post-Cross Qualcomm community that it shouldn't be, replication shouldn't be part of Post-Cross. And for that reason, there was Lone Distance Lone, trigger-based solutions, because if you don't plan to do it in core, there will be some tools all around. What they were doing, we will talk about trigger-based replication, but we can consider them as logical, because they were not sending changes as physically, but there are triggers and tables so it is considered as like based on logical replication, trigger-based replication. And then at some point, I mean at the beginning, we all were like, Post-Cross Qualcomm was focusing on single nodes fault tolerance. So if something happens, you can replay your role and turn to a point that is safe for you. That was the initial fault tolerance in Post-Cross. And then by the time we get to replication, it was considered that for you, if you have a cluster for with more than one node, then replication is best for you. But then, how we solved the replication, I mean how do we started to do replication? As I said before, we took the wall and sent over network and we have Post-Cross Graph replication point four, which is relatively new. We have logical decoding, which allows us to do logical replication. And in Post-Cross 10, it will be coming in this year. We will have logical streaming replication. And we will talk about the details a bit more. So that is the general view, how we were doing it, how state of the art is it is still streaming replication. But now we will have logical streaming replication and also we have logical decoding so that we can use this as well. I will show how do we use logical decoding. So we are not restricted only for streaming replication. Physical replication. The picture is showing what I am talking. So here we have master and client runs the queries here and then it is written to wall and then we will send over network to the standby and the recovery process on standby is replay the wall and we have standby basically. And when we say physical replication, we should thought that we are sending these files in some, it's using some protocols like SecureCopy, TCP, R-Sync. And when we say streaming, when we are streaming here, what do we mean that we have internal protocols for sending and receiving these files which are like sender and receiver processes and wall internal protocol. Okay, when we consider that there is a replication, we assume that there is a way of doing a standby. Not everyone is needing to use a hot standby. So there is in PostgreSQL, we can have warm standby and hot standby. So what is the difference between warm and hot? Consider that we have a base backup, that we have master and base backup and we are feeding this base backup with wall files all the time, then you have a replica. It is warm. If it's not active, I mean it is ready but it will be activated if it's needed, this is warm. But if it's already in the system and you are using it for read-only queries and it is allowing to query read-only while it is in archived recovery mode and file archiving mode, this is hot standby. The important thing is here, it we can promote it without interruption so your queries will continue to work and here you don't have this chance because you need to wait for it to be activated. I think this is clear. While I was thinking about there is a big relation with wall and the replication, I realized that there is one parameter, it shows it really, you can sum up really easily. That is wall level. Wall level is the parameter that it determines how much information you will store for having, for using your standby, for which reasons? How do you want to use your replication? What is your main purpose to do for doing replication? The default and minimum one, obviously, is minimal. This is only for that in case of crash, you can recover with this level of information in your wall. Here, replica, this is slightest for 9.6 before in the releases prior to 9.6, it was hot standby and archive but they are mapped to replica now but you can still use them. What does it does is you can do file-based archiving, you can run your read-only queries so that you have enough information about having this kind of replication. But then recently, as we talk that there is logical decoding now, so if you specify logical, then you are allowing to use logical decoding and then all the features comes with that. So it allows to, it continues to look more to have logical replication in this level. And all the levels also has this previous levels information. So if you have logical in your wall level, then you have this and this and if you are replica, you have minimal and so on. So this is, I think, very important and you can just see easily what they do and what is suitable for. Let's talk about failover and switchover a bit. As you can see in switchover, you have the control, failover, sadly you don't have. When your master dies and you need to recover it so that you can accept the rights, otherwise you can't. If you do it by control, basically stop your master and promote the standby, this is switchover. And this, you need to do something more. You need to do base backup or copying the files and I will show that how we will fix failover station better in a better way with PG rewinds. So this picture you should remember if you want to know the difference between the terms, switchover, you have the control, failover, you don't. Timelines, I think timelines in PostgreSQL is a really important term to understand what is happening, actually. Why do we need this term and why do we use it? In every recovery process, after every recovery process, PostgreSQL issues a new timeline so that you can know that the new timeline is not like overwriting the whole files that is written by the initial previous timeline. So that you need to diverge these timelines to overcome like switchover, failover, crashes, point-in-time recovery. So you need to be aware that we need different timelines to control this. And we will see this in details. So as you can see in case of failover, this is our master's timeline, the first initial timeline. And this timeline is increases here. So I assume that this failover is happening around here and your master, these changes are not flushed to disk yet. So this standby supposed to be new master doesn't have these changes. So in this scenario, you will lose all these changes. You can't easily, the old master can't follow the new master because there's no way to send this to this. So what you need to do is like you promote and then do base backup and then continue because there are outstanding changes in the world and this timeline changes is showing that there is something messy, I mean, there is a failure. So it can't be replayed. But in case of switchover, as you remember, here is the case that we don't have the control. But in switchover, we have the control. So there are no outstanding changes. Basically, we stop this and then we promote to new standby. But like I say, there's a new timeline issued here and easily the old master can become new master. I'm sorry, the old master can become standby to follow the new master, which is the standby, like usual promotion without any loss. And what does PG rewind is? Here, we are kind of, this tool is helping us after 9.5. There are still outstanding changes. It is more useful in the case of failure, for example. What does it do? It first check the wall files in the old master and the advantage of using PG rewind and comparing to PG base backup or some other copying method is in the usual way you need to copy everything even if they are not, even if they didn't change. So PG rewind only looks for the changed ones. So let's say you have a very big database and you have like proportionally small changes. So you don't need to go through all the unchanged files but this just checks and finds the changed files. So then you will have time. You will still lose this data. I will explain why you still lose this part. But at least your old master can follow the new master so that you can do it in online. You don't have this failure situation now. So what was it doing? It was checking the wall files in the old and then it was checking from the new standby. I mean, the old standby but the new master. It is checking to go into a checkpoint which I want to remind you it was checkpoint, was safe point. So like let's say there is this problem happens here but you have checkpoint. You turn from this, you come to this checkpoint by just looking to this one, supposed to be new master and now it's standby. You find this point and send, copy all the files from this one to this one so that you kind of rewind the things and copy all the C-LOG files, configuration files but not relation files. And then these walls are replaced by wall. This master, which is not really the tailor replace but let's not go there. So that the important point is you can just continue. PG rewind is helped. So this is another improvement. I mean, before there was not this opportunity now we figure out a way to overcome that. Synchronous replication. Let's talk about this a bit. That this one was greater than five nines available tenfold tolerance that I was talking. By default, Postgres do asynchronous replication which means that we replicate whenever the standby is available or how do we configure and so on. But we can force Postgres to do synchronous replication as well by configuring the synchronous commit parameter. Here, as you can see that you can force that at least your data is written to at least two nodes before you sense that it's okay, it's committed message to your application or your client. So you can be sure that it is safer but when I say that what we lose here we can lose a bit of performance because I don't accept anything before I got the message that yes, it is safe. It is the place that where you want it to be. So then you need to wait for that. And another thing that maybe we can say that when you said synchronous replication it doesn't mean that you only need to use synchronous replication but it is for transaction base. So you can set some transactions for synchronous, some of them asynchronous. So it is up to you. So that for some certain things you want to keep them very well and secure so that you can set for that transactions that's like synchronous but some of them it can be still asynchronous. You can just, you can decide yourself so that this trade of between performance and durability is up to you. Just a bit. So which parameter is deciding this? A synchronous commit. We will come to that but before we go to synchronous commit and how it applies synchronization in Postgres scale I want to remind you how commit is processed in replication in general in Postgres scale. We talk about it but this picture is helpful that and I will use this template to explain synchronous commit as well. So here this is our master let's say and this is our standby. We can think as upstream downstream doesn't matter. So here when a client execute the commit and this is written to wall, wall files and then wall sender is sending this to over network to and this wall receiver takes it and it's written to standby as wall and it's replayed and that's basically then your commit is following this pet basic. Okay. I think this template is helpful for basic idea and let's talk about the synchronous commit. So here this parameter is deciding if you will have synchronous commit synchronous replication or not. As you see, if it is off we don't wait for, we don't pay for the transaction record to be flush to disk. It's off and the second is a bit higher. This will be go to higher level of synchronous applications. So the minimum was off. It was closed and now it is local but does it mean that we can be sure that it is in local disk of the machine? So it's written here but this level is stopping here. So when I say, when I say remote write that I can be, I can be sure that I send this to the remote node which is standby but it is not flush to disk. It is in memory operations here. It is not here but at least it is better than I mean the level of off and the level of local. It go on there and then say that okay it sent a message that is here. So this is remote write. The naming is a bit different because when I heard remote write I was thinking it's written to disk but it's not. On is the basic idea. Synchronous commit is on which means that we do the synchronization by we sent the changes now from this level and flush to disk here we are now. And the extreme level I mean the tougher level is, top level is remote apply that I push these changes that I recorded these changes are visible on the database now so it is replayed. So basically the transaction is waiting for this happens. This event happens. So you can be very sure that it is even visible. This change is visible before that is turning to say message that is okay that I did it. Okay. So this part that I think it's really important. That's why I was playing. I want to show that isn't graphical way so that we can be following the comments easily. Okay. So now we can talk about what we talk. First we started with wall mechanism and then we say that how it is like how it is used in replication, physical replication. In physical replication you should think that it is raw and block by level. So all the changes needs to be replayed from the other side. So it is a bit strict. You need to replay everything. You can't have differences between master and standby. But when we say logical application you can just put this keyword in your mind. It is more flexible. You can apply this to more complex topologies. You don't need to have the same major versions. And this is also an advantage that you can do. You can by using logical application you can upgrade between major releases and so on. So logical replication I think is the future of the full tolerance and dependability in Postgres. But for some cases not for all. Of course physical streaming replication is still important and it should be used for lots of cases as well. So that was the general intro for logical replication. Why I, there is two major approaches for logical replication. The one that we talk a bit in the history that trigger-based replication and then we will talk about logical encoding and it is known as change set extraction in some other systems. So what do we do in trigger-based replication here? All the tables are, it is controlled by triggers. So all the changes are written to tables. You amplify the writes. So you write double times, the changes are written double times and then also it amplifies the wall of course because then you have double amount of writes and double amount of wall. That is the K-Watt in this trigger-based replications. And but it is of course predating the now logical replication even the physical replication. But this is the K-Watt that you need to remember. It is like, it's okay, it is working. So on this dance, Loni is doing this for years. But it has some disadvantages. Increasing the amount of work needed to be done and like increasing the amount of the size of wall and so on. Here, that's the main part that I want to show and it will be in Postgres 10 as well. The talk will continue. Logical decoding, which is since 9.4. What does it do? Remember that there was a table that we had wall levels and there was logical level. So when you use this information, what do you do? You extract the information from wall that you can use this information for into logical changes. Now you don't think about that is like block and byte level but you have logical changes in you now and you can use it more for selective replication. You can just use it for some other topologies and there is no right implication. This is referring to what? To trigger-based replication because then you need to have everything double times. You need to write double times. So here, this is the advantage of logical decoding. There is no right implication and there is another thing. When you use trigger-based replications, they are like using triggers as in the name but the ordering is also another issue. You need to order the things that you copied. I mean, you write. But here, since we are using a wall protocol, wall protocols, there is already in order. So your changes are in order. So you don't need to think about it in orderly. It is per-per-row. So let's say you write an update for 100 rows. There will be 100 changes. And it doesn't decode DDL. And there is an SQL interface which means you have functions and it is using wall sender streaming interface. It also cares committed transactions which means if you abort your transaction, this is not considered in logical replication. So based on logical decoding, this is very important because whatever I say here, it will be applied to logical replication and the extensions based on logical decoding that I will show you. So here, what I can say that is, here as you can see in the physical streaming replication, we were again using this wall sender and wall API processes. We still use the same protocol but we do logical replication now. And it is currently available as an extensions, PG logical and Postgres BDR. We will have a small talk about them as well. And obviously it is better performance than trigger-based replications which is the reasons that I listed before. And it is built on top of logical decoding. So that is the view. And it is also allowing a synchronous commit. Why? Because we still have this same protocol for wall answer. So you can still do synchronous replication. So what is PG logical? Since we have the logical decoding in Postgres now, we can have extensions and can do logical replication. This is an extension for logical streaming replication and it is available after logical decoding, basically 9.4. And it is optional synchronous. Why? Because logical decoding is allowing us in the previous slide we saw it. We configure it via functions and that is you can do row filtering and column filtering. That is the thing. You can do selective replication. Now your limitations are less. You are more flexible than the physical replication. And you can use it for many reasons for your BI's and you can have different data sets in different platforms and everything. So it is better in that terms, in these use cases. And the thing is that you can do major release upgrades because then in the physical replication we are limited to use same version. But here we are not limited to that. We can use from 9.4 to 9.6, for example, for upgrades. So this is physiological. This is the good news, good slide. Postgres固al 10, in Postgres固al 10 we will have basic built in logical replication in core natively and of course it is built on logical decoding. Again, you see the same sentence which is optional synchronous because what? We have this now. You can do optionally synchronous. And the difference is from physiological and Postgres 10. In physiological we have functions to configure the physiological and logical replication. But here it's in core. You can use DDL, so you will create publication and subscription. There will be publisher and subscriber notes. And it is based on physiological. So it implements some features of physiological but it will improve by time because there are like lots of actually stuff that we waited to be in core. And differently implementations of course but same logic applies. So Postgres固al 10 and Postgres PDR. This project was used for, is still is used for feeding logical replication development in Postgres固al. And this is multi-master. When we talk about replication there is also a difference between like multi-master if you have more than one master it's obviously multi-master system and then the traditional replication is like master standby. So BDR allows you to have multi-master and asynchronous. That is important. And the main aim was for BDR project that you can have globally distributed cluster. So you can actually distribute your load in your local systems. Let's say you have two server in America, two server in some other country, two server in Ireland. Let's say if you distribute the load in the masters because it is asynchronous so you don't need to wait for writing, you write and then you divide the load for locally and then they will get synchronous eventually. So it is eventually consistent cluster-wise so that you can have globally distributed clusters. This is all thanks to the logical decoding and the projects behind. And you can think that I mean if I write the same table if some other people write the same table what will happen in all cases of multi-master solutions conflict? So there are lots of ways of handling these conflicts and in case of BDR it is optimistic. This means that after commit it will be settled, this conflict detection and then automatically this conflict resolution will happen. I think for BDR it is last committed one wins. So this is important when we say asynchronous doesn't prevent concurrent rise. That was the reason actually. If we prevent the concurrent rise we can't have this locally distributed latency, right? We are trying to solve writes locally so that we don't need to wait for it. Usually when we do traditional replication what do we do? Mostly we are scaling the rates but for scaling the rise and distributing the load we need multi-masters if it's your cases like available for this kind of architecture. So this is BDR is again on logical decoding and that features that is allowed for us. For conclusion what can we say that? We can say that it is a long, long way that we came and we will still go a loss of way and that is either out of box solutions what is that that I mean? We hold this have wall mechanism which is like used for all the replication that we talk, all the decoding, logical decoding. So we have wall, we have replication, different types of replication now which is getting better and like more like variety and then also PostgreSQL allows us to use extensions so that you can write your own plugins and you can use it actually logical decoding has this C output plugin so that you can write your plugins sorry API so that you can write your plugins for like cache validation for example or like integrating with Kafka or some other tool. So you have the control which means that full tolerance and dependability is getting better and better and as I wanted to show that we come a very long way and it is going better for me and as a last point it's that I focused on logical replication in core will be the better and next leap in terms of full tolerance and dependability in PostgreSQL. So I guess that's it. Thank you. Oh so as a result are coming. So quick. Okay we have plenty of time for questions. So please wait for my phone. I'll explain. Thank you. So questions. Yeah hello over here. So I just want to check with me. I misunderstood but it seems to me that when you're talking about logical decoding what you're replicating is just the statements instead of the changes to the data files. Yes. Okay so. Not statement based but not that the files is. I beg your pardon. I mean it is not binary changes that we write and it is what was it? Row based. Yes row based. So it's not statement based but row based. For example when you write an update which is affecting like 100 rows there will be 100 changes. It's row based but not directly statement based. Okay but what will happen if you have like some information that will be different depending on which server. Like if you send an insert that has a timestamp and maybe the time on the two servers will be different. Will it still work? Will it just take the time that was inserted on the master and use that same time on the replicas? I don't know the details actually and there is the author of logical replication here. He can maybe respond, I don't know where he is. I don't know, I think it will replicate anyway so it doesn't mind the timestamp changes. So it actually replicates or decodes the changes that happened to the data. So if you insert timestamp or random or something like that it evaluates it, writes to the table and while and then what was written is decoded to the stream. So it doesn't, what he responded doesn't care about the timestamp but it already decodes the changes that you write to data. Okay. It doesn't do this. There was another question around here. I see one hand there. Here as well. Okay, okay. Also here. Cool, so you mentioned that the logical replication also supports synchronous commit to one of the secondaries. Does that mean that there is some sort of shared part of the protocol between the old school disk based replication and the logical stuff or does the logical decoder provide some way of blocking commit on the primary in order to, so that you don't acknowledge to a client until it's on two nodes? I think from my understanding it is using a traditional way to, because we have this wall and we are using these facilities it's coming from this wall process. So that's what I understand if I'm incorrect. I mean it doesn't have the extra thing depending on logical replication but it is using the same traditional way. All right, okay. Here and there was here. So you mentioned that the logical is like another level to wall writing like right head lock level. So, but there is also logical extraction. So do you actually like try to decode whatever was written like in right head or you actually write another type of right head lock? Yes, you save another type of information. Okay, so what's the extraction then stands for? Like what does it extract? Well, just it's just a read. There's more information. I guess I can answer this better. So it actually decodes the data that you would write normally. The logical level just adds some additional information that you need. For example, if you do update it stores the data about the previous row. So because you need to actually somehow identify the previous row, so it puts it there. With normal update, you would only get data for the new row but with logical you need to do old one. But the decoding then decodes to binary data from the wall to actual changes for the API. Okay, so it has more data too. Here. All right, thank you. So going back, my question, someone's already asked half of it. So the logical replication, as you said, if you've got like a million row updates, it will break down into a million one line update statements. I've seen that with other products, other database engines and it can cause perform the replica to lag quite horribly. Is there any sort of clever logic for handling those scenarios with logical replications? It was quite a while back when I saw that problem. I'm starting to use Postgres more and more. And I'm just worried that I'm gonna see that again from, you know, that problem I saw five years ago all the time with a, it was, you know, it wasn't a Postgres, but it was logical replication. And it was exactly that scenario. A million row update gets broken down into a million one line, million single update statements which just takes a long time to process. I'm not logical, the replication developer, but what I know is this leg is almost all the time the problem for any kind of scenario. So I don't know how they handle it in like for in production use cases. But what I know is it is, for example, better perform. There is the performance reports actually from my colleague Tomasz and he wrote in our blog for company blog that he kind of compared with replication in 9.4, replication in 9.6, physical replication and logical replication in use. So with benchmarks and so on. So it doesn't have this huge gap of performance that I see from the graphics. So I don't know how they handle it, how they manage it, doing it better, but there is a, yeah, they're working all right. And it's the obvious thing that from the theory that we can see at least that is in the presentations comparing to trigger-based ones, they are really good performance. But they don't have this huge gap of performance that's coming from physical or logical. So it is, yeah, it's in use and it's used. Because since 9.4, it has like loss of time for people to try and use it. But in core that we don't know yet. I mean, we know that it will be in part of process 10, but we will see how it will work. I don't have a performance test for 10. I have a question about the wall here. Ah, yeah. For how long is data stored in the wall? Let's say you have a table with a limited number of rows, but with very regular updates. Would the wall grow indefinitely based on those updates or is all data discarded at some point in time? How is that? You need to decide how long you want to keep your walls. So it is depending on your configuration, basically. You can keep it like unlimited time, then you will run out of this space. But you need to be sure about your dynamics. How much do you do checkpoints? How do you replicate? And so you need to configure it. Okay. If I understand correctly the question. Yes. Is the data also released back to the operating system when it's been discarded? Or does it keep the old records allocated but empty for future use? It is beyond my explanation, I guess. I don't know how it works really. Does anyone explain this better? I repeat, but just if it was wall, as well as... Okay. Yeah, so the wall is going to stay around for as long as you have it set up. There's some different commands that can be used to do kind of archive cleanup and also archiving. When you do archive wall through with Postgres, it's going to release that once it's done. But if you're doing logical decoding with it, then you need that wall stream for as long as you have an ongoing transaction or throughout a given checkpoint, at least. Anyway, you're going to need that anyway. So it's not really kept around. Eventually it does get recycled. So even inside of Postgres, as long as your archive command is working properly, assuming you're using an archive command. If you're not, then it shouldn't actually need to keep it around for longer than a given transaction is my understanding. Okay. I just get a question. As far as I understood the data fights between minor versions of Postgres are compatible. Where is the benefit of logical replication between minor versions? Or is that- For minor versions, it's already easy to do upgrades. That's why I emphasize major versions because recently it is harder. There are solutions to do this PGA upgrade and you can still do that, but it is better because you don't need to use the same major version that was I meaning. So for minor version you can use it, of course, but it's not needed. Okay, thanks. Yeah, you're welcome. Hello. Hello. With streaming replication, synchronous replication and replication slots, we have a new failure mode, which we encourage sometimes. When you have a slave, which for some reason becomes unresponsive, your wall partition is filled and after some time your master is also going down. Do you have some solutions besides monitoring for dealing with such situations? You need to, the basic thing maybe I can say that you need to monitor your application leg. So you need to be aware of before it got unconscionable. So you need to know that what is acceptable for you in terms of leg, I mean, in bytes or in time. So then you don't need to be like, okay, it's like everything is filled and it's like not catching at all ends. Like it is like too much legs or you don't need to do base backup or something start over. So before that, there are like lots of monitoring tools that I don't know actually externally, but you can find it. If you are a reader of Planet PostgreSQL, all the company blogs are going there. So you can find their tools. Also there is Check Postgres, which is mostly used I guess, the scripts so that you can just check and implement in your monitoring. Replication leg, you should definitely check. There is one question here. What is the difference between streaming and non-streaming replications from the point of implementation? Why streaming replication is more efficient and why PostgreSQL protocol, internal protocol is better than, for example, R-Sync or something like that? I don't know actually how the implementation in the code because I'm not the level of her, but what I know is that you can use this. If you want to send your files, you can send it because there's this data directory you can send your files with any kind of protocol. But if you use your, our internal world sender and receiver protocols, it's possible it has more optimization. It is more like you don't need to control it over. So it is streaming all the time. You don't need to ship your files. That is what I understand. And Steve wants to say something more. Why is better? If you're sending files, it's sent in 16 megabytes chunks. So you're always lagging 16 megabytes behind. And if you're streaming, you're getting good data in real time. So instead of like shipping them, like bytes, you'll just stream it. Yes, yes. The other thing is R-Sync is single-threaded. Yeah. Also this is multiple-threaded. There is one question here. I don't know if it is. Oh, sorry. Yes, sorry question. No problem. If I choose a logical replication, does it mean that I have twice all the information? So I will get the size of the world will be more. It will be more, but I don't, I ask this question before it won't be double. There's not that huge difference. Not the double, but. Yeah, right. There is something, I mean, not all the information is in the wall because it has also other stuff that's done inside, but your wall is not going double. Okay. This double thing, maybe you remember from the trigger-based ones, because they were writing a double time, so double writes is coming from there, but not in logical replication sense of this. Okay, thank you. Yeah, welcome. Question for you. You said wall is single-traded. Okay. Sorry, you said wall is single-traded and streaming and the multi-traded. Sorry, no, no. I didn't mean to imply that what we have today is multi-threaded. I was just pointing out that's one of the issues with our thing. It is something that we're looking at doing in terms of being able to support parallel archiving of wall. It's not there yet, but I've had a discussion with Magnus about it and I think it's looking promising. So in the future, the idea is that we'll be able to do parallel archiving. It's just not, no. So right now, no, but, you know, we can't with our thing at all, so. Okay, maybe one last question. Yeah. So the question is on, if you have replication and long transactions, do you log this before you commit so we actually can stream things while the transaction is, or replicate while the transaction is running or is this thing just going into the log when the transaction is finished? After commit, I know. Okay, so if you do like a large, oh, that's the follow-up question, with all the table type of commands, will I have to complete on the master before it actually starts on a slave or how do, okay. Okay, we have a few minutes left, but I think we had plenty of questions, so. Maybe one last one. Two minutes. On the change set extraction, it said that DDL wasn't supported. Are there any implications with respect to schema changes? Sorry, I didn't quite get it, I guess. There was. DDL, yeah. So on your slide with the change set extraction, it said that DDL statements weren't supported in the extraction. So does that have any implication in terms of your master and slave when you're doing schema changes? What kind of implication can be seen? Well, the short answer is that things are getting complicated. Yes, yeah, he's asking like in case of alter or something. Yeah, yeah. You need to sync everything and then do the changes at all places at once. And I think the idea is that POSCOS 10 will get the DDL replication as well, but it's not there yet. So if geological is far ahead then what's in POSCOS core? But it's, yeah, it's not in there yet. Yeah, so when it's in core, it won't be a problem. Okay, so thank you. Thank you again. Okay, in the next talk we'll be starting in 10 minutes at 3 o'clock.