 It's not a matter of if but when the next disaster will unexpectedly happen being related to hardware failure, human error or atmospheric causes which unfortunately are becoming more and more frequent. Planning for disasters, being prepared for when they occur, reacting properly and promptly in order to restore operations in the shortest time possible and with minimal data loss should be the premise of any business risk strategy. We might not open shop without it. Postgres, thanks to its continuous backup, point-in-time recovery as well as streaming replication operates in business continuity environments with low recovery point objective and recovery time objectives. Good afternoon, my name is Gabriele Bartolini and today I will be talking to you about one component of a business continuity solution for postgresql databases using an open source tool called Barman. The talk is, I will be moving if you don't mind, the talk is organized in four parts. The first one is an introduction to Barman because I didn't know how many of you already used it. The second part gives an overview of the main features. The third part, because I was requested to focus on this, covers some important case studies for Barman and Postgres actually and the last part will try to outline the new features that we want to develop and put into Barman. So I'll start with a history of Barman. Since I joined second quadrant in 2008, I've always been in desperate need for a tool that was capable of managing both backup and recovery of postgres databases. Can you hear me over there? Okay. We had several customers coming from Oracle and they all had our man in mind and they couldn't accept custom scripts. Said, ah, it's not acceptable that in Postgres, we have to use custom scripts. Fortunately, in 2001, in preparation for a conference organized by second quadrant and called Char 11, clustering high availability and replication organized in July, we conceived the idea of writing an open source tool for managing both backup and recovery. Thanks to a few customers and two of them were facing a large migration from Oracle, we were able to write a prototype of Barman and also thanks to funding from a European research project called Forcast, we were able to release the first stable version of Barman in 2012. Now, we just came out with Barman 140, which is the 13th version of Barman, which means that every quarter, pretty much, we are releasing a new version. Barman is written in Python. It requires you to accept the GNU GPL license, GNU GPL 3 license, and second quadrant maintains packages for both Red Hat and Ubuntu Debian Linux. We support and test Postgres from 8.3 onwards and currently it is only based on R-Sync over SSH. When developing and actually conceiving Barman, we've always had three principles in mind. The first one is integration, integrate with existing components and solutions in a business, then focus on developing an intuitive user interface and configuration mechanism for usability reasons and automate Barman, I mean, this tool, in as far as backup, monitoring and recovery are concerned. We believe that these three are the most important components of a disaster recovery solution. As the previous talk outlined, I want to introduce some backup types. Barman supports hot, full, physical, continuous and incremental backup. So it's a physical backup solution. It doesn't support logical backup yet, which is essentially a snapshot of a database at one instant. And it's done usually through PGDump and PGDump whole, but there's also an open source project called PG Backman. That allows you to keep track of logical backups. Physical backups in Barman rely on two concepts, essentially. Continuous wall archiving and base backups. These are concepts that even though Barman hides them from you, we always recommend at least studying them on the documentation or take one of our courses. Because having an understanding of how things work underneath will help you understand better Barman and also the architectures that you can implement with it and with Postgres. With Barman, we can back up the minimum information unit that we can back up with Barman is a Postgres QL server. Unfortunately, we cannot back up a single table. We cannot back up a single database. Table spaces are transparently backed up by Barman. And when it comes to recovery, we can also specify a different location for each table space. When we talk about disaster recovery, we have to briefly introduce two main concepts called two main metrics, actually, and measures. They're called RPO, recovery point objective, and RTO, recovery time objective. Let's consider a very simple scenario for disaster recovery. This is the most simple scenario you can have. You have a master server here and the backup server there. In terms of recovery point objective, which is the amount of data you can afford to lose, we already have a quite optimal value for the majority of the cases. Postgres QL continuously sends wall files to the Barman server in this case. And so the maximum amount of information you can lose is a 16-megabyte file of transaction. Depending on the workload, it could be even more. But at the moment, with wall archiving, this is what the amount of data you can lose. With Postgres, you can also configure that through the archive timeout configuration option. But even with the simple configuration, you can have a very good value of recovery point objective. However, when it comes to recovery, the recovery time is the worst you can have. Because you don't have an, when a disaster happens, you need to restore a new server, reinstall the operating system, reinstall Postgres, and recover the backup data. So the recovery time is very high. But depending on your business, it could be a very suitable solution. So it all depends on assessing RPO and RTO. So there's no one solution fits all. Okay. It all depends on your requirements. This could be quite, could work quite well for you. A basic disaster recovery solution involves the addition of a node for recovery. So this way, you still have a very low RPO and you can reduce the RTO from low to a very high value, depending on the size of the backup. So for instance, you have your backup continuously taken here and you have a recovery node that is once a week, for example, synchronized with the latest backup. So the recovery time is just a matter of reapplying the whole files from the latest backup to the start of the server. The good thing of this solution is that if you plan and if you are prepared for it, you can measure that. And of course, you know that backup works and you have people continuously testing in your team that the backup works and they get familiar with the procedure itself. Remember, a backup that is not tested is not a backup. You can have all the backups you want, but if you don't test them, they're useless. And let's go through a very basic business continuity solution for Postgres. This is a very typical and cost-effective solution that brings down the recovery time and which is based on high availability as well, because disaster recovery focuses mainly on RPO. High availability is more focused on reducing RTO, mainly bringing it to zero. And that depends on the uptime that you require, that your business requires you to set as a DBA or as a database architect. With this very simple solution, you can achieve fantastic results in terms of business continuity. It's very simple, and it's effective. So let's see some of the major features of Barman. So it's not a coincidence that both backup and recovery are part of the name of Barman. Can you guess why? Yeah, but that's not what I want to say. It's only through a successful recovery that your backup can be considered valid. If you don't recover, you can't say that that backup works. So this has always been the assumption when we designed Barman. So the word recovery must be in the name of the software. And this actually led to an architectural choice when we designed Barman. We wanted to have a separate server for Barman because in the simplest scenario, you could use that server for recovery. Imagine you have only two servers and if the master goes down in an emergency situation, you can restore the server on that node, on the backup node. It's an emergency situation, but again, it depends on your cost evaluation and also your allowance, your budget. So that's why having a separate server led to allowing remote operations in Barman for both backup and recovery. So we wanted to work remotely since the conception. Also, working remotely allowed us to concentrate more server backups on a single backup server. Any comments? I'm reading your mind. Yeah, that can be considered a single point of failure. OK, however, it's not a primary failure. Having this backup server down will generate issues. That's for sure. Will generate actions to take after reactions. However, you want you want experience a downtime. Unless, of course, you don't react. In that case, for example, the PostgreSQL disk can fill up with world files and it eventually stops. You will have to react, as I said, but it's not a primary failure. Also, you can have your backup file, your Barman files backed up on another disk in the cloud or on tapes. And we are working on adding geo redundancy into Barman and the import and export facilities and also S3 storage support. But I mean, this is my opinion. The coolest feature of Barman is the backup catalog. With the backup catalog, let's say, you take a weekly backup every Saturday. Over time, you generate weeks, months of data. So you can restore at any point of time within this range. And Barman keeps separated backups, which are like milestones, you know, like little flags you can put on the history of your PostgreSQL server. Separates backups from the wall archive, which is a continuous stream of wall files from the first available backup to the latest that's just been shipped. I don't know if you're familiar with how recovery works, but basically, when we want to recover at any point in time, we have to start from the closest, hopefully, you can start even from an older backup. But let's say you have to recover at. Record to this moment here, you take this physical backup, you copy that into another server and then you start replaying wall files up to that moment. And then PostgreSQL eventually reaches a consistent state and starts up and you can query that server. But you need both base backups and wall files. The cool thing of Barman is that it actually associates every wall file to the closest base backup. So if you, for example, erase or delete this base backup, you don't erase these wall files, but they are associated to the previous one, transparently. And if you remove the oldest base backups, all this stream of wall files up to here is automatically deleted for you. I was saying managing backups over time, of course, this backup catalog keeps growing. And following the principle of automation, you would like to automatically prune the backup catalog, specifying a way to do it possibly. You could have, for instance, storage needs or low requirements that tell you that you can't store any data older than six months, for example, for low requirements. So Barman can remove obsolete backups and by defining one option, essentially, that could be defined globally or overridden at server level. You essentially have two options, one based on the number of backups. So you can type redundancy four, which means that you can keep the last four backups. Or recovery window of four weeks, three months, and Barman calculates automatically the point of recoverability defined by this recovery window retention policy and automatically discards obsolete backups. So it's very easy to set up, highly configurable and once again, fully automated in case your database has a large portion of data that doesn't change between a full-based backup and the subsequent one, you might benefit from, you can see that well, but from incremental backup. So incremental backup has been introduced in one for zero, so last month, and it's a file-based incremental backup. It heavily relies on our sync and hard links and it can reduce disk usage, so you can save space, you can save network bandwidth, and we can reduce backup time as well. The average, of course, on the customers that have this, you know, the specific workload that I described before, is between 50% to 70% reductions. So I'll show you later on a case study about incremental backup. We actually tried to introduce incremental backup into PostgreSQL 9.4 for PG-based backup, but we couldn't make it, so we'll try and we'll try higher for 9.6. Barman allows you to set up compression of wall files. So, you know, the first step is to set up Archive Command, Archive Command, sorry, Continuous Archiving. Without Continuous Archiving, we don't have physical backup in place. So PostgreSQL continuously sends wall files as soon as they, the 16 megabytes of transactional data are filled, sends them to Barman in a special directory called Incoming. Then through the Chrome command, Barman Chrome, which is the maintenance command for Barman, if you have requested it, wall files can be compressed using Gzip, Bzip2, or your favorite compressor. And they are placed into the wall archive, the one we saw before. So when you perform a backup, which is a very simple command, it's Barman backup, backup name of the server, the R-Sync copies the base backups while PostgreSQL keeps sending wall files in the incoming directory. Usually, if you install Barman with IPM or Ubuntu packages, Barman Chrome is set to run every minute. So every minute, the maintenance job runs, which takes incoming files, compresses them and stores them permanently in the archive. Without monitoring, we believe that you can't have a business continuity solution. Sorry about that, but monitoring is a crucial part of managing a PostgreSQL database and any system. So with Barman, we wanted to be able to automatically detect any problem and immediately notify us about this promise so that we could act and solve the issue. So we didn't want to invent anything special because following the principle of automation and integration, we wanted Barman to be integrated very easily in the monitoring solution that your company is adopting. Does it make sense? So there's one command, which is called Barman Check. You can type Barman Check all and all the servers configured in Barman I checked sequentially, or you can type Barman Check name of the server and Barman will start checking that server and make sure that everything works. So in this case, SSH connection was not possible, so an error was triggered. So Barman Check returns zero if everything is fine. Otherwise, it returns another code. An interesting feature as well is the so-called Smelly Backup. So let's say your Barman, your periodic backup command fails. You set up a cron every Saturday at 4 a.m. and that Saturday you have a power outage and the backup is not taken. Barman, if you set backup maximum age one week, after one week, it will start signaling through the Barman Check command that backup maximum age is not okay. So your backup stinks. Okay, so you can actually react and manually start another backup. With 9.4, we have integrated Barman with a small patch that I wrote last year for 9.4, which is the PG Start Archive view. So Barman is able to immediately detect if the archiving process is not working. And it is also able to give you an important statistic which is the world generation rate of your server. We have also integrated Barman Check with Nagios and Itzinga. So if you type Barman dash dash Nagios, you have a fully compliant Nagios output. In this case SSH was failing so it returns two. So it's very easy to integrate Barman with Nagios. Other minor features is we can limit bandwidth usage globally per server or table space. Can you give me an example why reducing bandwidth for table space could be useful? Well, it's mainly for, let's say if you have a table space on an SSD disk, you might want, for example, not to impact on the production environment in terms of read and write operations on that disk or on the cache. So you can set it globally per server or per table space. You can also enable network compression. So if you have a Barman server sitting on the other side of the world, you can compress your data before shipping it through the internet. Or if you have a standby and you don't want to stress your master, you can install the PGSpresso extension and backup from a standby. You can use hook scripts. Hook scripts are scripts that you can execute before or after the execution of some commands, for example, a backup or the archival of a whole file. And we'll see some example of file listing. I mean, you can get the list of all the files of a backup. But let's get to the most interesting part of this talk, according to me, which is the I mean, I will unveil some important, I think, use cases for the first time. And they involved also the PostgreSQL database. They are Italian companies, but two of them operate worldwide. So I'll start with the first one, which is JobRapido. JobRapido is a job seeking website that is present in, I think, about 60 countries in the world. It was founded in 2006, and it's been one of the most successful startups in in the Italian recent history. This is their architecture. They have several PostgreSQL servers. I mean, through an initial consulting activity, they were looking for sharding that single database into several shards, each one responsible for a different country. So speaking with them, I'm talking to them, I said, why do you need sharding? I mean, do you really need it? Do you have intra-country relationships or way to do you need to perform queries between different countries? I said, no. So you don't need it. Why don't you shard by country, by having different servers? So by placing each country in a different database and play, I mean, move them according to the workloads. Doing this, they saved a lot of money. And they came up with a very simple architecture at the moment because they migrated in 2013. They only have master and disaster recovery nodes. They have 14 PostgreSQL servers, spreading two different data centers. They have 64 databases and around 650 gigabytes of data. So it's not a large database, but it's, you know, let's say it's a medium-sized database. Okay. And of course they have money's hurry in place. Money's hurrying. So it's Inga and Munin in place. So they have two backup servers here that constantly and continuously backup all the PostgreSQL servers. It's very simple. So the largest database is 104 gigabytes. So they backup this database daily. The backup time is two hour and 10 minutes. Uh, they use, uh, virtual servers and in some cases also physical servers in the cloud. So they don't have servers in their premises. They produce about 6,000 wall files a day. So about four, four and a half for an hour. No, it doesn't make sense for a minute. Yeah. No, two. Yeah. Yeah. Yeah. Yeah. So that, that's a mistake. And the compression ratio for wall files is 70%. So every wall file is on average compressed 70%. This is what they, they want to go, want to go. They want to add geo redundancy into Barman. So basically have another Barman node in this data center continuously and asynchronously, uh, cloned from the master. So this way we can, I'll show this feature later on. Then we have Navionics. Navionics is if, if you have a boat, I don't have it, but if you have a boat, most likely you have a Navionics GPS or an app, you know, a mobile app. And they are considered the leaders in this, in this sector in the world. They have the largest marine database and it's stored in Postgres. So in 2011, they decided to move away from Oracle because of costs, essentially. So they, they moved to 8.4 and, uh, they couldn't, you know, there was not, uh, streaming replication at the time. They mainly had one master and went into production with that. Then they called us and we started to work on Barman, thanks to them. But now they've moved to 9.4 and if you, if you have read the release notes of Navionics, of Postgres QN 9.4, you, the Navionics is mentioned. They, they waste it. Uh, they postponed their migration, which was set to 9.3, to 9.4 because of logical decoding. So then their main database, and now the cluster is 10.8 terabyte, but until, I mean, two months, last month it was 14 terabytes. Sorry. This is their architecture. So they have monitoring one master server and a backup server. So they are aware that if something goes wrong, they have to restore those 10 terabytes from scratch. But now that they have 9.4, they, they can think about the future. Okay. It's the next slide. But the server size is 10.8 terabytes. They backup the database weekly. Backup time is around 20 hours. And the duplication ratio is about 43%. So they save 5.7 terabytes out of 10.8 terabytes every week. So this is an example why in some cases file-based incremental backup could be useful. I, I struggled to say to Navionics, the file-based incremental backup is not useful when they save 20 hours a week of backup time reduced by 50% network bandwidth and also disk usage. They produce 90,000 walls per week, which is an average of nine wall files per minute, which is not bad. And compression ratio is about 30%. So these are numbers that you can quote if you want. You can use if you, if you need to, you know, marry the PostgreSQL cause, you can use these numbers. An interesting feature that we have developed for them and that uses hook scripts is the weekly backup on tapes. So it's a semi-automated process. Thanks to a hook script, they use the barman list files command joint with tar and they produce two files. One is a standalone backup, which they can pretty much copy in a, on a recovery node and start up and the PostgreSQL server starts on a consistent, consistent state of the moment the backup ended. Okay. And then all the wall files from the start of the backup to the start of the following one are placed into this tar file. Then, you know, tar works perfectly with tapes. They put that on tapes manually because they need an operator that does that operation, but that's fine. And they ship those tapes to Italy because it's much faster through express courier. So they have a copy initially of the database. The database is in India as I was showing before. So the future, you know, they have a very simple architecture, but obviously they need stand-bys. So they will add sync rep, evaluate rep manager. They will definitely work on logical decoding because now, I mean, that's the heart of the database, but then they have replication with londist that creates several publishing servers that are used by the applications. They will move everything to logical decoding. Then cascading replication. For example, they can use stream replication and have a server in Italy that is pretty much instantly synchronized with the master and then cascading backup. Now, subito, subito.it, if you stop an Italian person and you ask them about subito, they know what you're talking about. Subito, which means at once, you know, is operating the classified media advertising sector and basically they outruled eBay in the Italian market. So if you want to sell something, a bike, a house, or you want to hire somebody, you put that ad on subito.it. These are some numbers. This is the most visited website in Italy in 2014 in its category. And it's the sixth most visited website in Italy after Google, Facebook, and subito. Okay, this is the sixth website. Five million classified ads per category and geographic range. Over eight million unique users per month and 30 million page visits per day. Imagine how many shards they have. What's their architecture? So they've always been in Postgres since, I mean, using A3 and last year, we moved them to 93. They have read at 6, 1.4 terabyte. The peak is 400 transactions per second and at night, as you see, this is the workload. Fortunately, because it's a time zone influenced market, they can afford down times at night. So they don't operate 24-7. So this is their architecture. They have one master, a standby, a backup server, and then reporting and staging servers. And this is what I will be covering in a minute. So these two hours and 45 minutes is the backup time. 800 gigabytes reused every day. Five world files per minute is what they generate on average, considering also the downtime, the low workload. I will give you details about the automated recovery solution that they have implemented. They have a reporting database that is generated from scratch every day, automatically, using the last backup. So it's a totally disjointed server, so they can use it also for rights. And every day, it's completely erased and regenerated. This is the best outcome for Subito.IC. Subito.IC is 100% sure that backups effectively work every day. So we have implemented for them a post-backup script that, you know, after the initial checks, creates a restore point on the master server, stops the postgres server on the recovery node, recovers Baman to that target, which is also incremental because you are recovering on an existing data directory. They customize the configuration because they use different RAM and different disks, et cetera. They start the postgres server. They wait for postgres to exit the recovery mode. And then they customize the instance. They create users, they create tables, they create whatever they want. And then they enjoy it and they let the marketing department enjoy these database. It takes around six hours now, but we're planning to improve it. So again, they are thinking about moving to 9.4, adding more servers and the value-ex-sync rep and rep manager. So the last part, the road ahead. So this is what we would like to implement in Baman this and next year. And everything is subject to, of course, private funding. So we normally operate this way. We receive funding from private companies that want, you know, that have specific needs. We develop them and we put them in the open source product. You know, as a business, we have to sustain our activities and pay us, pay all the programmers. We would like to work for free, but I guess nobody works for free, right? So these are the first two features we would like to work on. The most important one I must admit is the PG-based backup copy method support. So rather than having only our sync, we want to implement the PG-based backup support, which unlocks PostgreSQL on Windows support. If we implemented the tar copy method, you know, each one of any of them will unlock S3 storage strategy. So whichever we implement first, we will be able to work on S3 storage. Then again, if otherwise, if we implement tar storage, we can have base backup compression. So let's say you're interested, David is interested in base backup encryption. You know, I said, okay, David, you are interested in that. The shortest way is for us to implement tar storage and then base backup compression, which unlocks base backup encryption. Otherwise, we can implement PG-based backup and then S3 storage, et cetera, if you're interested in S3. Another interesting feature is the get wall facility. You know, if you set up a stand-by server, you have to set up also the restore command. So currently, Barman has to ship every wall files before hand to the destination node. If a smarter way to do that would be to actually allow the recovery node to query Barman and ask for the get for the wall file that it's needed. So through the get wall command, we would be able to do that. And also, this would allow you to set up see Barman as a hub of wall files for all your PostgreSQL infrastructure. So you can reduce the wall keep segment, or you can use, you know, in a different way, replication slots. If you set this fallback methodology for recovery. So you can set it, even if you have streaming replication, if there's a, you know, connection problem, PostgreSQL tries the restore command. Save and the same way on another node. So there's a crossed synchronization. And also, this opens the door to cascading backups. Then you can export backups and reimport them again into Barman. This is another feature that we would like to implement using Tire, of course. But, you know, there's more window support if you want to use Barman on Windows. Keep and no keep of backups. So you can make sure that protect backup from retention policies, backup scheduling, the grandfather and father and son, backup scheme, recovery nodes, etc. So if you're interested, I mean, we can talk later on. I'll be at the second quadrant desk. Just a quick overview of some similar tools that you can find because Barman is not the only one. So if you're planning to use a disaster recovery, look all of them and decide which one suits better for your needs. All of them are open source, but as far as I know, is an enterprise DB product. I don't think it's open source, but I might be wrong. This is the DevOps team that continuously developed Barman. We implement, yeah, DevOps culture, as I call it. It's myself, Marco, Giulio, Francesco and Giuseppe. And behind there's our Kanban board where we, you know, measure the progress of every feature we implement. We have about 8,000 unit tests and 1,500 integration tests that we run every time we implement a new feature. And every feature is reviewed by at least three people. Because when it comes to disaster recovery, we can't make mistakes. Okay? My advice is to start simple then grow. This is valid for everything. Start simple, evaluate RPO, RCO, costs, and most importantly, risks. It must not be a technological reason, but assessed by all these four things. Have a disaster recovery plan. Update it regularly and make sure that everyone knows about it. Because when the disaster happens, you'll be on holiday. Okay? You don't want to be cold when you're on holiday. Always test and simulate. You can implement the Kers-Monkey paradigm. So destroy a master or, you know, shut down a master or, you know, I've seen it doing with Barman and restoring that. Because practice is the only way to mastery. So make sure that regularly you test your backups and you simulate a disaster. So finally, Barman is robust, I believe, maintained. And tested. We followed these three principles. I hope you got an idea of the three of them. Integration, usability, and automation. Monitoring and alerting are crucial components of your system. We also believe that Barman has fostered migrations from Oracle. It has reduced barriers for migrations. That's what we are continuously told by Leeds, especially. But most importantly, because we manage over a thousand database servers with second quadrant, it has standardized our support procedures. So we know that most of our customers have Barman in place. So when there's a disaster, all the engineers know what to do. And this is important because if you have thousand databases and each of them has their own custom scripts, when the disaster happens, you're under stress. It's unexpected. Could be at night. Yeah, there's a lot of emotional situations that occur. So you have to deal them as well. So I would like to hear about your story. You know, I'm here, as I said, all day. I would like to hear what you think of Barman, whether it's good or bad, like David. So I'm quite open to that. So I don't think we have questions. Do questions start? Can we have some questions? At least one. Any questions? Oh, you talk to me. Okay. Any questions? No, no, it's transparent. So the incremental is managed by your heart links. So you don't, you don't notice that. No, it's between the start of the backup and the end of the backup. You need all of them. Otherwise it's not consistent because the assumption is that a page, this page can change its content between the moment you start a backup and the moment the backup ends. Okay. And changes are written in the world files. That's why you need them. Yeah. The first time you can recall, if that's, you know, your first backup, the minimum time, point in time is the start of the backup. No, actually, no, no, it's the end of the backup. Sorry. You need a previous one to recover. Yeah. It starts from the previous one. Okay. No, it's not differential. It's basically you take, it's like you take the two snapshots and you compare the two of them, file per file. And if all your files have changed, you know, you don't share anything, but you share all the single files that have not changed between the two base backups. Okay. So if you touch all the files, like if you perform a vacuum or something like that, you don't have a file level. Yes. You can, it's driven by Postgres. So you can have time, transaction ID or target name. So the example I showed you before for subito.it before restoring, they tag. They place a placeholder in the database. It's called restore point. So you can specify that at recovery time and PostgreSQL stops there at replay time. Okay. No worries. No, at the moment, it's not stored compressed. It's on the storage method is plain. So you have plain files. But if you use, for example, ZFS, ZF, yeah. You know, it's transparent. That's the best way, I think. You know, all the, I mean, the three, I explained the three principles. The fourth is, you know, it's not to reinvent the wheel. So if there's something else that does that job brilliantly and it's well maintained, we stick with that. We don't want to re-implement everything in our software. You can't compress the wall files. You can compress wall files. And as I said, if you want to compress backup files, we need to have at least either the tar storage feature unlocked or the S3 storage. That way, you can store on S3 in a compressed manner and encrypt it. It depends. It's driven by private funding, you know. Yeah. Generally, we try to cut all those features in a three month period so that every feature could be a new release. So, and three month doesn't mean that it takes us three months to develop that because development time is usually quite short. If we, if there's 10 hours development, there's 90 hours testing. And plus you have documentation. And, you know, when, when we develop an open source tool, we have to emphasize on documentation, on writing manuals. You have all the mount pages. So if you type man, barman, you get, you know, all the options for barman, man, five barman, the configuration options. And so we try and write standard applications as much as we can. And, and, you know, easy to install. So if you want to install it on Ubuntu, you type APT, APT get installed barman and it's, it's installed. Or YAM installed barman and it's installed on Red Hat. And another important thing is that so far you can upgrade from any version of barman to the latest one and everything is compatible. So you don't need to worry about internal formats of data files or things like that. You can update transparently. And we always encourage to update to the latest version. So right now I use compressed PG dumps for backups. And so they, they're a lot smaller. I want to switch to using barman and, you know, get all the nicer features and not have custom scripts doing things. I should expect that I'll need a lot more disk space, right? Yeah, you have, the difference is made up by indexes. Okay, because indexes take zero makes five, five words in an SQL dump if I could take gigabytes on disk, you know. So that, that's mainly that the difference you get. But the difference is you have continuous backup here with PG dump. You have to take into, you have to consider RPO because the recovery point objective is basically it could be, in the worst case scenario, the time between two PG dumps. So, and but that's, if that's fine, but streaming replication doesn't, that's a, that's a, that's, you know, mainly the difference between disaster recovery and high availability as, as what's the name? But yeah, sorry. She was pointing out before with nine four, you have delayed standbys and they can protect you from human errors. If you have synchronous replication and you type drop table, you can have as many standbys you want, but it's yeah, but still, you know, if it, if it's all you type delete, delete from without way, you know, and, you know, it's a, it's a similar use case. So that's a disaster recovery solution with continuous backup also covers that. Replication doesn't cover that case. It covers and it's more, mostly a high availability feature. So high availability and disaster recovery together, you have business continuities. So this is, and you need both of them. And disaster recovery to conclude is the primordial way of doing high availability. Whereas with high availability, you can't do disaster recovery. Okay, so in the most simple solution you can have, you have a disaster recovery solution, as you've seen in these examples. So here you have some URLs and if you have questions, I'm, I'm here, as I said, at the second quadrant stand. I, these slides are publicly available and these are some commands which are very, very simple to understand. It's our sync. No, no, no, no, it's our sync that will do its best to finish the backup in the shortest time possible. Yeah, but the backup command needs to copy all the pages of the, you know, all the data files in the PGDASA and the table spaces. So by default, it's performed at the highest speed. But you can limit that through the bandwidth limit option, as I said before. So you don't, if you want to reduce the impact on the, mainly the operating system cache, you can, you can configure bandwidth limitation. You, you want to go, you want to eat.