 to you about Postgres how we do it right now very briefly and then we more interestingly talk about how we envision Postgres on Kubernetes why this is important for us we will hopefully show you and convince you somehow that Postgres can run in Docker and also can run in the Kubernetes environment. What we are talking about is basically database as a service at Salendo. We briefly introduced the UIs and the monitoring we have right now and then I will hand off for more technical talk to Olexi to talk about Patroni and the Postgres operator that is basically driving the future of the infrastructure. So let's take a look. Yeah, about me I mentioned it. My name is Jan Musler and the engineering lead which basically means I hand all the technical questions to Olexi later and Olexi is an engineer in my team who has basically spent most of his work on Patroni and the operator recently so I hope we can answer all your questions later. Salendo and Postgres at a glance it basically means and maybe some of you have heard that already before that we run a lot of Postgres databases in the data center then AWS came along and this was like appreciated when you run something like database as a service and everyone wants something else bigger disk smaller disk faster disk more memory and so on. I was the state right now Patroni is powering more than 150 production databases on AWS running on EBS running on local storage while this is beneficial we can briefly cover later and what's the interesting part is that we are transitioning hundreds of databases basically to Kubernetes. The reason for this is basically saving money putting more resources to more efficient use and also believing that the self-service we will show you later basically makes our life much easier in the database engineering team. Where are we coming from? We're running basically Postgres and LXC containers for a really long time in two different data centers spread across Germany. There was the Git repository with all the configuration that was cloned to the hosts and then basically scripts were setting up the infrastructure so this is always a good thing to do have your basically configuration Git somewhere do pull requests on this. We had something called the database discovery service already so REST API to find all the hosts and all the databases in production. This helps when you have like automated monitoring and other tooling that basically needs to know on the fly where the stuff is running. Yeah, what else is interesting we always had the second data center basically at a time delay like one hour of replication delay. This helps you a lot when you accidentally delete stuff because the streaming stuff is really quick. Here it still shows we have Postgres 9.3 and plus but actually very recently we migrated almost everything to 9.6 thanks to PG upgrade and minimal down times. This is really beneficial to everyone. Performance went up and the people appreciate the new features also. Next slide. Yeah, briefly I mentioned this already. We have the Git repository in our local GitHub setup and then we basically do push pull and so on and then it gets deployed. Not so interesting anymore. Let's go briefly to Postgres on Amazon. I said already it makes our life very easy. We can much quicker provision databases than before where we had basically to wait for hardware forever and this is way more fun on AWS. And what does it mean? We have more memory, more CPUs, different storage is available so we can tailor this much more to the needs of individual teams. What also was interesting that Docker was basically for compliance reasons basically in. So basically we needed to know what is deployed and where and which software. So basically Docker was or Docker images were our deliverable. I think there was some fighting in the team initially. I wasn't in the team at the time whether Postgres is supposed to run in Docker. But I think by now everyone agrees this works quite well. We expected more note failures on AWS. Everyone told us EC2 is unstable. Turns out it's not the case. So the cloud is actually a lot of fun and EC2 is pretty stable. It very rarely fails and if so you almost always get an email ahead of time. But this basically drove the team to build Patronius some time ago as a folk of governor. And what it actually is it's a Postgres management demon that tackles the problem of basically bootstrapping and more importantly the automated failover. So no one basically has to get up at night. And while I joined Salando seven years ago, spent two years in the database engineering team and really had to wake up at night. When I came back in January 2017 seeing basically Patronian action on AWS, it's pretty amazing for me to see how well this really works. It uses master election basically backed by HCD or you will later learn we by now use the Kubernetes API for this. Much more systems are supported. It's not pinned to AWS either. So it also works on Google and other implementations. It was one of the first open source repositories at Salando with a growing adoption. The first basically non Salando contributors, which we really appreciate people testing it testing it on different platforms and giving back. This is the whole idea why we basically open sourcing this to make use of Postgres easy. So more about Patroni later from Olexi. Maybe one question, why are you not using RDS or why are you not using Aurora? There's nothing really bad to say about either of them. Aurora is also pretty interesting technology. And we think at a certain size it makes sense to use. But in the end, we believe that having Postgres having it under our control, having a real super user having streaming replication in and out of the system between different accounts between data center and AWS is in the end beneficial for us. The integration we will show it also is basically better. So we connected to our I am solution, we will show this. And we are also able to use local storage. And this is basically the main instance type we now use for everything that is requiring performance IO. So we run thing on FML, I3 instances on AWS. This means we have like SSDs. And when the machine dies, everything is gone. But we have two replicas across availability zones. And we strongly believe that this is good enough for us. And until today, it has worked flawlessly. But let's talk about Postgres as a service. So why are we going to talk about this? Because in the data center, there was a driver ticket on AWS. There was still a driver ticket. And everyone who waits for a database like two days or so, he will go to Audi S immediately. Why? Because there is a button and you have your database and you are done. So the whole idea basically when Kubernetes came to live beginning of 2017 was, let's build something that is equally convenient to use, and makes it really give you a database within like 30 seconds in your cluster, make it easy to connect provisioned users and so on. And you will see it later. This is basically what we have right now. So fast provisioning of Postgres databases or clusters in our infrastructure connected to deployment pipelines connected to our platform I am. And let's look at how it looks like. Did I miss something? No. Yeah, so amazing UI work by myself and the team here. So everyone loves doing UIs and JavaScript. That's very powerful. Basically, you know this, you give this database a name, you select the version 9.6 and 10. And that's also the benefit. We have Postgres 10 rolled out. We're not waiting for Amazon to give us 10. What else is there? A number of replicas and so on. A replica load balancer so that you can basically do a read load balancing. And then there's one point I will show later is basically user and database configuration. Why I will explain later. So let's click a button provisioning more provisioning and databases ready. We are skipping the live demo. But basically, this would take roughly about 30 seconds for provisioning new clusters. An interesting part that is in there basically make it easier and especially hands on deployment is about role provisioning. So we are supposed to not connect to databases in production. No SSH, no P SQL, nothing. So let's look at what we can do to make this really like hands free. And this means like declarative user and database description with an owner and another role to connect your application. And then there's something nice in Kubernetes. I'm not sure. Let's make a show of hands here who is aware of how Kubernetes works a little bit or understands some stuff. Yeah, see, that's pretty amazing. The key part is that there is something called Kubernetes secrets and you put credentials into this and then you can use it in your application. And this makes the life of the developer quite easy. He doesn't have to worry about can you commit credentials and get to us to use encryption and so on. It basically never leaves the system. And there's one example down here would basically connect the whole thing. But this allows basically application deployment with a database created and roles and then fly away to kick in to provision everything you need without ever connecting to the database. And that is like the whole idea. So basically have a deployment pipeline on the Git repository with all the configuration and your application is up and running. Yeah, and the last one is basically encouraging this role split. I think more people here are aware of like good design practices than most engineers who just want to have a super user and do everything with this. So some idea here of encouraging a database owner and an application role. So let's look briefly at the other part of integration. We have this nice platform I am at Solano, which basically allows us to use JSON web tokens. So like 5600 bytes of signed JSON to authenticate employees against systems. And we thought, okay, let's stop using passwords in Postgres too because they had another password. Syncing passwords is hard connecting to LDAP from AWS is annoying. So in the end, luckily Postgres again, greatly extensible supports PUM and one of my colleagues, he wrote this module. So now basically we do the JSON web token verification from Postgres. We talk to some service and says this token is really valid. And if it's okay, the employee is able to log into the database. The neat thing is we basically get the Google MFA for free so employees cannot get a token without MFA enabled. And that is also very convenient. One example basically here is the set token that creates a token for you and then you connect to the database. Let's go to monitoring quickly. Where's the minutes left? Am I supposed? How does it look like the setup in Kubernetes? Briefly before Olexi goes into more detail. There is like a pod that's running Postgres and there is one extension from Aleksandar Kukushkin which is PGMon. It's basically an extension running in Postgres that gives us the REST API exposing PGSet activity. Why will you see in a few slides? And there is the scalar sidecar that streams all our Postgres logs into the cloud. And then we have PGVue which I will show. We have a setmon agent which is our platform monitoring for long-time statistics and we have the real worker who allows teams basically to execute SQL code against the database for like KPI and other kinds of process monitoring. PGVue, I talked about it, we have this REST API around PGSet activity. And basically what it does in the lower half it exposes everything that is currently running which is really helpful so people can go to a web UI, look at all the Postgres nodes that are running and see what's going on in the database. There is storage, there's IO and memory. And this was greatly appreciated when we wrote this out. Remember there is like no SSH anywhere so you can't go there and use the old nice journal tool anymore. It integrates into the other monitoring with some nice set of deep links so this is really helpful. People don't have to remember where to go for data. And in the end we have basically Grafana and KyroCB for storing all the metric data we have for the recent 60 days so everything you basically know about metrics is going there like IO, CPU, memory, whatever you can imagine. And we're also tracking the stable sizes, sequential scans, index usage and so on. Yeah this is then basically my part. I was really fast so this means you will get more technical stuff from Olexi in the next minutes. Are there any questions right now due to time and I think I can take one or two quickly at this point? None? Great. Okay, that's up to you. Yeah, you can also ask your questions later. So, Alexia, I'm supposed to go into details on everything and I'm going to describe a bit more details, our infrastructure and also because we had FOSDM I'm going to highlight what we have an open source and how you can actually use what we have to build your own infrastructure. Yeah, so we are building this so-called cloud native infrastructure meaning that our services directly interact with Kubernetes, know about existence of Kubernetes and can take advantage of can take over the benefits that Kubernetes provides. So I saw a lot of hands earlier when Jan asked who knows about Kubernetes but still I want to highlight why are we doing this. So we have one database it's really easy to SSH into the database and then run any DB in case of Postgres, configure it and run it and this is fine. If you have 10 databases you probably start to employ solutions to iCanceable or Puppet or Chief or something else that allows you to write scripts to once and then run them against your database cluster changing perhaps the name, the database size and some parameters and this is also fine but even if you have these you still have to go to the system team ask them to provision your server and this goes for every database. If you have thousands of databases you don't want to go to the system team for each of them. Instead you want a system that will recognize your demands in terms of for instance CPU, disk size and memory and provision servers for you and then run your databases on top of this fully automatically. What you need to do is to just declare your intention to do this using some human writable format like YAML for instance and then you expect the system to do this. Also as Jan mentioned we are running databases in containers. So what Kubernetes is is the open source Linux container management system it's developed by Google and Red Hat and many other contributors around the world and yeah it has a very rich metadata each object in Kubernetes can be tagged with the label. There are concepts of nodes and pods. I'm also quickly going through Kubernetes objects because this would be essential when describing how we use it. So node is your equivalent of a physical server. Pod is your application. A H pod can consist of multiple containers. Those containers share the network and the disk in a shared namespace and pods are scheduled on the nodes depending on the resource requirements. So each pod can declare which resources it needs and Kubernetes will figure out which node to run it and then it can automatically provide more nodes if your infrastructure supports this. Kubernetes can run on top of for instance AWS so Google compute cloud which can provide service automatically. It can also scale up your infrastructure automatically as well. So we also have so Kubernetes itself is basically a system which consists of a system node actually two nodes at least master nodes and worker nodes. So system nodes provide all the system software and worker nodes actually run your applications in the form of pods. There are also services and endpoints which answer the question of how do your application connect to your pods to your database pods. There is a service that basically points to one of your pods or maybe more than one but in case of Postgres database you really want for instance only a single master and there is an endpoint that actually stores the address of the pod where your application should go if it uses the DNS of the service. There is also stateful sets that are designed specifically to run stateful applications on top of Kubernetes because normally if your pod goes down and there is a volume attached to that pod the disk volume then this volume goes down and if your pod controller creates a replacement then it gets a new volume and has to initialize the database in you and if your database is big it doesn't work. So people from Kubernetes invented stateful sets that allows to keep the attachment of volume and the pod together so if your pod dies then the new one gets respond together with the existing volume and that means that your data directory is still there. So for an example there is a stateful set that defines three pods each pod asks for 100 gigabytes of disk space each pod mounts this 100 gigabyte volume as home Postgres pg-road and then there is a container inside this pod that starts it does a need to be on this home Postgres pg-road pg-data when it is a container zero when it's not a container zero it does a pg-base backup from container zero this is how you basically get your replication and then if postgres dies it just gets restarted by the stateful set so we essentially get some sort of have the ability for free from the stateful set idea. So this is shortly about kubernetes objects and now there are multiple ways to run postgres and kubernetes once we set up that we probably want to do this. So best stateful sets this is the setup that I just described there is a thing called helm that allows you to automate writing YAML manifests and basically write only essential stuff in it. There are solutions based on the bot pattern I will briefly touch what is it and there is a managed service like RGS or Google database cloud databases from Google or many other cloud providers that provide database postgres cloud-based postgres solutions. So there are stateful sets this is the simplest one you get the stateful set you get master port number zero if the port dies it gets respawned automatically by the stateful set and that is basically there are many advantages in it one of them is that it's very simple you can use whatever postgres image you need it works with any of them and basically it's easy to configure the service that points to these ports the service always goes to port number zero because port number zero is the master there are also disadvantages that we also came in in production one of the disadvantages is if the master port dies it takes some time up to a couple of minutes for this port to be recreated by the controller and during these couple of minutes your application is not available and this might not be what you want or what you mean by high availability in case of like heavy traffic databases you probably want seconds not minutes of downtime also if your volume gets corrupted which unfortunately also happens with EBS you're out of luck so you have to start in you and if there is a master corrupted then you have to do a manual failover and continue with so do a lot of manual interactions in your setup and it's also kind of hard to monitor because your port may be running but postgres did not start on that port and you have to employ some additional services in order to detect this so there is home which basically mentioned here because there are some home charts in order to provision postgres but home is actually just a provisioning system it does not have a separate way of running postgres on top of Kubernetes so you can use these stateful sets based scenarios with home as well it just allows you to write less YAML files so it's basically same as the other solutions and the raw managed database service this is the quote by Kelsey Hightower one of the well respected developers in Kubernetes community who says that if you want to run managed databases in Kubernetes then just look at if you want to run databases on Kubernetes then look at managed solutions so we basically read this quote but we did not obey it instead we still decided to run our managed solutions on Kubernetes why is that we don't want to vendor lock-in we want a lot of extent we use a lot of extensions and together with our own extensions and vendors usually does not allow you to install custom extensions there there's no super user access so something like pgdump won't work because it requires super user you can use new versions of postgres like alphas and bettas which are usually very stable it allows you to check new capabilities before the official release and perhaps find some bugs and benefit to the postgres community but usually vendors does not provide that and even in case of patches like security patches you want to apply them as fast as possible and not to wait for the vendor to do this and one of the main go one of the main disadvantages it's very very expensive it's rather more expensive than just running a bare instances on the same call provider so we have another thing we have something that is called that that is implementing the bot pattern called patroni and why do we need this well we need automatic failover as i described in case of the state full set if your port goes down then you have to recreate it and this takes time so instead we want the system to automatically figure out that the port is down and do the failover and the automatic failover can be done in their own way a lot of people are doing this in their own way for instance just doing two node two clusters master and replica if the master dies replica gets promoted but if the master is still there but there is a network partition replica gets promoted you get a split brain so this is not how we want to do this you want at least a quorum to decide who is the leader you want to learn what happens if the network is partitioned you want to account for that and you won't stone it should the other node in the hat so if your master gets demoted you want this master to not start ever again as the master while the other master is running so you want to wait split brain as far as much as possible for instance by using the watchdog so given that this is the schema of typical setup we have a primary that is replicated into the secondary and you have a quorum that is based on some consistency where in this case this is hcd a small application from course so the master always pinks the hcd and says i am the leader and the leader updates the leader regularly and the replica regularly asks who is the leader did the leader change and if the leader changes then the replica either initiates its own promotion depending on whether it's the house is not around or follows the other leader and if the master gets partitioned then it won't be able to ping the hcd saying i am the leader and in that case basically it gets demoted by itself but unfortunately posgras cannot talk to hcd or systems like that directly so for that you need a separate demon that runs alongside your posgras installation and talks to hcd both talk into posgras and decides at which at some time whether posgras has to be promoted and demoted and this basically is called bot pattern and we have the implementation of this bot pattern in python that is called patroni it's developed in open source by zelanda and contributors around the world it's quite heavily used by quite big companies and it supports hcd console zookeeper and it has recently been so recently a support for kubernetes has also been committed by one of our developers alexander kokushkin so we can patroni can talk to kubernetes directly and if you run like multiple pods on kubernetes each of them using patroni it will use kubernetes to decide who is the master and do automatic failovers in case your master has disappeared so we use such a thing as annotations it's inside the metadata in kubernetes so each pod has its own annotation saying providing some details about that pod we have an endpoint that points to the leader we use endpoints because endpoints are updated atomically so there is a guarantee that you want that only one leader is there and that another pod will not override the current leader because of the race condition and yeah we basically also use services to direct clients to the correct pods so there is also a compatibility mode that allows to use config maps and not endpoints in case you are running in some systems like open stack that does not allow easily easy modification of the endpoint and patroni is packaged in the docker image because kubernetes is all about docker or about other containers but we use docker so there is a spear that contains patroni multiple postgres versions some extensions and it has rather small size so it doesn't take a lot to download it on your server before provisioning one of the interesting things that also contains is the pro methodication for postgres that I will touch briefly later it's also open source so you can get it in github and this is how we run one server but the question is how we run soldent of servers how we run databases as a server we don't want to write manifests to provision each of our clusters each of these soldents of them even if you have patroni to do the failure we still have to write those manifests instead we have developed a goal application that implements a so-called operator pattern so instead of doing typical DBA tasks by hand we have a program that does this for us which basically starts our databases that maintains them syncs kubernetes subjects allows us to update configuration of this particular cluster and well ultimately it can also delete databases as well so we now use it for staging and production clusters in side-surrender and I think some other people are using this this is part of the HACVIC project by one of our my colleagues Murat Kabilov and it's also open source you can also get it on github so how it works it works well by layer starting from operator that starts spots with a spear image and provides environment variables for spear spear gets these environment variables and generates patroni configuration then patroni gets started and then using the patroni configuration patroni starts posgras figures out if it's the first node in the cluster and if it is the first one initializes the cluster if it's not the first one gets old spears and initializes replication and continues running quite stably and does fail over or if necessary so this is the schema basically how it works the database deploy deploys the manifest the cluster manifest goes to the operator port the operator port uses the cluster manifest and its own configuration and something that's called infrastructures to create kubernetes objects basically it creates the actual stateful set with spots spears inside it creates the service to connect client applications it creates the secrets so that passwords for the client applications are not propagated anywhere in the plain text but read from secrets I think I briefly mentioned this and what the developer needs to know is only the small yaml manifest this is how small is it so basically you have to provide a name the number of instances for your cluster version of your posgras and if you want you can provide users and databases that operator will create it for you and that is basically it nothing else you provide this manifest and every developer not even not familiar with posgras can do this you provide this and we create you the operator we create the operator will create you the posgras cluster higher availability of posgras cluster will be the same with the amount of instances that you asked and that will do at like followers for you so this is the operator configuration each operator runs in the kubernetes cluster and there are some configurations like what is the docker image that is used for all the posgras cluster right now we only support our own speedo image we cannot work with stock posgras image there is something like infrastructure roles that I will touch afterwards super user username replication username how often do we go to each cluster and check that all kubernetes objects are in place that they are not deleted and if they are not deleted that we will rethink it there is an API part that you can interact with the operator and ask for the internal state of it for monitoring purposes and there are some other details at the portage creator so we also can create roles for you and there are multiple roles some of them are described in your cluster manifest those are roles that will be created for this specific cluster there are also such I think as infrastructure roles for instance in your cluster you want to monitor your posgras pods and for that you need posgras to have a certain user that the money for indume connect to posgras and this user should be the same for all pods and for that we have such I think as infrastructure roles that gets injected into each posgras cluster that is created by the operator and how do you manage human users we have such a thing as the teams API teams API is configured in the operation configuration you basically get a URL this URL is OAuth protected so you need a secret inside Kubernetes that contains the OAuth token and this URL delivers you the list of members of a given team a team is specified in the cluster configuration and once this list of users is delivered they are created in your cluster and they are created in a special way so that you cannot authenticate using the password but you can also you can only authenticate using your OAuth token so how does that work we have a PEM module that is also written by Alex Kokushkin it's open source and it basically goes to an endpoint in OAuth and validates the password and token that the user supplies against this endpoint and either allows you to connect to the database as a node so essentially it's equivalent to very long passwords perhaps I think our tokens are like 600 characters or something like that and those tokens are expiring every hour so you never had an issue that you have a password that never expired and the colleague went elsewhere and he's still in the system and cataloging so this is automatically expiring very long passwords the only problem is not all not all Postgres clients support this for instance if you use .pgpass in your Postgres there is a limit I think of 255 characters for the password but if you use something like an environment variable .pgpassword then it will work so it also can supply some arbitrary other parameters to this to the endpoint for instance you can also validate that the user is indeed part of the team that it claims to be part of and it's configured in the imperative configuration so there is a PAM configuration that points to this euro where PAM should validate against and there is the PAM role so in pghba.conf the operator adds this line host as a cell all plus users plus means that the user has to be part of the users group then also we want to cover databases of hosts and PAM so this means that those users that are part of the users group will get through PAM and will only be permitted if this PAM module allows them so this is basically how we authenticate human users and we also do something else from the operator as I said it's a replacement for your DBA and for instance we can dynamically change the disk size when running on AWS the operator runs on top of Kubernetes Kubernetes is kind of platform independent but in case you are running on AWS as we are we can also dynamically change your disk size so if you in your manifest say something like the volume size like not one gigabyte but change it after you deploy your cluster to for instance five gigabytes then the operator will come to your cluster call EBS API to resize your volume and then get inside your every port and run a resize to a face on the actual file system to make the file system aware of this resize and you can grow your cluster for well almost for free you have to still pay to AWS for EBS size so it basically solves the problem of clusters right now of disk space and we also support such thing as Kubernetes cluster upgrades why do we need this as Kubernetes is a very very well creatively yarn technology people needs our system team needs to need to keep in pace for the like Kubernetes development so Kubernetes is developed very very fast and they need to upgrade our cluster to the new version of Kubernetes also very often but each upgrade means that you have to queue your nodes and then create new ones and that is very disruptive for Postgres databases running on these nodes obviously and one of the issue that we came across is that when the node is queued then Patroni will detect Patroni will do the automatic failover if the master node if the node where the master is running is queued but this automatic failover may land the master on another node that is still the old one that is still also requires to be queued and that means that during the cluster upgrade we may have multiple failovers which is very disruptive for client applications that connect to this database because they have to handle these cases multiple times so we try to minimize this and we actually succeeded with the operator and Kubernetes so we try to avoid multiple failovers and for this there is a convention that each node is marked with a special label that is configured in the operator is the node readiness label and then the operator finds out that the node readiness label disappeared for instance when our system team decides to rotate to to upgrade the cluster a certain set of the nodes have this label removed and then the nodes are drained meaning that all containers from this node are asked to be deleted and during this time the operator detects that the node has to be decommissioned and finds out all master pods running on that node and moves those master pods to the node that is not yet to the node that is healthy basically does the failovers to this master node of these master pods and in the master in the pods we have a definition that the pod can only be run on a ready node we use the feature of Kubernetes that is called node affinity saying basically that this pod can also run on the node that has this life cycle status ready label in our case and the operator also for each pod defines the pod disruption budget that says that if you have masters running on this node you cannot kill this node when it's drained so the Kubernetes life cycle manager this is our component that does all the cluster upgrades essentially waits until the node runs out of all master pods before killing this node and this is basically how we make sure that we have only one failover during cluster upgrades so this is basically it this is the list of open source applications that we have feel free to take a look use it and send your feedback and if you have any questions to me or to Yan then I will be happy to answer them now thank you so it's not a technical question but more on the human side how much time it took you to be this solution and design and how many people were behind this so the vision for deploying Postgres on Kubernetes started January 2017 so basically Kubernetes came around and we made the decision to go there we the the key part was this initial requirement that the whole deployment including an application and a Postgres database comes straight from Git so basically nothing else no UI in production nothing all from Git then a deployment pipeline and after that I think we had the first working operator in March and it was like two persons working on it kind of Olexia and Murat from January to March to see the first production deploy or not production deployments but the first Postgres deployments on Kubernetes and the the amount of features now from volume resizing to figuring out how to do cluster upgrade basically has grown until like autumn last year I would say with still basically two people doing the coding and the development does your automatic systems also make database backups yes we um basically Petroni acts with Wally so we basically do Wally and ship like a nightly base backup to Amazon S3 and we do continuous archiving of all the wall segments to S3 during the day and we store basically five days on S3 for every cluster we have so this is enough basically when you wake up Monday morning figure out you broke something on Friday you can go back and for other clusters where we need more we can store more yes and regarding the cluster is it the standard Postgres streaming replication or something else yes it's all standard streaming replication in Postgres and we are now giving like the latest feature we have is basically giving people this fork and clone and this means basically in the manifest you can say copy from this database at a given point in time and this will basically enable other users and team without talking to us to trigger a point in time recovery to any system they are running have you done any performance tests comparing to bare T2 or like some EC2 instances or our server less and something else no so we are not personally I'm not thinking that like every Postgres database Salando is running will be on Kubernetes end of this year like I mentioned earlier we have a lot of deployments using i3 instances because of high IO demand and Kubernetes doesn't support local storage right now so if this changes obviously everything goes to Kubernetes but if we can tackle the majority of databases like 95% where we know from all the metrics we have that they're either IO demanding nor anything else this basically frees us from an amazing amount of work and this is basically the goal of this so the remainder we can still run on high performing plain EC2 instances or whatever we need basically so we haven't measured this in particular do you deal with multi regions or no no which basically can still be done when you think about we do recover our clusters from the data center on AWS with S3 in the middle but we just don't do it right now where are your database clients are they also in the Amazon cloud yes most of them are well let's say most clients are where the database are so as you saw earlier we have applications on Kubernetes on plain EC2 and in the data center and they are mostly local to the database yes I think I'm lying I told earlier Alexi would answer all the questions but he gave this microphone away that's fine any advice for somebody running Kubernetes on bare metal that doesn't have that easy access to network volumes I'm supposed to say no I don't know maybe earlier I said we trust Patroni and post for streaming replication to run stuff in production on I3 which means like we lose volumes once a month and do a failover and we build a new replica automatically maybe this can help also in your basically local disk environment so run but basically run outside Kubernetes no you have to basically trust that your pods and local storage lives long enough postgres to do the base backup and recover and depends on your risk I guess okay thanks and you know the other people using your tools have experience running it on Azure or Google cloud I'm told on Google cloud yes we have also like a few contributors on the operator already we have a few discussions and issues with people around Google cloud Azure we have three more minutes for questions I guess and if there is more we can talk outside and probably we can also show some stuff if you're really curious if we have been like just showing screenshots here in the beginning you mentioned that you have lakes which are running one hour behind the master yes how do you do this I think there is a bash script wrapping the recovery command that just looks at time age of war file you can always you can also do this build in postgres since I think 9.3 and 9.4 with a way of running postgres classes with streaming replication that are duet against the master for a certain amount of time just in recovery command great then thank you for listening again or if you have any questions reach out on twitter email I think we're quite active and good in answering there too