 Welcome to Turning PostgreSQL into a Cloud Native Data Service, where a speaker will discuss feedback gathered from operating a full life cycle automation for PostgreSQL across all major on-premise and public infrastructures. How enterprise requirements can be met at scale using state-of-the-art open source tools, an open source-based toolset to continuously ship new PostgreSQL versions into production environments, and finally, strategies to organize zero downtime deployments, horizontal and vertical scaling, monitoring logical backups, and point-in-time recovery with off-site replication that works at scale. My name is Lindsay Hooper, and I'm one of the PostgreSQL Conference organizers, and I'll be your moderator for this webinar. I'm here with Jillian Fisher, the CEO of NE9s, who has dedicated his career to the automation of software operations. In more than 15 years, he's built several application platforms. Most recently, he's been using Kubernetes, Cloud Foundry, and Bosch. Within the field of platform automation, Jillian has a strong focus on data service automation at scale. With that, I'm going to hand it off to Jillian, and you can take it away. Welcome, and hello, everybody. Turning PostgreSQL into a Cloud Native Data Service is a complicated matter, as PostgreSQL itself has a non-trivial life cycle. So maybe let's have a brief look into how this can be done. So yeah, I'm from NE9s. We build platforms, among others. We also automate the hell out of data services. So that's where a lot of the inspiration for this talk comes from. In this talk, I will connect to a prior talk that has been held at the Postgres conference about two years ago. I will recap a few of the concepts so that for those of you who are new to the data service automation part of Postgres, we'll understand the term knowledge yeast and the ideas. So feel free to ask questions if things are not really clear. And we'll then, after looking into the concepts, we'll have a stronger focus on Kubernetes and look into Kubernetes, what are the building blocks that can be used to automate data services, in this particular case, Postgres. All right, so there are many ways on how you can do automation. So it makes sense to define a few constraints so that we have a better context for a talk like this. So first of all, you could automate the operation of a very large and important central database of your company. And that is very different from the idea that we are following this talk, where the focus is on operating a large number of data services and data service instances. So in this particular case, the context would be, for example, the modern application developer platform could be Cloud Foundry, Kubernetes, OpenShift, or anything like. In these environments, there are often a number of applications, hundreds of applications, or even thousands of applications. And application developers have demand for several different data services. A relational database is often in the top of the priority list, together with message queues, analytic servers, document database, and so on. So when looking into how such a database can be automated throughout its entire life cycle, it turned out that the leading paradigm is the on-demand provisioning of dedicated service instances. We'll get into the detail of that. But the basic idea is that each service instance is represented as a dedicated Postgres server. Whether it's isolated with a VM or container doesn't matter, whether it's clustered or whether it's a single instance, single pod or single VM doesn't matter. But it's important that we distinguish the on-demand provisioning of dedicated Postgres servers from, for example, having a shared Postgres server and then serving databases on demand by creating a database in a shared cluster. It's a different approach and the automation would have a very different profile and very likely also be vulnerable to bad neighborhood issues. So the automation should work across infrastructures. So when building application developer platforms for global companies, it is often the case that they will go into different regions of the world. For example, there are at least three different larger areas or regions that they will come with their own legal constraints. That would be America with North and South. There will be Europe and Asia Pacific. So in our experience as a company, we've seen that clients have due to legal reasons or due to the affinity of their customers have a certain affinity to certain infrastructures in certain regions. So, for example, some customers like Azure more than AWS, but both are rather popular in Europe and in North America, whereas, for example, in Asia and China, for example, AliCloud is very popular. So when building an automation solution, it will take care of the life cycle of your databases and you're part of a global company, then infrastructure agnosticism is rather important. Now, with the rise of application developer platforms such as Cloud Foundry and lately Kubernetes, the integration with these platforms is also very important so that there's a standard interface for application developers to utter their intents, to create databases, to perform version upgrades and guide those instances throughout the entire life cycle. Now, as I mentioned earlier, it makes a large difference whether you only maintain a few couple data service instances or whether there are thousands of them. So all the concepts discussed throughout the talk are actually meant and proven to work at that scale. And somehow the implication of that would be that striving for full life cycle automation is necessary for the application developers who want to create their database instances without waiting for platform operators or SysOps or DevOps or database administrators to get their databases ready like it was in the past. And on the other side, platform operators who often carry the responsibility, for example, to close global CVEs rather quickly. So if you remember Shell Chalk, for example, we had environments with more than 1,000 virtual machines. Today we have environments with roughly 10,000 virtual machines. So if there's something like Shell Chalk, you need to have the possibility and the automation in place to upgrade a lot of virtual machines in an automatic fashion. All right, so part one of this talk then would be to recapitulate the basic concepts of on-demand provisioning and make a few steps into a terminology. So as I already explained, the on-demand provisioning of dedicated database service means that whenever a client or an application developer requests a database, what he will get is a dedicated Postgres server represented as a pod, a set of pods, a virtual machine, or a set of virtual machines. This example is using Cloud Foundry. In Cloud Foundry, you use a command line utility called CF, creating a service, in this case a Postgres, with the service plan single small, and name the instance single Postgres 1. So that would talk to the Cloud Controller of Cloud Foundry, which will then delegate the creation of the service instance to a so-called service broker, which will trigger the automation. We will later see that in Kubernetes, this will work similarly. You can integrate service brokers into Kubernetes, but there could also be a different interface, for example, a custom resource definition. However, it is important that there is such an interface that is rather uniform and homogenous so that each application developer knows how to create a database easily. And if you're having more than Postgres alone, it would be nice if other data servers could be created similarly. So in this case, we are using, in the Cloud Foundry use case, we are using the service broker, which will then trigger the Postgres automation, which will then provision dedicated virtual machines. In Kubernetes, we will later see that the architecture will look a bit different. You will talk to the Kubernetes API. You will create an object of a custom resource, for example, that will then, in turn, hit a controller that will create Kubernetes resources, such as a stateful set, which will then create pods rather than virtual machines. But on the conceptual level, the approach would be similar. In this case, the Postgres automation will also use an automation technology called Bosch that will, you could compare it to Kubernetes, but using virtual machines instead of pods. So it's a declarative tooling that allows you to orchestrate a large number of stateful deployments. Another important use case is that you create a service and the service is clustered. So in this particular case, the service instance would consist of several virtual machines. In Postgres, for example, as in Corona streaming replication would be set up, the cluster would somehow have to be monitored in order to make sense of such a distributed instance. So in the end, this will lead to some kind of application runtime, or several of them. Could be several Kubernetes clusters. Could be a Cloud Foundry or a combination thereof. And a lot of applications will then connect to different dedicated instances, whether they are clustered or whether they are solo doesn't matter. But it's going to be a large number of applications and a large number of databases. So if you think about how can you achieve lifecycle automation or Postgres, the question is, what are the aspects of lifecycle automation? And in this talk, we distinguish two different perspectives. One perspective would be taking care of the automation itself. So whatever code you use that will then trigger a Kubernetes deployment, creating a stateful set, or a Bosch deployment, that code needs to live somewhere that's very likely to be non-trivial. And you also have to take care of this automation's lifecycle as well. This will include shipping the data services themselves with particular versions. So if new versions arrive, you will need to somehow update your automation and ship new versions of that. So that would be the lifecycle of the automation versus the lifecycle of the data service instances themselves. So if we think about that picture we've seen a few seconds ago with a lot of applications and a lot of data services, application developers will also have to create and upgrade their databases from time to time. So let's say a new Postgres version is released. In our case, I think version 13 is on its way. And application developers are, let's say, planning to use a new feature of a new Postgres version. Or a very old version is becoming out of date. And upgrades are necessary. They need to have the tooling at hand to, for example, create a major version upgrade for Postgres. Or if a patch level version shows up, they'll have to update their service instances as well. So the data service automation lifecycle can be imagined as a continuous delivery pipeline. So I just used a schematic from our internal structure. The overall goal of such an automation would be to minimize the delay in that delivery pipeline. So to be precise, if a new open source Postgres version is released, we want to be able to build a new version of our automation release as quickly as possible. So in our case, we do that automatically. So whenever there's a new Postgres version, especially if it's a patch level version, we wrap it up into a Bosch release, if we're talking about the virtual machine side of things, and run our excessive test suite that will provoke many different failure scenarios of Postgres in the context of our automation. For example, it will create single and clustered instances. And for clustered instances, it will simulate a failing primary to see whether the failover works properly. And so if, for example, in a new version, there are API breaking changes, the test suite going through all the major use cases of the automation, as well as inserting data from the using, so to say, simulating use cases from the application developer perspective, we'll take care of that no obvious problems will slip through the automated testing. Now, you can imagine that this will fail from time to time. This is when engineers will have a look at it for sure at latest. For major versions where API breaking changes are to be expected, the automatic build of releases is rather unimportant because it is very likely that people have to look into the differences between those versions anyway. But for CVE updates and patch levels, that first part of the automated build pipeline is rather important. So once we have been through that quality insurance, we've created a Postgres release, in this case, the AnyLines Postgres release, which will be located at the repository where the second part of the delivery pipeline will take over. So imagine we're selling this automation to clients and the client has several platform environments. These environments will watch the AnyLines Postgres release repository and download those releases into staging or into the test environment and run the tests there, which closes the gap between our infrastructure and the customer's infrastructure. So for example, we could be running on vSphere or AWS where the client runs on OpenStack. So in those cases, the client will run the test suite, find out whether it will work and qualify it to production or anything like it. So at the point this deployment has been done, the automation release has been put into the client environment. And from this part, application developers will usually be notified by the client so that application developers can then upgrade their service instances. Well, some clients do have different policies where the platform operators have more freedom. For example, if there's a patch level, they can trigger patch levels automatically, which can be scripted. So if you're building an automation for Postgres, I highly recommend having such a continuous delivery pipeline where the first part of the pipeline creates new releases of the automation and the second part picks up those tested releases and deploys them into the actual platform environment. So as I said, the automation lifecycle is one perspective and the other perspective is a service instance. And if you look into Postgres, there are a lot of different things you want to do with the Postgres. For example, you want to provision a database. You need to install the database into the virtual machine or into the pod. You need to configure it by adjusting configuration files. You may want to set up replication, which includes setting up a replication user, again touching configuration files and maybe invoking commands such as pgbase backup. If you want to add monitoring and alerting, if monitoring finds out that something goes wrong, deal with logs, for example, adding log swings, same for metrics, create database, add database users, and so on. So I'm saying Postgres isn't a particularly easy data service to automate. There are lots of different use cases that need to be considered in its lifecycle. As a general remark on this, we have, let's say, we call it a reference guide on data service automation. But when we start a new data service, we usually pick the most important use cases, start automating them, and then add more depth to those use cases over time in subsequent milestones and releases. So in this regard, it's automating a data service is like drawing a fractal by adding depth each iteration. In my particular experience, the choice for the right automation technology is essential. I think with Kubernetes and its gaining popularity, I don't have to spend so much time in explaining this because Kubernetes becomes an effective standard. But a few years back when we've been using Chef and Ansible and technologies like that, for example, switching to technology like Bosch was a very important step as Bosch introduced declarative lifecycle management rather than the more imperative side of things. And with Kubernetes, I'd say we are pretty close to declarative lifecycle management as Bosch did it with VMs but using containers, which, well, let's say, has its particular advantages and disadvantages, but we all come to that. So in my opinion, first of all, the automation tooling you'll use should be declarative because you can't keep track on how data service instances evolve over time by uttering a sequence of little commands towards a service instance. This will work for, let's say, a few data service instances, a few Postgres instances, maybe even a few hundreds. But at a certain scale, chaos will actually prevail and the expected state of the system will not be necessarily as the desired state should be. So let's say I'm using Chef and I'm running a sequence of cookbooks that make assumptions about the virtual machine state. At some point, the actual state of the virtual machine is a bit different from what is anticipated in the cookbook and things will go south. And there's a certain likelihood, certain probability that you end up in an unfixable state because making root-course analysis will be much too expensive. So the combination of having a declarative approach where you design or where you draft a system and you'll hand over the desired way that this distributed system should look like, whether it's in a Bosch lease or in a Kubernetes stateful set, makes things a bit different because it often comes with an increased repeatability. So for example, in Bosch, similar as in Kubernetes, when you describe a stateful set or you describe a postgres streaming replication clustered instance in a Bosch release with a Bosch deployment, then whenever you make substantial changes to that service instance, you'll basically redeploying it, which means that you're destroying the virtual machines and recreate them or you destroy the pods and recreate them. And on the first side, this looks like a wasteful activity because it takes a lot of time for a virtual machine that can take up to 10 minutes, depending on the speed of the infrastructure. For pods, this will be faster, assuming that the underlying container images are that big or that they are cached on the Kubernetes nodes or that there's a local registry with a very large bandwidth. But the point is that if you deploy a stateful set or you deploy a Bosch release, 1,000 times you will get exactly the same result because it has a very high repeatability. Another aspect and somehow related is predictability. So if you can be sure that something works, if you do it over and over again, it also is somehow predictable because it does the same thing over and over again. But predictability has other facets too. So despite of Bosch's behavior or Kubernetes' behavior, your own automation also has to be predictable. So for example, we've been through a rigorous testing from one of our banking customers. They've been doing relational databases surely longer than we have, but we have created a piece of automation that fascinated them and they were looking into that very, very closely. And they came up and said, we want your solution because it is so predictable. So whenever there was a failure case, they simulated, the system behaved in a very specific way every time they tried it. And so I learned in that particular customer relation that predictability is key, especially if you look into a scale of thousands of instances. So scalability is obviously a very important factor in cloud environments. So if you remember that we are talking here about the on-demand provisioning of dedicated instances, let's think about scalability for a second. And the point here is that if you create a shared database server and you'll serve databases as service instances, at some point, this server will be full and highly utilized. If instead you're handing out dedicated pods or stateful sets or virtual machines or sets of virtual machines, then where's the limit of that? The limit is your infrastructure. So there is no inherent limitation in the scale of on-demand provisioning dedicated service instances, which makes this maybe the most, the single most architectural pattern with the broadest spectrum of applicability. You can create cheap and tiny single pod containers as stateful sets, but you can also create full Kubernetes node covering availability zone spanning highly available replicating instances. And that with a single automation code base. So that's a nice thing to have. So just to remind you, dealing with state is essentially important when dealing with a database. And I think this is a common knowledge today, but for those who are just coming from or new to this, the most important paradigm here is to store state on a remotely attached block device. In Bosch, they are called persistent disks. And in Kubernetes, they are called persistent volumes. But it's basically always the same. It is a remotely attached block device that will provide it or that will separate the ephemeral virtual machine or the ephemeral pod from the persistency of the data. And so if you look at virtual machines, you will see that there's a virtual machine running some kind of operating system. And it will attach this storage volume that is very likely to be hosted on a highly specialized, highly available storage server itself. And in our recommendation, this also will be distributed across availability zones. So let's say you have a three node service instance, then each virtual machine will be residing in a different availability zone. And very likely, they will also refer to a different dedicated storage server in the background. With Kubernetes, that can be achieved by assigning certain affinities and tains to the pods so they ensure that they are distributed in your Kubernetes clusters in nodes located in different availability zones. So whether using virtual machines or pods, you'll have to do the same thing in order to survive availability zone outages. So I've been doing operations of service for my entire career, which started around the year 2000. And since then, I've seen several availability zones failing even in tier one data centers at tier three data centers where there's a lot of redundancy because there was a chain of unforeseen failures. For example, there's a simultaneous outage of the power supply and the battery backup units and the diesel generators because they haven't been used or tested in a while. So the data lifecycle has been decoupled from the virtual machine and the pod lifecycle. That's one of the biggest takeaways here is that the virtual machine and the pod becomes disposable. So in putting it into short terms, ephemeral VM and persistent disk or ephemeral pod and persistent volume, if you really go down the containerization road, you may have discussions like, do you need replication nowadays? Because the cell feeling of pods in a stateful set, for example, when a Kubernetes node dies, then the pod will be recreated in the same availability zone on a different Kubernetes node. And it can be a matter of seconds. So why would you deal with replication if this is just a hiccup for the application? And the answer to this is, well, if the entire availability zone goes down or the storage server loses its data, well, then it's nice to have a up-to-date copy of that data. So replication is still a thing. But the advantages are melting away to a certain degree. All right, infrastructure agnosticism, it is very wise to use a technology that makes this abstraction from infrastructure. And in this regard, Bosch is a bit, well, let's say, in the advantage because Bosch has a true support for multiple infrastructures. So in Bosch, there is a pluggable concept called the CPI, the Cloud Provider Interface. And this will basically talk to the infrastructure API, such as AWS or Azure. And, well, we'll create virtual machines using the API. Whereas in Kubernetes, you'll have to get a Kubernetes running on a certain infrastructure. And Kubernetes itself doesn't help you to do that. There are projects such as the cluster API that will provide you with an API that you can then use to create Kubernetes clusters automatically. But, finally, this will also run in a Kubernetes cluster. So you have a chicken egg problem. You'll have to get a Kubernetes somewhere. But Kubernetes becomes ubiquitous. So maybe this problem will be solved in the future. So the difference between virtual machines and containers is that virtual machines provide a stronger isolation and, therefore, a stronger neighborhood protection. Whereas the containers have less overhead, they start up way faster, which also creates faster development cycles when developing automation. But they also have weaker isolation. You can counteract that weaker isolation. So that weaker isolation is basically disk.io and network.io. So you could collocate two Postgres instances, two Postgres pods of a stateful set on a single Kubernetes node, and potentially see that they will impact each other. You can work around this by affinity, entire affinity, and other placement or influences of the placement behavior of the scheduler. But it's additional effort that is not necessary when using virtual machines in a non-overcommitted infrastructure. So high availability and cluster management involves replication. Well, I think you've been through that in several talks. So to keep it short, we are talking here about just asynchronous streaming replication. So that gives you a primary node and secondaries. And in order for that to work, you need to have a replication cluster manager that would do failure detection, trigger the leader election, perform the leader promotion, and therefore avoid split-brain situations. So without going into too much details, there are solutions for this already. So Petroni with Spillio, and even there's a Kubernetes operator, there's been much progress in this field. So the basic challenge what these tools have to solve is solving the so-called Byzantine generals problem where a node that sees a failure of another node, so let's say a secondary that sees a primary not responding, it's hard for this node to distinguish whether it's the network communication that isolates him as a node or whether it's truly the primary that has failed. So there are in distributed systems, there are solutions for that called consensus algorithms. The most two popular ones are Raft and Paxos. Raft is a bit easier to understand and also well presented. So there are implementations all over the place, ZooKeeper, ETCD and Kubernetes itself are implementing them so that to such as Petroni, they don't have to go down and build their own consensus algorithms, but they can just rely on components that are there. So there are certain failure scenarios you want to cover. For example, a failing standby would be, well let's say one of the easier ones. So let's say you have a declarative automation technology such as Bosch or Kubernetes in place and you have a replica set or a Bosch deployment that has three virtual machines or three pods and one goes away. This technology will recognize that and immediately resurrect the missing virtual machine or pod by recreating it. In Bosch there's no strong identity for each of those virtual machines whereas in a stateful set there is. So for example, if the secondary goes down it will come back as a secondary even if your automation is unaware of it. However, the more interesting scenario would be a failing primary because if a primary goes away then the application will see a difference with asynchronous threat application. Applications usually use the primary for both read and write commands if well they can do read, write, splitting sometimes but in most of the cases the applications will see an error if the master goes away. So in this case it is important that there's automation in place so we'll do the leader failure detection and a new leader is elected which requires to form a stable quorum a consensus about who should be the new leader. So you need an odd number of notes and a good consensus algorithm and new leader can be elected and then you have also to take care of how this leader promotion works. So some of the challenges you'll have to solve here is make a pointer point to the recently elected leader and ensure that it is impossible that the prior leader is still around and that they are still incoming requests because multi-master situation can result into data inconsistencies that only with human brains can be resolved and that's questionable whether this is possible for large datasets because it's a lot of work. So this is to be avoided and this is where your cluster manager is very, very important. All right, and one thing to take away here is that zero downtime updates actually use this upgrade, this failover mechanism to achieve zero downtime updates. So the idea of such an upgrade would be to remove a secondary upgraded remove another secondary, upgrade it, then trigger a failover so that the master switches over to one of the current secondaries so that you can take out the old primary as a new secondary. So that will reduce the service impact. The application will see just a few hiccups and it's basically just the same failure case as in the, it's the same use case as in the failure case. So the notes will go down, they will be recreated and if your cluster manager works properly these new nodes will join the cluster with their new anticipated roles. So this way the application can continuous writing and reading from the server all they will see is a few hiccups during the update when the notes are taken down or the failover is triggered. Well scaling obviously is with on-demand provisioning it's quite a natural thing. You don't have to explain a lot. You can create as many virtual machines and parts as your infrastructure has resources. The vertical scale out is a bit more interesting. For example, if you want to turn a small service instance into a large one, the vertical scale out will work like this in Bosch. You take the virtual machine and unmount the persistent disk. You destroy the virtual machine. You create a virtual machine with more resources and even a larger persistent disk copy the data and destroy the old persistent disk. In a stateful set when it comes to pods this would be the same when it comes to the persistent disk. It depends on the persistent volume provisioner you're using whether resizing of disks is possible and easy. So this is where Kubernetes might become a bit more tricky because it does not truly abstract from the underlying infrastructure totally or entirely. And therefore, well let's say there can be infrastructure related issues. For example, in the provisioning and resizing of persistent volumes. So you should be aware on what are the limitations of your volume provisioner. Backups, backups are important. So if you zoom into a service instance then you can collocate a backup agent that will be the actor that, for example, takes commands from a central place, for example, where your backup jobs are scheduled and then perform the backup in a similar way like this. So one of the ways would be to have a stream reader that will create a backup, put it into a filter chain that could do compression and or encryption and then write it to an external object store. If you do point in time recovery, well, this may look a little different but it's basically a similar approach. Yeah, so in this picture, you just see that there's a backup manager that will trigger the backup and ensure that the backup is streamed to the object storage. So the automation architecture in the case of Bosch somehow looks like this. So there's a Cloud Foundry environment talking to a Cloud Foundry Cloud Controller which will talk to the service broker which implements the open service broker API and therefore creates a standard interface on how to create services. Postgres could be Redis, could be RabbitMQ and the deployer will translate the incoming request which is basically a data structure with how the service instance should look like. For example, should it be a single instance or should it be clustered? What size should it be? And then talks to the underlying automation to actually perform the deployment. So this could be Bosch or Kubernetes. Now with the time left, we could dip our toes into the world of Kubernetes and see how data service automation can actually work there. Be aware, there are already solutions doing that. So this part of the talk is more focused on explaining what concepts are used so that you can judge. So either you develop it yourself or when looking at an existing solution judge the quality of the implementation by just looking at how it's done and you can make your own decisions on whether you like it or not. So the term service instance is defined in the service broker API. It basically is the database and with on-demand provisioning of dedicated instances. This would be a portal or virtual machine. The service instance can be built using Kubernetes natives as a composite of headless services, stateful sets, config maps, secrets and possibly jobs. So maybe talk about this a few seconds. Config maps and secrets in Kubernetes entities that you can create using the Kubernetes API and that represent simple key sets of key value pairs. So a single config map may contain several key value pairs. The secret is basically pretty similar to a config map but is dedicated to store sensitive data. Finally, secrets are per default not encrypted in Kubernetes, but I'm pretty sure it can be plugged in so it's always wise to use a secret. There are also special kinds of secrets such for example, secrets for storing certificates. So there's a bit of structured key value set so to say, but you can imagine config maps and secrets as key value pairs, sets of key value pairs. And in Kubernetes you can mount them into a pod either as environment variables or as files. So this is very nice. So as if you mount config maps as a file there will be the config map itself will be represented as a folder and each key will be represented as a file and the value will be its contents. And the nice thing about this is that this even reflects changes of the config map. So you can mount a config map into a pod, change the config map and if the process rereads the file we'll also reflect upon the changes. So it's something that you can use for example to slurp in configuration files and then mount these configuration files into your pods so that they don't have to be contained in your container image. Well, secrets are obviously necessary if you want to create let's say store the credentials of your replication user then you'll create a secret and you can also mount secrets as environment variables or as files. The single most important concept in Kubernetes for data service automation is the stateful set. It has been introduced I think in Kubernetes 1.15 I'm not really sure about this but it is, well, let's say a new concept to Kubernetes but it's stable, it's been there for a while but it hasn't been there from the beginning. In the beginning there were pods, replica sets and deployments. A stateful set is not using replica sets and it's not using a deployment but it combines behavior of replica sets in the sense that you can determine or define the number of replicas. Basically you'll have template for creating a pod and by saying you wanna have three replicas you will get three pods in the stateful set. But in contrast to replica sets, a stateful set they can perform rolling upgrades similar to deployments but a stateful set is much more than a deployment because they'll provide stable identities to the pods in the stateful set. So the pods in a stateful set they will have numbers starting with zero, let's say zero, one, two and in combination with the headless service this will generate a stable network identity in form of DNS service, in form of DNS entries. So from within a Kubernetes cluster you can refer to the pods of a stateful set by using individual DNS names that will be automatically created for you. Another part of the stateful set to mention is that each pod in the stateful set will receive a unique persistent volume and whenever the pod goes away it will be recreated with the same identity and it will have the same DNS entries and it will have the same persistent volume. So for example, if your stateful set is a static replica set with a primary and two secondaries the primary will always come back as a primary with a DNS entry, the network identity of the primary and its data. So the DNS entries are basically using services. In the case of a stateful set you're using what they call a headless service which is different from the standard service in the following way. The service per default in Kubernetes is a layer four load balancer. So the service itself has its own IP address and then makes a reverse proxy towards the back ends whereas a headless service comes without layer four load balancing and is only a DNS entry. So the DNS entries are performed against the internal DNS of Kubernetes and they are placingly fast. So that they can be used for example as pointers to pointing to the current primary which is interesting when you implement your failover mechanisms. Jobs is basically what the terms adjusts. There are jobs and ground jobs. So it's basically running apart several times or once or according to a certain time schedule pretty similar to what Kron would do for you. So with all these basics, native structures of Kubernetes it is possible to create a service instances. So you could create a Postgres stateful set and that stateful set would represent let's say a primary and two secondaries in a streaming replication. Wonderful, but how do you crown access to such a service instance? If you're familiar with Cloud Foundry and the Open Service Broker API you know that there is a concept called a service binding which is a resource in the service broker. It has a rest endpoint that will allow to create a link between an application and a data service instance. So you deploy an application and you want to crown this application access to a particular Postgres instance then you create a service binding. And this will create a unique set of credentials so that if you have two applications you have two service bindings and whenever one of the applications is somehow compromised you remove that one single service binding without affecting the others. Because this is an API creating access to data service instances is part of the automation. So when you look into existing solutions you should also think about application access that should be part of the life cycle management. In Kubernetes itself there is no equivalent to a service binding but there is a secret. And sometimes people ask me isn't a secret basically a service binding and the answer no it isn't because a secret is just a set of credentials and it doesn't create the credentials. It's just there to store it. So you can create a secret that has the Postgres replication username and also stores the Postgres replication password but it doesn't create it for you and therefore you need extra logic. So with Kubernetes you'll write plenty of YAML and you also need to place additional logic because just this YAML won't be enough. And the point where you want to create something that has additional semantics for example you want to create an entity that's called Postgres that will create a stateful set for you in a headless service and secrets and config maps. You want to give that thing a name and make Kubernetes manage it. So similar to creating a service instance using the service broker you can teach the Kubernetes API you can teach the Kubernetes API to accept objects of new types and that is called a custom resource definition. It's basically specifying new data type that will then be owned by the Kubernetes API. So if you have a custom resource definition like this which is here described as YAML file you can see that this particular one is called Postgres or PG. It is defining a new kind. A kind for example, stateful set is a kind of config map is a kind and Kubernetes and you can add a new kind and the new kind will be called Postgres. And this one is a very stupid Postgres implementation as it only has two attributes replicas and version. So if you execute that YAML your Kubernetes cluster will know to create objects of a new type. The pre-reaction would be PG's like Postgres' and the kind would be Postgres SQL. So if you want to create a new object you will create another YAML file and that would look like this. So you can tell Postgres please create me a Postgres object with the name PG1. It should have a Postgres version 12.2 and three replicas and it will store that object with its desired state. So we are in a declarative approach. We would declare and say that we want to have this Postgres but Kubernetes will store this information but it will do nothing with it. It doesn't know how to turn this thing into life and therefore you need a custom controller that will be triggered whenever a custom resource such as your Postgres entity will be created or modified. This is a piece of Go code that will add an event handler to handle an incoming request. You can see that there are functions for adding a certain object, updating it and deleting it. So the crud operations that would also be part of a service broker. A lot of this will be about comparing like the old structure versus the new structure. So let's say what has changed could be the number of replicas could be the version in our simple example and this will obviously need to be translated into modifying the underlying resources and triggering more behavior. The version for example could be rather complicated matter and changing the number of replicas from one to three also requires setting up streaming replication. So this is not a trivial task and it will also involve modifying the state of Postgres by executing commands like PG base backup. And this is where things become non-trivial. And in order to give, let's say a means to make this more or easier, the community comes up with solutions to give this operators or to give those controllers home. So the operator SDK has emerged and to overly simplify it, the operator SDK provides you means to generate CRD, YAML files and corresponding controllers in go. There are alternatives in the operator SDK to play with handsable and helm support but I think if you're striving for full lifecycle automation with a complicated data service such as Postgres, Go will be your friend because things will become complicated and you need the power of a programming language unless there's something very, very mighty. So I'm not saying Ansible and helm can be used but let's say my hinge here is that Go will be used rather frequently. So there are code generators that will help you create custom resource definitions and controllers which you could also create in other languages. A controller could be implemented in basically any language but the operator SDK will help you to do that. In a search generator, you will constantly have the same corrective loop where you compare the actual versus the desired status for example, how many replicas do I currently have? One, what is the desired count? Three, oh, I have to act. And what exactly needs to be done that will be part of the implementation of such a controller. So you remind yourself of the various different use cases and that this all has to be mapped into such a controller into a reconciled loop and you can also see where the operator SDK will yeah, say become challenging because this will require a clean structure of your code or you will drown in spaghetti code as this becomes more and more complicated. However, it can be done. They're already implementations. Zalander, for example, already have a Postgres operator. We do have an operator, so it's possible. However, if you think that there is a repeating task if you automate several data service and you don't want to write the same operator over and over again because a lot of the time all you do is changing and modifying Kubernetes resources then you may want to look into Kudo. Kudo is a generic operator, which if the operator SDK is Ruby would be, let's say similar to the Rails framework. Comparison is a bit simplifying, but you forgive me. So Kudo introduces higher level concepts that are reminding of workflows and it basically will allow you to describe a deployment of a service instance, for example, by applying several YAML files in Kubernetes in a particular sequence. Now, I definitely encourage you if you're working with Kubernetes and you automate your data services, play with operators and play with Kudo and make yourself your own opinion. The question here is Kudo expressive enough. So while in Go, operators, you might drown in Go code if it's not structured properly. In Kudo, you might reach obstacles where Kudo gets more in your way than it may help. So maybe you wanna try it yourself. Whether it's conceptually the right way or not, I'm personally still making myself my mind up because I think there's currently no way on describing the life cycle of data services in an elegant way without hitting ugly technological constraints which make your life hard. I somehow believe that this can be facilitated and in part will be the mission that NNINDS will be on for the next two years, for sure, making this easier. So we are building Yucca-Bernitas products for the NNINDS platform and we'll try to make this even easier. So maybe you'll try it yourself and make your own opinion. As I said, Patronis, Billu and the Pontroni operator, they've built a cluster manager, they've containerized it and they've created an operator. And regardless of whether you like it or not, it's definitely a good piece of work that can be used as the foundation to build upon. So credits for them. So if you look into a platform solution, let's say a Cloud Foundry on Kubernetes, well, that's where Kubernetes or Cloud Foundry is heading, that you'll use Kubernetes to schedule pods to run your applications, then you can see the architecture is pretty similar, but at some point the deployer will hand over work to the Kubernetes API because you create objects such as service instances and service bindings, which will come as custom resource definitions with their according controllers. So there in these controllers will be much more logic than has been before. And such a service instance to zoom in a bit will be a stateful set. In this case, one stateful set and each of those stateful set squares will represent a pod in the stateful set with different containers. All right, so if we sum it up, to turn a Postgres into a cloud native data service, that will run in the context of a large application developer platform. First, we need to have a standard interface to create and modify data service instances. This can be the open service broker API or Kubernetes custom resource definitions if you and then use the Kubernetes API. You need a Postgres cluster manager, for example, such as Petroni. And the automation will then turn it into or make those pods and virtual machine the femoral where your data is stored on those volumes. This will, together with your automation tooling, will open the box for on-demand provisioning of dedicated service instances, which is the key to make Postgres cloud native. So at this point, I would like to point you to our free Kubernetes tutorial at learn.kubernetes.anynines.com, especially if you felt a bit left behind with all those Kubernetes terminologies and terms. We'll also publish a Postgres tutorial. So the section of the stateful sets already uses Postgres as an example to build a naive stateful set that doesn't do replication. And in a chapter yet to be published, we go down a bit further and create a replication, a streaming replication based on Kubernetes. So if you are interested in the technical details, which are hard to cover in a one hour talk, you're free to keep an eye on this URL as new content will arrive and it's free. So we are sharing our knowledge here with the community and you can always reach out and ask questions and drop me your opinions and discuss Postgres automation issues with me. So thank you very much. Thank you so much, Julian, for such a great presentation. Thank you to all of our attendees for hopping on and spending an hour with us. And I hope to see you on the next Postgres conference webinar. Thank you very much and see you next time. Bye-bye.