 I'm Karen Jax. I'm a Senior Solutions Architect at Crunchy Data, and I'm here to talk to you about running mission-critical Postgres databases on Kubernetes. It's such an honour to be here to talk to you. I'm really amazed that there's such a crowd interested in database on Kubernetes. Just a few years ago that would have been unheard of, so it's fantastic. I've described this talk as suitable for beginners, so I will go over some basic concepts, but I'm assuming that most people in this room actually know more about Kubernetes than I do, so I'm not going to try and teach you Kubernetes. I'm just going to go quickly over and recap some of the basic concepts as they're relevant to the topic at hand. Is this going to work for me? We're good. No, it doesn't want to work. I'll go back the old fashioned. I did start my time, I think. It doesn't want to do that. Come on, you can do it. Right, I'll try again. This is the shape of my career, so far very database shape. That's why I'm saying that you probably know a lot more about Kubernetes than I do. I'm talking about the one subject that I know about, and that's databases. I don't really like my current title, Solutions Architect, because I don't think it really says much. I quite like the fact that I was a DBA. I knew where I was. I knew where I stood. So that's me just to give a bit of background. So as a Solutions Architect, I have come a little bit out of my database bubble, and I've come to recognise that there is an ecosystem that goes around the database, but I help my customers to implement database solutions. I'm still very much focused on the database at the heart of the system. And if anyone ever wants to talk to me about it, I'm also doing a part-time PhD in computer science. I'll give up on that. So we'll briefly look at the evolution of database architecture, how things were deployed, how databases are currently deployed, how things are changing, how things might change. Remind ourselves what containers are, why they might be useful, where container orchestration comes in, just so that we can then understand why, where and how we might want to deploy databases in general and Postgres in particular on Kubernetes. And then finally, the best way to explain these things is via a demo. So I'm not going to go back too far in the history of databases just about 25 years to the start of my career as a DBA. So I think that qualifies me as a vintage DBA now. So at that time, most databases were deployed on bare metal physical servers. We didn't really call it bare metal, it was just a server. If you wanted to deploy multiple databases, you could do that. You could deploy multiple databases on a server, but you had to accept that they aren't really isolated and they'll be competing for resources on your server. If you wanted isolation, you'd deploy each of your databases to a separate physical server, but obviously that gives you higher overheads. You've got many servers to manage. Deploying databases to bare metal is still a perfectly valid and well used architecture for databases. But then from about the mid 2000s, virtualisation in the form of VMs started to become popular for deploying databases. Virtualisation had been around already for at least a couple of decades by them, but it became popular for deploying databases. So now we could split up an individual server or physical machines to have multiple VMs. We've got isolation there, but we do have high overheads. We're looking in the order of gigabytes for each of our VMs. We've got the hypervisor and the guest OS to consider. That's not ideal. So if you want isolation between your databases, you can now just deploy each one to a separate VM. VMs are fairly easy to spin up and down and that's still a very popular database architecture, whether you're deploying on premises or in the cloud. It doesn't want to do it. So fast forward to 2023 and now most of the customers I work with or in fact all of the customers I work with are running some or all of their Postgres databases on Kubernetes, which, as I said, would have been absolutely unheard of even five years ago. So just a few years ago, most people thought that the idea of running a production database on Kubernetes was absolutely ridiculous. I remember at a database conference probably 10 years ago, someone did a joke presentation. They ran a database in a container and everybody thought that was hilarious, the idea that we might do that. So what's changed and why do a lot of database administrators now think that running Postgres on database, running Postgres databases on Kubernetes is actually a sensible architecture? So I'll just take a quick step back and remind ourselves what containers are and why they might be useful. So as we know, a container is a lightweight, self-contained, executable software package that we can deploy pretty much anywhere. We're using features like namespaces and C groups so that we can share the host OS. But we've got isolation between those containers. Containers are stateless. So we lose any changes made to our container when we stop it and they're ephemeral. So the data only lasts as long as the container. Now as you can imagine, those last two words, stateless and ephemeral, strike fear into the heart of any database administrator. I'm going to get crossed with this. So why have DBA started to look past these sliding conveniences and consider containers, containerised environments for their databases? Well obviously containers have advantages as we all know. They're lightweight. We're looking at megabytes usually instead of gigabytes as compared to a VM. They're scalable. We can just spin up and stop containers as we need to. We can cluster notes together. We can cluster service together to get more capacity. They're portable. We can move things around and we've got that isolation that we spoke about. Of course, once you start managing more than just a few development databases and containers, it's going to start to feel a little bit like herding cats. So you're going to want some kind of container orchestration. Other container orchestration tools are available, but given the name of this conference and given the name of this talk, obviously I'm going to talk about Kubernetes. It doesn't want to play the game today. So this is a list of some of the container lifecycle management tasks that a container orchestration tool such as Kubernetes can do for us. So it's looking after provisioning, deployment, configuring our containers, allocating resources, making sure things are available, monitoring and alerting, networking, security. It's doing a whole list of things for us and as luck would have it, that's actually a very similar list to the things that DBA might want to look after when looking after a production database cluster. So to make use of this new architecture, obviously the vintage DBA had to learn some new terminology. So it had to learn how to deploy to Kubernetes clusters rather than to physical servers or VMs, had to learn about pods containing one or more containers, had to learn about deployments with one or more replicas or copies of a pod and learned that those were interchangeable and that just like magic, if a pod's destroyed then another one will pop up to take its place. But what about the elephant in the room so to speak? I'm assuming everybody knows Postgres and knows that that is Slonic, the Postgres logo. So we're talking about running Postgres databases on Kubernetes. Obviously the most important thing about a database is the data in it. But I thought that containers were ephemeral and stateless so we don't have persistent data. But it's all well and good if a pod's going to be spun up again if it disappears but that's no good to me if I lose my data. Which is one of the main reasons that for many years containers weren't seen as a viable option for running databases. Well to the DBA's huge relief, Kubernetes already thought of that. So we've got persistent volumes obviously that allow the DBA to attach storage to a container that's going to remain available after the container no longer exists. And not all databases are created equal. What about standby or replica databases? In a production environment we're very unlikely to want just a single standalone database. We're going to have a primary database and one or more replica or standby databases that we can use for either high availability or potentially re-scaling. After all scalability is one of the use cases that we said of containers. Pod's in a deployment are interchangeable but our databases aren't interchangeable. We need Kubernetes to understand that the primary and standby databases aren't the same. Well the DBA was obviously relieved to find out that Kubernetes thought of that as well. But stateful sets can be used for this type of situation. So even though the pods are defined on an identical spec, they've got their own identifiers so that we keep the existing volume even if we lose the pod. So that's quite a relief to the DBA. I'm going to read this bit because I forget. So amongst other things the Kubernetes documentation tells us that stateful sets are valuable for applications that need stable assistant storage, ordered graceful deployment and scaling, and ordered automated rolling updates which sounds very much like what we want from a high availability database cluster. Sidecars, those helper containers that are tightly coupled with the main container in a pod turn out to be really helpful for database administration as well. In addition to our main database container we can also have containers for things like metrics exporting and for database backups. So why might you want to deploy Postgres on Kubernetes? We mentioned some of the advantages of containers in general. Our customers have lots of different reasons and use cases for running Postgres on Kubernetes. In no particular order these are some of them. So multi-tenancy particularly where they've got a database as a service type offering, where they have microservices often because they're already using Kubernetes in their application stack because they know Kubernetes and because they want to automate things and enable them to deploy at scale. One of the anti-use cases that I haven't mentioned is where you just want to use Kubernetes and we do see a lot of our customers doing just a lift and shift of their existing multi, sometimes multi-tens of terabyte database into Kubernetes without making any changes. That's not one of the use cases that I would recommend. I also quite like the answer that Joe Conway gave in a blog that I read a couple of years ago of his where he says resistance to containers is futile. So I've left the link there in the slides but I won't go into that. Okay so you've looked at why, where can you deploy Postgres on Kubernetes? Well pretty much anywhere you like and certainly anywhere that you can deploy Kubernetes. So on-premises in the cloud as a managed service, as a self-managed deployment using your choice of Kubernetes distribution and platform. So once you've decided where you're going to run Kubernetes the obvious question then is how are you going to do that? To run Postgres on Kubernetes you need a lot of knowledge or expert knowledge in two different domains. You need expert knowledge of Postgres and you need expert knowledge of Kubernetes and it's hard enough finding someone with expert knowledge in one of those domains let alone someone that knows both of those things. Since Kubernetes doesn't naturally natively speak Postgres we need some way of interpreting what we want. We need to tell Kubernetes how to manage a Postgres database is we need to tell it how to do things like replication, high availability, backup and restore monitoring. So you could write some logic to some code to handle all of that yourself but to the DBA's great relief once again Kubernetes has already thought of that. So we can create an operator to extend the Kubernetes functionality and allow us to manage our Postgres databases. This will mainly use custom resources and the control loop to watch the state of our cluster and make changes as needed to bring it closer to our desired state. So as we said a Postgres operator combines that detailed Postgres and Kubernetes knowledge into one place extending the functionality of Kubernetes so that it speaks Postgres and it automates the management of your cluster life cycle. So some of the things that you'll want it to do are here. So we refer to it often as knowledge in code or a virtual DBA. So it's doing things like creating, managing and dropping databases, taking care of your database storage, performing database backups and restores, implementing database, high availability, failovers if there's a problem with your primary database, monitoring the cluster and alerting if something goes wrong and automating database upgrades. The easiest way as I said to explain a lot of these concepts is with a demonstration so I'm going to demonstrate deploying Postgres on-cuminaties using one of the available operators. There was a talk this morning about the database operator landscape. There are multiple operators both for Postgres and for other databases and I know that the data on Kubernetes group is working to put together a matrix of the features of those to make life easier for people when they're looking at those and choosing which one to use. I'm going to demonstrate the one that I know because it's the one that my company develops and uses but the basic concepts are the same, the different operators will implement things in a different way, they will have slightly different functionality but the idea is that it's managing your database lifecycle for you. So we'll deploy the Postgres operator, deploy a high availability Postgres cluster, scale up and down, perform a minor Postgres upgrade and deploy the monitoring stack. So I'm not brave enough to do a fully live demo. I've attempted to record the demo in advance and I will just explain what's going on now. Let me see how this is going. So is that visible? I'm hoping, okay it's very small I'm afraid but I'll just explain what's happening. So I've got a three node Kubernetes cluster that I'm deploying to, it the operators cloud agnostic it really sorry Kubernetes platform agnostic it really doesn't matter where you're deploying or what you're deploying to I've just got a three node GKE cluster there. So I'm just I forked the Postgres operator examples repository and I have just cloned that. Oh I apologise it's gone back to the beginning. Okay so I've cloned my repository you'll see how bad I am at typing and that kind of thing. Okay so we've got some example script in there we've got helm charts and we've got customized manifest because I know customized and it's there in kubectl I'm just using that. First of all I'm just going to create myself a namespace that I'm going to do my deployment in so I've created myself a Postgres operator namespace set it as my default because I'm lazy and I don't want to have to keep typing that and then I'm doing a server side apply so that's now installed the Postgres operator it's created my custom resource sorry I'll pause briefly whilst that I'm trying to explain. It's created me my custom resource and the various different things that I need to run my operator I've got my operator pod there and I've also got an upgrade pod for when I want to perform upgrades later. I'm just going to leave that get pods running on the left there so this is my manifest where I'm going to create my Postgres cluster custom resource I'm calling my cluster kubicon I'm just setting that to Postgres 15.1 just so that I can just so that I can demonstrate a minor Postgres upgrade later and I'm setting replicas to two so that I've got a primary database and a standby okay so now if I apply that I should on the left hopefully see my cluster deploy I say hopefully obviously I I do know because I did it yesterday okay so I've got my repo host that's my backup repository you might have noticed in my manifest I had a pg backrest section so that's the backup tool that we use and I've got two instance pods there the first one is already up and running and now it's initialising my second one which will contain my replica database and it will start me an initial backup as well no let me pause if I can no that's not going to work so the operator itself is a deployment and the Postgres cluster uses a state force set okay so where are we up to okay so that's me deleting my primary database pod you can see oh and I apologize for the terminology here the the the master and replica comes directly from Petrone they've updated the documentation but not yet the code itself so you can see that straight away did a failover so that I had a new primary database come up and then it rebuilt the replica based on that new primary so I can scale up if I want to add more replica databases I've just changed replicas to four now so if I apply that it will create me two new replica databases obviously this is all fairly instantaneous because I haven't actually put anything into my database I'm not trying to hide anything from you recreating a replica database will take longer than that if you've actually got the reasonable amount of data in there so it's initialising those and bringing them into our cluster as new replica databases so I'm just going to scale that back down to two again so that I don't have to wait as long for the the next bits of the demo to happen okay so here I'm just um execing into my primary database pod and selecting the Postgres version so you can see we're currently on 15.1 I'm going to change the image so that it uses the 15.2 image I get the associated PG backrest image there so if I apply that it's going to do a rolling upgrade of Postgres so it'll upgrade my replica database first once that's been upgraded it will do a switch over and then it will do the other one so there I can see it's taken my replica database down so it's upgrading that okay so that's back up and running and now it's already failed switched over to that one and I can see that we're on the version 15.2 of Postgres and then it will do the same on the replica and bring that back in trying to remember what I did next okay so in the backup section we can set any of the or pretty much any of the PG backrest settings that we want I'm just going to hear I'm going to set a backup schedule so I set it to do a weekly full backup and daily incrementals and if I apply that I should then see those in my crontab and then the final thing that I wanted to do is show how easy it is to set up a monitoring stack so that there's a Prometheus and Grafana monitoring stack there which has some initial metrics and alerts configured for you you can set your own dashboards and alerts metrics etc so we can see those now we can see the associated pods what I also need to do is to add the exporter side card into my database pod so I'm just going to add the monitoring section into my manifest and that again will do a rolling upgrade of my sorry a rolling update oh there's one pod up there with an error I just wanted to show that's just the initial backup that it took sometimes the initial backup still happening whilst other things are coming up if that's the case it will just run another one to make sure that we've got a successful initial backup did I just no we're good so up there I've just patched my Grafana service because by default the service is our cluster IP I wanted to make it load balancer so that I've got an external IP address and I can connect to Grafana just to show you some of those default dashboards um and whilst I wait for that I'm just generating some data in my database I'm using PD bench which is a simple client tool that allows us to initialize some tables and then run some activity against them okay so that's generating some activity otherwise the dashboards aren't very interesting to look at if there's nothing happening please ignore my appalling lack of security I'm using the default admin admin username and password I'm not resetting it when it suggests I'm just skipping that bit obviously I wouldn't expect you to do that in production so we've got some dashboards that are already set up obviously you can use Grafana as you please to to create your own dashboards based on any of those metrics that are exposed so because I'm almost out of time I will stop that and then just move on to the conclusion slides so um Kubernetes can give you a flexible scalable database architecture I highly recommend that you benefit from the expert knowledge of Postgres and Kubernetes and use a database operator and the great thing is that your day-to-day DBA tasks can be automated which obviously makes things more reliable but also frees you up to do more interesting tasks keep getting that doesn't want to work so thank you very much