 Hello everybody and thank you for joining us today and welcome back to another all things data. Open shift commons briefing today. We have special guests to give. Off is joining us again. Our chief storage performance instigator and storage. And Greg Smith and from crunchy data are joining us today as well. We really are excited to hear about all the performance testing that has been done between crunchy data and the open shift container storage team. So please take it away. Good morning or evening or afternoon everyone. Let's just start with the slides. We're going to do very 60 seconds brief intro that is not needed for post-quest. Folks from crunchy are going to talk about the crunchy operator. I'll talk a little bit about why use open shift container storage for with crunchy. And then I'll talk about the testing that was done talking about the layouts, numbers and all of that. And we'll leave a few minutes for questions if there's anything. So post-quest very old and trusted database really no need to introduce massive community of source developers that are actively contributing and one that is not owned by a company on the west coast and support for sequel no sequel very strong concurrency. And yep everyone knows post-quest. Now let's I'll give it to Greg and he talk about the crunchy operator. Great. Hello everyone I'm Greg Smith. I'm the director of open source strategy for crunchy data. We like to give people just a brief outline of things because if you're not working inside of the post-quest community it can be a little confusing as to what's going on. So we like to give people an overview from our perspective and why why we think it's good to know all this for Kubernetes and this type of deployments. So crunchy is the top company supporting pure open source post-quest and by that I mean we don't have any commercial or closed sourced projects it's all open and we've expanded to where the majority of new community post-quest code passes through crunchy on its way to commit and then we have a stack of additions to that base available you know which goes all the way up to the operator and then features on top of that. The the thing that we think makes it particularly well suited uniquely suited really for this type of project is that post-quest is free software in every way. It's BSD licensed it has a democratic process for changes. I like to think of post-quest as being the most like a Linux you know there's a lot of different companies working together and they all review and commit each other's submissions and then that community creates a shared core kernel commercial vendors spin their own distributions based on. Now Red Hat ships a perfectly fine post-quest in Fedora and it had Enterprise Linux there's nothing wrong with that. The idea is if you use crunchy certified post-quest instead you add some auditing and security features and then if you add our operator that's the next step to picking up crunchy best practices for administration and scale out. In the last few years our post-quest spaced approach has really gelled well with all the people are looking to do cloud database migrations to shrink their data centers and that sort of thing. The the kubernetes has worked particularly well for that and operators are the preferred way to set up and manage that sort of cloud. This is real stuff in production. We have production customers using hundreds of crunchy containers doing work in and out every day you know Fortune 500 companies you'd know by name if I could talk about them and what everyone likes is that by being true open source software that makes our approach great for any kind of hybrid proud people what you know maybe some of it's on AWS maybe some of it's on their local servers even customers in secured locations with air gaps and stuff like that people that build only from like trusted source codes they're able to pull all of our code in and recompile it because everything's open source there's no kind of gating bits to the whole thing. We think it's an important point because as we see it kubernetes just dislikes commercial licensing for the fundamental infrastructure pieces and in a cloud environment where the exact number in distribution of servers deployed is is constantly changing. Commercial database licenses in particular can be a real hard fit. What's nice with Postgres is we do have a wide range of features you know all the way to things like the no SQL Segi was just mentioning you know we have a very wide range of apps you can deal with in Postgres and it's in open license you never have to worry about that when you're doing your competition setup. As for why to use an operator at one point we really spent a lot of time trying to package our database products into kubernetes. When I first read the initial kube blog that was introducing operators back in late 2016 there was really a light bulb moment for us. I said this state-driven architecture has immediately struck me as the right way to organize this type of problem and we've jumped right into it as fast as we can. We started out by covering the most urgent database foundations so we think of that as being replication and backups you know things where especially in the kind of less reliable than bare metal situations the cloud gives you you really need to make sure your backups and everything are great. Step one before you do anything else and then we work our way up the stack from there. Lately there's been this another burst of commercial open source softwares that seems like they're going the other way. I mean some of the some of the projects like Redis and Mongo are kind of re-engaging their customers with cloud-based license restrictions. The whole thing's kind of weird but the the point that we like to emphasize with Postgres there is no one company that owns the entirety of the Postgres code it's owned by the group that does the development you know crunch is as big as anybody but even we can't just go in and change Postgres or take it over re-license it or anything it's a true community project like Linux and that makes it a reliable base that people can build on without having to worry about it going away because you know some VC person changes their mind one day. We don't have that problem in Postgres. So what the operator is let us do we've had three years of development on it with a busy team and we don't just publish the code you know we're doing the full eating your own dog food style here many of our projects are operator-based we're doing production customers on the operator and we keep adding more and more user interfaces the hard parts of the database like setup and monitoring you know and once you've got all these basic things together that everybody wants I mean everybody wants high availability provisioning failover all this other stuff and once you get them all that then that lets people move to a higher level of actually building real database applications and the way Postgres works because it's all open you can make all your own modifications to any level you want which is how we get to being able to make substitutions for the storage underneath and have Postgres just be fine with that and that takes us right back to the rest of our talk so back to you thanks for your time. All right thanks Greg I guess you got a better coming up. Yeah we have another beta I mean we're just we're starting to get into more complicated features like there's a lot of security related stuff where people are going oh we want you know secure connections to work over this type of thing but then you discover the connection pooler doesn't work I mean these are the sort of things you discover after you build the initial demo and go okay wait a minute you know there's some things that just don't work together quite like you expect them to so we just we just keep cranking out releases moving over as many of those hard parts as we can the idea being once we get them done you don't even need to know about these weird problems with you know TLS protocol and how it interacts with the two connection pullers you know we're just trying to hide higher and higher levels of code with each new release. Cool I just noticed the hippo as your logo so let's continue to talk about the performance testing we have done and the first just a little bit why we have done this so the first reason is is that in OpenShift Container Storage 4.3 we are actually introducing as a technology preview a feature called Director Touch Storage and this feature basically connects a physical device whether on bare metal or virtualized on the cloud into the OSD pod that is actually using it and providing the storage for other applications. So the testing that I've done here are basically on OpenShift Container Storage 4.2 kind of a you can say almost 4.3 version with this Director Touch feature. Why use OpenShift Container Storage for Crunchy if you notice a few of the slides mentioned have availability features from the Crunchy operator well the the main feature is the main reason is that you might not only run the Crunchy operator you might not only run a Postgres database you might run various different types of a database or you have other needs that needs a persistent storage and needs resiliency and so OpenShift Container Storage basically provide a this resiliency to all applications that need their persistent storage among them Postgres via the Crunchy operator of course OCS4 fully integrated with OpenShift 4 and that's why we have kind of the two companies partnered together to bring this up all right so let's not talk about the testing layout all of this was done on AWS only six nodes IPDI OpenShift clusters three of the nodes what I call the worker application nodes these are of type M5 for X they are dedicated to the applications that the cluster will run in our case these are the the Postgres databases each of these nodes 16 CPUs 64 giga RAM we have a another set of nodes three nodes of the type I3 and the 2x which are from the OpenShift perspective worker nodes but they are in my testing considered to be storage nodes meaning they are running OpenShift Container Storage 4 and that's the only thing that they are running of course besides the any other minimal pods that needs to be run by OpenShift each of these nodes is with 8 cores 64 gig of RAM each of these virtual machines I should say and they each of these VMs have two direct attach NVMe devices so we have basically a total of six devices for the OpenShift Container Storage cluster or the CEPH cluster that is being used in this configuration each worker application node is running three Postgres or three crunchy single instance clusters so we have a total of nine and the tests were ran in two configurations so there's actually two clusters one on a single zone on AWS and one of the strengths of OpenShift Container Storage 4 is the ability to do cross zone storage application so single zone versus multi-zone each a crunchy instance or each Postgres database was restricted to four CPUs and 16 gig of RAM via Kubernetes resources which means we have a total of 12 CPUs and 48 gig of RAM per worker application node so we left about 25 percent of the resources or 20 or 25 percent of the resources in each of these nodes for anything that might you know that OpenShift itself or Kubernetes itself might need benchmark tool usually most people with Postgres like to use a PG bench but I've chosen this bench a few reasons it has emulate a more realistic let's say a web application form of IOs with a lot of random processing of all kinds of data it's very easy to from the storage perspective it's very easy to make the workload run in a manner of the 70 30 70 reads 30 percent writes IO pattern which is usually what most people do for general testing of storage subsystem it is one thing that is actually not written here it's also very easy to then compare these results to the other database that the sysbench was which is my sequel and another thing is which we will discuss in future talk is that it is very it's easier to show on on this bench resiliency under load mainly meaning that sysbench can sustain errors and continue to run and in this way in the future we're going to show together with the crunchy people that are actually working on the code how the crunchy operator together with OpenShift container storage behave when losing a crunchy pod or when losing a storage node or an OSD pod things like that each database is 100 gigabyte in size it's 400 tables one million rows each each sysbench was ran with six threads so each two connecting to each postgres pod and the usual about five minutes of a warning followed by 10 minutes runs of 10 minutes each with the rest of 60 seconds that's about the layouts let's talk about the this is another half a day for me work to create these nice graphics but that's basically showing the difference between the single availability zone and multi availability zone showing where exactly each of the storage nodes and application nodes and sites all right so let's talk about a little bit about the numbers on the right side is a table that shows an average of 10 runs from all the databases the middle column is the average transaction per database which means we aggregated all the transaction divided by the number of runs number of databases the total cluster transactions is for that particular run what we were able to achieve and as you can see there's we are showing the latency per database average and even more important than 95 percent latency for these ones from the you know performance perspective in on the graph side we do like to see things are more on the steady side and not too many spikes on the line which is really good so this is on a single zone the performance numbers these two graphs that I'm heading and showing here are talking about a little bit on the inner level on the left side is basically the these rbd one two three are basically the persistent volume claims that each node as you can see we have three nodes that these are the application nodes these are the pvcs that each of these nodes have each nodes have three pvcs attached to because each node is running three post-course databases you can see clearly how the ratio between the reads and write is you know roughly around the 70 year 30 on the right side these are the actual set numbers or the usage or utilization of osd devices um and from the three storage nodes so if you remember we have two NVMe devices in each of these storage nodes and and this is the utilization of between reads and writes important to note this for from self perspective the more devices self has the better it distribute the load and better distribute the data so this was single zone we'll go to a multi zone and again it's like a the previous numbers that I shown single zone you have the average transaction per database and the total transaction per cluster what is uh what you can quickly see in here and and you should be able to get all these slides and compare the numbers is actually that um multi zone on my testing is actually behaving faster than the single zone um and now it's not uh that there's any kind of an open shift container storage magic in here it's just one of the basic rules testing on on the cloud is that the performance when you are stressing the the VMs that you're using the performance is just not not very consistent um logically the single zone which should have better latency between all the nodes should behave faster than nodes that are on three different zones that also should have higher latencies between them but again and I always add that in my conversation about performance don't believe my numbers go and go and check it yourself but again a steady number even across three availability zones and as I said better numbers than the single zone and the last slides here is the same thing as previously on the left side these are the ratio of the read writes of the pvcs of each of the post-quest databases and that was on the left side and on the right side these are the read writes ratios of the NVMe devices between the nodes that are providing this storage and I think that if there are any questions we will be happy to answer them now I had one quick comment I just wanted to make for people watching this which is to note um I mean you implied a little bit how you work to set this this whole benchmark up but I really want to emphasize to people that you picked a database that is a large multiple like 16 17 times the amount of RAM in the database and you know you were doing a bunch of other things that are very high latency operations like the replication among pods and that sort of thing so those numbers if people are looking at them and going wow that doesn't seem like particularly good transaction per second numbers I would suggest think of them more as being like when vendors give you worst-case IO IO per second IOPS numbers you know these are more like that because you went out of your way to force the thing to like struggle and seek with this and whatnot you didn't just run it and let it go with you know everything in memory you know which which a lot of smaller databases do operate like that yeah exactly and and just going back to here also I don't have these numbers in here but I can add them to the slides there are utilization percentage for for example for the NVMe devices and they are in the higher 80s to the lower 90s on each of the storage nodes and again like Greg said and that is the difference basically between these two graphs on the left and the right while the PVC that the Postgres or the Crunchy Instance is using is doing a single write of course with a OpenShift container storage because of resiliency we add two more writes because there are three copies and and and that's why you actually don't see this ratio on the right side of the of the graphs of between like 70 and 30 70 reads 30 percent a writes a ratio but and and and also repeating Greg's point yes the goal here is to actually measure storage and not to measure and the memory or RAM of an AWS virtual machine and that's why the ratio between the size of the database and and the RAM of each of these Postgres database is such we are forcing actually IO work to actually go down into the IO subsystem and and there's nothing wrong with that I just want people to make sure that's what you're seeing and you understand the difference between it I it's a great job focusing at that thing it's just I know it can be a little tricky when you're your first time you stare at this graphs to make sense of it so that that's how I think of your test and I think it's very valuable to have gotten that information so we appreciate your your digging through this for so many weeks hi this is sure I'd like to add another comment here so this is a great solution if anybody is looking to run single instance Postgres with a replicated storage but with crunchy operator you could still have Postgres HA natively available with your replica so if you're looking for a read replica that is also possible with the operator so if you combine OCS with the Postgres application you have best of both worlds you could have read replica plus storage replication so I just wanted to add that well thanks drug and Greg and Ziggy that was excellent just seeing those numbers it just makes it more real Ziggy do you have instructions on how to run this on your own the performance yeah I need to compile this and I mean compile these instructions and and I'll put it out how to run this and then other people can can do the same thank you all right so we'll do that as a follow-up to this to this video is instructions posted to the OpenShift blog so look there for that all right and thank you all again for another all things data and same time next week thank you