 So we are ready to get started. I'd like to thank everyone who's joining us today. Welcome to today's CNCF webinar, Democratizing Analytics with Cloud Native Data Warehouses on Kubernetes. I'm Valerie Lancy. I'm a Cloud Infrastructure Engineer at Lyft and I'm a Cloud Native Ambassador. I'll be your moderator for today. And I'd like to welcome our presenters, Robert Hodges, CEO at Altenity and Vladislav Komenko, Senior Software Engineer at Altenity. Great. Thank you very much, Valerie. We're delighted to be here today. We'd also like to thank the Cloud Native Computing Foundation for putting on this event. It's just a real pleasure to share the work that we've been doing both on Kubernetes as well as data warehouses and putting these two things together. So I'm going to dive in, just give you a little bit more color about our backgrounds. I'm CEO of Altenity. Our business is to offer services and support for ClickHouse, which is a very popular open-source data warehouse. We'll talk very briefly about how that data warehouse works to give you some background. For the purposes of this talk, the relevant part of my background is I'm also a database geek. I've been working with databases since 1983, have worked at various different levels with over 20 database types, and I've been working on Kubernetes since 2018. Vlad is, as Valerie mentioned, a Senior Software Engineer at Altenity. He has over 15 years of database and application experience, and most importantly for this talk, he is the main designer of the ClickHouse Kubernetes operator, which is the main focus in this presentation. What I'd like to do is just take a couple of minutes and introduce ClickHouse to level set because I suspect that not everybody on this call is a data warehouse specialist or even deeply involved in data. So ClickHouse is an open-source data warehouse, and the simplest way to explain this, if you have any experience with databases, is to think that we combined MySQL, which is a very popular open-source database, and Vertica, which is a very popular proprietary data warehouse, and conceptually, ClickHouse is their child. So it doesn't inherit code from either of them, but it has many of the ideas that are present in these two products. So for example, both from MySQL as a relational database and Vertica as a column store, it speaks SQL. From the MySQL side, very portable runs on bare metal to cloud. From the Vertica side, it has a shared nothing architecture, that is to say there's a bunch of nodes that each have attached storage and with closely connected compute. They talk to each other through a network. From the Vertica side, we store the data in columns. So you can think of that as having the data arranged in a bunch of arrays, and every time you touch the data, you're reading it sequentially down the column, or you're writing it sequentially, again, down the column. So this is optimized, it helps you optimize your IO. Again, from the data warehouse and Vertica side, we have parallel and vectorized execution. By vectorizing execution, we mean the ability to break up these columns into pieces, efficiently farm them out to all the available CPUs and cores on those CPUs and use things like SMD instructions to get the maximum speed of execution. We also parallelize very well across many nodes. The database runs on everything from a laptop, which I'm running right here with Ubuntu, excuse me, and all the way up to clusters that include hundreds of nodes and petabytes of data. And finally, it's open source. It's an Apache license, so same licenses we have for most products or in projects inside the Cloud Native Computing Corporation. And one final note, it's really fast. So if you're into data, it's an interesting data warehouse to look at. And one of the things that we have been very interested in, of course, is bringing this to Kubernetes and preserving all these qualities. Now, as we begin to look at this for Kubernetes, it's also useful to think about what kind of application does this really look like on the ground? And what's particularly relevant for Kubernetes is that ClickHouse is also a distributed application. And of course, Kubernetes supports distributed applications very well. So a typical application would look like the slide that we're showing right now, we will have data sharded. So a shard is a subset of data, and we'll have data replicated. So what you see here, the shard one is replicated across multiple servers. Shard two is replicated across another servers. The replicas are spread across availability zones. And then when people run queries, there are ways that you can actually run the query and have it automatically hit all the shards and then aggregate the results and hand them back to you. We also have ZooKeeper, which is used since this is a distributed system. And there is some state where everybody has to have consensus about what is going on in the data warehouse, like what is the list of things that need to be replicated. We use ZooKeeper to store that date, that state. So this is actually an application that it is beneficial to have it on Kubernetes, but at the same time, we benefit from having the support in Kubernetes to support it to you, to run this kind of distributed application. So what I'm now going to do is given that background, I'm gonna go ahead and dive in and talk a little bit about what this looks like when we're actually building a cloud native data warehouse, what this looks like from a user. So the first thing is that even though I can say lightly that it's great to move this onto Kubernetes and how, because Kubernetes has all these great features, but the fact of the matter is that when you're actually running on Kubernetes, as many of you know, Kubernetes itself has a rich set of resources. The application that we're bringing onto Kubernetes has many moving parts as well. So what we get is a relatively complex set of resources which we have to stand up and manage on Kubernetes. So this is a problem. If we have to just write a bunch of deployment files and hand code this, this would be very, very difficult to even get it to run in the first place. And then of course, to make any changes, upgrades, adding things to the system, taking things away. Of course, this would be very difficult to do even if we scripted it using something like Helm. So what we do instead is we use one of the new features in Kubernetes, which is called an operator. And they are specifically designed to help encapsulate these kinds of complex deployments. And the key thing that the operator does is it creates what we call a custom resource definition which contains all the information that is necessary to stand up the cluster. And that custom resource definition can be loaded into Kubernetes. It has a specific, it becomes a new resource type when we install the operator. And then the operator when it sees this will look at the specification and then take appropriate steps to create all the resources like pods, stateful sets, persistent volume claims, all those kinds of things that are necessary to have the Clickhouse actually run in Kubernetes. It not only takes care of setting up the resources but it also does what we call a best practice deployment. So it's one thing to set up pods but of course, within the pods you need to have configuration files. You need to handle schema correctly. You need to set buttons and knobs correctly to run in the environment that you're in. You need to add users. These are additional things on top of the basic resource management that the operator is capable of doing. So operators, this is nothing unique of course to Clickhouse and data warehouses. These have actually become very popular as a way of managing any type of application has state and there are now dozens of them available. So that's what, and a lot of our work then of course has been to implement this operator and then use it to stand up data warehouses. Now, when you're installing an operator, it's quite simple and these are, I won't belabor these commands but we basically just grab an installation file which contains a bunch of parameters that define the operator as well as the custom resource definition. That file I'm just pulling from Git. We can go ahead and apply that file and then it pops up, the operator is encapsulated in a container, it's actually just a pod that can accept commands. We normally install into the cube system name space. It pops up and then you're ready to go. And when you're done, you can just do a delete and take the operator away. So it's relatively simple to manage and bringing it up takes just a few seconds. When you're working with Clickhouse, you also need ZooKeeper. As I mentioned in the introduction to Clickhouse, that's used to store the state where we need consensus across nodes. So there's a variety of ways to set up ZooKeeper. It has its own operator right now. For demo purposes, I tend to use Helm. There's a pretty good chart for Helm under Incubator ZooKeeper which will just pop one up for you very quickly and have something ready to run in about a minute. So when you're actually using Clickhouse, you'll do these two steps. You'll install the operator. You'll set up at least one ZooKeeper cluster. And then at that point, you're ready to begin adding data warehouses. So what I'd like to do now is briefly look at what this custom resource definition looks like. And the idea here is that instead of having 25 or so different files that define all the stateful sets, pods and everything like that, we have a single file and we want to make the top level decisions in this file as simple as possible. So this is what a typical Clickhouse installation resource looks like. So we have a name, for example. The clusters are all gonna have CNCF in the name. We're going to define what are called clusters which are basically layouts of data. Inside the data warehouse, we have one called replicated in this case. And then we're going to give some simple directions of how many shards would we like to have and how many replicas on each of those shards. And then finally, we have a little bit of information about how to find ZooKeeper. What this is doing is pointing to the ZooKeeper service, which will then is actually a load balancer that has three underlying nodes. So this is the top level and for very simple configurations, you don't have to add much more data than this. Now, in a real production data warehouse, of course, you need to have much more information. You need to have configuration data. You need to have users. So there are additional sections in the Clickhouse installation resource file where you can do things like add users. And this is an example of how we would add a local user that will then be created on each of these nodes. The user is called demo. It has a password demo and some other things like which networks can access, which quota it uses, stuff like that. So that's as simple as just adding this information. And actually what we can do is with the resource files, you don't have to make these decisions up front. You can set the, stand the data warehouse up. You can add this later, apply the file, and then it'll automatically go out and add those users. And then finally, we make use of templates. One of the ways that you get simplicity in systems is you wanna have some way of expressing defaults. We do that through what we call templates. So for example, we have volume claim templates which give you, give direction about how to lay out storage. The specs for these are largely similar to a persistent volume claim. We have pod templates, again, which allow us to specify the version of the pod and other interesting information about that process. One thing I'll point out, which we'll bring up a little bit later is that storage is a really interesting question, obviously. We're dealing with data. And one of the key problems that you have when you're setting up any kind of system on Kubernetes that involves data is you have to be very careful about configuring your storage because otherwise you will either not get what you want or you may actually set up configurations where you can lose data quite easily. So this is something that will stress a little bit in this talk as we go forward. And then finally, one of the nice things is in addition to making configuration changes and adding and subtracting users, we can quickly scale up and scale down by modifying the layout. So for example, if I wanted to take that previous layout, which had X number of shards, Y number of replicas, if I wanna now make it three shards and three replicas, all I have to do is change these numbers, reapply the file, and the operator when it receives it will figure out the difference and take appropriate steps to increase the scope of the resources that are allocated to the data warehouse. So this has been kind of dry code. What I'd like to do at this point is just show you how this works because one of the things that's really great about Kubernetes and I think gives where we feel a lot of optimism about how well this is gonna work is Kubernetes allows you to do these operations very quickly. So I'm gonna go ahead and for those of you who know Rocky and Bullwinkle, this is the famous Nothing Up My Sleeve line. So what we're gonna do is go over to a cluster that is running in Amazon and go ahead and set up the system. So let me just make sure that things are completely clean. Well, you're all familiar with the Kube cuddle command. So do Kube cuddle get all, just make sure there's nothing in the environment other than the Kubernetes IP address. Gonna make absolutely sure they don't have any storage leftover from previous demos. So get PVC, nothing out there. Great, we're ready to go. Let's do the following. I wanna just show you this file that we have. So it's very similar to what we just showed you in the slides. We have our ZooKeeper configuration. We have a cluster, it has a layout. It's just gonna start with one short and one replica. We have defined users, and then we have some templates just as we discussed earlier on. So this is the file, it's not particularly complex, but this is all that's necessary to create a relatively powerful data warehouse. So what I'm gonna do is I'm gonna go ahead and I'm going to apply that file. So let's go ahead and apply it. And what we're gonna wanna do is just keep an eye on, run a watch command, and let's just keep an eye on what's going on inside the cluster as this comes up. So what you can see here is we've already created a stateful set to manage this cluster. We actually create one per pod for reasons that we'll discuss in a minute. We've created, because this is running on Amazon, we've created an internal load balancer. So you can see that if you're, this is a cops cluster, by the way. So if you run cops, some of this stuff is gonna look pretty familiar. And then we have the pod, which is already up and running. So this is one of the really things that I really like about Kubernetes. I started working at Kubernetes at VMware. I was used to working on VMs. Kubernetes is really fast, as particularly if you're doing sort of demo test type things. So that cluster is already up and ready to go. Let's go ahead and log into it. And so I've gone an exec, using kubectl exec. I've now connected to the pod. I'm actually inside it. And I'm going to see if I can connect to the cluster. And there we are. I'm connected in. I'll do a show tables. This is if you're a SQL person, particularly if you know my SQL, this is gonna look very familiar. I'm just checking to see what tables are available. And what I'm gonna do is just go ahead and create a table. This is gonna be a replicated table. So I'm gonna copy in the table definition. Boom, I've just created a replicated table, but I only have one replica, so it's a bit boring. I'm also gonna add a little bit of data into it. So let's go ahead and add one line of data. Again, very trivial example, but quick. And if I select it back, there it is. My one row of data just came back from my replicated table. Let's quit out of this. And what I'm gonna do now is I'm actually gonna scale this cluster up and add an additional replica to it. Get fully out of the pod. And I'm now going to make a very small change to that custom resource definition that we described. Let's just go ahead and diff it. I'm just gonna change the replicas count, which was in there for one to two. That's all I'm gonna do. And what I'm now gonna do is apply that file. So let's go ahead and do an apply. And we'll watch as this comes up. And what we see is the existing pod continues to run. We see there's a second stateful set here. And now what we have, what's happening is, we're basically firing up the pod, creating the container, so on and so forth. We'll give that a second or two to finish. I wanna just look at a couple of, just show a couple of other things which are interesting when you're working on these systems. It's always important when you're dealing with data to have a quick look and make sure that the PVCs are correctly created. So those are the persistent volume claims that then cause storage to be allocated. So let's look at the PVCs. Sure enough, we see that there are two of them. They're bound. They have volumes that they correspond to. We can go ahead and check that the volumes are created. We actually have five volumes here. Three of them are used for ZooKeeper, which was already set up. So basically this cluster is up. We see the storage has been allocated and correctly bound. That's always, if you're paranoid, and if you're dealing with data, you're always paranoid, so it's good to see this. And what I now like to do is that pod should be up. So let's go ahead and we'll quickly jump into it. Let's just do it again. I'll make sure the system is fully up there. It's running. It's fully up and running. So now what I'm gonna do is I'm going to go and exec into the second pod because this is a replica. And what I am going to expect to see there is it's actually already set up that data that we created in the first pod. So let's go ahead and exec into this using kubectl exec. Here we are, and we'll connect to the server. So let's do, okay, ta-da, we're gonna do show tables and we don't have a table yet. This is great. This of course worked beautifully. Let's just see if it needs a minute or two more into interesting. Okay, the second here. Okay, this of course, in every single demo, there's some, it looks like it's briefly hung up here. In every single demo preparation that I did before, of course it correctly replicated across and I was able to view the data. Let's just do a show databases. Let's see if there's any other problem here. Nope, default, show tables. Okay, it's not replicating across. Every now and then demos don't quite work right. So this seems to be one of them. But normally what you would see if we weren't having some sort of mild system problem here is you would see this table replicate across, including the data. That's something that we correctly sync the clusters and ensure that they can talk to each other. I think some, it's possible because of multiple rehearsals of this, I've screwed something up in the system and you're not seeing it here today. Let's just assume though that I'll debug that and be able to get that working shortly. I'd like to show you one other small thing which is how we upgrade these systems and that's very simple. So what we can do is I'm gonna go ahead and in the wrong place, let's go ahead and let's just run the upgrade. So what we're gonna do is make a very small change where we're gonna switch the templates for the pods and we're gonna go from running Clickhouse version 19.6 to 20.1. That's the only, I have separate pod definitions, two different templates, just by switching the templates and then applying the definition, I can then cause this to come up and run. So we go ahead and run the upgrade and we're gonna watch and what you'll see here is, and we won't wait for this to run because it'll take a couple minutes, what it's actually doing is going in terminating pods, recreating the clusters, attaching them to the storage and then bringing them back up again. So basically you get a new version of each pod and once this process is done, in the meantime, we design this so that the pods, so that the data warehouse stays up and then is able to and users are able to continue accessing it while this is going on. So that's it for the demo. Let's go ahead and switch back to the slides and I'll bring this up and at this point, what I'd like to do is go ahead and invite Vlad Klamenko to join us and talk about what's going on inside the operator that helps these operations complete successfully in most cases. So Vlad, over to you. Hello, everyone. So let's dive deeper and take a look on how operator actually works and what does it do? So how it communicates with Kubernetes and how it builds the Clickhouse cluster from these YAML definition file. So let's take a look into this slide and here we see the communication between major parts involved in this process. So all things starts from Clickhouse resource definition. This is blue file that you can see on the left side and here is these YAML file that we've just walked around and so you provide this Clickhouse custom resource definition through Qubectl apply and it just jumps through Kubernetes API into HCD database. Actually, inside this HCD database, there are definitions of all objects that are used by Kubernetes that are really created and Kubernetes handles with native controllers all resources known to Kubernetes. So when you create a pod or a stateful set or a service, there are native Kubernetes controllers which know how to deal with these type of resources. But then all of a sudden there is a new type of resource, these Clickhouse resource definition, this new type of resource. And Qubectl apply, it's just puts this new resource description into HCD and here we can see it in yellow that there is a custom resource, Clickhouse installation and Kubernetes by itself has no idea on how to deal with this kind of resource. So what it does, Kubernetes by itself, it places this custom resource into HCD with hopes that there would be a custom controller that could handle all of these custom resources and actually materialize this custom resource into what would be a Clickhouse cluster. And this controller is an Clickhouse operator. So here it is in the bottom right corner and Clickhouse operator is actually custom resource controller. So Clickhouse operator knows how to deal with custom resources. And the operator communicates with Kubernetes again through API, be directionally. From Kubernetes API, Clickhouse gets some events, notifications about changes, new resources that have come and Clickhouse operator can do actions towards Kubernetes API. Such as create new resources, instantiate new types of what needed to be made to make Clickhouse cluster. So actually we can see that out of this available blocks Clickhouse operator can create an actual Clickhouse cluster. So these, all of these parts are made of like Clickhouse custom resource definition and let's take a look in more details what part of this custom resource definition can be made or available in our YAML files. So there are five main parts that create Clickhouse installation specification. First of all, it's like a default. So in this part, you can describe global defaults that will be used all over the Clickhouse installation. Second part is configuration. This is the big part where you specify cluster topology, zookeeper locations, all other custom specifics, like custom configuration files for Clickhouse, custom configuration files for environment. And there are three other parts that specify templates used by the Clickhouse installation. First is like service templates. These templates that used to instantiate services. Then it comes both templates and operator instantiate pods out of these templates. So there is three steps to instantiate a pod. First of all, we have the spot templates. Then they are applied to build a stateful set. So actually Clickhouse operator manipulates pod through stateful sets. And Clickhouse operator makes stateful set based on these spot templates specified. And the third part is storage claim templates. So up again to the stateful set, storage is bound through storage claim templates. And thus we have like stateful sets which can operate with spots, can operate with persistent volumes, and also we have services available. And all of these parts actually are bricks out of which Clickhouse cluster is built. And let's move a little bit further and take a look on how actual cluster can look like. The main idea is that Clickhouse by itself is under a very active development right now. So new versions are coming. And also sometimes you need to build a cluster out of like a commodity hardware that you have. And very often you have different hardware servers or you have different like Clickhouse versions running simultaneously on your cluster. So that's why there is like different variety of all parts of hardware and software that can be comprised of the cluster, the whole cluster. And also there is another notion. Because Clickhouse is under active development, there are new versions coming all the way. And you will need to have a canary test. So you can set up a new Clickhouse version and take a look on how these particular version performs in new cluster. So actually we should split like notion of healthy note in terms of Kubernetes and healthy note in terms of Clickhouse and your like business aims. For example, you may have a note that need some weeks to run before we can safely name it like ready for real production usage. Because for example, in real life there are like different business scenarios. And you may want to run this new version of Clickhouse through all business scenarios that you may use it for. For example, maybe one per week you have a like grant report build for your data analytics department or maybe like any action by weekly is launched. This is all very business specific for your particular task. But from Clickhouse cluster point of view and Kubernetes point of view, this means that you have to have cluster made out of different versions of Clickhouse and different hardware. That's why we are moving on the next part. So actually Clickhouse operator treats like one pod per stateful set. So actually we have as many stateful sets as there would be like Clickhouse instance. And we are often being asked why these approaches used. And main idea is that stateful set is very good. Of course you can like bundle all the same replicas into one stateful set. But the stateful set has some limitations about that we definitely need to have different versions of Clickhouse software running at the same time in the cluster. That's why that's one starting point. The other point is that we need to spread these cluster over boundaries of availability zones. So we have to have completely distributed system and taken into consideration that we have to have this system running of different versions of software with different sets of persistent volumes of different storage applied. And also running on over multiple availability zones we decided that we would go with not picking all pods into one stateful set. But through this approach when we have one stateful set and one Clickhouse instance running in each personal stateful set. So okay, so let's move on and take a look on how actually operator operates. And so what actions it performs. So first of all, we have new Clickhouse resource definition coming into operator. So these Clickhouse resource definition passed through Kube-Catel, through Kubernetes API then through HCD and again, finally lands in a Clickhouse operator. And now Clickhouse operator has to compare this new resource definition. This is actual state that should be achieved and current resource definition. So what we actually have in the moment in our Clickhouse cluster. And then the main idea is that Clickhouse operator builds action plan. So what actions should be done to move our Clickhouse cluster from current state to the desired state. To the state that end user won't clickhouse to be in. And there we have an action plan and having this action plan, Clickhouse operator starts moving step by step one pod at a time. So actually we are trying to minimize downtime of all pods. That's why one pod at a time, Clickhouse operator starts rebuilding those stateful sets remounting those data volumes. And finally, when we have these like sequential pods upgrade completed, we will have new state of the cluster. Now let's move to the next slide and take a look into another like area. So the cluster upgrade, it's a complex procedure. So it involves three parties, Clickhouse operator, Kubernetes itself and Clickhouse. So actually, for example, if we are adding a new replica to the cluster definition, all three parts would be involved into this procedure in adding a new replica. And let's take a look on how exactly it, first of all, Clickhouse operator, it receives these requests for new YAML specification with a request to add replica. So after building this action plan, Clickhouse operator understands that, okay, we need to add new replica. So Clickhouse operator process, this add new replica request. Actually, Clickhouse instructs Kubernetes what object should be launched, what object should be created like stateful sets, pod persistent volume claims, config maps. So Clickhouse operator requests Kubernetes to instantiate all objects that are required for this new replica. Then when we have these all parts up and running, Clickhouse by itself starts to boot. So first step, we have these Clickhouse instance running, but it's not yet included into cluster. Here, Clickhouse operator moves on. So next step, Clickhouse operator can configure monitoring because we need, we won't like to monitor all this cluster, but we'll talk about monitoring a little bit later in more details. Third part, send schema to all new replica nodes. So actually, Clickhouse operator add tables and adds specification what cluster to join. And at this point, Clickhouse new instance of Clickhouse sees these distributed tables and actually it can join cluster. So all three parties are involved and at the end of the section, we have a new replica joined into the cluster. Now let's take a deeper look inside this small rectangle about monitoring. Actually, operator does not only installs the Clickhouse, it tries to maintenance its full life cycle, like install, upgrade, delete, and also one important part is monitoring. So actually, Clickhouse operator completely encapsulates all the monitoring of the whole cluster and Clickhouse operator provides one simple promissive endpoint to fetch all metrics from the whole cluster. Main idea is that a cluster is a dynamic entity. So you can add new replicas, you can add new like shards or maybe you can enlarge or shrink your cluster. And if in most cases you need to add or remove new replicas from promissive. So for example, if you would like to enlarge your cluster and add new replicas, you have to add those replicas into promissives. But Clickhouse operator completely automates all this part because operator knows what replicas this cluster consists of and that's why Clickhouse operator can completely encapsulate all this monitoring inside like a black box providing one single entry point. And this entry point can be used by Prometheus and then by Grafana, for example. The optimization and the automation is of hiding all these dynamics behind the operator's back. So actually you will have one single point and all monitoring, all metrics come through this endpoint. That's very good automation for monitoring and we are really proud about implementation of this part. Okay, so let's take another step and speak a little bit about storage. So actually with storage we have options what storage can be used. And generally operator can communicate with two general types of storage, cloud storage and local storage. In cloud storage there are like options like Amazon or Google or any other cloud providers. And strong sites like advantages. It's very simple to use cloud storage from Kubernetes or from Clickhouse operator. You just simply specify persistent for the claim template that you would like to use storage, persistent storage that big and cloud storage will simply provide it for you. Downside is like this is network access and it's not as fast as it can possibly be. So in this case, if you are trying to squeeze all the performance that you may can achieve from these drive from your storage system, you have to go with local storage. Actually many of Clickhouse operator users go with local storage. And here we have again some options because Kubernetes provides several local storage like templates to be used. And first of all, it's like empty there. It's very simple to specify, very simple to use, it's as fast as local storage is. So no problem about that, but backside of this, it's durability. So empty your local storage, it lasts as long as your pod lasts. That's not good enough. So actually we are moving on to the next storage type. It's host path. Host path, it's like a storage type when you can specify as path to your local storage. And that's more like durable approach than empty. Actually data written to this host path are durable as long as these local path is available. So actually that's good, but on the other side, it's very complex to how to manage all this part because main idea is that this is pass on your local drive and all pods that are running on this node can actually write into this host path. And it's up to you or actually up to operator on how to set up all these parts. So multiple instances will not collide and will not harm this data located into this host path. Actually Kubernetes also is moving on and providing like a next level of abstraction. It's like storage class local which actually is host path with new some advantages about like simplifying usage of host path. So these are main storage options. Actually, if you would like to go simple and like quick, you would like to use cloud storage. If you would like to squeeze any performance you can achieve, go with local storage. And let's move on. So how to define what storage classes are available in cloud provider? First of all, you can simply ask your kubectl to describe all available storage classes and you can see the set of storage classes provided by your cloud storage operator or your Kubernetes which storage classes are configured. So and you can simply bind to default storage or to some kind of specific storage in our like click house persistent volume claim template. You can request for particular storage type that you would like to deal with. Okay, let's move to the next slide. And next slide will be about where would you like to place your data? So for example, you're going to build a distributed cluster that would keep some data in different availability zones. So you would like to have one replica in one availability zone and another replica in another availability zone. And click house operator handles these already so you can specify two different templates for each specific replica. And you, for example, can specify that the first replica would use these templates that would be located in two A availability zone and the other replica would use another template that tells to be in two B availability zones. And click house operator has very convenient shortcuts on how to manage your location of your pods among different availability zones. Okay, also templates add additional portability. So it's much easier to manage your cluster when you have templates. So you do not change layout of the cluster. You simply can switch templates and go from test site, test cluster that can be run in mini cube within one node up to the real multi-availability zone deployment running over spanning over several availability zones. And layout of the cluster will be the same. The only thing you are going to change is like template that is used in cluster specification. Well, there are plenty of stuff already done but we already have a lot of stuff to be done in future. So what our roadmap, what would be the next steps to implement and click house operator? First of all, and it's very well asked by many users, it's backup. So we are actively working on how to provide a convenient way to make a backup of the whole cluster. So another step is like reclaim storage. That's very interesting functionality as well. So when you would like to delete a cluster but keep all storage intact and attach this already existing storage in a new cluster, that's very interesting functionality as well. So also we are working about with services. How to provide services external or internal and third node access that can be done with services and last but not least security. Next big step, it's like proper certificate management across nodes and using encrypted storage. Well, I hope that we will be continuing very intensive development on click house operator and implement all these items of our roadmap quite soon. Okay, now I will give a word to Robert. Thank you everybody and we'll talk about how to use Kubernetes. Thank you so much, Vlad. And I just wanna say there's some great questions coming up. I wasn't able to answer them fully because I was actually making sure I had the slides moving properly for Vlad, but keep on doing them. We've only got a couple more minutes to go and then we'll dive in and answer questions because I think that what you've seen so far does raise some interesting issues. I wanna talk about just one final issue in conclusion which is the possibilities for using Kubernetes to build analytics solutions. And this is really just a sketch. These are things that are going on right now but I just wanna give you a sense of what the possibilities are because we're very, very positive on how Kubernetes can help people combine analytics with their applications. So the first thing that you can get with Kubernetes is Kubernetes makes it very easy to build up applications from separate kinds of services. And this is just an example of a typical solution for analytics which would include not just ClickHouse but you'd have content sources. You might have one or more Kafka consumers. You might be running a Spark machine learning pipeline to run ML on the data as it comes in from Kafka and enhance it or clean it up and then put it into ClickHouse. And then finally you'd have Grafana which is a great open source solution for dashboards. And the point here is that this is all fairly easy to set up inside Kubernetes particularly as each of the data related solutions has increasingly run by operators. And that's important because real solutions at the business level for analytics are not just a data warehouse but they're a data warehouse combined with other things. The fact that Kubernetes allows us to stand these up and connect them to each other efficiently as well as securely is really important. So going forward, I think this is going to be a, this is clearly one of the big advantages of Kubernetes and one of the reasons why we wanna be in that environment. There's a second advantage which I think is actually bigger. If you look at the history of data warehouses they have traditionally been monoliths. The first SQL data warehouses were systems like Teradata which were not only monoliths but actually ran on their own hardware that was then, so it was a completely integrated stack all the way up to pretty much the endpoint that user applications talk to. That has been broken down of course with virtualization and as people start to do less custom hardware for database and data warehouse solutions. But what's really important about Kubernetes is that you have a common set of resources and Kubernetes allows you to divide them up into pieces very efficiently. And what that means for data warehouses is instead of having these monoliths we can now have individual data warehouses that are scaled and managed and configured for the application for each individual application service. And this is, I call this sometimes breaking up the monolith but I think it's a really important step forward in the evolution of data warehouse because it basically allows services to have analytics plugged into them, so high performance analytics plugged into them and run locally in the same way that we expect services to have their own copy of MySQL, their own copy of MongoDB, their own copies of Redis that are encapsulated within the service. So this is another benefit and something that I think long term, the combination of data warehouse on Kubernetes offers a lot of interesting possibilities for users. So just overall then coming back to our original theme and title, we're interested in Kubernetes and we're working on it because we do believe it democratizes data warehouse access. These are, we see this as a way of providing data warehouse capabilities to applications that previously would not have considered them because they were too complex to manage or uneconomical and the combination of having open source solutions as well as the ability to carve things up really opens up a lot of doors. And as you saw in the presentation, the operators are really key. This is just a really enormously important innovation in the Kubernetes ecosystem. They allow us to do this data management effectively and I think from the user slides as well as Vlad's description of the internals, you can really see how much it's actually doing. And of course, using the operator then it is the key thing that enables this ability to add these performance, these high performance analytics to any application. So with that we're done and we have a number of, Valerie, I don't know if there's any other extra things but we have some open questions which I'd like to cover and if you have more questions, please feel free to type them in and we'll gladly answer them. Looks like we have just a few more minutes for questions. Great, okay, so we've got about seven minutes. Question here about sharding which the, yeah and Vlad gave a partial answer but I'd really like to enlarge on that because it's an important aspect of the way ClickHouse works. So when we talk about shards, what we mean are effectively disjoint sets of data and the way that ClickHouse works is that it doesn't make any attempt to rebalance shards. So for example, if you're familiar with Cassandra, when you stick an extra node in the system, what Cassandra will do is it will automatically rebalance data so that the data is roughly is approximately evenly distributed across all the nodes available. ClickHouse does not do that, at least for shards. So if you want to have the data moved, you actually have to move it. And this has pluses and minuses. The pluses, you know what's going on. The minus is that you actually have to take pieces of the data, we call them parts. And one thing you can do is just copy them across and so they arrive on the machines for a different shard and then delete them in the old one. But it's a manual process. You can of course script it. This is, as Vlad has pointed out in one of his answers, this is something on our to-do list and actually it's one of the biggest asks overall for the ClickHouse community. Replicas on the other hand, well at least when the demos are working, all you have to do is when you add an additional replica, as long as the table schema is correctly created, it just automatically replicates the data. So you do not have to do anything special to spread the data across the replicas when you add them. I hope that answers it. It's sort of the way ClickHouse works now and over time we expect this will be answered. So here's another question that we didn't get to yet. How does Kubernetes handle ClickHouse high availability? So that's a good question and I think goes back to some of the things that we touched on. So ClickHouse, the fundamental way that ClickHouse handles high availability is through replication. So the replication driven through ZooKeeper, we didn't do a lot of details on it, but it's basically a multi-master, eventually consistent replication scheme. You can go to any of the replicas, add data, and it will eventually and usually pretty quickly find its way to the other replicas. And so what you do is this is baked, this is already baked into ClickHouse, what you need to do then of course, is you need to then integrate this with the Kubernetes features. And just to give you an example of this integration, one of the things that Kubernetes does pretty well now is that it supports availability zones quite nicely. So there are annotations inside Kubernetes. If you set them correctly and use affinity rules, you can actually cause your pods to spread out nicely across availability zones. And as Vlad gave in the example, we can actually force them to particular locations. What we didn't show is those templates actually have detailed affinity rules, which then push the pods to specific availability zones. So the key is use the, and this is true of any database, use the mechanisms for creating replicas that the database has, but just then correctly configure your affinity rules so that you get them far enough away that if you have a failure, your replicas are outside the blast radius and aren't destroyed. Let's see, checking the replication status. There are system tables that allow you to do this and this data is fed into Prometheus. And so you should be able to see the number of parts awaiting replication that should be available. I don't have it in front of me. Vlad or actually Alexander Zaitsev, our CTO is actually on the call. Do you guys have a quick answer to that? How we track replication status in Kubernetes? I assume that's available through Prometheus. Yeah, I guess, sure. So, Drator has a bunch, provides Prometheus integration. So it can export metrics from Clickhouse installations and post to Prometheus and there are metrics to check and track replication status as well. So Prater itself doesn't do any particular actions on that. It's a responsibility of human being to decide if anything is wrong with replication, do some actions and you can certainly configure alerts based on certain thresholds, how you can react when something goes wrong. Great, thank you very much. And Alexander, by the way, just joined this talk and to see how we did. He was the one who cooked up this whole idea of using the operator and has been sort of behind the scenes helping to drive this whole question. I'll just review a couple of questions that came up and were answered live. One was data placement work only on Amazon or do you have the same template for GCP? And the answer that Vlad gave is in general, the templates are cloud agnostic. So particularly for data placement, anything that's built into native Kubernetes like support for zones, you can just use it. And so you can write affinity rules where the one thing you'll have to do is obviously change the names of the zones. But generally speaking, these approaches work the same in each area. There are differences, of course, between cloud providers, if you go and look at the annotations for things like services, what makes a service internal versus external, those annotations tend to be specific to particular providers. So you have to check the documentation, play around with it. We have a lot of examples of that. There was a question here, is ClickHouse available as a pass offering on cloud providers? Or is it designed to be run only on Kubernetes? And the answer to that is yes. We actually do have a pass that we're working on right now. If you're interested in that, feel free to check in with us. It's not public. But just in general, I think like a lot of other people, one of the really great things about Kubernetes is it gives you the ability to run everywhere. So this is not just the longer term vision of Kubernetes overall, not just data warehouse, is that you'll be able to build services for managing data, for managing applications, where users can just decide where they wanna use them. It's anywhere you've got a Kubernetes cluster, you can run these services. And we're very much in line with that model. So we expect to, and we have customers who run this both in the cloud as well as on-prem. And we've seen a lot of community usage in both places as well. So Valerie, I believe we're at the top of the hour. We've even gone a little over. So I don't see any more open questions. So we can probably close it down. Thank you so much for sponsoring us today. It's been a pleasure and an honor to present in front of people in the CNCF. And if you have feedback, here's our emails, please do contact us. We'd love to hear what you think about this. Thanks Robert and Vlad for the great presentation. So since we've cleared all the questions, the webinar recording and the slides will be online later today. Thanks everyone for joining. And we hope to see you in the future CNCF webinar.