 Hey, thank you very much Christina it's great to be here today and I'm really excited to be talking with you along with Chris Bradford and our goal for today is light on slides heavy on demos if I'm right so we'll see how we do with that and we're going to be talking about unboxing Kate Sandra the data layer for your Kubernetes powered applications. So first of all, we want to start off from the very beginning. What is Kate Sandra and before I before I kick Chris off on that. I'm going to have you take a look at the logo and see if you can kind of guess here you see a little bit of the nautical element with the sextant right and then there is this star here and there there's a little bit of a clue there so but once you unpack it a little bit for us Chris. Yeah thanks Jeff so. So Kate Sandra is a bit of a mash up here we have Kubernetes right as well as Cassandra or Patrick Cassandra. And we're going to talk about running Cassandra on Kubernetes, but also some of the pieces that enable you to run a production like system. So specifically, not just the database I think it's pretty easy to just spin up a pod or container and in the case of Cassandra a few of those, but also the supporting technologies that surround it. Yeah, well that's so I mean we do got to start with the foundation of this and we've already tipped our hat that it's Cassandra on Kubernetes is the basis of this so you know we're going to assume some familiarity with Kubernetes here but we want to we want to kind of mention why, like, what is what is the problem that Kubernetes really solves. And, and why did it when it was competing with other orchestration systems for containers right so. There's some aspects of this that I think that will make it a natural to go with Cassandra but let's unpack that for a second. What do you think Chris. Yeah, definitely. I think one of the things that we want to look at specifically is. What is what is what is Kubernetes solvent right. It's really a platform for deploying a number of containers across a fleet of servers. I've heard it described as the operating system of the data center and I think that's pretty apt description. When we talk about the system like Cassandra which will do here in a second. It runs across multiple servers already. And so it's interesting to see how multiple distributed systems kind of come together to less into a scalable self healing. If things are configured correctly platform for deploying your applications. I think that it's removing a lot of the TV from the work that I used to do in the knocker on individual machines. Awesome. All right and that brings us to Cassandra. So Apache Cassandra is has a lot of similarities to Kubernetes and that it's a distributed system right. So at the surface at the high level we have these common terminology that are common to both as distributed systems of nodes and nodes that are formed into clusters. But then there is a point at which the similarities stop and there are a few differences so the mapping maybe appears really easy at a high level, but then at the lower level there are some details that we need to work out and we will kind of talk through that later on in the presentation but let's go let's let's step back and say like why Cassandra like what makes Cassandra a great solution for cloud native architectures. Sure, so one of the things that I think Cassandra sun really well is it's, it's a peer to peer database. There's no leader and follower there's no concept of like a right replica and read replicas. So any node can answer any query and it will route that to the appropriate actual instance that posts the data. So it's, it's a really interesting fit here when we look at a system like Kubernetes where traditionally it was really solid for state workloads or sorry stateless workloads. Those are pretty easy to scale you just spin up more of them and make sure they're added to your load balancer in a way you go right. So if you look at stateful services like Cassandra or really any database, things get a little bit trickier you can't just add a new node and expect it to be fully functional right away. What if a read request goes that well it doesn't have any data when it first starts. So there's this process that you have to go through to make sure okay this is ready to start accepting traffic or if note goes down it comes back up is there data there to operate on or to drive off the other notes in the cluster. So it's, because Andrew is a really interesting fit because it does some of those things out of the box. And what I like about it, at least in the realm of Kubernetes and data on Kubernetes is that it kind of aligns with the expectations of the Kubernetes has a bit of its containers, and it brings a lot of the behavior of the, the stateless where you can scale out pretty easily right and you have high availability, your self healing of no goes down it can come back up that you would get with your stateless workloads you get that with your stateful workload with Cassandra. But do you want to, Jeff, do you want to go into a little bit about Cassandra you have a experience or you've written a book on the topic. Well, that is true. I mean, I have written a book on Cassandra. One of the things that I found interesting over the past few years. I've actually done two editions. I did a Cassandra book for a Riley and this I did the second edition and the third edition and went about five years ago when I was writing the second edition. It was very it was clear that I needed to add a section on running Cassandra in containers in Docker, and I ended up putting a note in it was basically like yeah you should totally try this out and use it in your dev environment but I wouldn't be proud. So that was kind of like the my word of record for three years until I did the third edition of the book and by then the landscape had completely changed. And you know I was like, yes, you could play Cassandra in containers and it's going to work fine. And then it was kind of like, and this area of deploying in Kubernetes and managing with Kubernetes is emerging. And again, it's probably almost a year and a half since I wrote that. So now, we're in a very different landscape here. And I think you've had kind of a, what do I call it conversion experience of your own. I mean that we had a blog that just went live yesterday where you kind of shared your story but why don't you nutshell that for us. It was actually really fascinating, just at a high level. If you ask me, just a number of years ago, probably up to whether you should be running databases on Kubernetes I've been like, No, no that's a horrible idea. It's interesting to see like how Kubernetes has grown to accommodate staple workloads from the primitives with persistent volumes, persistent claims, staple sets, and then starting to leverage those from the database side and say okay, this is a little bit more reasonable. So we'll get that link out in the chat here in a moment. So you all can take a look at that. I don't want to go too deep into the conversation there. But I was a former skeptic and now whole heartedly embrace the data on Kubernetes approach. Absolutely. Good. And I think that to the point we have a question from John Doe. And I don't know, John Doe, John Smith anonymous friend that is asking if we're if we have anybody using Keith Sander and prod yet. And do you want to talk to that a little bit like I think we have different pieces of the Keith Sander ecosystem that are in various stages of adoption. As well as kind of a whole assemblage that we're about to unbox for you so. Yeah, I think it's important to note so we started with with our journey into Cassandra on Kubernetes we started with an operator, and we'll talk about all the other operators during the second but we started with just an operator. And that's grown to containerizing and productionizing multiple components that make up a cohesive stack of the platform. That's what Keith Sander is. It's the collection of all these technologies to run a production ready workloads. There are people that are running Cassandra on Kubernetes today in production with Cass operator. And some of the other technologies, as far as the packaging of it all under one name. That's something that's currently in flight. Great question. Yeah, that's I mean that's a relatively new we had a one dot release couple months back and we're dropping a one one release this week with some enhancements that maybe we can talk about in line here as we get to it in the presentation but one of the things as we talk about unboxing right and looking at what's inside this case and or distribution. One of the architectural goals that we had was composability. So we are saying when when you get a release of Kate Sandra that we're saying this is a blessed configuration that we know and are confident saying this all works together and is packaged together well. And yes you can customize it and swap other pieces in and out so I feel like we've been teasing the unboxing part for too long now let's let's push on. Let's dig into it so the first component to really look into is is monitoring any production system, well any system that I want to shift production, maybe it doesn't always happen but I try to make it happen should have observability for monitoring. And so, Kate Sandra has taken the cube for me to stack helm chart and brought it in, but we actually handle all the wiring of Hermitius and the running Cassandra cluster so the case and distribution is going to make sure that we enables the metrics collector for Patrick Sandra inside of each of the containers. And then it spins up the service monitors and the dashboard starts to wire this in to Prometheus and Grafana running inside of your, your Kubernetes infrastructure. Now, you might already have that in your infrastructure. I'm seeing a number of vendor backed Kubernetes just rose actually already shipments reinstalled it's centralized centralized monitoring is something that you would see. And under the banner of composability, you can just turn this off like you can say, I already have my own for me this already have my own to find it just don't bother with that. But we have the hooks in place that if you do want kids center to still push out the service monitors which provide the, the plumbing between for me thesis great targets and the actual notes, we can still provision this for you without setting up an entire right and that's a that's a really nice and flexible configuration. One of the things that I love about Grafana, you know as a as a platform for graphical and Prometheus as well is, you know the fact that I can build a complete observability solution for my application, all the way down the right so I can have graphs that are showing the workload that's coming in on my microservices and my application tier but also put that alongside graphs that are showing what's happening with underlying standard database and I can you know compose my own dashboard that really kind of gives me a top to bottom picture, which I think it's just amazing. I love the composition of looking at your application metrics and looking at looking at your database metrics. Okay, where exactly is this going sideways can really give you the whole picture. I think it's right, though, it's some people might say well what if I don't use Prometheus, what do you think about that. Well, as we've been talking about the, you know that that format of the Prometheus, the way that it represents metrics is becoming some of somewhat of a de facto standard and so we're seeing a lot of other monitoring stats provide compatibility with that so I think that's our short term answer for that right. Yeah, even if you don't use Prometheus a number of like the monitoring solutions out there can still scrape those those metrics and ingest them into their systems. Alright so monitoring knowing what's going on is one thing, but there's definitely some operational tasks that Kubernetes itself isn't going to magically just do for you so I feel like we should talk about some of those. Yeah, and I think one of the key things that comes to my mind is well what happens if some monitoring is great but all of a sudden I just don't have notes. What do we do then. There's a stray command point in the wrong namespace and now I don't have my cluster what can I do. It's not just human error too right I mean we have a worker nodes that can fail in a Kubernetes cluster and it's designed around that. But now what if those notes have data. Yeah, and that's that's where the stateful conversation gets interesting right and so we've we've taken the, the Medusa project out of the last pickle and I think it trying to remember if that came out of Spotify or if that was the repair system but in any case. It, it, it powers backups via Kubernetes custom resources to a object store so you can instead of having to kind of orchestrate this by hand you just say hey I want to send your backup. This is alright. It will reach out to all the nodes trigger the backup operation snapshots the data files, then ships those off to an S3 compatible object store. And that's part of the, the next release that Jeff into that earlier is compatibility with men IO, and as, as an example, an example as to have a store. Yeah, but what do you think some of the other use cases are for this kind of backup and restore functionality Jeff. Yeah, we're starting to see a lot of interest in incorporating backup and restore as a tool for building CI CD pipelines so in other words I want to spin up. You know, an instance of my application stack in order to run integration tests against it let's say and we want to as part of that to have the database there as well. So maybe I want to have an initial state that's represented in the database so I can actually use, it's pretty handy to be able to restore a data set and then, rather than having to do like a bunch of data loading tasks, or run Kubernetes jobs to do that, you can actually just do a restore and you, again, because there is an API for this and Medusa, you can easily do you know just hit that API do the restore and script that as part of your pipeline. Yeah, it's pretty cool. And it's all done with cube CTL to so there's no extra COI or any any interfaces you have access to custom resource inside of your environment. That's a great example. You're using Cassandra friendly terminology and abstractions that are built on top of Kubernetes at that point. One of the interesting things about it should be the system though is is not is that data can sometimes get out of sync and we do everything we can to make sure that doesn't happen. Cassandra has been around for things north of 10 years now. And so there is we call repair. I like to refer to as anti entropy things are going to get a sync though whether that's maybe a GC ran too long, and Mr right, or the node was just down when I right came in. And, like I said, there's a layered approach to resolving these issues. But I think it's worth noting that there's one that's called the out of band repair process. In this case we're using a process called Reaper, which handles calling the API endpoints to trigger these repair processes. We have a best practice where you want to do this every 10 days across your entire cluster, so do a full repair. Again, ideally it's very minor repairs that need to happen. And there are other systems in place to assist with this. But what this tool does is instead of you as an operator having to log into each server and say all right, survey it's your turn today, running the repair command. This actually breaks it up in the smaller chunks and runs it over over that entire 10 day period. Right. Yeah, this is one of those things that's, you know, one of those fine print kinds of things that a lot of people forget to do, you know, in order to maintain the health of their system until all of a sudden they realize that it's a problem. So it's not that big of a deal if you start doing it from day one, but then if you, you know, if you don't follow the proper operational procedures and ignore it then that you're not in a good situation either. And for what it's worth, I like the phrase always be repairing. And even when you're doing your capacity planning you should make sure you have repairs turned on and running. So you can account for that operational overhead when you're trying to figure out how many nodes you need for your cluster. Yes, yes, of course. I mentioned here a second ago. Taking this away from the human operator operators kind of an overloaded term in in Kubernetes land. Taking away from the operators with the DBA is right. And instead of having them run this, we have a helpful tool here but when we talk about running a distributed system. It's very tricky and automation helps with that significantly. I mean, I've, I've seen clusters that had hundreds of nodes that were all configured by hand. Very quickly moved over to an automation tool I think that particular case was Ansible, and it was helpful right you can still log in and say okay I want to deploy Cassandra to all these servers. And then go through to each one install the packages and start it up and then repeat throughout the rest of the cluster. But I tell you what when you get in the hundreds of nodes that can take hours can take a lot of time. And so one of the things that we've created. So here was Cass operators, it's a Cassandra operator, and it does a really good job of translating these like logical concepts of Cassandra data center and Cassandra racks and the size of a cluster in the topology and all this into Kubernetes resources. And so it'll say okay I know that a logical rack inside of Sandra is equal to a very stable set. Yeah, you describe that rack to me, I will, I being Cass operator will go ahead and say hey Kubernetes API, can you make a stable set for me. And just like we mentioned with with repairs and with with backups. And there are other operators as well as the Medus operator and the repair operator handle that translation for you. But they're also Kubernetes controllers that do reconciliation so in the case of a pod going down for Cassandra. Kubernetes will restart the pod right and try to bring it back online, but in the complexity of running a distributed system maybe we want to restart pods in a certain order. So basically the common operations is a rolling restart of the Cassandra cluster, and so you can just in your customer resource say I want you to restart, and the operator will handle terminating pods letting them come back up in a rolling fashion build the cluster performing upgrades that kind of thing. Yeah, that's the upgrade case I think is super interesting. I mean, basically it's a, it's an instance of a rolling restart with a little twist. We're just going to change the binaries on you. I feel like we've been given team we've given a lot of love to the operations side here Chris I don't know. You should finish your point but I want to talk about devs to you. Oh yeah sure. I think one of the things that that is worth calling out though is, and we go to do something like a, like an upgrade you want to make sure that everything isn't just going to go sideways. I would I would hate to say okay switch out all the binaries into the cluster to not come back up. One of the key features that we really don't talk about too much is we allow for canary upgrades so we can target a single node, or a selection of nodes to upgrade before moving on to the rest of the cluster as a whole but to your point Jeff I do think we need to switch gears and start talk about how the heck do we actually talk to the system so to just run it. Well that's right I mean if we have a perfectly running database that no one's putting data in or getting data out I don't know that's much use so. Personally in a Cassandra database we have the Cassandra query language. That's very similar to you know classic sequel that that we're using. One of the things that we've found is that a lot of companies have begun building out their own API layers on top of Cassandra and not necessarily letting every application developer just write raw CQL queries. Definitely. And so, you know, the purpose of this stargate project that we started this is another open source project that's kind of a, you know, a companion or fits nicely with Kate Sandra, and we've incorporated stargate in the Kate Sandra. So it is CQL compatible access to Cassandra but also provides rest graph ql and document API's so the idea is you know, no matter what flavor of API, you're most comfortable with interacting with as a developer that's that's something that's available to you. Yeah, and you might have an existing application that already speak CQL, but one of the things that can be tricky at least it was for me when I first started running Cassandra back in 2013 I went to Cassandra summit as a buddy had an extra ticket. And I spent the second day, just trying to use Cassandra and being so angry because I was like this looks just like SQL, and it's not, there are certain things you're not allowed to do because it's a distributed system. That's fascinating, but I tell you what if I had a rest interface or document interface, I would have been able to go a lot further that day. So let's take a look at what that looks like. Jeff, do you want to dig in a little bit into kind of the, the interfaces that are. Sure. I mean, this this slide can you know appear pretty busy. I'd encourage you to look at it and two halves the top half on the bottom half. So the idea is that starting it isn't an extensible architecture. You can add in new API's so existing right now or rest graph ql and document API and of course the center query language. So there's a Kafka integration in progress pulse our integration on the roadmap, and then on the back end. It's also designed so that you can plug in a compatible data store so right now, the the ones that we have to date our Cassandra compatible databases. So open source Cassandra 3.1 311 the 4.0 release which Cassandra 4.0 is about to go live here imminently there's a release candidate out there right. And so we'll be upgrading case Sandra is just as soon as that goes live. So Kate standard would ship with the official Cassandra 4.0. And then of course the data sacks, the enterprise distribution as well fits. But the idea is that you could plug this in. You could do the work and want to if you wanted to plug in a different database engine on the back end. So very composable that's the architectural principle that's at work here. So one of the fascinating things I think is kind of interesting here we've done some testing the space is that you can use Stargate almost as a think about the traditional separation of compute and storage people talk about inside your data centers. You can do something similar here with with Stargate it's, it is stateless layer in front of your, your, your back end database so if you have a large number of clients, rather than having the actual notes that hold the data handle like switching context and things like that. You can still that layer independently of your data layer. We've seen reductions in latency, while reducing the total number of notes in the cluster which is kind of fascinating. This is especially great for read heavy workloads, so. So let's look at what it takes to actually install and Kate Sandra and actually fire it up inside of your, your communities environment so it's it's pretty straightforward. You add the helm repo for Kate Sandra. You go to make sure you have the latest version of the chart, and then you just install it. And given that we have a helm install command here I think it's probably a good time to transition actually shown you what this looks like. Why don't you just show us yeah. Yeah. So I, I've actually run the same command here inside of my terminal. I have my helm install here I'm overriding a couple of values that are unique just to my machine. It's nothing crazy I just told it to use a specific storage class inside of my Kubernetes cluster called local path and the data volume I want to use this is one gigabyte in size. When this, this takes about two minutes per node to spin up but rather than waste your time here I've already run it so let's go over here to lens though. I make this. I cannot make that a little bit bigger that's okay. We, let's take a look at the components here that have been deployed so if I go back and I look at my, my customer resources. We can see there is a Cassandra data center that's been provisioned called DC one. And if we go and actually look at it. There's, there's a bit more here's this is all templated out with helm. The fields tends to be a bit verbose, but you can see we've configured password authentication. We've set the number of replicas. This is a single node installation because I'm running here on my laptop. But if you're in a larger environment, let's say you have a three node cluster. We could set the replication factor to three. We describe it in this case we have ways to override certain pieces inside of the standard data center. So in this case we're configuring Reaper. We mentioned that part of Kate Sandra is we wanted to have a cohesive platform. And so we we handle setting up things like a net containers for you, instead of having to roll that yourself. But let's say you have a special system for handling secrets or for SSL certificates, you can actually add your own a net container in here to request those certificates and add them to configuration. So this is secure by default. I think that's an important thing to note is that when we deploy Cassandra and in Kubernetes here. I mean, it's, it's not just wide open access right. Yeah, and for what it's worth. I don't know the credentials yet we're going to find those out here for how to connect to the cluster. But we can see we have helpful status as well. I believe we just added to the docs it's really nice one liner where you can just say hey I want you to wait until the Kubernetes until the Cassandra cluster is up and running. But let's go back to our pods view and take a look at what's going on here so here we can see a number of components first we have the actual CAS operator that we mentioned before. There's the operator for Meteos Reaper and Medusa is not enabled on this particular instance. But if we had enabled Medusa we'd see the operator running as well. There's a job that ran to configure the schema for Reaper which just is for tracking the progress of the repairs. We can see our Grafana. And then these two are the really important ones. In my opinion we have our stargate pod so single stargate instance, as well as as the Cassandra pod. And what's fascinating here is we mentioned a little bit earlier that Cassandra racks are analogous to stateful sets. And so when we when we talk about like the topology of a Cassandra cluster what's the purpose of a rack here Jeff and go into that a little bit. Sure, I mean in Cassandra we use racks because we want to achieve high availability of our data by using replication. So the idea is that a rack is a representation of kind of like a physical server, or a failure domain, such that, you know if we have a piece of physical equipment that's running our node in our Cassandra cluster that goes down, we don't want all of our Cassandra nodes to be, you know, on that piece of hardware that goes down we want to have some distribution of Cassandra nodes that are running on different hardware. So these concepts that are built into Cassandra's terminology like data center and rack are used to manage the topology of the cluster such that the different copies of each piece of data that are stored are actually spread out across multiple failure domains. So if I was looking at a cloud environment, would it be safe to say that a rack is like an AZ and a data center is more analogous to like a region. I would say that that's the kind of the default assumption I think you know if you want to have additional logical data centers for different reasons like you want to offload some read heavy workloads or something people do do that, but I think the the layout that you described is kind of a good default. Okay, so looking a bit more at the at an instance of a Cassandra pod or which is the same as a Cassandra node. We see a number of containers first there's this and it container called server configured and what this does is actually takes a JSON structure of the configuration and this matches the. The config block that we see here in the Cassandra DC. Custom resource, and it actually takes that and it renders that that into a sweet of configuration files that you would see traditionally under like Etsy Cassandra on on your virtual machine installations. So this applies a number of defaults but for the most part this is this is what's used to actually build the configuration of the individual pod. The credentials just sets up the reaper and super user JMX credentials. Now we get to the interesting containers so those were net containers the actual running containers the first is Cassandra so this is the actual Cassandra process. We can see has a number of ports exposed include those for from Atheist, the management API which is used by the Cassandra operator for certain tasks. It's interesting when we think about multiple distributed systems growing up kind of in the same time we mentioned that Cassandra is going on 10 years old, or might not be 10 years old. Oh yeah it's over man it's like 12 or 13. Oh my goodness I'm so out of date here. And, and Kubernetes, which is what five going on six years old. They've kind of, you see a bunch of similarities between the components, but where some other items have differed. So one of the things that's interesting about Cassandra is when you go to start notes the cluster. You want to start them in a particular order. And with, with Kubernetes you can sort of do that but what we do instead is we schedule all the pods, all the pods start, but they're just running this management API. And then once we see that all the pods are running, and they're ready to start, then the Cassandra operator actually reaches out over this, the secure interface and says hey, go ahead and start your Cassandra process. It's just a way to orchestrate some of the differences between how these two technologies have evolved and like I said they're similar but sometimes they're a little different. It's not a little bit of scaffolding you need because, you know, in, in, in Kubernetes it wants to treat every pod the same, you know that's of a given image. In Cassandra, we know that, you know, it is a, it is a peer to peer architecture where there is no one node that's calling all the shots but it doesn't mean that there isn't some sort of coordination, you know there is there is that kind of communication that nodes kind of need to negotiate with each other as they come up so that, you know, that's a little bit of that behind the scenes scaffolding that it really takes to operate Cassandra effectively in Kubernetes. But for what it's worth based on like what I mentioned before starting up hundreds of nodes could take hours right. Yeah, doing a rolling restart to take hours. The nice thing about the operator here is you say hey, I want you to start the cluster, you can go get a coffee right you don't have to check to see if like certain hosts were skipped because something like that happened. You can just come back and your customers are they're running or it's reconciling and working to make your cluster run. And like I said if you have a local registry, the start time per pod is a couple of minutes. And it's, it's really refreshing to have something else handle all that work for you, instead of instead of having to babysit an SSH terminal. That's right. So looking back here though we have another container here called server system logger. And this is just, we mentioned that the management API we wanted to make sure that there were two log streams so you could still use qql logs. The server system logger is actually the output of the main Cassandra process, and the output of the Cassandra container is the management API process so you can look at the separate log streams individually. We have a number of volumes we really don't need to go into that but we can see we have a running node we have a running stargate everything screen across the board so let's take a look at how we actually talk to this is pretty interesting so if you go over here to the network. And we'll get services whether there's a whole suite of services whether you're looking at the monitoring interface which we will take a look at here a second, or the, the Reaper UI. But there are a number of, of headless services that represent your Cassandra cluster so your application would point at maybe the DC one service, where your monitoring might point at the all pod service, because that one shows pods that are not yet. But there are a number of endpoints there's also the stargate service which is where I would point my applications. Finally, we want to get into the actual secrets so this is how we actually we communicate with the cluster so let's take a look at the super user so Kate Sandra has created our cluster, it's also created our user account. So here the name username is Kate Sandra dash super user. And this is just a random string. You can create the secret yourself and use predefined credentials. But just for the sake of the demo here we want with the default which is just a make it for me. So let's go and kind of explore the interfaces a little bit. I'm going to go back over here. So I want to connect first to our monitoring system. So this is services. And we go to get Senator fauna. I should just be like this. And there we go. Yeah, now I'm not taken to our. Did you put forward that to expose how did you get that access so quickly. Oh lens is awesome so the Kubernetes lens tool if you click on the port will set up for reporting for you. Awesome. And it's admin secret. No, no, go away. All these plugins are now like we want to save your password. So you're going to see this says that there's nothing here it's misleading. The dashboards are already installed here we go. And we can see we have for pre installed for you to go to standard overview. The cluster is up and running we can see the status of our node will run some queries against it here in a second. But we, again, you didn't have, I didn't have to do anything here to set this up. All the automation spun this up for me. The really interesting thing is if I change the number of nodes say from one node to three, they'll automatically get added to the monitoring system for us. And that's not it's like another item in the checklist that it didn't accidentally for you to do again. Right. This is just handled for you, which is refreshing. And so, besides just Cassandra we also have a stargate a separate stargate. dashboard a condensed dashboard. And if your application is using metrics or if you see centralized logging, you can pull this into your own structure. And that's the monitoring interface. I do want to just mention that. Meteos, one of the one of the health checks that I like to do to make sure that my cluster is healthy as I'll go and I'll look at the targets and make sure that a that they exist here. And be that we're seeing all the labels come in so you again you can see that me this is scraping the metrics from both my stargate node, and my, my Cassandra. That's a great tip. Yeah, if something isn't showing up in here I always go look at Prometheus first. I've gotten into plenty of edge cases for things just kind of going sideways. But let's talk about let's see where are we on time. I want to make sure I don't. We got about 10 minutes left for the main the main piece. I wanted to interject a quick question about where you're running this as you kind of transition here. Is this on your desktop or are you running in a cloud. Does it matter. It doesn't matter if we test and run on a number of Kubernetes versions. In my case I'm running on K3D on my local laptop so I'm running a single a single node with with a load balancer as part of K3D we test and kind with our integration tests. And I think that's kind of the current build out we're building out a whole suite of integration tests that target the major cloud Kubernetes vendors, but you have to go to the project and see. Yeah, a number of Kubernetes versions and test against those. This is where I think the composability of this is really helpful because I also mostly run. I was talking to Sandra on K3D on my desktop and so that's kind of like why you have that configuration where the Medusa is turned off because maybe if you're just doing dev work locally, you don't need backup and restore capability. Okay, so just don't, you know, turn that part off. Yeah, and I was, I was doing some work just the other day where I didn't, I didn't really need repairs locally either so I turned that piece off, just to slim down the amount of resources that were needed just for my development. So, John Stajinowski has a great post on the Kate Sandra IO site about like what is that minimum development configuration, because we know a lot of people like to run, you know, just the essentials locally so definitely look for that that post on the on the blog on Kate Sandra that IO for that minimum config. There's a fascinating conversation around, do we go with a single single node or do I run multiple notes in my development environment, and that way you can simulate what happens when two pods go down and maybe you have a reduced consistency level. Yeah, what's your recommendation. Personally, I like to just keep things really minimal on my local machine so as I wrap up features or as I do my testing, I'll run against a cluster with with multiple nodes, because I do like to test this failure scenarios. But I also am cognizant of depending on the size of the data maybe a single node is right. Um, this this is kind of related to you I don't want to be really too much sorry Chris, but there's a there's a pertinent question from Eric Rodriguez here, what are the hardware and storage type considerations if any for running Kate Sandra. In other words like what instance type and backing storage you want to use in your various public clouds I'm paraphrasing the end of the question there to broaden it a little bit. Sure, yeah so it's it's it's interesting. We just use a storage class to to provision our storage so here I'm using the local path because well that's what comes with with rancher and with K3D it's from rancher and it's pretty easy to use, but if I was going to deploy into GKE. I may use something like a PD SSD. If I'm if I'm looking at EKS I'd be looking at like elastic box storage. One of the nice things about Cassandra is it does a lot of the shuffling the data for you so you can, for instance, leverage those ephemeral volumes which are super fast, but the downside is. Yeah, they're pinned to the server right and so if a Kubernetes worker goes down what happens that it well, you have no guarantee that it's coming back. So let's think about the operators in case Sandra is well if you're moved to a new Kubernetes worker. We actually are aware that you're using a volume that can move or not and if it comes up and it says oh this volume is empty, well then we'll reboot strap with from off the other nodes in the cluster. Now let's say you're using something like EBS where the data can follow that pod to another worker. And then it will happen and then we won't have to do that bootstrap that that full bootstrap process will just bootstrap the data that we already have and rejoin the ring so it's a great question I think it's highly dependent on your use case. You're super latency sensitive I'd be looking at it from volumes. If you have a little bit more flexibility and want the, I would call a more available story, it's misleading we're not. It's not highly available using local volumes it's just the rescheduling and spending back up where we move volumes a lot faster than re hydrating like an entire terabyte of data off of existing replicas. And to peel back the covers just a little bit we are working on more extensive documentation and recommended deployment options per cloud AWS GKE. Azure, right so there will be more coming in that area with even more specific guidance. Definitely, definitely, and I expect to see a number of outputs in that area to select like we saw before with the Cassandra data center from a storage perspective, you just specify the storage class and then the size of the volume. It's right here. So we're asking for read write wants the local path provision. So if you're on a cloud provider they may already have a source class name or you can create a source class with the name of their provision. Again, it's highly dependent on the product you're using or if you're on time. Okay, so I'm going to take a look real quick. I'd like to show, we've talked a little bit about Stargate and I think we should see if we have a second to just show what that looks like. So let's go forward Stargate. Using qctl go over here. All the API is exposed. Yeah, just a single command would be nice right. And so we're going to copy our password for the Kate Sandra super user. So we're just going to look at that copy. If I go back here to postman. The first thing that we do is we authenticate with a token to retrieve a token. There we go. So we send this. And it comes back it gave us an off token I've taken the liberty of stashing it here inside of an environment variable so it's easy to use. But from here, so I'm using the document API so document API has a flexible schema you can just kind of send it whatever JSON you want and it'll work. So here we can see these are all the namespaces that exist. But I don't have one from application yet so we'll go ahead and create one. So let's say create a new namespace named my app. It created the namespace. And then if I go back and I read that name space will see it exists. The next thing that I'd like to do is, is create in this example I'm talking about surveys it's created a survey. There's no survey table you can see the URL is namespaces my namespace name my collection. Here I'm going to create a survey, and in this case it's just a title and an array of questions that's empty. And this comes back with the document ID this one takes a little bit longer because it's actually creating the tables in the background said it didn't have tables before now it does. And so here we can see the document ID, if we go back and list out. We can see this is the first document I've submitted. And the second thing happens if I want to create a new, a new collection this case questions. I'm submitting. What is your favorite color, and I've just, I'm using environment variables here to keep the IDs in place so now I have a new document ID, I can actually update my existing document with that new ID. You might be saying Chris this doesn't. Why, why don't you have like nested documents inside of here that's a fair question. And one of the reasons I'm talking about it in this kind of context is I like to play around with my data model here with the document API and then start to turn that into more concrete tables that I use with regular rest API or even with CQL. And, you know, for those that might be less familiar with Cassandra, this is, or even those that are right, not what Cassandra interactions have looked like historically. I'm just going to experiment and play around with data model and try some different document formats. It's not how things have worked to store. So this, this document API that that Stargate provides is a bit of a game changer that really kind of changes the developer experience for working with Cassandra. So, like, what you just saw is something that five years ago would have been like you did not just what I like that I can just start adding fields willy-nilly and not having to like go into CQL and say alter table and add my columns and think well do I need to add this as part of the key and all all that stuff. Right. So, I think this is, this is a great point I'd love to go into a bit more but I think this is a really good time to transition into talking about questions that have come in. Oh yeah, you should. Let me just tackle this thinking about it. I started writing an answer to it. I was like, wow, this is too long to type. So, Sharvesh is asking so our enterprise is using Oracle. And we're ready to move most of our apps to Kubernetes clusters, but how do we present Cassandra as a good option for cloud native apps so I this, there's multiple levels on which to answer this question. So let's think about it in terms of developer experience. So one is what is the skill sets that your application developers have versus what do you want them to have. If they are prepared to learn CQL. I'm great in a lot of cases that might not be what they want to spend their time doing it all depends on the needs right so having something where you have flexible API is like provided by Stargate. That's a good developer experience story. What I want to look at is the cost story. So the cost of operating this system. Are you able to tune and provision the resources to exactly what you need not only for peak but can you also have an elastic system that scales back down. And, you know, so that there's those considerations as well. There's there's also operational cost of how much labor, do you have to invest. There's a lot of maturation in in this Kubernetes world for a lot of different databases there are a lot of different operators out there. Especially on the relational side it's almost like a wealth of choices and you have the interesting problem of having to evaluate multiple different operators and pick which one you're going to use in the Cassandra world right now. There's a smaller set of those but we've been working with this CAS operator project to kind of take the best of all of the the existing Cassandra operators and kind of bake that into kind of one operator that the community is really rallying around. So those are the two main ways I would look at it is from the developer skill side and then the operational cost side. I hope that's helpful. And then also I would just say you need to, you know, experiment and try this out for yourself and give us the feedback. You probably heard that we don't have everything answered yet. This is very much like an emerging area that we're putting a lot of work into you and a lot of collaboration and a lot of input is required so we. We love getting the feedback about what works and what doesn't. And the questions that you have the questions that you're asking are honestly gold. Yeah, so Edward has a question all the key standard features are free to use or the full version is paid. This is an open source project, you can download it you can run it there are no licenses involved but there's the Apache software license but that's, that's what the code is distributed So there's, there's no issues there. If you're looking for support there are some, there's self support through the repos. There's also paid support through data taxes Luna offering. But there's no, there's nothing stopping you from running us today. Yeah, do you want to take this next one to you about if you if you already have a cluster running with CAS operator. How do you just integrate the rest of the key standard elements kind of on top of that in place you could do that right. So we're putting together the documentation on this it's. There are some assumptions that are made with that Kate Sandra it plugs into the, the existing CAS DC or into the CAS DC. I want to make sure I understand the full implications of what that would do to existing CAS DC the seamless way that I would pursue is I'd set up a new data center with Kate Sandra, and have it connect to your existing data center. So that's being managed purely by CAS operator. Then you can have it migrate the data into it and spend on the old DC after you perform migration. It's, it's, it's zero downtime. It's, it's pretty painless we have some documentation coming out about this year and in the coming weeks, but rest assured we have been looking into that. We would love to the specific details of this use case so please reach out to us. The community page on the case and on the case and or died I oh site lists out this way as they can get in contact with us probably the best way right now is our mailing list but of course file in on the on the GitHub repo is also fine. We just have a question like if you have a specific issue on issue, and then we are working toward launching forums so we can have a more responsive turnaround on your question to see the answers to that's coming live very soon. But there's another question of how can I add these dashboards to an existing Grafana. So with Kate Sandra, we have a flag that if you don't want to deploy the full keep me the stack you just want us to provision the dashboards will create those in a config map, and then you can wire those into your existing Grafana installation. So if you have a type of installation that doesn't support that kind of deployment of dashboards. We have links to those in the project, and they're coming from the metrics quite a bit for that you can send your project that. So that that's also available so if you can't use the config maps. You can just have the JSON files in URLs that you can put in fairly certainly Grafana pod supports downloading dashboards for URL as well. There's there's a couple ways you can bring these into your existing Grafana environment. There's a question here Jeff about the performance. If we're putting in additional layer of distractions that I think the, the, the person Robert is referring to stargate between your clients and the data nodes and. Certainly, we would say hey just point your clients directly to Xander cluster maybe it'll skip an extra hop or it knows directly about which needs to go to. Do you have any, any feedback around that performance question. Well, I think it's it, it really is going to depend on the deployment that you choose I mean one of the things that we're we're doing here is the starting nodes are deployed very close to the Xander nodes so that is going to cut down kind of on your network hop latency. I'd like to consider the there's definitely some trade offs here so you know one of the things that Stargate nodes are really doing is providing a layer that that can help you scale out reads a little bit better and and take some of that traffic that's off of the core Xander nodes that maybe can focus a little bit more on handling the right load so it's it's it's sort of like a matter of how do you configure this and what's your ratio of starting nodes to old school Cassandra nodes. We, one of the things that we're working on is a blog about how you kind of run your own performance tests using something called no sequel bench and another open source. It's a tool that that we've been working with. So look for more content on how you do that here in the near future. There is a question here about in the architecture we're showing both GraphQL and SQL. Do we need both of them. No, it that's, that's there so you can choose the language that you want to use whether you want to communicate with GraphQL, or just simple rest calls or the document database like we showed there's also the native protocol. It's really about flexibility so if you have no need for the GraphQL endpoint what you just don't call it. And it's not an issue, but there are some architectures that like the GraphQL interface and when you start looking at things like federation for GraphQL. It gets pretty interesting as you you can then query across multiple days. That's a whole another webinar. We're running out of time so I wonder if we might just kind of put up some of our last little promos and contact info here and we'll try to get to the questions that we didn't get to and work on on figuring out how to follow up with that. We are doing a workshop at KubeCon Europe. I mean that's less than a month away now. Man, that's coming up fast. Myself and Alex Wachnev. He's a fantastic developer advocate and leads a great workshop. So I'm excited to do that with him free workshop, and that's on May 4. I just shared out the URL if you if you're interested in registering for that. Awesome. I appreciate it. Jeff Christina, you want to take us out. Yeah, sure. Thank you so much to Chris and Jeff for their time today and thank you to all the participants who joined us. As a reminder, this recording will be on the Linux Foundation YouTube page later today. We hope you're able to join us for future webinars. Have a wonderful day. Thank you so much.