 Alrighty. Thank you for that wonderful introduction. I hope it's all clear to everybody. Keith and I have done webinars a few times together. And so as noted, we are happy to get questions throughout the course of this. I know sometimes you wait until the end and answer things, but we are happy to get questions along the way. We're going to give a little bit of a presentation upfront, but we're going to get deep into, you know, we're going to have some fingers on keyboard and show you. A demo here as well on this, on this topic. This is something that is very near and dear to my heart. I'm, I have been kind of in the Kubernetes community for a while. And I'm a real advocate of all things that are happening in our community across lots of different ways. But in particular, you know, running workloads in Kubernetes that that use data and then use state and, and these sort of things. But more importantly, in this context, how do you do that globally dispersed? You know, how do you, you know, are you running multiple clusters and then you're doing asynchronous data replication between like, there's a lot of challenges. I think that we have, and it really comes down to, you know, I think of that as federated applications. You know, the SIG Fed group in the Kubernetes community has had some fits and starts, but it's always because there's like some really difficult things to solve. And I hope what we present here today is valuable to everybody and can help give some context into some of the challenges that that are faced when we, when we do get geographically distributed in our Kubernetes deployments and how do you run our applications. So we are going to use CockroachDB as an example of how we're solving for this problem. Well, that's what we do. We are a database that was kind of designed for this world, but, and that's what we're going to talk about. And sorry for not capitalizing Kubernetes here, you guys, I'm so sorry. I apologize to the entire community. But by means of introduction, Keith, you want to give a quick introduction of your experience in the, in the community and kind of where you come from. Yeah, absolutely. But before I do that, I'm going to give you, I'm going to give you a little bit of, of crud here, because you didn't tell me I had to be polite on this call. At the top of the call, they're like, you got to be really polite and nice to people. You know, that's not what I do, Jim. Why did you do it all the time? You invited me to that. I love you. So Keith McClellan. So I have been in the greater cloud native space for what, seven years now. I started in my, my career in containers at Mesosphere. So, you know, kind of a precursor to Kubernetes. I have, I have been kind of leading our Kubernetes efforts here at Cocker Slabs now for a little over two years. What the thing that kind of brought me to Cocker Slabs was one of the things that's hardest to do in a cloud native ecosystem is manage state, right? So that's kind of why, why I joined the company, right? It's, this is a really hard thing to do. And Cocker's TV quite frankly is the easiest database I've found to run regardless of where you're running it, which makes it perfect for running in Kubernetes. Yeah, in pods on Kubernetes. And yeah, and likewise, and Keith, I go to Keith all the time with some really deeper technical questions. So y'all feel free to ask whatever you want of Keith. We can go to lots of different levels. I originally got introduced to Kubernetes. I was at a company called CoreOS and I, I too saw the problem with some of the state, like I'm a next developer application needs database. How does that work? Right. And so that's how I got introduced to Cockroach. And I'm happy to be here now. It's, it's interesting in things. So let me give it a little quick introduction, five, 10 minutes on it and some slides and we'll get right into demo, you know, Cockroach. And I think what Keith and I both saw was this is something that was architected for the cloud. But ultimately I believe it was architected for Kubernetes. You know, Cockroach is direct descendant of Google Cloud Spanner. It was actually the Google Cloud Spanner white paper, which was the kind of the, the, the, like how we got started here. Three Google engineers, Spencer, Peter and Ben are three founders actually started this company to try to build Spanner, but not tied to, to, to Google infrastructure. How do you build an open source version of that? So that's really the, the, the foundation of where we started, but this is a relational database and all the beauty that you get with the relation database with familiar SQL, with consistent, you know, isolation, a strict, strict sterilizable isolation. But more importantly, this isn't a document model underneath where you're kind of accelerating complexity when you start to change things or the more model, the more documents they have, the more difficult. Let's use the, the simplicity and the elegance of, of a relational database, but make that distributed. And that's really what cockroach delivers, but we implement, we are implemented as a series of nodes. You can imagine each one of these running in a pod, spin up a new node, point it at the cluster, and it just takes care of rebalancing and dealing with kind of where data lives in that cluster. We'll show you that. We could scale further. We could scale across regions. And when you start to scale across regions, this is where Kubernetes starts to have some problems, right? Are you going to have, can you run one Kubernetes cluster across multiple regions and clouds? That's difficult to do. And federating a single control plan across all that, that's some tough stuff. Lots of companies out there trying to figure that out right now. And I think it's interesting. But for us, we just federate at the data layer. We just create one single logical database across, you know, all the different regions. And we're going to show you that in cockroach. But again, being able to do that, it's not simple. You actually have to deal with the speed of light. It becomes a big issue. Now we are going to show you today, not only can we do that across regions, one single logical database. Well, we could actually deploy a single logical database, ask any node for the data that's going to access the data wherever it lives across three different cloud providers. And I, and you're going to see a diagram of what we're going to do, but we can do that as well, which there's a lot of questions. There's just such thing as a multi-cloud application. We have a multi-cloud database, like one single logical database. Any endpoint in cockroach can service reason rights. But cockroach in the name really comes from our resilient nature. You know, depending on what you want to survive from an HA point of view, whatever your failure domain is, is it a node? Is it a rack? Is it a server? Is it a rack? Is it a data center? Is it a region? Is it a cluster? And so there's a lot of stuff that we do within cockroach that allows you to survive the failure of a node without service disruption. What we're doing is we're ultimately writing data in triplicate across multiple different nodes. And then we use raft. It's very similar to the way that SCD works in, in Kubernetes. In fact, we contribute to upstream, to upstream SCD. And a lot of the stuff that we've done is actually under the cover is actually helping Kubernetes. A lot of the things that we've found to fix an SCD. So if you look at raft SCD slash raft, there's lots of innovation that we've done there. And that's a lot of the area we contribute. This is a, our code base. I like to think of as a, a PhD in distributed systems. There are some really interesting things that we've solved for as we've kind of battled the speed of light. What's really interesting, I think, when you start to think about multiple different clusters and the way that cockroach works, because we are a series of different nodes, you can simply set up a cluster and it's one single logical database. Any node can service a read or a write. And that's really important because each node is, is the same binary. In fact, if you wanted to bring up our management UI or DB console, which Keith is going to show you, you could point that at any cluster and God, man, Keith, I hope you don't kill the node that you're actually doing it on later. But that's it. You're going to get into that dude. We've, we had some fun building this demo. I think you're going to love this demo. But our UI is actually service from any one of these you can literally. So like it's, it's, it's kind of a core distributed principle of what we do. We can ask for data, but find it anywhere within that cluster. And that's not a node that's in the U.S. West, but if that data lives over in Europe or U.S. East or whatever, it could find that, right? But we locate data. We allow organizations and customers to pinpoint or pin data to a particular location so that we can beat the speed of light so that we can actually locate data closest to users. Now people love this for other reasons, like can I comply with data privacy regulations to say all German data lives on German servers, that sort of thing. So there's some really unique capabilities and cockroach, but it's very well aligned with the distributed nature of Kubernetes. And in fact, built within the same core principles, the same way that pods work, the way that we survive, the way that we can scale, very, very similar. In fact, the same kind of, you know, at CD and RAF type stuff, right? And so some core principles at the base of what we're doing, very, very similar to, to, to Kubernetes. So that's a quick intro. It's actually an important intro because it's actually going to allow us to explain some principles that are going on in Kubernetes underneath when we get to kind of the networking stuff and everything else. So Keith, you want to walk through, you know, disaster her and set up the demo a little bit. Yeah, absolutely. So when we start moving to the cloud native, actually leave, let's stay on that previous slide for a couple of Oh, sorry, buddy. I was going to the QA and I got lost. Sorry. No, that's okay. So when we start moving to, to cloud native disaster recovery, the, we need to treat a different class of events as if they're an everyday experience. So when we're managing your own data centers, the most common type of failure domain that you're managing for is rack failure where you have, you know, you'll have a switch at the top of your rack that switch might fail. That'll take out all of the communications to any of the servers that are in that rack. So generally speaking, you want to make sure that you spread your applications across multiple racks so that you're not so that a single rack failure doesn't bring down all your infrastructure. Right. As we move to the cloud, we've had to start treating either availability zones or regions as kind of the failure domain, depending on what we need to be able to survive. And that means that we need to automatically heal from that type of a failure, because as soon as a human has to be involved in a, in an event, then that's what basically what makes it a disaster. Right. Because humans are disasters and computers are highly available. Right. So, so when we, when we start talking about this, we want to make sure that detecting a failure is autonomous. That the healing process is, is automatic. That we don't lose any data. That if we do have a service disruption, that's RTO recovery time objective, that it's measured in seconds, not in minutes or hours. Right. We want to, and we want to make sure that the people that are tasked with fixing the problem are the people that are also tasked with building the solution of the problem. So in a lot of cases in traditional DR, where you have two sites, the application team effectively created the problem, but the solution is really on the storage team because the storage team is then restoring back from backups and tape drives or, you know, offsite storage or whatnot. So instead, what we want to do is we want to be able to detect a failure and continue to operate. And we're going to demonstrate how a one of the various ways we could do that here today. I think what's interesting here, Keith too, by the way, I love the process owner. Where do we think about these things? And it's actually extending up the stack from storage all the way up into the application layer. And I think we architect for disaster and we are, we architect for DR now pretty much at every layer. Right. Like I think there was another version of this. We were talking about it from a security point of view and certificates. I mean, there's lots of different ways or points where you want to think about DR in a cloud native world. Right. Yeah. And so what you really need to do is you need to change your perspective on how you think about DR. So, so legacy thinking on this is, is designing for failure. Right. Right. This thing has failed. I need to get it back up and running. How do I do that? What we're doing is we're designing for survival. We saying, we know this thing is going to fail. How do we make sure we continue to operate even when that happens? And that she slight change in perspective informs an enormous number of design decisions around how you build your system and where you invest your time and energy. And what the, the side effect is that it ends up actually being less expensive to design to survive. Then it does the design to fail because for, when you design to fail, I've got to have a complete cold standby of everything. So I have to buy two times my infrastructure. Right. Just to be able to recover because I don't know, you know, there could be a fire in my data center. Right. That's the type of disaster I might need to survive or recover from. If I'm designing to survive, what I'm actually doing is I'm in our case, we're splitting across three data centers. We're going to jump to this here in a second. If I need, if I lose the data center, I just need to be able to continue to service all of my requests. With two of the, with my two remaining data centers. So instead of having two times my infrastructure, I really only have 1.5 x my infrastructure, right? To survive a data center failure. So I can actually bring my, my costs down by being more distributed and using a database like cockroach DB to enable that. And then, and then avoid all the, the manual labor costs of manual remediation of an event. I mean, that's not even active system. It's not even the labor costs. It's the mistakes. Yeah, exactly. I mean, so when I was setting this thing up this morning, because this demo's got a lot of moving parts, which is why we're, we're going to show the, we're going to show the, the Julia child's, uh, finally, final baked piece and not the, I like to think about it. It's one of those fishing shows where they have a scuba diver down there underneath the water and they put like a fish on the hook. Yeah. Very much, very much like that. Um, you know, when you're designing, when you're designing to survive these things, um, there, there's just a lot of stuff that can go wrong. So when I was setting up this cluster this morning, I put in the commands in the wrong order and I had to blow everything away and get started again. And that's the kind of thing that happens when you have a traditional DR, a human runs the wrong restore command or a human does like most disasters are human created. They aren't actually infrastructure created. Like I think I don't remember what the exact statistics are, but it's like seven out of 10 disasters were caused by a person doing something they should have either not been able to do. Or even if they should have been able to do, they should have been double checked before they were able to do it. They should have been able to do other types of things that cause disasters. Are there like me, they fat fingered it. You know what I mean? Like it's hardly everything, anything go around and always does. Right. So, so designing to survive when someone does something stupid, um, like we all do on a pretty regular basis is, is really the most important thing. Um, so, um, if you want to jump to the next slide. Yeah. Um, I just, it's just a really point. It's funny, Keith. I often what's the trade off of, you know, in my opinion, I think, in the new world, the model world, we designed for things to be available and optimize for that. And disaster becomes something you don't, you don't, you don't react to. It's actually you designed into the system almost, right? And so it's interesting. So yeah. So, here's a picture of what we didn't do. I didn't set up three cockroach DB clusters and three different sites. And then kind of try to set up some asynchronous replication application across them because I might lose data that wouldn't be globally consistent. This is this is actually really hard architecture to manage because this is typically what people do though, right? I mean, and, you know, with with instance of postgres or, you know, like what we insert database, right? If you were to do the try to span multiple sites with a traditional database, what you would do is you'd have instances of the database in multiple locations. You'd have some sort of synchronization process. You also have some sort of a sharding process to be able to kind of make this stuff distributed, at least partially, it'd be really painful. It'd probably break a lot unless you had some really smart engineers who basically built a distributed database for you. But if we move to the next slide, I'm going to start show even if you're doing this and you're distributed in a single environment, you're still not solving the problem of latencies at scale across multiple clusters, right? And I think that's, and you're still managing three clusters, but you're managing three databases as well, which operational nightmare. So, right. And well, and the reality is, is your failure domain there is a pod or exactly node, right? And so, um, so what we did was, um, we created a single logical cluster across three different cloud providers, um, we're, we're using the different Kubernetes distributions in each of them actually. So we've got EKS in Amazon, GKE in Google and AKS in Azure, um, mainly because those were the ones that were convenient for me to spin up. Um, I think, you know, Jim, I'm a big open shift fan. I would have loved to have made one of these, uh, an open shift region. And now that Rosa is, uh, is publicly available. I think I'm going to add a fourth region to this demo, but I didn't get a chance to do it this week. Um, under the covers, we had a couple of network peering options that we could have selected. I think folks, if, if anyone that's on the call has, has been on one of our previous webinars, you've seen us do this with something called, uh, Submariner, which does an IP sec tunnel across the sites. Um, it's probably what I recommend for, for most production deployments today, it's a pretty performance solution. Um, uh, but to, to kind of highlight, uh, a different way of doing this, I chose to use something called scupper. So scupper is another, uh, open source project. It creates something called a van or a virtual application network. Effectively we're doing TP, TCP over HTTP. So, um, so scupper creates these kind of TLS tunnels between each of the sites and, um, and then uses that to encapsulate the TCP, um, packets. That, uh, would normally be transported between our nodes. So our nodes talk over, over GRPC, right? So, so the GRPC traffic is then going over these scupper tunnels to, so that we can get, uh, communication between all of the nodes. So Keith, this is like solving. I think when we think about federated clusters, one of the biggest issues is networking, right? Cause you have different networks and then that, that's what you're solving here. The other is security, right? And so I know we didn't do this in this demo, but you have solved for that as well in the past as well, right? Yeah. So in this case where, um, I am pre-sharing, um, TLS certificates. Sure. Um, one of the great things about, um, another open source technology called vault from Hashi corp. Um, is it also uses Etsy D, um, and uses the same kind of underlying protocol as Etsy D uses raft. Um, so you could, um, we could have just as easily set up scupper and set up vault and had a single, um, certificate authority, right? Sign our certificates. Um, that, that's something that we'll probably add to this demo over time. Um, yeah, we have shown it at to your point before you with Submariner, um, Submariner is a little bit more sensitive to having that as a requirement than scupper is. Scupper lets you pretty easily inject like a sub sign CA, which is what I did for this demo. Um, whereas, um, uh, Submariner is, is going to do its own. Um, it's going to reach out to cert manager and actually request certificates, which is the right way to do things. Scupper hasn't got that feature yet. That's the only reason I wanted to highlight the key that we talk about, you know, designing for disaster, um, and actually thinking through and, and, and it's, it's part of the overall architecture. There's so many different layers to think about federating of clusters is not simple. You're still going to have to manage three clusters here, but what are the things that you can do in this distributed system to do it across multiple? And I think that's a, you know, and I think the, the networking thing, we're getting closer to solving this at scale with, with lots of different solutions here, but we're talking about networking and three different cloud providers here. So I don't want to underestimate the magic of what's going on in this, this diagram, like that's pretty phenomenal dude. So yeah. And, and we're peering all three of these networks across three different cloud providers where the infrastructure for the cloud providers doesn't provide any capability. That's right. They don't want to, by the way. They really don't want to. Yeah. Yeah. And so like, yeah, like you talk about multi cloud, uh, or even hybrid, like if you want to run OpenShift on-prem and you do it, that's just a multi-cloud problem in my opinion, multi-hybrid. It's, it's multiple, multiple regions. You're going to have this problem. Right. So yeah. Um, yeah. And I think it's, it's, uh, this can, this is happening. Is it easy to do this, Keith? I mean, um, I set this up, this, this demo up this morning. So it's not that hard. Um, the, it's, I wouldn't say it's easy. I think Scupper is probably currently the easiest way to do this. Um, I think Submariner is a bit more robust, but it's a little bit more difficult, although there's some big changes that have come to that project in the last month that make it a lot easier to install and working with, uh, one of the guys on the team over there to, to update our Submariner demo, to use the new install pattern, which I think might get it closer to this. Um, what I really like about, um, Scupper is that because it's over TLS, it's over HTTP, I don't have to do any kind of, um, I don't have to get the network team involved and get, uh, firewall rules set and whatnot. Um, the downside is that it's, um, it's got more overhead because we're wrapping all of our TCP packets in HTTP, right? So, um, so it, it's better for environments like this where we want to kind of try something out and see how it goes. Um, whereas Submariner is probably what I would use today if I needed to do this for production workload. Um, what, what's really cool though is both projects are making huge leaps forward, um, in a very, in a real rapid way. Um, I've already been, I've been interacting with the Scupper team over the last few weeks when I was trying, like, when we started talking about this demo, what three weeks ago, Jim, they already put two bug fixes in a Scupper for me to make sure that this demo was, was working for us. They've been phenomenal. Uh, I have every confidence that this is going to be every bit the production grade solution that, um, Submariner is going to be, and it's going to have the advantage of, um, being more, uh, being more flexible in its deployment pattern than Submariner. Um, but I don't, I don't, I also think Submariner is probably going to end up being the most more performant solution. Well, and I think, you know, look at the tradeoffs for whatever and simplicity or speed or whatever, you know, overhead, like, I think, you know, there's different, and this is the beauty of this community. I think there's different approaches that are emerging, and I think the right approach for each implementation is going to win for that implementation. And so I just, I'm just excited that you were like Keith, I know how hard you work and this, this, the, the concepts are here to actually do this. You just need to go out and find them and do it. And that's the funny thing. And that's what's interesting about this community, but let, let's move, let's get out of slides, dude. So let's talk about what we're going to do here in the demo. Yeah. So if you move to the next slide for me, um, so we're running something called TPCC, which is a workload. It's a, it's a standard space benchmark, TPC.org. Feel free to go read about it. It simulates a warehouse workload. We're going to be running not that big of a load because I'm going to be doing some kind of neat. Just a hundred. Well, you're like just 150 warehouses, right? Yeah, we're running 50 warehouses per site. And that, that workload client is just kind of, you know, generating some load against the cluster. Yeah. And then we're going to use something kind of neat. Well, one second before you do that, by the way, the TPCC workload, we actually package that in the binary with Cockroach, a little commercial. We package a couple of different workloads within the binary that if you wanted to start using Cockroach, Cockroach demo, you can actually run the TPCC workload against our, our database today. But yeah, and we test Cockroach to be up to 140,000 workloads. So that's like three, roughly three orders, four orders of magnitude larger than what I'm doing today. I was this is just kind of a, this is more of a functional demo than than it was meant to be a performance demo. And then we're doing something called using something called KubeDOOM, which is literally what it sounds like. It's doom, like in software 1997's doom that's they released it as a GPL project and someone hacked it to do chaos testing on Kubernetes pods. So we're going to be killing some Roach demons in KubeDOOM and it's going to kill pods in our AKS region. So a couple of which is kind of fun. And we're going to show the database keeps running. It may end up killing the workload generator in the east because it is not monsters don't appear differently based on what the pod is. That's right. So I might, I might act as a pod of monsters, a monster. I might shoot the TPCC demon. It actually is not thread safe either. So it might, I might kill KubeDOOM itself, which could be fun. So it's a little bit meta using doom to kill its own process. But Keith, I mean, the trick here is like, even if you were to kill that, Kubernetes is going to survive that pod. The pod is going to come back. Right. I mean, I just have to have to reconnect exactly. But in the trick here is, and you may actually kill the admin UI that we were hitting, or maybe you're doing it from another region. No, but the point is, is like, once a pod dies, you're killing it. We're killing the pod, not the instance of cockroach. We're killing the pod. Kubernetes automatically is going to restart that pod and cockroach is going to start in that. And the database is not going to have any sort of. Well, the database will continue. The database will continue to run regardless of how long it takes for those pods to recover. Exactly. The UI, I think I'm actually reading the UI out of the Google region. Oh, good. So you won't kill it this time because we had a couple of fun. Yeah, so I shouldn't, although I am using load balancer. So one of the other things to keep in mind, this is this is a specific discover. So a van uses forward proxies. So when when I pull up the screen, each region can see all of the pods for all three of these regions. And so in the east region, it's going to see the central and west pods. Those are actually proxy containers. They're just all they're doing is acting as kind of a transport layer for the for the community, the nodes to communicate with each other. We can end up killing a bunch of those too. So why don't we go ahead and if you could stop sharing your screen, I will start sharing mine. OK, buddy, I mean, we'll go kill some stuff. Love this. So I also already used some cheat codes so that I can actually kill some demons. There's something like 40 pods running in this environment. So so there's a there are a number of. Hey, Keith, real quickly, there was a quick question there. I mean, one of these regions could have been on bare metal, right? I mean, it's there's no I mean, whatever, like, yeah, regions or regions, the clouds, the cloud hybrid environments. It could have been it could have been equinex metal. We just chose these three because that's what we had. It's good. We chose these three because I had accounts that spin up infrastructure for these three. I could have these. We could have just as easily done this in digital ocean or done this on bare metal if I under under your desk. Yeah, I think we probably would not have wanted to run it on my laptop, but we could have fair technically. I mean, I I tested Coob, do my laptop. It did work. So left hand side of my screen, the first three boxes, the red, green and blue boxes. Those are just the workload generator running in the background. So we're processing transactions. I kicked this off about 45 minutes ago. So we've been processing transactions for a while. The black area of the console screen is showing you the current status of the pods in East. So this is an Azure. So you can see the the cockroach DB dash East dash zero through eight pods are those are the actual local nodes that I'm going to potentially be killing. There's also the internal proxies. These are the outbound proxies for for the cockroach DB pods that are local. And then we have the inbound proxies, which are marked as central and west. OK, there are also a couple of other things going on here. We have a client container, which is the what's actually running TPCC. We have the service router and we have the service controller. The only thing you're not seeing, because this is showing the cockroach DB namespace, is the coob doom pop, which is running in the coob doom namespace. So I didn't switch to that. So you can see we're running. 500 queries per second. I didn't optimize the performance of the queries here at all. So ignore the P 99 latencies. If you've watched any of the demos that Jim and I have done in the past, you know, I can get these latencies for TPCC down to, you know, 100ish milliseconds for P 99 have just didn't choose to do that for this particular demo. I did make sure that we could dig into each of the regions and take a look at each of the nodes. So you can see all of the nodes that I'm going to potentially be killing are all running some queries. OK, so I've run queries running across all of the nodes in the cluster. All right, so now for the fun stuff. Going to go over to doom. I'm going to make sure that I can still fire walls. I've got no clipping set up. So you can see the names of all the cockroach DB pods. I'm going to go and kill some stuff. I should still be kind of immune for a little bit. Oh, wow, Keith, really. Bloody pod killing. Yeah, bloody pod killing. You're going to start to see some of the pods. See on the left hand side, one of the East Proxies died and restarted. We got all kinds of stuff is dying. Oh, I ran out of and then you're terminating Kubernetes, just bringing them back. Meanwhile, the database is not having any sort of impact. Now I'm shooting them with shotguns so they're dying faster. Because before I was using the Gatling thing, I can switch to the one that like will probably kill me to the BFG. Oh, finally killed the cube gym pod. Well, so we just saw a bunch of stuff die. You can see we're still creating containers. We killed the TPCC process that was local, but my queries are still running on the West and in the central. See, three nodes actually still haven't recovered yet. They're recovering for us. We don't have any unavailable ranges, which means we're continuing to process data as we go. And now that this suspect node list is down to zero, our under replicated ranges will go back to zero and the cluster will be completely healthy. So Keith, one of the things that we didn't kind of describe to the audience in cockroach, when we write data to a cluster, we're actually writing data in triplicate. So, you know, the replicates set is, you know, these three things that are running. So while you're doing that, Keith, so we're writing data in triplicate. We're writing across multiple different nodes, physical servers and that so that we can survive that. So an under replicated range just means that one of the replicas of that raft group is missing. And so the database just knows that and it's just telling us, oh, wait, you're missing some data here. What's interesting is that at no moment in time was there an unavailable range. A range you think of as like a tablet or a shard. We automate all that underneath the covers. So the data, no matter where it was being accessed in West, East or in central was accessible. And that's the critical thing, or we didn't have any impact to the availability of that data. But we did have, because we killed some pods, that stuff went away. Now, when the pods came back, we're just using stateful sets to actually bring that pod back to its current state, reconnected to the storage it was using. Somebody was asking, you know, what's the best practice for deploying cockroaches that, you know, well, how does that storage work? Well, each node is actually has its own storage connected to it and its own PV claim. Right. And so ultimately, stateful sets is managing that in Kubernetes. And as you kill the pod, it just comes back and then the range is back, that little piece of data. Right, Keith? Yeah, that's correct. Yeah. So. So did you want to show the impact on the some of the metrics, Keith, or no? Yeah. So I am waiting for the cluster to decide that it can get its metrics. Oh, right. And I will reshare. This is, you know, what'd you kill? What did you, you killed CubeDoom? I killed CubeDoom. I killed the TPCC container. I think I killed all of the forward proxies in East as well as most of the database pods at least once. Right. It looks like I don't have anything left. I have looked. So for East, I killed half of the database pods. So anything that's here that says 4-H, you're no longer sharing, by the way, Keith, did you know? Oh, sorry. Yeah. There we go. Sorry. So for anything here that says 4-H didn't die. Yeah, because we're running for four hours. Right. But the ones that said there. So we killed a bunch of the forward proxies, both inbound and outbound. We killed, it looks like three or four of the Congress DB pods, and I managed to not kill the Scupper Router or the Scupper Service Controller, which is good. Which is good because that would have killed all of the proxies. That definitely would have taken, that's effectively like taking the entire region out of service. The database will survive that too. Let me ask you about that, Keith. If you had done that, I mean, it would have basically, I think we did this last time we ran this, right? It would resolve over time, correct? I mean, it would just basically automate and heal, correct? That is correct. Yeah. The challenge that we're having right now is I think I was wrong. I think I actually killed the pod that was hosting the admin. Oh, so you got to go to a different pod. Yeah, so. You have a different IP. So again, what Keith can do is just point our, point the browser at a different node that's running. Because I didn't set up a global load balance. Yeah, that's right. Yeah, otherwise, yeah, it would have been, there we go. Oh, there we go. Yeah, you're back. Let's see. Yep. There we go. So you can see, we definitely had an impact to our queries per second while this was going on, which is not surprising. We did just take out a third of the cluster, basically. But the database state available, it was always processing queries. And it's starting to recover right now. Because one of the load generators died, it won't get back to full unless I restart that load generator. But you're not going to kill the load generator in real life because that's a bunch of people hitting your app. Which is different. Yeah, normally that would have a load balance in front of it. So it would have, it would have continued anyway. You know, our latency numbers kind of jumped around a little bit, which is not surprising. But the point is, is that the cluster survived. Right. And healed itself. Right. And I really didn't have to do much anything other than talk to you for a couple of minutes. And this is one of those things where it's like, you know, we've talked to customers and are like, well, we didn't even know an event happened until we went back in time. And because stuff just basically survives, right? And I think that's the trick here. It's like, you know, you got to actually go find when you have issues. Our biggest challenge is when customers want us to show failure in Kubernetes, it's kind of the reason why I wanted to build the demo. It's actually hard to do without, like, straight up deleting the stateful sets, because Kubernetes will regenerate the pod so fast that we don't even have, like, a substantial service average. Yeah. Yeah. All right. Well, Keith, I love your demo. I'm going to stop sharing. Yeah. I do, I do like doing chaos testing with Doom. It brings me back to my video gaming heyday. So it's always fun. Keith, there were a couple of questions in chat. And if anybody has any questions of anything we went through, I'll go through a couple of them. I think Jim answered a lot. Thank you, Jim, for answering these. But somebody was actually asking about a project. I think they were asking about Submariner. It's just Submariner I.O., right? Isn't that the one? Submariner.io. Yeah. And then scupper is scupper.io? Yeah, SKUPPER.io. That's right. That's right. Yeah. So if you're looking for that, I think those are the two projects. Let's see here. Jim, there was one question I didn't do a very good job answering about U-Limits and Docker and Kubernetes and kind of how we deal with that. Keith, are you familiar with U-Limits and whatnot? Yeah. So we don't run into a lot of problems with U-Limits generally. This is definitely a low level Kubernetes thing. When using the managed service providers, Kubernetes, we don't have a whole lot of problems with this stuff. Sometimes we run into it with self-managed. Generally speaking, you fix them outside of Kubernetes and then restart the Kubelet on the POD host and then that gets inherited. But that's not something that we normally need to even really worry about. We do have some instructions on what to set those U-Limits to and our docs to support the database because we do have minimum requirements there. Right. So another question that came through, and this is a real basic kind of cockroach question, Keith. How are pods distributed across nodes? Since those staple sets are used, does this mean it's restricted to one POD, one node? What happens if we want to score horizontally scale? Kind of a softball of a question. Yeah. So by default, we do use anti-infinities to stop multiple cockroach DB pods from launching on the same physical server, largely because we need to treat the server as a fault domain as well. And that's how. So now in a multi-region deployment like this, that's less important because we're treating our top-level fault domain. So from a replication perspective, the top-level fault domain is the region or the cloud. So we're already not going to have multiple replicas on the same physical server by default anyway. But you can enable running multiple cockroach DB pods. Generally speaking, you're going to get better performance in that scenario if you run larger individual cockroach DB pods and then scale out by having more POD hosts. Yeah. That's generally the best practice. It can be done. You can disable the affinities. But then you have to be very careful about how you set the locality flag so that we don't put multiple replicas of the same data on the same physical node. And Keith, I mean, that's a pattern to think of regardless of cockroach. I think that's just something that as you're building your own applications and services and you're running on Kubernetes, that's really good advice. Because I think it's one of those things that you don't know until it hurts you. Yeah. For a stateless application, running five copies of the same service on a node is not a big deal at all. But for stateful applications, generally, that's not a best practice. That's not to say that there aren't scenarios where we might suggest it for certain requirements. But in those scenarios, we have to. So we didn't cover a whole lot of cockroach DB architecture stuff today. But when a node or a pod joins the cluster, it announces its location in the topology. So in a scenario where I'm going to delete all of the anti-affinities from our stateful set config, I need to add the host name of the pod host as part of the hierarchy for the locality so that the database knows not to put multiple replicas on the same physical hardware. That would be the only thing that would have to be done there. But reformers is generally going to be better if you just use larger cockroach DB pods in that scenario. Well, cool. I think we actually hit all of the questions. I mean, Jim, I'm sorry, Jim, too. I got to call you Jim, too, live, buddy. Thank you for jumping in. I love that we had the voice. I think we should call him the hatch. The voice of Jim came in there for a second. It was great. Jim, did we hit everything? I think we hit everything, yeah. The few questions left. But I think we addressed most of them. Yeah. All right, well, if there are any other questions out there, we're happy to talk through. I think your code, the repo, it's all available, right? Keith, you've published all this to GitHub? Yes, although not quite yet because my notes are still it's in GitHub, but it's not public yet. But we'll get it out there in the next week or two. Oh, there's a question. There's a question about scaling down cockroach DB that I would like to answer. I missed it. I'm sorry, buddy. Yeah, it's brand new. So someone was asking about scaling down and seeing it as failing nodes. And is there a way to do it gracefully? Yes, we have a node decommissioning process. Right. So it is this is a pattern that is a little bit more challenging to do in using our current config. We are working on an operator because, effectively, that's like a day two operation, not a day one operation. So the stateful set config is great at the day one ops that will handle gracefully scaling down your cluster in a way that's not going to create data unavailability. The other option is to just do one node at a time and make sure that that under replicated range count is at zero before continuing to shrink your cluster. That's a bit more of kind of a hard muscle way of doing it, but we'll survive that as well. You've got to be careful. You can't just take out. This is true with any stateful service? Any, I mean, again, this is a truth for all things deployed in Kubernetes. You go kill half the thing. You're going to have some problems. Yeah, so we need to have a quorum of the replicas for each of the ranges available for us to be able to answer queries against those ranges. So if we have a quorum, if we have a raft group of three replicas, we need to have two of those replicas available to be able to respond to queries. So that's why we would only recommend scaling down one at a time. If all of your tables were a replication factor of five, you could scale down two at a time. And at seven, you could scale down three at a time. It's just that's just kind of the nature of consensus-based replication. That would be true for any stateful application running Kubernetes, not just cockroach to be. So Keith, let's shift the topic a little bit. And there's a couple of questions that are coming in related to something. And this is a day two operations thing is what we're talking about here. How do we scale down on these sort of things? And we get asked about operators fairly often. And I know you've do a lot of work on that. What's the current state of our operator for Cockroach? I mean, because honestly, Cockroach didn't need an operator to do any of this stuff that we were showing today. It's literally kind of aligned perfectly with this whole the Kubernetes architecture, but we do have an operator. So what is our operator delivering then? Yeah, so our operator is making it easy for users to do common administrative tasks against the cluster in a way that's safe. So all of the things that the operator is doing, you could already do with the existing stateful set. But it might involve you updating the animal files and mounting file systems. So a rolling upgrade, for example, we have a manual process for doing a rolling upgrade in Kubernetes today. It's pretty straightforward. But the operator is going to manage it in a way where we're going to do things like fully health check the cluster and not just the pods before rolling the next pod. We're also going to do things like make sure that when you upgrade, you're upgrading to a valid version from the upgrade that you're going to and you didn't skip a feature release. It's not that you couldn't already do all this stuff, because you couldn't already do all this stuff. We wouldn't be able to build an operator to do it. It's a matter of making it so that you can't accidentally fat finger a config and then break your cluster. So it's those types of operations that we're working on. I'd like to think of you as making things more graceful. Well, I see the operator. Like the operations. I see the operator as a way for getting how to operate CockerCB on Kubernetes out of my head and into everyone's hands. Exactly. That's the reason why I've been spending so much time. And by the way, that was the original intent with the operator pattern way back years ago when it started was how do you take the knowledge and expertise out of somebody who's an SRU as doing these things and create some sort of framework within Kubernetes? It's exactly what it is. So there was a question on if we're going to support MySQL alongside of PostgreSQL. We're wire compatible with PostgreSQL. We've gone a long way to do that. There is no plan immediately to do anything that is like MySQL. We have to make a choice. It's a lot of work, by the way. Well, so the other thing is that we're serializably isolated. We didn't really talk about database at all. We're serializably isolated, which is PostgreSQL can run in serializably isolated mode. Exactly. MySQL can't. And so it would be much harder for us to provide a MySQL compliant endpoint, because there's no way for us to signal to the client that we're running in serializably isolated mode. Whereas with Postgres, we can do that. Yeah, I learned something new. Thank you, Keith. I love when I learn things new, especially from you, which is all the time. Or never. Or never. I know 100% of the time I'm making it up 90% of the time. Right? I don't even know the quote. I can't even start. So well, let me go back to the slides. We'll stay on screen here, Keith. We can go back to this is it. So where was I? So I just realized 52 minutes into the webinar that because I'm using the fancy camera, people can see that my headphones have messed up my hair for the day, which I think is awesome. Yeah, and the problem with a fancy camera is like I have to shave, and, you know, anyway. So we are a database that was designed with Kubernetes in mind, y'all. I hope we showed that to you today. But I think there's a lot of principles that we've done in our code base that might be applicable to what you're doing in your code base or what you're actually trying to implement. I like to think a cockroach in our code base is a little bit of a course in distributed systems and some of the cool things that we have done. Keith talked about deploying and making things a little bit easier to do. We do have a Kubernetes operator that's absolutely available to all. Lots of companies using us today to do these sort of things. And Keith, what's the percentage of deployments that are on Kubernetes that you see? I mean, you work with a lot of customers. Yeah, so we know of the clusters that have telemetry enabled, we know that roughly 40% of them are running on Kubernetes and another 30 of them are running in containers, but we don't know for sure if they're running Kubernetes. So anywhere between 50% and 70% of cockroach DB environments are running Kubernetes today. These are long-running clusters. So we define a long-running cluster as one that's been up for more than seven days. So not something you run on your laptop and shutting down at the end of the night. Also 100% of our cockroach cloud clusters are running on Kubernetes. So there's a lot of internal dog fooding around our use of Kubernetes. Yeah, and so yeah, so you could spin up, start running cockroach cloud today off our website. It's all available. Any of this stuff that we went through is available on our website. But I think most importantly, and Keith, oh my god, how do we go an entire session with and out saying nice things about docs? I can't believe we made it this long. That's because there weren't any docs on how to get cockroach DB running over scuppers. Well, that's true too. There's this now. But our docs does do a really good job of, if you want to dive into how we do our transactional layer, there's questions around how we do, typically in our docs it's there. If it's not there, we do have a community Slack channel where people are always on and always answering. So if there are any follow-up questions or other stuff, I mean, we're pretty open and transparent about everything that's going on here at Cockroaching. We're happy to engage and we would love for anybody to be part of this community and to tell us what they're doing or if they have any questions about how this stuff works, Kubernetes and Database or without. And we're happy to engage and a proud member of this community. So Keith, thank you as always. Thank you. Yeah, and that was, it's a wonderful demo and I really, really like it. So I hope this was valuable to everybody today. We tried to make it a little entertaining. I love the CubeDoom thing. But this is something I've never seen before. I've never seen an application run across multiple Kubernetes clusters until I saw this demo and Kudos, buddy. I mean, it's super impressive. Thank you. Thanks. All right, so Marisa and Linus Foundation, we are done. Thank you very much as well. Awesome. Thank you guys so much for that wonderful presentation. Keith, Jim and Jim as well. That's Jim too. And thank you. Yeah, Jim too. And everybody of course for participating and joining us here today. As a quick reminder, the recording will be posted to the Linus Foundation's YouTube page. Should be up by the end of the day today. Okay, we hope to see you guys back here again for some future webinars. Okay, have a good one. Thank you. Bye. Thank you.