 I want to welcome you to the architecture of a geodistributed SQL database. My name is Lindsay Hooper. I'm one of the Postgres conference organizers and I'm going to be your moderator for this webinar. I'm here with Jim Walker VP product marketing at Cockroach Labs and Chris Casano Solutions Engineer Cockroach Labs. Let me tell you a little bit about them. Jim is a developer who spent a large part of his career helping people understand complex emerging technologies. He's passionate about distributed systems, data and open source technologies. He's a startup, a holic and loves building apps and new communities, and he has helped build companies like Hortonworks and Core OS. Chris prior to Cockroach was a senior solution engineering manager, Audara, a solutions engineering manager at Hortonworks and a principal sales consultant at Oracle. Welcome Jim and Chris. We're super excited to have you on the line today. One quick logistical note for you. I've put you all on mute to avoid background noise, but we do still encourage you to ask questions. Questions will be answered towards the end of the training, though pressing one should be answered in the moment. You can ask these questions through the chat function depending on what kind of computer you're using. I'm going to hand it off to Jim and Chris, and you guys can take it away. Enjoy. Awesome. Thanks, Lindsay. We are excited to be here with everybody today. I hope everybody in the audience is healthy and staying safe. These are some interesting times, but we're excited to talk to everybody today. I'm excited in particular for this webinar. Actually, we do a fair amount of them at Cockroach Labs. Because I think, you know, the Postgres world and being part of Postgres and that community, you know, I've been definitely involved in building on top of Postgres for a long time. You know, I just, I'm just so amazed at how big and vibrant the community is. And so we're excited to talk to everybody today about, you know, what is a cloud database and what is these, what do these things look like over the next couple of years? How we think about scale, how we think about resilience, how we think about meeting the expectations of, our users, no matter where they're at. And that's globally, how do I, you know, have like a great experience for everybody? And I think that's, you know, one of the challenges around applications is really data and transactions and how we need these kind of, kind of diverse workloads. You know, Mark, Mark Adrian is an analyst at Gartner, who I look up to as somebody who's, somebody who I feel is a pretty, pretty smart, great, great analyst. And he believes that the future of what we're doing is smart, great, great analyst. And he believes that the future of all databases is cloud. And I kind of agree with the guy. So that's kind of quick intro. So why another database? You know, we have relational databases. You're all obviously, you know, familiar with Postgres. You kind of know there's no SQL databases that things like, like Mongo and some of these other things like Cassandra and some of these other databases, you know, with the inception of cloud and with kind of the requirements to scale, the requirements to be resilient. And really kind of with distributed systems, you know, the real is an opening for a different way of thinking about a database and not taking legacy systems and making them fit into distributed systems, but let's just build a database that was really from the ground up architected for distributed systems. And that's, that's what we're doing at Cockroach Labs. And that's what we've been working on for, you know, a fair amount of time, you know, really since the start of this company, 2015. And so we fit in, and if I think about relational workloads, some of these kind of transactional system of record type workloads and applications that we're building as they move to the cloud, having a database that is not only SQL, but will scale and build in this resilience is kind of critical. And really kind of what we focus on. Lots of people find Cockroach Labs because of, you know, they are building out kind of a microservices based platform or application. They're breaking down a monolith into these things. And so if I think about this, you know, like the traditional way of thinking about a database, if I think, you know, back in the days of like Oracle, the way we would get redundant systems and get coverage, like global coverage for databases, you know, we have instances of Oracle, we use Rack, we use Golden Gate, it's Baltly, it's costly, it's difficult to get going, right? I think AWS has done a good job about pushing forward into kind of cloud database. The problem with AWS and Aurora is it's not really a cloud database. It wasn't really built to be a distributed system. I think distributed storage is awesome, but in a true kind of world of distributed systems, what we think of typically is shared nothing. And you know, that's something that we've gone a long way to actually have this kind of more matrix type database where each instance of Cockroach CB is its own single atomic unit, is its own kind of standalone piece that coordinates and takes part in a larger system. We don't live on top of anything that's shared, and I think that's one of the things that allows us to be a truly distributed system. And I'm going to go into how we did that in this webinar today. That's all we want to talk about. Like, what does it mean when I say don't take legacy tech and move it to the cloud and try to make it kind of distributed? What does it mean to actually tear apart a database and think through, like, how do we become a distributed database at the core and how do we build something from the ground up to be so? For me, if you're going to be a distributed SQL database, and I think about this as kind of a new emerging group of databases, well, number one, and I think everybody on this phone would agree, you've got to speak SQL. Like, I'm not going to require y'all to go out and learn a new language. I find it incredibly important that we should be able to use the interfaces that we have today. You know, we're wire compatible with Postgres today. You can use, you know, there's a world of ORMs that we've built if you're into that, if you like to use ORMs. And we're going to work the way you work today. And I think that's actually really critical. Requiring people to learn something new is not realistic. I think, you know, our application speaks SQL today. And I think, you know, gosh, I was learning SQL a long time ago and I think it's still applicable today. You know, a distributed database should ease operational scale. I think we can shard databases. I think a lot of probably a lot of people on this phone today have gone through this whole manual sharding process. It's difficult. There's a lot of operational overhead. Distributed systems should automate scale. And so how do we make that happen in the database? That should be a requirement. These databases should be always on and resilient, right? Another kind of like a core tenant of distributed systems. And if you're going to be a database and you're going to store data, well, you can't lose data, right? That's kind of important. So you got to be asset compliant and allow for transactions to happen. We, for instance, we implement serializable isolation. So this isn't like snapshot isolation and eventual consistency. This is true consistent transactions. Difficult to do a distributed systems. But I'll show you how we do that as well. And then this last requirement for a distributed database is tying data to location. Now, sounds simple, but it's really difficult. Typically as application developers or people who are dealing with the database will actually build a layer at the application layer to basically tie data to location. So all records of this type can only get stored in this set of servers. That's typically left to the application layer. If you're going to do that, you know, we believe that that should be part of the database itself because ultimately if you're going to be a distributed database, you're fighting against the speed of light because latency between data centers is no joke, right? And if you're going to do, you know, if a transaction takes four different phases or 10 phases and it's got to go back and forth between three servers and three regions, you know, latency is add up. And, you know, users are not going to wait, you know, more than, you know, 100 milliseconds or 200 milliseconds for a transaction to finish. So, you know, tying data to location is actually really important from a performance point of view to give the right experience for the people who are actually accessing that data for the users of your applications. You know, we like this because it allows us to do things like, you know, we can, we have a feature called geopartition. It allows us to basically tie data to location to also meet, you know, privacy requirements as well. But I think it's a, it's a nuance that this is really about performance and really a critical thing. And so how does the data basically, how does the database automate that? Now, again, CockroachDB, this is a unique architecture. I'm starting at the top level. Each instance of CockroachDB, I could spin up an instance, right today on this, on this, on this laptop that I'm working on. I can point it at a cluster and it's going to participate in the larger cluster. Every node in the database is a consistent gateway or an endpoint, if you will, right, to the entire database. It's actually pretty important, right? So while the data is going to be stored and triplicated across multiple different nodes, and we're going to show you that, like, I can ask any node, I could have a thousand nodes and it's going to know where to find that data, which is kind of cool because I could spin up a node here. I could spin up a node in AWS. I could spin up a node in Azure or GCP. And actually, I'm going to have one database and every node is going to have access to all the data that's going to cross all those nodes. And that's truly a shared nothing experience because I have the complete instance, the complete binary is the same at every single node. And, you know, we've gone a long way to build out all the management and optimization of this thing, you know, all the enterprise security capabilities you need. You know, we actually built a, a location-based cost-based optimizer. So filming with that and how to actually, you know, that two performance around CDO. We built that actually in a database too. It's kind of cool. Now where it gets really cool, CockroachDB is this unique distributed architecture that because all these nodes are their single, you know, atomic unit, they actually coordinate with each other so that we can actually coordinate transactions because we're going to write data in triplicate and I'll show you how we actually coordinate transactions. But, you know, if a transaction comes in at Sydney and a transaction comes in at New York on the same record or the same row in a database who wins, the database actually deals with those conflicts and ensures that you're going to have no overlap, right? And ensures you're going to have serializable isolation for things like that. And so it coordinates for that, but it also coordinates in that there's no more manual sharding. You know, if you want to actually expand the database, just keep throwing nodes at it. The database is going to worry about basically spreading that data out in an intelligent way so that we can actually deal with performance and utilization of servers and whatnot. And then, again, we can attach data to a location because of performance and because of privacy concerns and this sort of stuff. So lots of cool things that we're doing here, but let's get into basically now the architecture of how this actually works and the differences here. And so, you know, lots of databases are databases, but some databases are actually built on other databases and, you know, sounds kind of meta, right? But, you know, we actually built a database on top of a KV store. So if you're familiar with KV stores, a key value store underneath the covers here, we actually use the database called RocksDB and that's between us and the storage layer because every database can be broken down into the language, the execution layer and the storage layer, right? And at the storage layer, we're actually using KV to store data to disk. And it's actually pretty important. It allows us to do some really cool things. We've completely re-architected the execution layer to do all this coordination. And then we've also, you know, we made sure the language is SQL as well. So across these three things. So let's just start with every table. Every table in RocksDB can really be ultimately at the storage layer thought of as this monolithic logical key space. And we order everything in here. Here's a dogs table if you will. And we're just ordering everything by name of dogs. So you can see it's basically in alphabetical order here. And so basically every table ends up getting that looks like this. Now it's a little bit different. A typical database, you just keep adding records to the table. Say it's an inventory table. And in order to find these things, we set up indexes. There are other tables that actually are meta and allow you to do some things. We don't need this because actually we're actually ordering everything based on key, right? So the keys are all ordered in CockroachDB. So it's a little bit different from the typical database structure, which you're going to find in Postgres and some other things. It allows us to do some really, really cool things. Now the sequel, the problem with this is that's KV. How do I deal with tabular data? How do I deal with this relational data so that I can actually run queries? Like that I select star from customer. That's a little bit different. And so the traditional way of looking at a table needs to be translated to the kind of KV layer, right? So we do that. This is part of actually, if you're ever like looking to solve insomnia, right? Like you go read the white paper about Google Cloud Spanner. I think this is one of the really cool things that, you know, Spanner really led the way with and the designers there that really drip this up. Really cool stuff. And this is really what allows for this distributed nature of our database. So the Google Cloud Spanner actually, that white paper that lays out a lot of stuff. So at the KV store, what we're actually doing is we're storing, we're going to translate that stuff all up into SQL. This allows us to do some really cool things. It allows us to efficiently distribute data. It allows us to do dynamic schema changing and allows us to do like this really cool thing around geo-partitioning as well. But I'll show you all that, right? And so let's just think about a table. Here's our dog's table. It has three fields, ID, name, and weight. Typically table entries kind of look like this. Now what we're doing at the KV layers, we're actually translating this when we store the data. Like this is the way it looks on the right hand side. This is how it looks at the execution layer, right? The SQL execution just thinks about it as a table. But at the storage layer, we're doing something kind of cool. We actually each take, we're going to each take each one of these rows and we're going to break it down so that the K, the key, right, is going to be basically an index, right? It's going to be the table name. It's going to be the ID, and then the column that I want to store. So in this case, it's the dog table, ID number 34, and name, and the value is going to be carved, right? And then weight is going to be 10.1. So where I had one entry where I was storing in the traditional way of the storage layer here, I'm actually, I have two entries. Now this allows us to cool things because we can actually change the primary key. You know, we can do some interesting things in this key, like this 34 can actually be a consolidation of multiple different fields so that we want to order things. But ultimately, if you look at this, we can just order these things. And now I know exactly where to put the next record in here into this table. And the KV is what allows us to basically keep this completely ordered pair of key values that that are going to translate back into the SQL table. So this is actually a really kind of cool trick of what we're doing here. I'm not going to go through this example, but just think through how we sort things in, in, say Excel sheets, you know, I could sort out one column and I sort out the next and I sort out a third. We actually can like embed this sorting. So this really kind of cool way of sorting data that allows us to do some really cool things, right? We have this monolithic logical key space. Now we take this key space and what we do is we, we break it up into 64 megabit ranges, a range you could think about as a shard. And instead of us having to kind of manually choose the shards, the database is just always doing this, right? It actually is always thinking about the ranges and we're going to use raft, which is kind of cool concept to actually show how these things get then distributed, right? And so here we have this table, we're going to break it down to 64 megabit ranges. Here we had just three ranges, right? They're small enough so that we can move them around quickly and we could split them. But they're also kind of the right size so that we can actually gain value in terms of how we index them and how we think about them. So if we're going to break a table down to this, we actually need a layer that's kind of allows us to understand where to find things. And we're just going to update this thing every once while as we kind of add more table and more data to this basically monolithic key value storage. This is very much like a B tree, right? If you're familiar with that concept. So order keys, they allow us to do cool things. So we don't have to do these crazy like told table scans. We're doing these very efficient race scans. So if I wanted all dogs between muddy and Stella underneath the covers in the database, it knows it only needs to go to these two ranges. Now these ranges are going to be physically stored at different servers. And so this actually allows us to be very efficient in how we use network and bandwidth and the various different servers that are in the thing. It actually really, really key concept, especially when we're thinking about distributed systems. So what happens when I want to insert a record into a table? I want to insert sunny. That's my friend's dog. And so I'm going to put sunny in there. Sunny's going to go. Let's see in between Stella and C. So I'm going to ask that range. Do you have the updated? You have space. Yes, great. Let's insert that record. We're going to insert that record. It's going to be updated in a couple of different spots. What happens when I want to insert and there's no range. Well underneath the covers, what happens is Cockroach TV actually automates the creation of a new range, right? We've split that range in two and we now have more extra space. So we can actually go ahead and insert that record into that range. Now we're doing this in real time. We actually will allow a range to go a little bit over in real time and then come back and actually fix all this stuff. So we don't affect performance like I don't have to actually split the range while the transaction is happening. I'll just note that, Hey, this range is a little bit full. I better split it. Great. And so this is where we get this whole sharding capability, right? Like cause, and I'll show you how we're writing data to the various different nodes, but think about what this means now from the way we think about like shards and manually doing this process. The database is just doing this, right? And so no more manual sharding. It just basically automates this whole like it's, we just split ranges, right? And so a new range range index was created or it was maintained. I could find that data. If I delete data, it'll actually consolidate ranges together. So it'll actually retract as well. So we'll both expand as it will both expand and retract the database too. And so that's not a really key piece of what we're doing here. And that's underneath covers. I'm a notice that the developer, the database just dealing with all this stuff. And it ensures basically this easy scale and kind of resilience. Now, when we write data into the database itself, like we're going to actually write on physical nodes, like physical locations. Like so I'm going to write, you know, I'm going to have a cluster with, with six nodes in it. Those are six different machines. If you think about, and I'm going to actually write data three times. Like by default, the database is going to write data three times. You're going to be three replicas of each of the ranges. So here we have four ranges. We're going to actually write the first range, Carl, Downey, Figment and Jack. We're going to write it three times across three different physical servers because we want to survive certain values, right? We want to actually distribute this data out in a more comfortable fashion. We use something called raft to do this. And a raft group. A raft group basically takes these replicas, like that, that, that first range is going to write it in three ways. And that, that little group, they communicate with each other and make sure everything's cool. Right. And they're, they're really what ensures that like, there's this atomic replication of command. The atomic replication basically says like, for that group, one thing is happening. Like, you know, it's like an asset, you know, a is atomic, like, you know, that atomic unit, that happens for that group. Right. And there's, there's this concept called a leaseholder. You might hear about it. If you know raft, like it's a raft leader, that's the, that's the unit that basically in this, this group of three copies of this range, they actually understand really what's going on. They're like the leader, most things go through them and you're insured, you're going to be right, but they basically kind of coordinate everything for each raft group. So in the implementation of a database for Cockroach DB, Cockroach database, there's thousands and thousands and thousands of these, of these raft groups going on and lots of coordination going on amongst all of them. So, so let me just, I'm going to, I'm going to fast forward a little bit just because of time and get into the stuff that's really important. So that actually, I'm going to have Chris show you guys how this actually works in the database. But when we write data, here's our table, right? Our dog table. We're going to actually write it to physical location. So here we are. Like this is what I was just talking about. The Carl through Jack range that shard is being written three times across three physical nodes, node one, node two and no four. You know, range two is going to be written on node one, node two and node three, range three, node two, node three, node four. Now what we've done here is we've actually distributed the data across physical nodes, the replicas so that we can actually survive certain values. I want to survive value moments like a disk or a rack or a data center or region. We can tune the database. Actually, you can use a command called alter table. And we can actually change this change this in real time in the database. The database will actually change where all this data lives. So you can fail different things, but it really depends on what you want to do when you implement something like Congress database, you got to think about two things. You got to think about latency and resiliency. What do you want to what kind of failure do you want to survive is a key thing. When you're thinking about the table itself and then, you know, what is the latency? Do I need data to live in certain locations? I'm going to show you how we do that as well. So we're going to distribute on this sort of thing. So if you think about this, well, this is kind of to even out storage, right? And utilization of disk basically across the cluster. We basically wrote the same amount of data on every single node. Sometimes a range, however, is accessed more than other parts of the table. And so the database is smarter. Also understand that utilization of each node is actually really important. So I can isolate a range into nodes that have you know, that are that are straightforward for a particular utilization so that that workload is going to have the same performance or optimize performance because I've placed it in an area where there's not a lot of other ranges, right? And so I can basically take heuristics of how that data is being accessed. You know, muddy is a really popular dog. That's my dog. Obviously she's really great. And I'm going to actually take that blue range there and I'm going to put it on nodes without a lot of other data so that I can optimize from a load point of view too. And the database is automatically doing this underneath. So not only are we placing data based on load or you know, by, by, by utilization of storage, but we can do it by load. Now we can also do it by geo partitioning. So the same concept that we had here. Here what I've done is I've actually overloaded the key with a value from the table for location. So, you know, you can imagine that table had an added column for country or whatever that is, or location. Here's the EU US East US West. And so now these shards or these ranges are going to be tied to servers because each server, each node is tagged with the location. And so we can actually geo partition data as well. So not only can we distribute data to survive failure, not only can we distribute data to, to work through utilization issues, but and access and workload, but we can also do it from a geo partitioning point of view. And all of this can be changed on the fly. We can actually change the primary key. We can change this, this key, the K and the KV store. And so that once we change it, the database is going to automatically start to move things around and make it correct for everybody. Right? So kind of cool thing about what we're doing here. And kind of unique and important because it actually sets up some of the key capabilities of what we're doing. So really quickly before the demo, again, we can rebalance replicas. So here we, here we had a, a four node database, a four node cluster. We just introduced a new node. The database is smart enough to actually just start moving data around to, to, to, to balance things. Right? And so I just, you know, they basically, it decomp, it writes the data, it recopies it in the new node and then actually decomposes the, the previous replica, yet raft still maintains the, the connection between all of those, those, those replicas, each of those shards, if you will. Right? And so that's kind of done. And we just take care of that. We can actually survive a failure as well. Right? So often a node will go down and Chris is going to show you this in a demo. Right? Here we have node three went down and had two replicas on it. Each of those replica sets is smart enough to understand. That's like, wait a second. I'm missing one of my replicas. I've got to have three. Like I can't, I can't live without having three men. Like what am I going to, so it's far enough to actually understand like the leaseholder is going to say, wait a second, I don't have a third replica. Let's create a third replica on a node that's available. And make sure that I have that in the, in the database. So you'll, you'll see something in the demo, like under replicated ranges is what we call that. And that's really when a node fails, you know, the replication factor has gone down a two for a moment until the database can actually recover from that. But all the while, any one of these nodes because each one of them is an endpoint could have taken that query and found the data no matter what, right? So you're never going to lose access to the data, right? It's active, active mode. We don't need active passive in the world of this. We can do distributed backup and restore for those sort of things for those kind of trouble issues, but we don't, it's just, there's no more like active passive system where if it fails and I bring on this, I bring the passive system online and you know, that thing's going to run for a while. And then basically I bring back up the, my original primary and how do I actually consolidate records across these two databases? Now that's tough, man. Like those problems that RPO RTO issues that people typically think through just doesn't happen with us because everything is active. Right. So this really simplifies a lot of things from an operational point of view for developers. Right. It's going to be sequel number one. So that's great. But this whole like this sharding thing goes away. The resilience thing kind of gets solved as well. And so we can also survive a temporary failure. So it isn't as harsh, but I wouldn't. So Chris, do you want to jump in and just show a little bit about the resilience in the, in the scale part of cockroach TV? Absolutely. Can everyone hear me? Okay. I can hear you just fine. And then by the way, everybody, there is the chat window. I am monitoring chat. So if there's any questions, do, do please feel free to do that there. So Chris, I believe you have the floor. Great. Hey, everyone. Good morning. Good afternoon. Good evening. Good to meet you all. I'm going to go through a quick demonstration or really around the fault tolerance of cockroach TV and how you can maintain performance by, you know, being resilient. And you can repeat this demo if you, if you'd like, if you go to our website, you know, docs dot cockroach labs dot com, we have a section over here called tutorials and there's some great tutorials on how you can learn how to use the database. The one I'm going to go through today is around fault tolerance and recovery. What we're essentially going to do is stand up a six node cluster actually already stood up. So we don't have to spend time. You know, you see me set up things in AWS. But we have a cluster set up. We're going to, you know, give it some workload. And then we're going to start removing nodes. And I'm going to show you how the replication factors work. I'm going to show you how the resiliency works while maintaining performance. Okay. So let's go over to the UI. And you're actually going to be seeing some new bits here too. So I'm actually showing this within one of our beta releases. Usually this is a solution engineer. No, no. As far as, you know, demo, demoing things on new software, but I've tested this out a couple of times. It works great. So I wanted to show you some of the new things that are coming within our next release, which is the 20 dot one release, which you should be seeing hopefully towards the end of this month, maybe if not beginning of next month. Okay. So here's the administration UI. You can see I have six nodes that I've created in the database. If I go to the metric section, you can see that I actually already have some load going on here. All right. If I look at the SQL queries, you can see that I have a lot of selects and updates that are happening. And over on the right hand side here, it's giving me an idea of the queries per second that are being performed under the current cluster. There's other things I can look at too. Yeah, Jen, go ahead. Hey, Chris, what's the workload that you're running right now? Oh, yes. This is the YCSB workload. It's the Yahoo benchmark. So we package a bunch of different workloads within the database. The one I'm showing today is the Yahoo benchmark. And if you want to see it running with the database, yeah, with the database binary itself, there's actually a couple of different workloads that people can use to test out the database as well. Right. So that's all in docs. I guess where I'm going. Exactly. I can even show that too. If you can just do a search for workload within our documentation, it will take you right to Cockroach workload. And you can use a variety of workloads here, such as TPCC. A lot of folks are, you know, are definitely familiar with that. There's a whole bunch that you can run there. But anyways, yeah, I'm running TPC. Excuse me. YCSB. You can see we have load there. We can look at the hardware metrics too. You can see that we have a lot of, you know, CPU utilization. The nodes I'm using are actually kind of small, because this is a demo environment. But you can see that we have workload happening. Now what we're going to do is if I go back to the overview, you can see we have six live nodes and we have 51 total ranges. What I'm actually going to do is I'm going to actually remove one of the nodes and we can see what's going to happen with performance. So let me come over here. I'm going to do a little command. Let's say stop. A particular node on my cluster. I'm going to stop node six. And I could have done this more crudely as well. I could have logged into the machine, you know, do a kill command and, you know, it'd be a little bit more abrupt. I'm going to do this more in a gentle fashion. So as you can see, I stopped the node. You can see the node is under suspect here. I have one node that's, you know, the cluster identified as not being available. If we go and look at the metrics, you know, there is a little bit of a dip, but you're going to see this come back up and the performance will maintain. And on top of that as well is that, you know, the, you'll notice that the database is actually very patient. Right. So we have a couple of different statuses here. We have a healthy status. We have suspect and dead. We're actually, because we can't communicate with a node, we don't just eliminate it from the cluster. We'll wait for it to come back. There's a default time of about five minutes where we, where we will mark something from suspect to dead. But if the node does come back up, it will just reapply itself to the cluster. And that's what I'm going to do here. I actually sent the, set this command actually to, to mark a node dead after about a minute and a half. So I want to get it back in there. And what you'll see is that eventually these nodes that, these ranges that look like they're under replicated are actually going to go back to being replicated because I just added that node back into the cluster. So, you know, and just another second or so you'll see everything be helping. Now right now I'm running this with a replication factor of three. So, you know, I can tolerate a single node failure. If I want to just tolerate more node failures within my six node cluster, I would need to up the replication factor. So that's something that's pretty easy to do. And you can do this even on a, on a, on a live database. So I'm going to show that here where. I'm going to come in and I'm going to change the, I'll show you this command in just one minute. It's basically an alter command that Jim was mentioning, mentioning before actually know before. Yeah, I'll run this and then I'll show you what this, what this zone configuration is. So what I'm going to do is there's a default zone configuration. And you can change this to specify how you want things to replicate what your ranges want to look like. If you want your ranges to be in certain regions. Or the number of replicas that you need that, and have an affinity towards a certain region, you can do that within, within this database. So I'm going to click, kick this off. This is actually going to up the amount of. The replication factor to five within the database. And you'll notice here and in just a few minutes, yeah, you'll see that I'm actually under replicated, right? Because now I only had, I only had three replicas and I need to up that to five. So the database is aware that, hey, this, you know, a configuration change was made. It's going to identify the ranges that are under replicated and start moving them out into the other different nodes. So you'll even see here that like the number of replicas that you're seeing in the nodes are going to go up as well. So we'll just give this another minute or so, so that we get into a healthy state. And then I'll show you how we can eliminate, you know, two nodes and still have the resiliency and performance under, under load. So it makes sense. Jim, any questions coming so far? Uh, no, either, like, we're that good at explaining things or we forward everybody to tears. No, I'm, I'm joking. No, it's, it's pretty scary for us because this is awesome. Yeah, that's good. You can see here, we did have a little bit of a dip, but everything was, uh, went, went back to his ride of rain. Um, you know, during the, uh, during the node loss, and you can see here that the service latency or, you know, the node, um, you know, fell out of service during the, during that time. So let's go back and check. So yeah, we have about, you know, one more range to get cleaned up, and then we're going to do the same thing. We're going to eliminate two nodes and we'll see how the performance looks. Uh, while we're here, you know, I, you know, you can see, we're good. We're in a good shape. While we're here, uh, there's other things that you could do within the UI that are actually really useful for diagnosing things that are happening or monitoring activity. You can look at latency of statements. You can look at the jobs that are happening in the database. Um, you can see there was an import job that was run here to actually see the database with the workload that I'm running. And there's a lot of advanced, um, debug features. There's a really rich rest API that we have so that you can see a lot of things that are happening within the database that all power this, um, that all power this, uh, this UI. Okay. So last part of the demo, let's go just do, um, now that my replication factor is up, I'm going to stop two nodes. Let's stop. We'll stop no six again. And we'll stop node five. And because our replication factor of five is up, you know, we can incur a two node failure. Um, and again, you'll see that we have two, um, two suspect nodes that are here. Um, and the metrics will go back to that as well. You'll see that pick up eventually as well. Um, and what we'll do too is we'll actually have one node go into a dead state and I'll actually bring back one of the nodes so that you can see what's happening here. I know we're getting a little bit tight on time. So just give us just a second. Um, take your time and people, people like to see, people don't like listen to me, dude. I'm just slide guy. This is much better money. Cool. All right. Well, I'm going to bring back node five. Um, just so folks can see this happen. You know, we kind of saw this in the last scenario where we have a suspect node that's going to come back and be attached back to the database. We'll see that suspect nodes go from, yep, from two to one. So we have five live nodes and I'm going to let that other node just die. And you can see how, um, you know, the database will continue on and continue to perform. Uh, my workload should be going for another couple of minutes. So we should be fine here, but yeah, after about a minute and a half, um, you'll see that this, this node, uh, that's in suspect will actually go to dead. That's pretty much it. I mean, that's, that's all I really have to show from a performance perspective, but, uh, it's just a very patient database in dealing with adversity. Uh, so when you do have failures, dis failures, node failures, and so forth, it takes a pause. It looks at what's going on. And then it, you know, patiently will say, all right, you know, I need to replicate over here. I need to balance myself out so that I can continue to, um, meet the workload that, that, um, that's being asked for me to do. Yeah. So Chris, a couple of things, um, I, you know, I think it's really cool. Like, look at this service latency, the 99th percentile for P 99 latency has actually gone up as you may changes. Another key thing, right? Um, you know, we're constantly looking at P 99 latency. That's one of the things we're looking at. Actually 23 milliseconds is pretty high. We can do lots of things to actually get that even further down. There's lots of tuning. Typically we're seeing those things under five millisecond. Right. And so, um, I think there's a lot of rich stuff in this UI. One of the questions that came through Chris was, um, How do you, you know, how do you run multiple nodes with the web UI? I mean, is there fault tolerance for the web UI? I guess like, how does the architecture of the web UI work? Right. Like somebody asked me, is this part of cockroach DB core? Yes. This is definitely part of the free version of our database. You can download. You can actually be up and running with this. Like I can do it today. Um, but how does the web UI architecture work? Like, where is that running? Is there multiple instances of this? Like, how does that. Yeah. Great question. So since we're shared nothing, uh, set up, you can run that, uh, the admin UI on any of the nodes. So, um, you know, if one of your nodes dies, you can easily just connect to another node and spin up the admin UI. There's nothing you really have to do from a setup perspective to make that happen. Um, it's not like you're, you know, you have to think about my application being on two different, um, you know, two different management nodes, make sure that they have a shared database to have all the metrics and so forth. It's a much easier setup. You just basically say, I want the admin, you know, UI running on this node. If, if, you know, that node dies, I can just spin it up on the other node and you're off and going. Right. So a node is a node is a node. It's always the same binary. It doesn't matter. And that's like a true, like really important concept in distributed systems. Right. And I think that's, you know, it's kind of one of those things. I think this is actually really important. Now one of the other, one of the questions came in, how compatible is this with, you know, Postgres command line, Chris. Yeah. It's very similar. So you, um, I can even bring up the command line here. So if you're familiar with, um, if you're very familiar with, uh, you know, Postgres, um, I can actually connect to one. Um, here we go. Yep. I'll go cockroach, SQL, uh, all this equals local host. Um, Oh, I forgot my one more flag. And yeah. So if you want to do, you know, show tables or, um, if you need, uh, help on a particular command, um, the syntax is very similar. Um, now we don't support every single, uh, postgres item that's out there. There's a few things like stored procedures and, and triggers that we don't support. But, um, for a lot of those features that customers are looking for, there's ways that we can support triggers through using our changed it capture option. Um, sometimes folks can move their stored procedures into micro services. Uh, so we don't support everything that Postgres supports, but the, you know, the wire postgres wire compatibility is, uh, is pretty spot on. Cool. Um, what happens when two nodes go down and say in a three node cluster, what happens that it is, is stop accepting connections? Is it, how does it gracefully handle that Chris? Yeah. So in the event that two nodes, that's a tough one, right? Yeah. So that, that's, that's actually a tough one. Right. If you have a three node cluster and you have a two node failure, um, what's going to happen is you're actually going to, you're going to lose quorum. You're not going to lose the data, but you are going to lose quorum. Um, and what happens is those Rangers are going to sit out there. Our support team is very good at bringing those databases back up to health. Um, but if you really want to prepare for resiliency, as, as Jim mentioned, um, you're going to want a more, you're going to want more nodes. You might want five, you know, with a replication of five, if you want to make sure you can fail two nodes. So anytime you're designing a distributed system, um, especially with cockroach DB, you really want to think about your resiliency factor. Um, what do you want to survive through? Um, so, you know, you can certainly set up cockroach with, you know, with three nodes set up as long as you're, you know, you want to be able to tolerate only a single node failure. Um, you know, that's, that's the best way to think about this. Uh, like I said, you won't lose data, but the data won't be, um, accessible because we don't have a way of doing, uh, quorum reads and writes. Exactly. And I think that's actually pretty important. So again, I always think about the, I usually with the database, I'm thinking about the logical model. You kind of got to think about the physical model of like where the data is going to be, what you want to survive. That sort of stuff, right? And I think that's like one of those key questions. That's a little bit different about this than, than a typical database, right? And I think, you know, going through that with, with customers. And we talk about, um, I think on the website, Chris, we call them topology patterns, right? I think that's what we call that in our docs. And I think there's a whole bunch of, yeah. And so we, we walked through that with people, like how to configure the database. There's, I, I've worked at a couple of pretty cool companies. I've never seen a documentation as good as ours. And I'll just say that because I, not because I love my documentation team, but, um, we get a lot of comments. Like there's a lot of really good information about how to, how to configure the database to just survive these things on, in our docs. Um, another question, Chris, and I'm just doing QA right now. You guys, I'll come back and finish up with transactions at the very end. Um, how does band work? How does bandwidth work for synchronization, right? So I have, you know, replicas, I have, you know, I'm writing data in triplicate, right? And so for each replica set, let's just say I'm in a single region. You know, there's data that's being written in the single region. But what happens when it's like cross region? I mean, you know, how does that, how efficient is it? Is there costs associated with that when you're, you know, let's talk about single region in that multiple region. Um, in terms of those, those sort of network effects. Yeah. So great question. So anything in a single region is pretty straightforward, right? So I'm actually, we have a chart here in, in the admin UI that shows the latency between the nodes. Um, so you can see here, the latency is really low, right? Because these nodes are all close to each other. They're in a single region. So you should expect, you know, um, you know, reads to be a millisecond or less and writes to even be two, two milliseconds or less. It should be very fast. Once you start getting into geo distributed clusters, you definitely have to, you know, you're constrained by speed of light. Um, there are a lot of things that we built into the database. Um, you know, such as parallel commits and so forth. That allow you to do a single round trip for a particular right transaction. So that's really, you know, and it's a two phase commit. So it's a really, um, fast way that we could do writes, um, even though they're, they're, uh, even though their latency between the regions might be, you know, so much far, you might be going from the, you know, the Eastern part of the US to the Western part of, uh, the EU. And you're bound to there basically by the round trip of speed of light. Um, so yeah. And Chris. And Chris, every node can accept a right. This isn't like Aurora where there's a right node and it's in every region and you're kind of stuck with that one right note. Every single node is an endpoint. Every node can take a read, right? It comes back to like the atomic unit, everything is, it's all, like, like we were talking about people. Like every node is the same. Yeah. That's a great point, Jim. And you can be really particular too, as far as where you want to pin data so that you can make your reads and writes very low latent if they're being utilized in that particular locality. So great point, Jim. But yeah, and so all this stuff, I mean, you think about database, think about network diagnostics, would that be in a normal database? Um, you know, that, that's not something that I would think about to be in the UI for a DB. Um, but it's actually really critical when you start thinking about a distributed database. Uh, and so we, we actually include all that it's kind of table stakes to, to what we're doing here. So, um, I've gone through a fair amount of the questions that we're in the chat. I'm just going to jump back into, um, just to talk about geo distributed application. Um, um, so the last thing I did want to talk through is, you know, Chris talked about being geo distributed and, and how do we actually take care of transactions, right? Cause if we're going to write data in triplicate across the cluster, well, man, like I got to ensure that that stuff is consistent, right? And so we do implement acid. I mean, when we think about the eye in, in acid though, um, we're real heavy on the whole, you know, serializable isolation. And so we've actually implemented that for a distributed system. So, um, we're pretty excited about that. And so I just want to go through kind of how that works as well. Um, really quickly, this is a, like, looking, I'm going to give you the best case. There are a lot of use cases, a lot of like corner cases here, you know, multi-phase transactions, transactions that deal with like 15 tables. I think it's really complicated. Our team has done a really good job of actually re-architecting the execution layer, the database to take care of these things. But let me just show you just, like some really kind of core concepts. So what we actually had to deal with here, let's just do an insert into the SOG stable. I'm going to take two records and I'm going to insert into this, this hypothetical table. Now, uh, the lease holders here are those, uh, ranges that are, uh, would have a heavier outline, that gray outline around there. Right. And so every SQL statement in a database actually gets broken down into multiple different transactions. Right. And so this case, the first transaction is begin transaction, which says, Hey, uh, I'm going to start this now. And then the second transaction is right. So I'm going to write sunny. The first thing I'm going to do is I'm going to find the range, the lease holder for the range that I need to write that data to. It's on node one. I'm going to tell that node. Okay, great. I'm going to do this. It's going to append a record that basically just says, Hey, I have a transaction and it's pending. So right now I've got this, uh, range. And so nobody can really do anything else. Like cause why I'm going to, I'm going to do some work here. So it's going to be pending. And what it's going to do is it's going to say, I'm going to, I'm going to temporarily write this data in, and I'm going to go tell the other replicas of this data, that we might not even know it's like, Hey, temporarily write this in. Okay. And so they're going to go ahead and do that. And it's going to, like, one of those notes is going to return back on a knowledge, maybe before node four, node four is a little bit further away. And basically at this point, I have quorum on the first part of this data of this, of this insert. Right. And I've actually now, I can actually wait for this thing. Okay. The app comes back. The transaction now is also going to say, Hey, wait, I've got to write the second row. And that's going to go into the second range Aussie. And I'm going to write that record. It's going to go do the same thing. It's going to say, all of these records at this point are kind of like penciled in. That's why they're all yellow. Now what happens is the one comes back and I, as soon as I say commit this transaction, this, this temporary transaction record that I had kind of pointed out, it actually goes to commit and in going commit what it does, the communicates out and just says, okay, everybody commit this stuff. All of the records are now committed. And I'm going to send back in an acknowledgement back to the originator. The application is asking for this. And now I've actually done this distributed transaction. Now I just went through a whole world of complexity in about. Two, three minutes. Doing this is not simple. Lots can go wrong. But I just want to give you a little bit of a sense about what core is, right? Two nodes out of three actually have to write data before we can commit it. Right. So I want to tell you a little bit about the core. There's a really odd number in terms of replicas, you know, Chris talked about, Hey, we can increase the number of replicas. That's important, but it's got to be a number, right? Cause I need core. I need an, you know, so two of three does if it's five, it's three of five. No, it's had to write that data. And so we go through that. We actually deal with that, that transactional layer. And this is how we actually are able to, to, to ensure that that transaction are happening serially in the database as well. I have four minutes. So Chris talked a little bit about pipeline and parallel commits. So typically when you write a transaction, there's lots of back and forth. I'm gonna start this transaction, it's pending, I come back, I'm gonna write Sunny, I'm gonna write Aussie back and forth. But what we're doing is we're actually taking the Sunny, the Aussie and this transaction and we're doing it kind of all at once. And we're giving it a staged, basically status if you will, right? So we can actually start doing all these things all at once and we're actually pipelining transactions. This is one of the ways in which we improve performance. And actually we can actually send this all over in one package. So the question about network and bandwidth, all these other things, we're actually doing some really interesting things underneath the covers to actually conserve the resources that are actually utilized in the database. Now, once all those three things commit, we can actually come back and have a commit happen and we've done this in say half the time, right? And so there's delta between what a normal transaction looks like in the serial mode and what we're doing from a pipeline mode greatly increases the latency and what's happening in the database from a performance point. And so that's my last point. That was the last slide. I am at, let's see, 150A here. I talked a little bit about the requirements at the very beginning. What is distributed SQL? A, it's gotta be SQL. B, I hope we showed you how we can simplify this scaling of a database. I hope we showed you how you can actually have a very resilient database without thinking about other systems. Let's let the database deal with that, right? I showed you a little bit about transactions and how we can actually deal with serializable isolation. And then finally, and we didn't really get into the demo, but I kind of showed a little bit how we actually able to dive into a location as well. So we are at the top of the hour. There's lots more. Our docs is a great place to get started. Go download CockroachDB core. It's on our website. We give you ample opportunity to go find it. You can start a cluster of minutes with Cockroach Cloud, swipe the credit card, get a three-node cluster, up and running pretty quickly. But I would implore everybody, please do. There's a survey link here. It's cockroach slash 2VBW2GW. If you could, please do that survey. That would be really, really helpful to us. We'd love to get feedback in terms of, I'm gonna dump this into the chat really quick here. Let me just grab this guy. I'm gonna dump that into the chat for everybody so that you can actually do the survey if you like. That would be awesome. If you like, from me to everybody, to everyone, not just our organizers. If you all can take that survey and let us know how helpful this was, it would be amazing. And then we will send out a recording everything to everybody as well. So thank you, everybody. We are at the top of the hour. There was one last question about transactions. I will absolutely answer that. We'll get that out to you in an email to this person who asked. And I believe that's the end of our webinar. So if there are any other questions, please do feel free to reach out to us. I think info at cockroachlabs.com. I'm Jim at cockroachlabs.com. Chris, are you Chris or Chris C? Chris C. Yeah. He's Chris C at cockroachlabs.com. So we're happy to answer these things, but any of the questions that were here in chat will make sure to get answers for everybody. So with that, thanks again, everybody, for taking your time out of the day right at two o'clock, well, at least the East Coast time. Listen, have a great day. Stay safe. I hope everybody and their families are well and everybody they love ones are all doing well in these crazy times. And thanks for taking the time to learn about us and have a great day. Thanks, Chris. Thanks, everyone.