 Welcome to Database Isolation Levels, Data Issues, and Global Transaction Consistency. I'm joined by speakers Chris Casano, Solutions Engineer at Cockroach Labs, and Jim Walker, VP Product Marketing at Cockroach Labs, who will discuss common data consistency issues, transaction isolation levels, application best practices for retries, transactions in a distributed environment, and Cockroach DB and PostgreSQL compatibility. Let me tell you a little bit about your speakers. So Jim is a developer who has spent a large part of his career helping people understand complex emerging technology. He's passionate about distributed systems, data, and open source technologies. He's a starter baholic and loves building new apps and new communities. Chris is a Solutions Engineer and has been with Cockroach Labs for a year. Prior to this, he ran the East Solutions Engineering Team at Hortonworks, and he's been working with databases for 20 years. So with that, I'm gonna hand it off to Chris and Jim. Y'all can take it away. Thank you very much, Lindsay. We're happy to be here today to talk about some interesting topics here. I'm isolation in a database. I think it's one of these concepts that not a lot of people actually pay attention to sometimes, but I think are actually pretty important when you start to think about some of the problems that can go wrong in a database. So we're gonna talk about this today. Where am I? Oh, there we go. Again, I'm Jim. This is a picture of me. And that's my friend Chris there. Hey, Chris, how are you doing today, buddy? I'm doing well, Jim. I'm doing well. I'm just making sure you know where the mute button is. You got me. Yeah, you know, I think that all future presidential debates should be done on Zoom and the moderator should have a Zoom button. What do you think, Chris? Love it. So... Kick people out. It'll be awesome. Exactly, right? So please do ask questions. We're happy to take them throughout the course of this. Anyone, Lindsay, was nice enough to encourage that. We doubly encourage it. Chris will be definitely monitoring the chat as I'm talking. I'll be monitoring the chat as he's doing a demo. We're gonna get into CockroachDB a little bit and show you guys a little bit of that today. But engage with us in the chat. We're more than happy. We'll try to get to everything that comes up in the chat. It's the chat today. I don't know if there's not a QA panel. So again, like Lindsay did, I also encourage it. Audience participation is great and help us help you. So today we want to talk about isolation levels and then what transactions mean when you become in a distributed database. And just to set the context, I have a couple of things to talk about. And I think if I could start in one place, I think this is my friend, Marv Adrian, who I've been friendly with for probably about 10 or 12 years. I too was at Hortonworks with Chris. And I remember talking to Merv about lots of different things. And recently the team at Gartner is really pushing towards the concept of what is a cloud database. As most workloads kind of start to move to the cloud. I think we're probably about 30, 40% of the way completely moved to the cloud across all organizations. The move of the database into that environment has proven difficult for some, but there's lots of offerings out there. I will contend that not all cloud databases are equal. Just running a database in the cloud, using RDS, using Postgres, that's one thing. It's a single instance. It's different than what you have and running it on your own server or whatever, because well, they host the infrastructure and whatnot, but is that a cloud database? To me, a cloud database actually answers the call of the cloud is scalable, easily scalable, is resilient, it won't go down, it won't die. And these are the things that actually become really, really interesting, especially as we start to move legacy technologies, legacy databases into the cloud. In a cockroach lab here, we contend that in order to truly take advantage and be cloud native, you have to re-architect the database. And we'll talk a little bit more about that in a few slides. Why do we need another database? We've had these relational databases over the past 10 years, no SQL databases came along. You're all, this is Postgres call, so I don't need to talk to you about relational or no SQL, if you think of the MongoDBs and because standards, I guess these databases, the no SQL stuff, they were optimized for read and they were optimized for global access more like the way a CDN would my work, but they took compromises in how they implemented transactions. Really neither of these two solutions were architected for the cloud and architected for not single data center cloud, but multiple data center and being able to survive across regions and these sorts of things. And so there is room for another database and this is the concept of distributed systems and the emergence of distributed systems over the past 10 years, really five, seven has really transformed the way that we think about our applications and our infrastructure that supports them. I love this concept of, if there's a guy by the name of Brandon Phillips, he was one of the founders of a company called CoreOS. It was really kind of a core kind of component of driving the whole Kubernetes community. Yeah, I think he was the top man because he was in a TOC, real brilliant guy. He used to refer to these new environments and this new approach as GIFI, which is Google infrastructure for everybody. Cause what we've seen happen at Google over the past, a couple of decades, decade or so is that they have actually figured out distributed transactions. They published a paper called Google Cloud Spanner, which is definitely a distributed database running completely in GCP, but it's a wholly new database. They could have used big table and tried to slap transactions into that, but they didn't. They actually, we architected it from the ground up and in so doing, they published a white paper. As Google does, their Google exhaust is fantastic. And Cockroach Database in Cockroach Labs really is the genesis of that, is in that paper. And so the concepts of what is a truly distributed database, what is distributed SQL? A lot of those core concepts were outlined in that paper. There's a couple of other approaches out there, but ultimately what we're doing here is architecting for really this cloud native world, truly being able to take advantage of cloud infrastructure and all the promise that that delivers. And so there is an emerging set of database and it's called distributed SQL. Now, why can't you do this with some of the older systems, the legacy stuff? I mean, you could kind of get distributed with Oracle. You can use GoldenGate and you have Oracle Rack and you're distributing data, it's expensive. It is a lot of configuration. It's basically taking other tech and augmenting it with augmenting a legacy database to make it do replication. It's ultimately monolithic. Now, I will never say anything bad about Oracle. That database is incredible. And being built over the past 30 years, it's an incredible piece of technology. I would say the same thing about Postgres. It's maturity in terms of what it can actually accomplish. Phenomenal, are they actually built for cloud? Are they built to actually take advantage of scale? Can they be resilient? Can you handle transactions that are gonna be distributed across multiple different regions so that you can actually deal with latency issues? I think one of the biggest issues when you start to go into cloud, you start to think about transactions is, well, there's one big challenge and it's called the speed of light. How do you actually deal with that? I think what AWS has done with Aurora and they have a Postgres version of Aurora is that what they're doing is they're taking the storage layer and creating distributed storage underneath a layer or instances of Postgres or of MySQL. Now, is that distributed? Well, it's kind of distributed in a single data center. And you can kind of get some rights. You have asynchronous rights of the incomplete distributed storage layer across multiple different data centers, these two blue boxes being different data centers. So you can get kind of distributed rights and you can get data available in lots of places but you can also get that through Bongo or distributed in a SQL database. The trick is rights. And this presentation is about isolation and guaranteeing transactions in distributed environments so we can take advantage of these new worlds. And so with a single write node, you still have a bottleneck and you haven't addressed the latency issue with actually accessing that write node. So for instance, in this situation, I have two instances of Aurora are to implement it in two regions, one US West, one US East. I still have the latency issue going across the country to get to that write node. And worse off is I have a single write node so it's a bottleneck. What we've done at Cockroach is architected something completely different. And in Cockroach, what we find is a completely distributed database, completely matrixed where every node can take a, is an endpoint, can take both reads and writes and it can service the data no matter where it's found within the cluster. Now data in a Cockroach cluster is replicated three times. So it's not on every node, but every node can access that data, hit as it can find us. We get these massively distributed data centers and we can actually start to deal with things like latency in a globally distributed world. Now the complexity here is transactions, right? The rate transaction through a single write node, well, that's pretty simple and straightforward because I have a single write node. Even if you're doing something like multi master in Aurora, well, it's only gonna be two write nodes. And by the way, if you're doing that, you're only allowed two write nodes, no extra read nodes and it's only in one region. So it's really only two nodes. What we're talking about is spin up as many nodes as you want and have a database span multiple different regions. It could even span multiple different clouds. And by architecting from the ground up, Cockroach Labs has really done something completely different and shifted away from this monolithic and re-architected the entire database to be distributed to take advantage of the thing. This is really an evolution of the relational database to meet the needs of cloud native. I will say though that if you're gonna be a SQL database, well, you better implement standard SQL and ease scale, be resilient. But I think these last two are really important to think about. And this has come to the stuff that we wanna talk about today in terms of what are isolation levels and what does that mean for you in your database? Is the database as a compliant? Are you using standard SQL syntax? Cockroach were wire compatible with Postgres. It's one of the reasons why we love this community and are involved. But if you're gonna be a distributed database, latency has become a really big issue. And so the ability to tie data to a location or have data follow a user, if you will, where those requests are gonna be. So I'm not writing data to every single node on the whole planet, right? I'm only writing data to three nodes. And so I need to tie data to locations to help us deal with these latencies that will actually come up as people kind of move around the planet. Or I don't wanna actually deal with this speed of light issue that I was talking about, right? So I think that's a core thing. And again, cockroaches is fairly unique. Every node is a single atomic unit. It's an entire database within each node. It isn't different pieces of the database. It isn't a storage node and a transactional node. Everything is the same. Every node is a consistent gateway to the entirety of the database and can implement everything. We're gonna show you the management UI for CockroachDB in a little bit. That UI can be spun up across. You can point it at any node and it will spin up the UI. There isn't like an admin UI node somewhere. I know it's just actually included in every single node. So this allows us to basically deploy Cockroach in ways that are very unique. We can get them everywhere, right? And the magic of Cockroach really comes down into this. How do we deal in a distributed architecture? How do we coordinate and gain consensus for queries and transactions? And we'll walk through kind of how that works in Cockroach. We use something called Raft. We use, you know, we have consensus, distributed consensus protocol. That allows us to basically get quorum, right? So two of three copies actually have to commit before a transaction gets committed. And so you're typically gonna wrapper your SQL transactions in your code to make sure that they actually commit all the way through, right? You want retry logic around those things in case there's failures, in case there is conflicts across multiple different nodes, right? But we deal with basically these transactions inside the database in any sort of conflicts. Nodes can be added or removed from Cockroach and everything gets rebalanced and repaired. We're gonna actually show you that. And then again, we can tie data to a location so that we can not just meet the needs of these of latency issues, but it's also used by many to think through compliance. You know, I want all German data to live on German servers, right? So there's some really cool stuff that we can do in Cockroach. But what we wanna talk about today, as I know that's kind of an intro to Cockroach, it's just kind of databases and isolation levels. Now, you know, I coded professionally for quite some time, used several different databases. And I gotta tell you, most developers I know don't even think about database isolation levels. I think it's just one of these things. I spin up a database and does it work? Okay, great, that's wonderful. I think we're becoming much more familiar with these things over the past couple of years. I think we've had some high profile issues with databases with downtime or with some graft, if you will, people stealing money or whatever it is. But isolation levels are actually pretty important. And every database has different isolation levels and you can think through that. There are really kind of four that are typically thought of when we think about these things. There's variants in this and some databases. But they really are read uncommitted, read committed, repeatable read and then serializable. And you set this in the database. You set the isolation in the database and the database itself then actually make sure that each transaction is gonna meet the requirements of these various different isolation levels. And this is the I in acid, right? Atomicity, consistency, isolation and durability, right? And so this is the I and then the isolation level of acid which becomes really important. Now in read committed or read uncommitted, basically what you're doing is just putting a lock on the database until the end of transactions. So reading from the database never involves any sort of locking, right? Like I can just read and whatever that is at that further moment in time, that's what I'm gonna get. Now, things like MongoDB by default are read uncommitted. I believe you can get up to read committed with them, but, and I'll show you some of the issues that you can actually have out of the database for that, right? And it's really kind of back, you know, these are reads, but actually think about this now in the context of transactions, right? So read committed is locks are required for reading and modifying the database. And then they're released after the read, but on modified objects, if you're writing or updating it's gonna be until the end of transaction, right? Repeatable read is much like read committed, but there's additional locks that are placed on other structures like indexes, where, you know, like if you have a B tree that's actually helping you understand where data is, secondary indexes, all of these things are released after reading. So that's a higher level and it actually protects and gets some things. Serializable is kind of the top level and basically the gold standard of if you want guaranteed transactions. So you're gonna use serializable in times when you need a system of record, you know, the data that you have has to be right. You know, it's financial transactions or it's inventory. And honestly, it doesn't require that much more. So you might as well just do it. I mean, from a performance point of view, there is, yes, you're gonna have some issues because there's these locking and other things that are happening. You're gonna have to put retry logic around a lot of your transactions because sometimes transactions fail because of the serializability nature of them. So there are certain things you have to do, but it does actually deliver basically the highest quality data in the database that you can get, right? And so there's these kind of isolation levels which you have to think about when you implement a database. I'm not sure if everybody here does this or not. Again, as a developer, I didn't think about these things, but I think it's very important to actually understand and think through. I think we read about them but I'm gonna never use them in production, but they can lead to some really big problems. There's these kind of three dark horsemen of bad issues that can happen to data. Dirty reads, non-repeatable reads and fan error reads. So a dirty reads is when you are reading table from, you're reading from a table or reading from your database, data that has not yet been committed. So I have a transaction happening, when a right transaction is happening, I don't have that thing locked. I'm reading that data. Now, what happens if that right transaction failed in the middle or something got violated? I now have read incorrect data and I'm using that in my transaction. Now that can cause issues. Non-repeatable reads happen when sometimes you can read the same row twice and get different values every time you read it. So what happens when you have a very complex transaction where you're reading from multiple tables and you're going to one table, you're doing some calculations on that all within a function and going back to the same table to get something and you have different values, right? How can that actually cause issues in your application as well? And then there's this concept of a phantom read. And a phantom read is when two queries are happening, but the data that's being retrieved by both of them is different. So I have now two instances of my application hitting the same database, but I'm getting different results back. What if that happens twice in a single transaction? Again, there's some really kind of big problems that can happen. And there's ways of gaining basically your backend database to basically leak it of data if you want it to do. So in all cases, what you're gonna do is you're in an uphaving is eventually consistent data where it basically makes up for these errors. But really what that means is you have incorrect data in your database. And really the only level that is gonna deliver and protection against these various different issues is serializable. So if you look at read uncommitted, yes, all these things that happen, read committed, you're gonna protect yourself from dirty reads, but you still have a non-repeatable and a phantom. And then repeatable read, of course you can still get a phantom read out of that. And then with serializable the highest level, you're gonna protect yourself from these data issues that can go wrong. Now, lots of things can happen in data issues. You can imagine things that happen in your transactions. I find it interesting to think through what are the repeatable queries that I can run over and over again as a hacker through some sort of data API to actually leak your database of money, if you will. And I think that's the biggest threat. And in fact, there's an entire topic, there's entire paper on this exactly, this exact topic. Guy by the name of Peter Bellis wrote this, Todd Worsowski and Peter Bellis. It's up on this website that's here. It's a pretty interesting read if you really wanna geek out about transactions and some of the things that can go wrong. He came up with this concept, this exploit called acid rain, which people have actually used in the past to mine Bitcoin and basically steal money from Bitcoin because of the way that the data underneath is protected from an acid point of view. And so there's lots of things that go wrong. And these are things that I believe that most people should be thinking through. And so to think about isolation levels, they're there for a reason and they're there to protect us from issues that you can have in your data. Again, this paper is one of those things that I'm completely compelling and kind of interesting if you wanna go study up on more advanced topics. So with that, I wanna call on Chris. Chris, are you still there, buddy? I haven't put you to sleep yet, have I? Sorry, what? No, you're funny. So I know you are. So please do, y'all, if you have any questions about those things, please do ask in the chat. We're happy to answer these things. And then I just wanted to get Chris to show you a little bit about cockroach and what we're doing and from a distributed data point of view. So Chris, it's all yours. All right, great. How's my screen coming in? Looks good, buddy. It looks great, actually. Okay, great. So yeah, Jim did a great job of explaining a bit about the isolation and consistency levels of cockroach. We have a lot of documentation on this, on how this all works. If you do search for life of a distributed transaction, we give you a lot of detail of how all the underpinnings of the database work for it. In our doxel world-class, you can find a lot of things here as far as even just getting started. If you want to install this locally on your machines, really easy to do. In a Mac, you can just use Homebrew. You literally can just take the binary and off you go. You don't have to do a large install process. It's very easy to get up and going, and I'll show you how that works today. One thing I should have done today when I prepared a demo is that we actually have a demo on serializable isolation. And if you wanted to check this out, you can go, again, go to our dox, search for serializable transactions. And we have a nice little tutorial that shows you how to do this if you want to see the database in action. And it shows you the same thing within Postgres and how you can change the isolation levels in Postgres, but not necessarily within Cockroach. All right, so just something that's out there. If you're looking to get started. And the last thing, if you really wanna get really going, you can always go to Cockroach University to learn more about how to use the database. But let me take you through a quick demo. So I'm on a Mac and I've had the binary already downloaded and so forth. If you wanna just get going, we have a nice command that you can use called Cockroach demo. And this is the simplest way to kind of get started. You can set up nodes on multiple clusters and do all the fancy things as far as setting up nodes on different clouds, on different availability zones and have them all connect to each other. But you can also just do this in a single binary and simulate it. So we have a command, like I said, called Cockroach demo. I'm passing two parameters in here. One is called Geopartition Replicas. And that's gonna set up a nine node cluster across three simulator regions. And then it's also gonna have some load in the database too. I can see transactions happening in the database. So let me just kick this off. This will take about a second or two. And like I said, it's gonna bring up a simulated version of Cockroach. It gives me a URL where I can go look at the admin UI. So let's go take a look at that. And here you go. Here's our admin UI, pretty straightforward. You can see I have nine nodes. It's still loading up some information. If I go and look at the metrics, you can see this is starting to build, but I do have some SQL queries running because I did that with load option. Also, if I go back to the overview, you can see that it's created a simulation of what a geo-distributed cluster would look like. I have three nodes in US West, three in US East and three in Europe West. So again, I can start playing around with, hey, was it like to work with a geo-distributed cluster? One other thing that this does is it also gives me a command show that I can go and start playing around with. And we do give you some sample data. So, since this is a Postgres conference, there's a lot of Postgres commands that work within Cockroach as well, since we use the Postgres wire protocol and the dialect is almost 100% compatible. Not quite there. There are some things that are unique to Cockroach, but a lot of the Postgres commands that you used to show tables, you can easily just type in show tables. I can see the sample tables that are in the database. I can use the shorthand commands like backslash DT, same thing as show tables and off I go. So what's great about this is that because I have all these different zones or where data can be distributed amongst all these different regions or different data centers, is that allows me to pin data or database objects into those regions. So I'm gonna give you an example. So let's look at one of the tables here that's in our sample database called promo codes. All right, and you can see we have a table here called promo codes. We have an index or we have two indexes on this table. One that's identified for West and one that's identified for your West. And you can see they look exactly the same. And this is one of our interesting topologies that we have with what you can do with Cockroach. This table might live in one region, but we can have the indexes or multiple secondary indexes live in other regions. And the reason why you would do this is that you wanna speed up reads for locality. So if I have users entering the database from the West and from your West or I have US users entering the database from US West, they can get local data from a single table. And like Jim mentioned, when we transact anything in this promo codes table, this is at a serializable isolation level where we're committing transactions into all of those indexes that are geo-distributed as well as the table. And that's all done in a serializable fashion. And you can see here when I, you know, the statements that I have here where I'm altering the where the index is positioned, I'm saying, hey, I want one of my secondary indexes in your West and I want the other secondary index in US West, right? And this has me covered across all the geographies so that I can have fast reads no matter where my users come in. So that makes sense. Hopefully that's pretty straightforward for folks. I mean, I can show this in a pretty simple fashion as well. I think we have a little bit of time here. So why don't I just do this real quick? Yeah, let me create a database for you. Let's create a database called test. And I'm gonna alter that database so that all of my objects in that database is gonna be located in the US East. And this is a really simple command. All I'm doing is I'm saying, hey, for the test database make sure that all of my leaders, if you will, my leaders or replicas that are gonna make up all those pieces of that database always lives in the US East region, right? Or at least where I want the leaders today. And then the replicas, they can all, you know, be distributed amongst other regions. So that's how I'm gonna set this up. And you could be really particular about where you position things within the database. So now that I have this database set up, I'll create a table there, right? Here's my table, I'm just creating a simple table with two columns, an integer and a string. And then the last thing I wanna show you is where the shards or what we call ranges within the database where they exist, right? So right now I have one table. Each table is broken up into different shards. And we call them ranges on the cockroach side. And with those ranges, you can see I'm listing out all those ranges here. In this case, this table only has one range because I just create one table. But if I had a lot more data here, you would see a lot more ranges. And in this case, I'm telling the leaders for these ranges to always exist or to have their leaders in the US East region. So the US East region would be identified by one, two or three. US West would be four, five, six. And Europe would be seven, eight, nine. So you can see here, I can assure that my lease holder is always within the US East region. I can see my replicas are in the other regions as well. So I have more resiliency within the database. And like I said, this is the beauty of cockroach is that I have a distributed, geo-distributed database. And I can be really particular where I wanna put my replicas, where I wanna put my leaders. So that I can have high resiliency, but also have locality for my end users so they can have low weight in the queries. Hopefully that makes sense for folks. I know I went to- That's awesome, Chris. And we're gonna talk a little bit about lease holders and transactions in the next couple of slides, but just to make it really clear to people how kind of raft is used to get distributed consensus, which I think is actually a really, really important part when we think about distributed databases. I think there was a good question here though, Chris. And I'll just lob it into you now while we're talking. Is cockroach support snapshot isolation? And is there a benchmark for the different isolation levels in cockroach? Yeah, it's a great question, David. So one thing we can do is we have a topology called follower reads, which allows you to do a stale, but consistent read. It's kind of like snapshot, it's pretty much on the snapshot isolation level, but we're only thing that we're tampering there with is on the read side. There's nothing that you can tune within the writes. So- That's great. So we just simply do not implement snapshot isolation. We have basically built a database that is optimized for really two things, correctness and latency. And correctness, it's serializable isolation. You can't tune it down, right? I mean, there's no option to do so, right, Chris? There's no option to do so. The best you can do is just on the reads. You can, if you want, you can call a read to just do a stale, but consistent read, right? Right. And it just really depends on what you want to accomplish from a table to table point of view, right? Because you can do follow reads based on the table, right? Isn't that how it works, Chris? Yeah, you can do it based on your SQL query. So you can just say, hey, for this query, I want to be able to read back in time. And right now the staleness that we guarantee is within five seconds. I think it's 4.8 if you want to get really technical. But a lot of use cases can tolerate that sort of staleness and it allows you to read from all the replicas that are local to you into your distributed database. So you can get all these fast local reads that are only about five seconds behind. So if that works for you, for your use case, you know, you're kind of golden with cockroach. Right. Yeah, so I mean, if your table data, like for instance, you know, like the promo codes and the example you were using before, those things don't change all the time. So you're going to be okay if those things aren't exactly right in the exact immediate, right? Like it just depends on what that data is and what you want to accomplish. I think that's where these things become really important to think through, right? So, and then, you know, the other part of that was, you know, just benchmarks and isolation levels. You know, we do a pretty good job about using, you know, industry standard benchmarks for everything that we do. We don't make up our own benchmarks and run it against our database. I call that benchmarking, honestly. You know, we try to use TPCC for a lot of ours. We're actually running a couple of the other TPCC benchmarks against our database. We do it nightly actually. We're also running, I think, YCSB, right, Chris? We're running a couple, right? Yep. And all of these, we publish everything. There is actually a performance page in our docs as well. Our docs, I have to say, they're phenomenal. So if you really want to go find some things, we do publish a lot of our benchmarks. You know, we've run TPCC up to 100K warehouses. We show results for the 10K warehouses. We're doing these tests nightly. And so at Cochris, like I said, we have aired on the side of serializable. So we want the data to be always correct, but we're also fighting ultimately. Our biggest competition is the speed of light. So for us, understanding these things is actually really, really important. And understanding how close we are from a performance point of view. And if small little commits that we made during the day have any sort of effect on that, we want to know. And so, yeah, here we go. Thanks for bringing this up, Chris. So we do publish all this stuff. We're usually pretty good about showing updates on this, but I know we're going to go through a release cycle here and we'll have lots of updates on this stuff too. So I hope that answers your question, David. But again, you know, the funny thing is, Greg, Chris, we go right back to our docs every time, right? Yeah, the source of all information, and they're just fantastic. And yeah, hats off to the education and docs team for putting together some great materials. Yeah, it's funny. Chris and I were both at a company called Hortonworks, which some of you probably know, and just the whole Hadoop community. And God, I remember trying to figure out things and go into the docs. And I would just give up, but I'd go find a committer somewhere and ask him. It was just, oh boy. So, all right, let's do the second half of this, Chris. I'm going to walk through basically how transactions work. This is the global transaction side of the, you know, of the promise here of the presentation. So let me get this going here. So how do we implement serializable isolation in a distributed environment, right? Because serializable isolation in a single monolithic database is kind of easy. Well, it's nothing's really easy, because you're in one instance, and you could kind of take control of that. Well, in a distributed environment, what we're doing is we're writing data in triplicate within a cluster of nodes, right? So imagine, you know, 15 nodes, I have to write the data in triplicate. So I have to write it three times. So what we're doing is we're going after something called a quorum write. Two of three of the replicas have to commit before I can actually say, hey, this is cool. I've committed this transaction. So this is how we do basically distributed transactions. Now, in cockroaches, there's this concept of raft. And, you know, raft is a distributed consensus protocol, replication protocol. If you haven't studied raft, check it out. It's pretty cool stuff. There's some great stuff online that talks through it. You know, if, you know, without raft, there's another protocol called Paxos. You know, these distributed systems are all built off of these algorithms, these technologies. And what raft basically does is it allows us to basically, we take every table and we break it down to these. It used to be 64 megabit chunks. We're actually doing 256 megabit chunks. And that's a range. Think about it as like a shard. But each table will have thousands of shards based on its size, right? And each of these shards is written three times across, you know, multiple. It could be 15, it could be three, it could be hundreds of nodes. And so these replicas of the data is gonna be spread out across all the database. So you can imagine the data for a table is written, oh man, it's all over the place. But the replicas, the three copies are all tied together by something called raft. And that's a raft group, if you will, right? And so raft is a really, really key concept to what we're doing. Now, in the concept of this raft group, so like all records between, you know, Casano and Conway are part of one raft group, if you could imagine, if they're alphabetically in the table, right? You know, those sets of records, you know, they're written across various different nodes. Well, one of the replicas is actually depicted as the raft leader. In our case, we call that a leaseholder. And the leaseholder is kind of like the master for that replica set. So for the three copies, it will actually understand exactly what's going on and maintain control of basically what the state of all the replicas should be. And it handles basically transactions to that set of records within the table. And basically we're controlling the transactional consistency through the leaseholder. And so the leaseholder does play a really key role here. Now, something that Chris was talking about was, you know, we need to place leaseholders in the right place because you know what, if I'm accessing, you know, my data, my last name is Walker, right? And I'm in New York. Well, maybe I want to put the leaseholder for all the records that are in New York, in New York or in Virginia where I have low latency. And that's what kind of what Chris was talking about earlier because, you know, I'm going to make sure I have that. So I can actually, maybe one of the copies is out on the West Coast, but two of the copies is here. That way I get quorum writes really fast. Yet I could still survive failure of a region and whatnot, right? Because I still have a copy of the data somewhere and that replicates that's going to actually sort that out. I actually, you would want like three data centers to do that. If you want to fail a region, typically five, whatever it is you want to do. But there's lots of different ways in which we actually talk through these things. And cockroaches will actually show you scale and resilience in the next quick demo. So transactions in cockroach are our asset and we are implementing serializable isolation within this context. So, and raft is where it really allows us. So let's just think through an insert into a table. We want to insert these two records into the dogs table. Now imagine the dogs table is a bunch of listing of a bunch of dogs and it's alphabetized by dog name. And basically the ranges or the shards, if you will, are basically every 64 megabits or every 250 megabits of that table is a different chart. And so this table in particular has about, I don't know, it's about 15 records. So it breaks down into three shards, three ranges. And they are distributed across multiple different nodes in a cockroach cluster. So you'll see I have copies of the various different data in different nodes within this cluster, right? So this last range of pine top to Z, well, it's written three times. It's a node one, node three, and node four where the blue one is in blah, blah, blah, right? And so, and if you look at this, the leaseholders which are depicted by this darker border, they're on different nodes as well, right? So they can actually place where those are and Chris showed that. When I wanna write data, now an insert actually breaks down to a lot of different transactions in the database. Underneath in Postgres itself or in cockroach, it's gonna break it down into, this is gonna actually end up being four different transactions. So the first command is begin transaction. I'm gonna do that. And I'm gonna write sunny. What I'm gonna do in the leaseholder, it's gonna find the leaseholder. Now this gateway, this thing that it's going through, this green thing. Now this could be any node in cockroach. This could be node one, it could be node four, it could be node three, it doesn't matter. Cause every node actually understands where to find the leaseholder for this range. It's just commuting to that leaseholder. So I'm saying it's green, it's a gateway. So my app can connect to any one of these things as an endpoint and it's gonna find this. So this green basically mirrors the green of any one of these nodes. I could have asked any of these nodes to do this transaction. So this is arbitrary, but actually a pretty important note because every node in cockroach is a consistent gateway to the rest of the whole database. So when I first write sunny, what it's gonna do is like, okay, let's create a, we're gonna actually create a pending transaction here in the leaseholder. And I'm gonna temporarily write that record. Sunny is the new record. I'm gonna temporarily write it. I'm gonna reach out to the replicas and say, hey, can you just stage this thing as well until we commit the whole record? And so they basically get written to that range with kind of a temporary mode, right? Now, as soon as one of those things writes back, it's like, hey, I'm good. I temporarily wrote this. I'm like, great, okay, cool. I can now start the second transaction or the second part of this write, right? Because I'm writing two values. Now what I'm gonna do is I'm gonna go out to the other leaseholder and this transaction is still pending. I'm gonna do the same thing. As soon as I get an acknowledgement back, I can actually send an acknowledgement back to the gateway. And now that first leaseholder can actually change this pending transaction to commit. Now what happens is the database now says, okay, everything's cool, communicate and flush out to all the different replicas and commit all that data and now send an acknowledgement back to the requesting application. So what we're using, we're using something called a quorum write. So two of three of the replicas have to actually commit. Have to say, I'm good before it can actually commit that transaction. So two of three. So it's always a negative amount of replicas. You can have five. So it would have to be three of five, right? It could be seven of nine, right? And so five of nine, right? Yeah, five of nine, excuse me. But that's how we're actually doing this. And because we have quorum and because this is all coordinated through the leaseholder, the leaseholder, if one of the replicas was out of whack, it would say, hey man, I'm the leaseholder, you're out of whack, you need to basically have the same state as me and make sure that that data is up to date. And so that's how we're actually doing distributed transactions in cockroaches. One of the things I kind of glossed over is like I can add notes to this, it's going to distribute it. I can actually survive the failure of a note as well. We want to show you that as well. So Chris, we have what? I guess we have 15 more minutes. But at the top, do you want to show the last demo then as well? Sure, sure. I think we have plenty of time to do this. Yeah, just bring this up. Okay, how's my screen looking? It's looking good, buddy. Good. All right, cool. So I have a quick self-driving demo. So I'm going to do what we call a resilience demo. And this is going to bring up a six node cluster real quick. And I'm going to show you kind of the same thing I was showing you before. I'm going to show a workload that's running. I'm going to show a little bit about the replication factors of all these ranges that Jim was mentioning. And I'm going to show you how you could be resilient with cockroach. And a lot of the resilience really depends on how much you want to stretch out that replication factor. It's not the only way to do resilience, but it's one of the major ways to think about it. So as this database comes up, you can see I have six nodes here. They're all listed down below. I have a couple of ranges that are already being set up. You can see the database is still getting populated. That's because I have a workload running. And the workload is YCSB, which we talked about before. It's a Yahoo benchmark. It's been in the industry in the Node SQL LAN for quite a while now. Maybe 15, maybe 20 years. And you can certainly use YCSB. We have it built in into the database itself. If you want something called Cockroach Workload, Cockroach Workload, if you want to run it, or if you want to grab the YCSB job jar file and test it out that way, you can do that with Cockroach as well. In any regard, we have this workload running. We have about 50 queries per second happening. I think this is doing workload B. So it's more read heavy than write heavy. But what we could show is that because we have the YCSB set up, we can show you how we can be resilient with the database the way it is. So right now we have a single table. We have a user table within the YCSB database. We can see that those ranges are actually on three different nodes. We can see they're on N2 and 5 and node 6. That's because we have three extra applications set up. And what we could do is we can actually take a node out of service. So I'm going to click here, and I'm actually going to remove one of the nodes right from the database. Just take it right off line. So this can be, this is simulating either a planned outage or even an unplanned outage. What I want to show you is what happens with the database when a node goes missing from the cluster. And right here you'll see that the UI tells you that, hey, we have a node that's suspect. It's not here. What's nice about cockroaches is that it's very patient. It doesn't overly react and start sounding off all kinds of alarms. But it is let you know, hey, something's not right. The database, and I need to get healthy. The way to signify that is that we can see that some of the ranges are now under replicated. So we only have two replicas out there instead of three. So after a given period of time, by default, that period of time is five minutes, we'll take that suspect node and move them to a dead status. And once we do that, we'll start up replicating these ranges to make sure that the cluster is healthy. So you can tune that to be five minutes. You can tune it down to, I think, a minute and a half for as high as 10 minutes. A lot of times you're doing that because you're doing maintenance. I want to be able to take a node down, upgrade it, or repave that node, put a new OS down, install patches, and bring it back into the database. So we allow you to do that in cockroach rather seamlessly. And because we have this kind of clustering effect and being able to replicate data across multiple nodes, it allows us to do things like rolling upgrades really simply, even for major version upgrades. So let's bring that node back online. And you'll see the database, again, is very patient. It will say, ah, this node's back. Let's add it back to the cluster. Let's have that leaseholder or that RAF leader look at the other replicas to make sure that they're caught up and we get everything in sync. So you can see that the node was added. Right now the RAF leaders are checking out that the replicas are in sync. And then once they are, good. We have nothing under-replicated and the database is healthy. Now what I showed there was just a single node going down. If I have multiple simultaneous nodes go down, I could have a problem. I could have a loss of quorum. And I might want to avoid that. So one way I can avoid that is by increasing the replication factor. So I'm going to do this as an online transaction. Kind of what I showed you before in the first demo with Cockroach demo, I'm going to increase the number of replicas to five. And what you're going to see is that, again, in the admin UI, we'll signify that some of the ranges are under-replicated and we'll start up replicating them. And once we do, we'll now have a replication of five. And if I go and look at how the data in the user table is distributed, you can see that I'm on all the nodes except node one. Okay? And this time, I forgot to show this before. When we killed one of those nodes, you can see that there was no disruption in service, right? We can see all the SQL transactions have continued on even when we lost the node. But this time, let's remove two nodes. So I'm going to stop two nodes and kind of the same thing. You're going to see the resilience here in that the database can continue to transact, right? We'll also go back to our node status. You'll see that two nodes will be in suspect. And some of the, you know, identify that some ranges are under-replicated, but the database continues to transact, right? You know, there's not a, you know, there's not a loss of quorum. There's not a loss of writes happening or a loss of reads happening. You know, there might be a little bit of a small blip because we're actually taking two nodes out of service. So you think about that with six nodes, you're taking, you know, a third of the capacity away from the database when that happens. But you can see we can continue to transact like there's, and you know, to the end user, nothing happened. They didn't even know there was an outage or something happening in the background, all right? So that just gives you kind of the beauty of the architecture and what we can do with resilience. And like Jim mentioned before, there's a lot of things we can architect for, you know, for architecting against region failures or data center failures or multiple node failures. There's a lot of nifty things that we can do at Cockroach to make sure that we're always resilient. That's right. And Chris, all this stuff, basically you're running this, you're just using Cockroach workload, right? So we have workloads for like, I think somebody was asking about, you know, running benchmarks against isolation levels, you know, we show you what we run, but this is all packaged into the binary itself. It's all via, it's Cockroach workload, not Cockroach demo, right? Yep. So yeah, Cockroach workload will allow you to run a series of different workloads. Mm-hmm. So you can see that they're in this. Yep. TPCC, YCSP are very popular. We have our own Mover one that we've created, KV, we use a lot for internal testing, and then there's some other ones here if you want to poke around. Yep, cool. Well, cool, Chris. Did there's anything else you wanted to demo? I mean, we are good. You know, we've covered, I think, everything that we wanted to cover. Are there any more questions out there where we'll be happy to answer them? I'm gonna just do a couple last things here. Oops, that's not what I wanted to do here. So just to wrap it up, and Chris, if you can just see if there's any QA because I've actually, I'm not on a single laptop here, so. Yep. You know, Cockroach database was kind of architected a little bit differently. You know, we like to think about isolation levels because for us, ultimately, it is about getting that serializable isolation and that guaranteed correctness of data. And we did that for a couple different reasons. I think where we feel that Cockroach fits is for the workloads that need this kind of thing, but we also kind of think that anybody who's using a database shouldn't even have to think about these things. And if there's trade-offs from a performance point of view, what if you're gonna basically build the database from the ground up and think about those things before you actually even implement the thing, right? Before you actually go out and code the thing. And so, you know, that's what we've done. And to get serializable isolation is also really important in the distributed environment too because just the nature of how these things work. And you know, the way that we've implemented these things, I think there's a lot of kind of good, representative code examples of how we actually do this for the database itself. You know, we're doing some things in here that, you know, like things like parallel commits where we're actually forward committing transactions and nodes, you know, across the planet before, you know, something's actually committed in the other side. So we can actually start to bend this, well, we aren't bending the speed of light, we're changing the photons. You know, we're dealing with performance issues that are really, truly interesting from a code point of view. If you're ever interested in basically researching, you know, how these kind of things work in distributed systems, you're building a database or not, there's certain kind of aspects of how these things are done that are kind of interesting. And, you know, our code is open source, it's available, you can investigate, go look through. So our team is incredibly helpful on these things too. You know, we would implore everybody here to, you know, go download and try Cockroach database. You know, we'll give you a free cluster on Cockroach Cloud if you just reach out to us. We'll give you a code for that if you wanted to try the database today. You know, Cockroach DB is available as an open source free download as well via Cockroach Core. You can use all the stuff, everything that Chris showed you today is actually, you can actually run through with our open source free product. And you can learn more. You know, we are all about teaching and hopefully this stuff is valuable to you all what we did here. You know, I mean, it's a little bit of a commercial but I hope it was valuable in terms of we've actually given you some useful information and stuff that's interesting for you as well. And you know, our docs and the education side of the university team does a great job. And so, you know, we have some tutorials out there. If you go to actually it's university.cockroachlabs.com you can actually take some coursework that'll teach you through all these things. So, you know, Will Cross does a great job and that team does a phenomenal work there. So with that, I don't know if there's any more questions, Chris, was there? No, nothing coming to the chat, pretty quiet today. Cool. Awesome. Well, I do hope that this was valuable to everybody. Chris, thank you so much for joining again and you know, I always like doing demos. It's always good to see product on these type of things. I hope everybody enjoyed and found something today useful and was able to pull something valuable but we thank you very, very sincerely for taking an hour out of your day to join us. I know these things are, you know, time is valuable as we all know and we hope this was valuable too. So, Chris, thanks again, buddy. Thank you, Jim. Thank you, everybody. Alrighty, and thanks, everyone. Yeah, I think you did a pretty good job. So, you took the words out of my mouth. Thank you so much, Jim and Chris for spending part of your afternoon with us and regardless of where you're located, whether it's morning, afternoon or evening, I hope you have a great one and we will see you back here next week for more webinars. Cheers.