 Welcome back to yet another OpenShift Commons briefing. Today we're going to kick off a series we're doing with folks who have built operators that run on OpenShift, but we're not necessarily going to talk about the operators specifically. We might talk about them a little bit and get you guys to tell us what you did and what you learned building them. But today we're going to do something kind of special, actually. The title is Database Transactions in Kubernetes and OpenShift, but the guests are from Cockroach Labs. We have with us Spencer Kimball, the CEO and co-founder, and Jim Walker, who is the VP of Product Marketing at Cockroach. And we're going to have an interesting conversation with them because Spencer comes out of Google, has a lot of the genesis of Cockroach DB, comes out of Google. We're going to talk a little bit about that. We're going to talk about distributed SQL. We're going to talk about working with OpenShift to make all that work for you and kind of the impact of cloud 5G serverless and all those kind of things. So it's going to be an interesting conversation. Introduce yourselves, Jim and Spencer, and we'll have live Q&A at the end. So ask in the chat wherever you are. That is great. And thank you, Diane. Thank you for having us sincerely. This is a little bit of a homecoming for me as well because I was a member of the CoreOS team as well. And so CoreOS being part of Red Hat and part of OpenShift and this whole shift to Kubernetes and everything else. I'll introduce Spencer. I'll let him introduce himself, but I got to say the very first time I ever met Spencer, actually, I was working for CoreOS. So here we were driving down the Kubernetes route. And Spencer and Alex Poli, who was with CoreOS at the time, wanted to actually demonstrate a database that would survive the failure of pods and the failure of systems. And this wonderful demo, I think. But when I found this database, I was like, wow, that's what's needed. That is right for this modern architecture. And so my name is Jim Walker. I am VP of product marketing here at Cockroach Labs. So Spencer, do you want to introduce yourself as the fire engine gone by yet? Yes, I believe it has. That's the risk of conducting this interview on top of Roof in Manhattan. So I will try to mute if it happens and hopefully it doesn't. Yeah, so yeah, I've been in the industry now wrestling with databases for about 30 years. It started even when I was at university. I wasn't very interested in databases, but the reality is to try to build anything. You run headfirst into them and I did a dot com startup and then ended up at Google in 2002 and as if for 10 years, and it was really a front row seat on the evolution of Google's interest in databases. And one of the first things I worked on was around the AdWords product and they were using MySQL and they ended up having to shard it, which just means that you have some customers on one MySQL instance, but it gets too big. So then you employ another MySQL instance and then another and then another as you grow. By the time I left that project, it was about 32 shards, and eventually went to about 1000 shards before they replaced it. Then Google built Bigtable just super interesting. That's like about, hey, we don't want SQL or we do want SQL. It's not really the point. We want scale. We want elastic scale. And then they built something called Megasource started to introduce transactions a couple years later and then finally spanner. So I actually worked on some of that distributed infrastructure, a product called Colossus, which is Exascale distributed file storage is kind of the successor to GFS, but also worked on applications in between. And so I got this. It's a really nice cycle, right? You work on applications. You decide, well, there's some things that are really missing from what the infrastructure is providing. Then you work on infrastructure and you get to provide those things and then you go back to applications and realize that's great. I've got that now, but I need these other things. And so the cycle continues when we left Google and I say we because I've got two co-founders and cockroach and all of us worked at Google for those 10 years together. We wanted to build the way Google built and not everything that Google had internally was available and open source at that point in time. In fact, open source really felt about 10 years behind. I think the world's caught up to Google quite dramatically. And that's part of what cockroaches, right? It started as sort of a manifesto. We weren't trying to build a database when we left Google. We were trying to build a private photo sharing application. But it became pretty obvious that what Google had evolved internally was exactly what we needed for our startup. And then we got acquired by Square and we realized, wow, Square needs a lot of the same things that we needed at Viewfinder and that Google needed for the 10 years we were at Google. And then you look around, people I knew at Dropbox and Pinterest and at Yelp and everyone needed a database like this. So that's where we started the open source project and the rest is sort of history. It's been five and a half years now at Cockroach. And we have some of the world's biggest companies and some of the world's smallest companies as our customers. It's very exciting, but the right way to think about Cockroach is it's a relational SQL-based database. In fact, it looks like Postgres. But instead of being monolithic in the tradition of systems like SQL Server, DB2, and Oracle where you're really scaling up an instance that sits in one location, Cockroach is fully distributed from the sort of most basic parts to the overall system. It's fully distributed. And what that really does value is what we're going to talk about fundamentally in this interview. But it's been a long journey and I never was interested in databases when I was at university, but it's what I basically spent all my time on since. So Spencer, thank you for that. That was like basically the first half of our conversation in a nutshell. I love it buddy. So there are two things I actually want to actually ask you about too. In university, you also had a pretty popular, I guess it was an open source project or are you in one of our founders actually started something I think a lot of people are familiar with. What was that? You never mentioned I got to pull it out from you. Yeah, we've had a long association. So Peter and I have been working together now for 27 years. We met in 1993 at Berkeley. And we were, you know, coming out of in 93. It was everyone had Windows or Mac Mac OS on these earlier versions of my class. We had Windows 3.1. I can't remember exactly what it was at that time. And we got to Berkeley and it was, wow, look at these Unix systems. And, you know, by today's standards, they weren't so good but you know we're using these stun Solaris workstation things and we're so impressed at the Unix ecosystem and all of the free software and what really was missing though is photo editing, which we were used to using Photoshop and the like. And there was X V and there's something called X paint, and they were, you know, pretty pretty sad compared to what Photoshop was at that point in time. And so we kind of decided one night after a couple beers I guess that you know hey this would be a wonderful thing to build as sort of free software. And so that's where the idea of the GIMP came from and GIMP was, you know, titled as the new image manipulation program but it was really inspired by Pulp Fiction, which came out that year. We just thought that character was funny and you know bring out the GIMP when you've got a difficult photo editing task. So we worked on that for basically our entire undergraduate career is often skipping classes and not doing a great job on some of our projects in lieu of working on this like full time. And then when we left college, the beauty of it is that we stopped working on it in 97. And it still exists and it's going strong. Right. So it's been since we stopped working has been 23 years that's it's hard to imagine but 23 years that open source community has inherited it and has promulgated it and improved it continuously. That's the beauty of open source. And that is the beauty of open source. It's exactly right. It's years and years later and here you are still doing open source as well. But the question that you and I always get no matter where we go, what's with the name, right. So Cockroach DB I think you kind of described it but like a little bit about the name I think people always ask us that right so how did you come up with the name. Well, as you can tell, I think Peter came up with the name and Peter and I have a similar sense of humor. And so we like things that are a little bit darker maybe is the right way to say, just kind of tickles our funny bone. You know cockroach, the name was one that I chose and it was really because when we were desperately figuring out what kind of database we were going to use in our viewfinder architecture which was the private photo sharing startup. We kind of hit on the idea after examining things like HPACE and Cassandra and MongoDB and, you know, all the normal things like MySQL and Postgres. We said, you know, we want something that's like Google Spanner, but it should be open source. And, you know, we were going to think about how it's going to work. It's like these individual sort of pods or nodes that are all greedily optimizing but making sure that the data they have is replicated elsewhere sort of greedily managing that. They give them more space, they colonize it. Right. And so this idea, you know, just kind of the evocative concept was a bunch of cockroaches. I live in New York City. And I don't like cockroaches. I hate them. I mean, I hate them probably more than most people. So it's a little ironic I chose the name, but I never really thought I'd be explaining it to big audiences. And here we are. Well, funny enough, I think, love it or hate it. People remember it. That's for sure. It's definitely got, you know, it definitely has that going for it. But let's talk about Spanner and Google a little bit and kind of the gen, because that is really the genesis of CockroachDB was, I mean, the Google, you know, the Google Cloud Spanner, that white paper that they published, right. And in some of those sort of things, you were kind of front lines in terms of like they had big table. You mentioned this in your intro. They had big table. They had a couple other solutions. Why did Google need to build yet another database on top of Borg and everything else at the time? I mean, I know you weren't on that team, but you were kind of adjacent to that, right. I mean, I think you were working, were you working at within like the classes team at that time, like the file system, correct? That's right. That's not quite a policy, but think of it as a blob storage system really. I mean, they built some file system stuff on top of it. Yeah, you know, Google has had a long and storied history of databases and I'm sure it's continued in the eight years I've been gone. You know, Google is not afraid to do pretty involved R&D. And part of the reason was that they had to. Like if they wanted to realize their ambitions and if you think about Google in 2002, scale was a, you know, unbelievably pressing concern for them. They had the entire world basically starting to do daily, maybe hourly, maybe by the minute, you know, actions involved their systems. And so they needed very large scale databases. And at the beginning it was really read only databases or write once read many times. The systems that were, you know, really just read only indexes and then they started moving to something that kind of gradually layer in additional changes called the RT server. And then the indexing pipeline, you know, was kind of its own database and index and so forth. It was very custom purpose. But then they started branching off into other things like AdWords, for example, right? So they needed something to manage all the creatives and, you know, the places where people come in and that's where they started using MySQL. And but at the same time they started building other things like started storing data for every, you know, if you had a cookie, and you started searching on Google, they wanted to associate data with that so that they could build better search. Right. So they knew what you're, what you were probably looking for based on past things. And so there was, you know, a huge need for, you know, massive scale data. What's interesting is that original MySQL AdWords project, because it started to scale way beyond what instance of MySQL could handle. And, you know, I got put on that project to help because it started to fall over when it got to more than four shards. You know, just too many connections coming in from too many application servers and it kept crashing the databases. They had this ads war room, because every morning we'd be in there. And Jeff Huber was running it. He said quite a career. And, you know, we'd look at the problems that happened since the last day. And we'd be like, figure out how to solve them, how to do the short term fixes is what the medium long term fixes were. And that went on for months, because it was such a big problem. And we got it somewhat stable. We built lots of interesting things. As I mentioned before, that went to 1000 shards by 2000. Wow. So it went, you know, it was hard enough getting past eight and, you know, getting the 32 they went to 1000. And if you actually add up all the engineering involved in that over that 10 year period, it probably is enough to build a couple databases. That's, that's interesting, but this is a pattern that repeats itself at Facebook, for example, they have hundreds of thousands of shards of my secret. And not a thousand hundreds of thousands. And Facebook has spent engineering millennia, knitting those together into a big meta meta database, very custom super snowflake technology only good for Facebook. But, you know, but cockroaches building probably wouldn't ever be able to apply to Facebook if their scale is so enormous and they have, they have a careful balance of consistency and eventual consistency. It's a very complex system, but what's interesting is Google's experience with AdWords in my sequel and sharding led them to create a moratorium on that kind of an architecture. I said, no, we're not going to do this again. All right, once is enough. All right, so if you want to build a large scale system, you've got to use something else. And around 2004 is when big table came out and big table is really geared towards that problem I was talking about of associating data with cookies and search. They're just massive scale and big table wasn't like, you know, we don't like sequel, we don't want to use sequel. Well, sequel is really complicated. And we've got a bigger problem right now to scale. That's big enough to not just scale but elastic scale. And so big table was a direct consequences of the challenges and the opportunities that Google was facing. Like how do you, how do you, they didn't buy mainframes or, you know, sun micro system, you know, huge multi processor systems. They were doing commodity hardware, they're kind of like the proto cloud, right. And Google's cloud and it felt like a public cloud internally, right. So Google like 10 years ahead of the, the rest of the ecosystem back then. And so how can you knit a big database together out of all of these commodity servers. And that's what big table was and it was groundbreaking and that paper was, you know, definitely one with red papers at the time, and spawn the entire no sequel movement. Yes, I was going to say that they started that that movement as well right yeah so go on sorry. They started a lot of them. And it was internally it was very exciting to an amazing piece of technology. And, you know, everyone was pretty jazzed about it. And I remember it was an ask early on of like from, I think the infrastructure team. Okay, AdWords, you know, you guys having trouble with your my sequel sharding just use big table this will handle your scale issues and there was a huge pushback because they said, Okay, guys, we have like, I don't know what it was at the time probably like a couple hundred tables. We need transactions we're talking about people's financial information there's essentially a ledger happening. Like, this is not. There's a huge impedance mismatch between us coming from a relational database system, even if we have a very awkward sharding mechanism to going to big table which is just not set up. And it's like big tables great for certain kinds of tasks, but AdWords required something that, you know, fundamentally sequel relational database have been evolving for 40 years by 40 years right so you can't just replace it in one fell swoop. And so, interestingly, Google responded to that infrastructure teams responded back and they created something called Megastore. And Megastore came out two years later. This is very interesting because, you know, MongoDB didn't get transactions until like Google had transactions of limited form of them to be fair in 2006, because the big table kind of fell flat for certain music. We need to have some kinds of consistency. The big table or Megastore introduced that and also introduced the idea of consensus based replication so big step forward to Megastore this two years after big table. But quickly after that people realize you know what, there are more things we need to do. We want general purpose transactions, not these limited ones that Megastore brought, and we want to sort of, in some ways, you can think of spanners is sort of closing the circle. Like the circle got opened. Right. It's like we need to deal with massive scale and then it's like, okay, started to close. We need to get some kind of transaction. Now we need really general purpose transactions. And then eventually they've created something called F1, which is we actually want to bring sequel back because we want to replace AdWords. We want to replace that my sequel sharded mess with a new next generation system. But in order to do that we can't just make them rewrite everything for some weird new API. It's actually should be sequel. It should look enough like sequel that we can make that transition. So they came full circle. It's very interesting. And so, you know, when we left Google, we realized, well, you know, we want a relational database too. And we also want these capabilities and we know it needs to be open source. So we had the enviable opportunity to fast follow Google, which is a lot easier than doing it all for the first time. Well, I think we're all benefiting from that period of time at Google. Right. I mean, today, Spencer, you and I, we get in conversations with customers and it's like, they're almost like your conversation right now is some of the stuff that I'm hearing in some large organizations. Well, we've scaled this database and it's falling over and we're having problems with it. We can only go so far. And then they're trying to figure it out using legacy tech and knowing that, well, there is actually prior art. There is actually somebody who has solved this. And, and I, you know, to me, this is the, you know, board, turning to Kubernetes, that whole movement, just the wherewithal and the vision at Google to actually understand what the future of compute was going to be. You know, resulting, what are we, 15, 17 years later or 20 years later after that journey, you know, really, really kind of took off, right. 15 years at least, yeah, from like the major technological advances. And it's one thing that's really interesting is just that you've mentioned board and Kubernetes. I mean, for a while when Kubernetes was new and didn't really do stateful workloads. There was like a non-trivial contingent of people that would opine frequently that, well, Kubernetes is really just for stateless services. Stateful isn't going to work. That's not what it's for. Well, you looked at Borg back in 2005 and six and believe me, Borg ran everything. It ran Bigtable, it ran Megastore, it ran Spanner, it ran Colossus, it ran D. It ran stateful services and it ran them with ease. And, and that's of course what orchestration needs to do. It's not just the stateless stuff, right. It's everything. So there's a, there's a, there's a really good match between orchestration technologies like Kubernetes and database systems like cockroaches. And the two are made for each other. So it's good to see Kubernetes evolving and maturing and being able to handle that much wider, you know, spectrum of use cases and infrastructure. Yeah, that's right. And it's, it's a brand in Phillips a long time ago once quoted this out. It's like, it's giffy. It's Google, Google infrastructure for everybody, I think is what he wanted to have. And I think that's where we're headed. But you know, but one of the things that had come out of that whole Congress, I remember this like four years ago was the whole concept of operators and actually simplifying the life of how all this stuff works. How much did it take to actually make all this work back then? I mean, just sort of, just a pure resources point of view Spencer, like if I'm going to have, okay, I'm going to go from four shards to a thousand shards at Google or at Facebook, we have teams and teams of people that have to manage this. Right. Is that, I mean, we're involved. I mean, it was massive. People times time, right? Like that's why I was used like things like engineering years and engineering centuries and engineering millennium. And I think the reality is that Google, I don't know about Google AdWords and that started my SQL system. If it, if you could linearly extrapolate how much time we spent on it. I mean, it was engineering centuries. You know, I'm sure of it. I don't know if the system matured to a point where they didn't have to do much work on it at some point in that interim. I really, I didn't follow it that closely. But I am aware that Facebook has spent engineering millennia on their database systems and that's, that's not something to take very lightly, right? That's a, that's an eye opening staff. And if you think about it, obviously Oracle had engineering millennia put into it. So did DB2 and, you know, it's like databases take seven years to mature and stone breakers fame this sort of rule of thumb. And, and then they have a lifetime which can go decades beyond that. And that's right. They're such a vast surface area to relational databases that follow the sort of modern SQL standards. And it's a kind of an unfilable pit, right? You just kind of get poured. It just keeps going. And, you know, the thing gets better and better, but you just look down and like, wow, we can't even see the cement. We keep pouring it. You know, it's a, it's a huge task, but it's, it's absurdly valuable, right? Every service and application in the world is backed by some kind of a database. And many of them are backed by relational databases. I mean, you know, we have a customer that's one of the big, big financial services companies. Just in one of their arms, they have 6500, just like one of their business units, 6500 externally facing applications. And that's just like mind blown. Yeah. Yeah. And the amount of databases that you hit in your day thus far between waking up till this moment, and it's only a couple hours, it's just insanity, right? Like, I joke about that in the legacy databases all the time. But, you know, I love the concept of engineering millennium, and it's like, well, there's DevOps millennium, and there's SRE millennium as well that's putting me these systems. And I think people really understand, like, what it actually takes to get going. Like, one of the reasons I love that I'm on here with what the OpenShift team is doing at Red Hat and simplifying the operations side and that side of the world. But we talked a little bit about how, you know, CockroachDB is aligned very well with Kubernetes, Spencer. And just being a distributed database on top of a distributed orchestration platform, right, like that. It is absolutely what originally attracted me to this company. If you look at the future of the database, to me, I believe it's got to be distributed SQL. But when you combine that with, you know, I think what is, is it Stonebreaker that says it takes seven years for a database to fully gestate and mature, right? Yeah, so, like, building a database takes seven years, right? Building a distributed system adds a wholly other layer on top of that, right? Like, that's complexity on complexity. You know, I guess going back to the beginning of what we've done at Cockroach and architecting from the ground up. What were the biggest challenges to actually get this to work, right? Like, I mean, I think we had a really good sample in the Spanner white paper. But, you know, technically there's some pretty hefty challenges that we faced early and we still face today, right? I mean, we've been building for five and a half, almost six years now. So what were some of the biggest challenges at the very beginning? Well, the distributed transaction model is, you know, required a lot of work, also the consensus replication. I mean, people, it's not unusual for undergraduates to implement raft, which is a variant of access is what we use. You know, one of my co-founders Ben Darnell likes to quit that it took him a week to implement raft and, you know, probably another several years to make it actually work in production properly. So that's just it's kind of shocking to hear that, but it's it's incredible where just how much of the devil's in the details in building these things. And the transaction model is another fascinating thing that we worked on, because Cockroach isn't just distributed in terms of, hey, you can shovel in more commodity hardware within a data center. Cockroach is meant to be distributed across data centers, for example, within a region like different availability zones on the US East Coast. That would be, you know, how you really want to do the geo replications, you have low latency but consensus replication, but it's also built to be replicated across continents, like vast geographic distances. So how do you build a transaction model that gives you serializable isolation and minimizes all of the round tripping. And that's, you know, SQL is a very chatty protocol. You open up a transaction, you read stuff and then you write stuff. You know, all of the every time you do a read, you're actually hitting the database. That certainly you never ever want that chattiness to go, for example, from Australia over to Virginia. That would that would absolutely destroy your your transaction time. So how do you actually build, how do you plan for that? What is the topology look like of the cluster and how does the transaction model work. And we have, you know, anyone that's really interested in the details of this, there's a lot of interesting posts on our blog that describe sort of the state of the art in terms of how we're managing that. But, you know, those two things transactions and replication, by far, are the stickiest points and they continue to be honestly that there's there's a lot of stuff on the drawing board for how we're going to really make global transactions, something that everybody can use trivia. That's right. And I think it's, I always, I always point people to our docs as well, Spencer, because I think there's just like, if you really want to get into it, what Jesse team does on that side is this it's tremendous right like it's there's such really, really good stuff there. So, but coming back to distribute the transactions and you also mentioned serializable. You know, and I think that's a, that's a, that's a big task to take on right because, you know, our ultimate competition here is the speed of light right like, I mean, if we're really going to do this at global scale right like I always think about that is like, you know, one of our engineering systems up, you know, here's speed of light that's our that's our end goal that's what we're trying to get after right. That's a tough way to get really like that Jim, I really like that like the next time someone, some investor asked me like you know tell me about your competitors and say well you know our real only real competitors. Well really. Well, I use it all the time because I mean, look at building a distributed transaction model from the ground up is not simple. I think there's a couple companies that are trying to do this as well. It's not a simple task it took you know how long at Google I go through the beginning of everything at Google how long it took to get to spanner Spencer because it is engineering millennia it is a lot of expertise because these are not simple things to do. You know building Oracle from the ground up from way back and what an incredible database I mean what I mean it is incredible technology I still think like that Photoshop I think are to the most incredibly complex. I always describe Oracle is like the absolute evolutionary peak of the monolithic architecture database. It's amazing. A fantastic dummy system. No question I mean it's running most of the world's high value use cases, at least the plurality of them. It's definitely extraordinarily successful company as well. There's, there's, there's some things that, you know, people don't love about their their vendor relationship with Oracle but you can't argue with their success. It's great product. Yeah, I mean our society is, you know, a lot of the advances that we have are due to that because this this this transactional model and being able to actually implement serializable like our banking every day like our money is basically run through systems like that and so doing serializable isolation in a distributed transaction model presents some fairly difficult like complications right Spencer I mean like there's a couple things that we did I think that are interesting here a like the way that we implemented it but also geo partitioning is also kind of part of that whole conversation as well. Can you just kind of describe just a little bit about I mean I guess you know we're only a half hour and a little like a little deeper into kind of how that works with raft and everything and like how the geo partitioning works. I guess, I don't know at the 5000 foot level. Yeah, I'll give it a shot. I mean this is there's a lot of technical detail and you know yeah high level. There's a lot of pieces that are moving. This is this is how you become a marketer see because like engineers will laugh at you like what was that you just glossed over like 80 things. I'll try to I'll try to put on my marketing hat more than my engineering hat. You know one thing that's just really important there's a there's a dichotomy in the database that I think people sometimes conflate the two things that there's geo replication and then there's, or just replication and generally doesn't be geographic but and then on top of that is the transaction model so when you write a single key, or even a group of keys that are kind of close together. It will be written atomically as part of the rap or Paxos doesn't matter replication protocol. The problem is that when you have a truly distributed system that's quite large you've got lots of different, you know, for lack of better term shards we call them ranges but they're essentially chunks of the key space that are replicated between say any three nodes you have but you might have 100 nodes and so you've got lots of different replicated it shards or the key space. In order to actually write to multiple parts of that key space that live on different nodes in the larger distributed system. That's where a distributed transaction comes. So at the low level you've got replication through something like Paxos or rap. And at the top level that's where you have a distributed transaction protocol and those two things have to work very well together. So, you mentioned serializable as probably most of the people on that are listening to this are aware from their experience with databases in the sequel ANSI standard there's a there's a number of different isolation levels. These isolation levels are really difficult to understand because they're not based on something that developers are thinking about what they're really based on is what database developers have been able to create in terms of trade off between how much how much latency there might be in a transaction versus you know how much isolation that you're getting and so you can have less isolation and you know really let the database blaze through in other words transactions that might be overlapping or intersecting in some way. They don't really care about each other. Some people call that like the question is the word but you know it starts with an F but you know F it mode transactions, let's say, you know that's that's like the read uncommitted. And then it goes through these different levels, all the way up to serializable. And when people talk about acid transactions that I isolation that means serializable let's just be honest right like all those other modes. They're about trading off isolation in order to get more efficiency in the database. So, you know what we decided to do like one of our sort of overall mission for cockroach lab just to make data easy. We came from square before and there's this make commerce easy kind of got a little easy there but it's actually a great mission because you're only ever going to ask them to approach the data is not easy ready very very much not easy. So, by having that mission we always know what our North stars can we make the database simpler for people and I talked about how to make it so that people can truly do global transactions easily that's that's kind of one of our North stars but you know in the early days of cockroach. It was like how can we make it so that when you're using distributed transactions, you're not trying to understand the distributed transaction model so you don't screw things up and still make it efficient. We said what we said is, there's going to be one mode of isolation and copper serializable and we started with two actually it's natural isolation as well, we ended up with just one serializable. We want that because that's the actual I an acid transaction. And so we said serializable only but we need to make that fast. So fast that you don't miss the other modes. And that's a work in progress still, but we made a huge amount of progress in the first three years that really allowed us to only release with serializable and to make our customers happy with that. And it's great because as a developer, trying to understand what it means if you do repeatable read versus read committed versus snapshot versus serializable is very difficult prospect in fact people never get it right. And there was a paper about this that came out of Stanford it's called acid rain, and it analyzes available open source e commerce applications which actually happened to run more than 50% of the e commerce on the web. And we analyzed it with this very intelligent tracing mechanism to see whether there are what are called anomalies due to weaker isolation levels and the transactions that the e commerce systems were doing. I mean, some of these e commerce systems weren't even transaction some of the time so it's like, there's some pretty shoddy programming out there. But even when they were trying to do the right things they often didn't. In ways that you could take these e commerce systems and for example you could put multiple things in your checkout card, just by like sending a concurrent request, because this concurrent request would come in and the database wouldn't have the right isolation level, and we'll let you do things that should be illegal. If you had the right isolation level, and that allowed people to check out multiple items and pay once, or to use coupon code codes multiple times. And so the reality is that this is just like a small sampling of e commerce things that happened to be very heavily used. But I mentioned before, there's, there's probably hundreds of thousands, maybe millions of database backed applications out there where people just use the default isolation level which is usually read committed. Right and are basically these these applications out there are just filled with whole I mean they're leaky buckets. So any, you know, and, you know, just to give you a practical example of the risks here. There was a Bitcoin exchange. I don't want to say the wrong ones. I can't exactly call what it was, but it's in the paper that was completely emptied out, and it was a it was a currency based acid rain type of tap. And so this isn't, this is actually a huge risk and there's really money in it. So, you know, this is something that we wanted to remove that difficulty of cognition from the developer. It's like, no, I use cars. I always get exactly the right isolation level I don't have to play games with it trying to make my thing faster. So that's, that's a, that's a, that's kind of a, I think it's a state of mind that we enter like the product sort of ideation. And, you know, what are we going to set our standards how are we going to set them. And it's just very helpfully aided by what our true north is it's just really to make data easy. That's right. And don't make compromises because of we want to actually have the serializable isolation. Right. We want the data always be correct because honestly, Spencer, I, I don't know too many developers who actually think about isolation levels in their database. They basically just been up a database and turn it on and they don't even think about that configuration. You know, they want, they want it to work. They expect it's going to work. I mean, basically work really well. Right. The problem is that people have set these isolation levels is they don't want people to do a load test and, you know, if they defaulted serializable which would be the same thing to do and let people off out of that. Then, you know, what would happen is people like this database sucks. It's really slow. Right. And so it's really just, I know it's a little cynical, but in the end what you end up with is a lot of people that don't understand isolation levels use the default isolation level and are leaving a big gap in the security of their system. And that's, you know, that's, I think a fundamental problem and it's something, you know, we wanted to let users, you know, off the hook for that extra knowledge. You know, we do want, we want developers to do what they do, which is like, hey, I've got to solve this problem. I want to build this application. I expect the database to work and it should work well. That's right. That is exactly what they should think. They shouldn't have to understand isolation levels, but other databases either force you to understand them if you're going to be a very responsible programmer, or they let you do the wrong thing. That's right. Yep. Yeah, and it's, you know, one of the reasons I'm proud to work here is we've actually dialed the knob all the way up on the speed of light and serializable isolation. And it's forcing some really incredible software engineering out of the team because when you have those two things pegged, you're actually forced to do some really interesting takes on how to solve these problems. You know, like parallel commits, which we came out with last year, which allows us to basically kind of forward commit transactions. There's stuff on our website about this that I think people can check out. But I think it's true. We just had a paper in Sigmund, which actually is a wonderfully concise description in 10 pages of a lot of people are talking about. So people that are interested to read that. And if you really want the gory details, the blog posts from the engineers that worked on these things are really explanatory. That's right. And if not anything just to explore kind of the inner workings of Cochran's database, which I think is really interesting. But as people go and build distributed systems, there is some really good examples of how you actually think about these things from a software engineering point of view as well. I think that's one of the reasons that, you know, being part of an open source company contributing back to a community is we do share all these ideas. And I think some really kind of novel ways of thinking through these things is it's that's what this banner paper did for us, right? It's the only right that we also publish these sort of things back. But coming back to that, and so, you know, people can use NoSQL to get scale across the planet, right? But they can't get serializable isolation, right? And that's one of those at AdWords, right? Way back in the day, Spencer. That's why they couldn't use Bigtable, right? And so, you know, how are organizations using CockroachDB today? We have, what, a couple hundred customers, we have some really huge, massive implementations of this, right? Some great brand names. Some really kind of, you know, high-powered use cases. You know, when people ask you how are people using us today, what do you typically respond with? Well, I think there's, you know, the easiest answer is that Cockroach is a system of record. It's a cloud-native relational database, and it's for any application that, you know, needs a system of record, which is most of them. And it's particularly, I think, an appropriate choice for workloads that are very high-value. And I mean, there are certain workloads where you can absolutely use something like Cassandra. It's not necessarily a bad choice or MongoDB. I mean, these are good systems in their own right. But, you know, relational database, high-value use cases, the right kind of consistency, that's what Cockroach is really geared towards. There are three primary capabilities or differentiators that Cockroach is bringing. And, you know, some of our customers need all three. Many of them need just two out of the three. And it's kind of somewhat rare that you just need one, although that does happen. And all three of these differentiators are fundamentally born from the distributed nature of its architecture. And that's why it's distributed, right? Like, you create a new architecture, not for the heck of it, but so that you get new capabilities. And there's three are scale, resilience, and global, or multi-region at least, right? And I'll just kind of go through them. Scale is pretty simple to understand. A relational database that's monolithic, you can scale it up. But it's a super linear cost curve with a ceiling that's actually not very high when you look at sort of global workloads. And so, you know, as an example, you know, if you were a food delivery service in COVID and we have a customer that moved, you know, they were using Amazon Aurora. And, you know, Amazon Aurora can scale reads out, you know, relatively well, but it can't scale rights. And so you get like the XXL size and, you know, if you kind of start hitting the limits there, your database starts to cavitate and then it falls over. And so they became, it was, you know, code red, and they had to get off Amazon Aurora and they tried cockroaches and cockroaches can scale the rights out arbitrarily. Right. So doing that, getting that elastic scale, which is a no sequel thing that no sequel brought to the world to the market. But doing it in the context of relational database, that's special. We're not the only ones that do it right now. Obviously, Spanner can do it. There's some other sort of smaller competitors. You know, but it's significant and it really matters. Like, especially like I think we're entering the era of what I call transactional big data. Right. If you think about, you know, the history of what databases have supported it sort of, you know, back in the 90s, it was enterprise scale. So like within a company, what are the kinds of customers that an average company used to have? But then the web kind of hit this web scale idea. And that was kind of, okay, you know, in a web scale, you can get 100 million customers that are banging on this every day. You can get a billion. Right. You can even go beyond that. But there's only so many human beings that are interacting with computers and interacting with the databases that are backing the services and applications. I think the next era that we're already entering is really when you have non-human entities with some agency that are connecting to services. As an example, IoT, right, or virtual agents that aren't even like physical pieces of hardware, but it's just virtual things that are checking, you know, the prices on something, but they're doing it at machine speeds. You're talking about not like sort of order 10 billion human beings banging on keyboards and mouses and things like that and interacting. You're talking about, you know, 100 billion, maybe a trillion, like you're talking about orders of magnitude that are going to increase. And so scale is going to become a much bigger problem even than it is today and it's already becoming one for many companies. Resilience, you know, kind of mentioned this about geo-replication. Fundamentally, you want to be able to think of a data center going away, not as a disaster recovery scenario, but as IT resilience. Like a data center going away should be, okay, we had three seconds of additional latency, no data loss, no postmortems for our application team. Everything's coming along. We'll get that data center back up. Things will re-replicate and then we'll be back in sort of the nominal green. If you look at Google and their high value use cases, they'll put it across five data centers. Even across like the different power grids in the United States, because they want to be able to say, okay, we've got five things, everything's nominal, everything's up and running. Lots of redundancy here. Things are working fine. Okay, we need to take a data center out for planned maintenance. We're going to replace something. All right, that's just not going to be available and we know about it. We're planning ahead. We're going to take that out. Now we're down to four. With consensus-based replication, as long as you have a majority, so if you have five, three out of five, if you've got three data centers for replication, then it's two out of those three. As long as you have the majority, then you're never going to lose data. You're going to have forward operation in your system. So if you've got five, you have this amazing property where you can take one out for a planned maintenance and then all hell breaks loose somewhere else and you lose another one. Now you've only got three of your replication sites. You still have complete business continuity. It's a new way of thinking. It introduces latency, so it does come at a cost. You can manage that. You can put the three replication sites very close to each other, like on the East Coast of the United States, or you can spread them across. If you spread them across, you get better sort of non-correlated failure domains, but you have more latency. You pay a price. And when I was talking about our transaction model, we're getting it down so that we have absolutely minimal latency on these even geographically distributed transactions. You can do a huge distributed transaction, modifying hundreds of different tables, and we can get that all down to one consensus latency. Just one, not even a two-step, which is normal for a distributed transaction. You're able to hide that second phase of the transaction from the end user. And you mentioned parallel access, which that is. And then the third thing, and I mentioned global or at least multi-region, this is the most exciting one. This is like 2020. This is going to be on everyone's roadmap. But the question is, how do you build a global data architecture? And what I mean by that is you have, even as a startup, the opportunity to have customers in Australia, customers in Brazil, how do you give those customers a first-class user experience? I'll tell you what you don't do. You don't put your entire application in one availability zone. You can't have Australian users hitting Virginia every time they want to do something on their app. It's going to be like a half-second lag at the minimum when they hit a button half a second. You can notice that very easily. In fact, the Department of Defense figured out is 100 milliseconds or less for command and control systems. That's the threshold for instantanity. So if you can get less than 100 milliseconds, you can make it feel completely real-time and interactive. And obviously this matters for AR and VR and interactive media and gaming and self-driving. But I think it also matters for any kind of application going forward. When you start to have an experience with your mobile app where you hit a button and instantaneously you're seeing interactions with other users. You can design new things that we haven't seen yet. It's going to push the state of the art forward and companies that are doing things the old-fashioned way. They're users in Brazil, they're users in Australia, they're users in Japan. Those users are going to feel like this is a kind of antique app. It's kind of like when the iPhone got bigger and you had those black bars on the side because the app maker hadn't updated their application. So this app maker is kind of cheesy like everyone else has kind of gotten with the program. These guys obviously they must not have enough engineers because they're not using my full screen and it's not as good an experience as it could be. That same thing is going to happen. And part of that's driven by 5G, you mentioned that earlier. But the other big part of it is not just latency. It's about data sovereignty. You've got things like GDPR. You've got states like California and New York threatening to create data privacy regulations that are different from other states. We could have vulcanization of data privacy laws in the United States. But Vietnam requires you to keep a copy of the data there. South Korea's got its laws. Brazil's got very stringent laws. China and Russia, they require that all your data is almost out there. Obviously, if you're Facebook, you can fight those but smaller companies can just get drummed out of business pretty quickly. There's a lot of liability right now in the world in terms of, hey, I've got a business and I'd like to embrace a global user base. But can I do that responsibly? Can I do that in a way that meets any kind of legal requirements that I have? But also, what's the user preference? Let's say you're a SaaS business, right? And you would like to have customers in Asia. You can bet that their preference is going to be that like if you're storing their employment records or their sales data or something, you can bet that they want that store in their legal jurisdiction. And if you're going to compete with a local, a more local operator that's Asia based, that is going to give them that guarantee. You're not going to have a very good story to tell, right? So like how do you build those holistically global data architectures? And Facebook can build them. Uber builds them. Netflix builds them. Google builds them, right? But can a startup? Obviously not, right? We put it in one availability zone because that's somewhat, you know, tractable. And what's interesting is, can the companies in the global 2000, kind of random financial services company that might have more than a billion dollars in revenue, can they build a global data architecture? And the answer is they can't because that's not their expertise. They don't hire those kinds of people that are kind of on that cutting edge. So what we fundamentally think of ourselves as doing is, okay, it's very complex building a truly global application of service, but the database is the hardest part of that. No question. We want to make that easy, right? We want to make that at least if it doesn't start easy. It's tractable and then we make it easy as we go. So those three scale resilience in global. Yeah, and that's exactly right, Spencer. And I think what we're looking at is the beginning of a massive huge transformation in a way that all of us think about applications, right? I mean, it's like, you know, you and I joke about 5G. It's like, you remember when we went from dial up to DSL and how fast the internet was at home. It's that same type of aha, I think is going to happen with people. But it's going to be on their phone and it's going to be everywhere. And how do you actually meet the needs of your consumers from the SLA, right? What do they expect out of you? I think the hundred miles rule is hugely important for organizations as they think about applications at that scale. But this resilient thing too. And I think, you know, I think this is why a lot of people are moving towards Kubernetes and doing this whole thing. It's like this whole concept of having this, you know, this orchestration platform that's doing that for you. But then surrounding it with the enterprise capabilities that they need to basically manage and monitor these things. And I think that's where, you know, our hosts, Red Hat, OpenShift are doing such a great job, you know, pushing down that path. How do we make it easy to install things? How do we make it easy to manage things? How do you sort out a rolling upgrade of software, right? You can do that now, right? Like in production, it's cool. And I think, you know, operators and this whole marketplace and everything that's going on, it's awesome, right? That's incredibly exciting because what all of this stuff represents from the public cloud to all of the, you know, new orchestration technologies, Kubernetes, OpenShift. These things are, you know, fundamentally allowing companies to not spend those engineering centuries of this dev off the centuries. Right. That is, that is, that allows everyone to do more with less, but it, you know, ultimately what you find is pretty interesting, like the cloud is a lot cheaper, right? And, you know, taking infrastructure as a service, the total cost of ownership is actually less expensive. But ironically, people end up spending more. But it's not necessarily a bad thing. It's kind of a growth mindset, right? You can do more with less. And so you have additional engineers, additional capability, which means you can build another service in that same amount of time, right? You can further improve the experience for your customers. And I think, you know, that's sort of the world that we're entering. It's extremely exciting. And fundamentally, it's, I think, once in a generation that you see this kind of a shift. And it's a paradigm shift. And it's a big paradigm shift. It took me a while to figure out Kubernetes and how the whole thing worked because it was, I was used to the old world, you know? It was a different way. And I think there is priority. There's, you know, there's organizations like Google and others that have kind of done this sort of thing. So Spencer, one of the questions that came in was about backup and restore. And distributed database is a different thing, right? Because you're dealing not just with like the backup or restore the database, but like this data sovereignty thing comes into it too, right? Yeah. It's incredibly, you know, textured actually. You know, it is one thing when you had sort of one bin log coming out of MySQL. Now we've got potentially, you know, 250 nodes and they might be in different countries. And they've got this, you know, geo partition. Some data might be replicated only within Europe. Some data only replicated. You can control all that, the row level and a table and cockroach. So, yeah, that vastly complexifies the burden on the backup and restore. And, you know, what we actually end up doing when we found to work best is to, you know, make the backup restore policies definable on the same level as we actually do the sort of domiciling policy. So where the data is stored, you also need the backup and restore to work that way. We didn't start that way, but it's moved in that direction because exactly as you say, Jim, you know, if you have data that's only supposed to live in Europe, you definitely want to back it up only in Europe too. You don't want that to stream over to the US for lots of reasons. And so, you know, but one, one, one I think really important point to bring up here is that we don't have to solve all these problems on our own. You realize that in this new ecosystem, all of your customers, not all of them, but most of them have access to all the other cloud infrastructure that's out there. So really what you really want to be able to allow them to do as an example, this is just one of many things you could do, but you can have for the European data that could be backed up to an S3 bucket that is sitting in Europe specifically. Right. Right. And so you kind of you're actually using what AWS is providing geographically spread out around the world in order to augment cockroaches capabilities and to complement them. And that's I think that's critical because if you try to solve this all, you know, in the database, the amount of work and the complexity. It's counterproductive right and it's ultimately not what customers want right that you kind of want to use the best of breed approach. So we do try to integrate with everything in the ecosystem. Yeah, and to be multi cloud and run everywhere and take advantage of whatever it is and each of the public cloud providers I think is a big piece so I'm Dan Mike. Was there any other question that you guys wanted to throw in here. There was one question about life cycle support. Yeah. Yeah, that's a good one. Yeah. The question is software like life cycle support. How does the software, how do you do a software upgrade without downtime. So, you know, that's a good question. It's actually not just the software upgrade. It's also things like backup from your story don't want to lock table schema changes. So we do online schema changes. It's very, you know, cockroaches, you know, very predicated on any kind of thing that you do to manage the database, whether it's an upgrade or a backup or, you know, change the underlying data model. All of those are do not require downtime and including scaling the database up scaling the database down moving from one cloud to another. All these things can be done live. The software upgrade. I won't go into too much detail but basically what what you're allowed to do is have multiple nodes and successive versions. You can have a mix of versions and like certain kinds of features don't mind that there's a different version because they're not really predicated on the version. But sometimes you have a new feature where it's not going to ever work if there's a mixed version situation. So what will happen is when you you have that mixed thing you can't use that new feature yet. It's available on the nodes that that are the newer version, but even on those nodes you're not actually able to access that feature. And then what happens is you kind of you get to all of the nodes being the newest version. And then at that point you'd make sure that the system works you can run it that way for a while but doesn't work. You can downgrade back to the previous version. And again it's really just about restarting each node sort of in a rolling restart fashion. But if things are working nicely and you don't need to downgrade, then you kind of flip a switch you say okay now we're going to make the minimum version for this cluster the new version, and that enables the new functionality. So that's just a little bit there's more details than that. And so when you do that other I mean there's there's some concerns within the application itself of how you build that to be backwards Lee compatible to features right and so I think one of the benefits of red hat open shift and Kubernetes is that you can do these rolling upgrades across compute right and like think about cockroach DB is just a simple application running in a pot and Kubernetes you all like that's how we get deployed on Kubernetes I mean we actually fit this environment very very well. But I think you know just for people who are casually listen to this there is things you have to think about inside the application itself to actually make that work and work very well. The orchestration platform or surely do this is beautiful rolling upgrade for you which is like what a huge massive awesome kind of capability so there was another question in here Spencer it's a little bit in the weeds. I don't know is there one that you're looking at in particular I was going to do the atomic clock I was looking at the benchmark Kubernetes storage solution. Yeah, this is from Wally Shari says, Have we benchmark Kubernetes storage solutions what kind of storage prefer to run on top of. I'm pretty sure that that some of our customers have been smart storage solutions. I don't know the actual details there. Unfortunately so I can't answer that part of the question but you know the storage we prefer to run on top of the ones that we have the most experience with and right now that's GCP's persistent SSD and AWS EBS and I think we've tried a lot of the different EBS ones. And we've had different experiences be just usually like a cost first performance trade off those those persistent SSDs and the EBS. The really beautiful thing about those storage solutions versus using raw SSD is that if you if you actually experienced some sort of downtime from one of your nodes let's say you just lose a node, you know that that would be attached normally to raw SSD. That usually would require cross data center bandwidth in order to re replicate that missing as raw SSD device when you use something like persistent SSD and GCP. You actually rely on GCP to do that within that data center that saves you any kind of cross data center bandwidth so it's very nice that's how we run cockroach cloud which is our hosted service. When come when we have customers that sell posts. If they're running it in AWS or GCP they can do the same thing we're doing the corporate cloud, if they're running in a private data center. I actually think the right thing to use this is probably some sort of Kubernetes storage solution that's doing something similar, right that's actually doing a distributed file system that can handle intradata center and if a node is lost. So, you know, I just don't know exactly what the best solutions are there I know we've got a fairly large number of customers that are doing things in private data centers with a variety of solutions from raw SSD to distributed storage systems that can be run with Yeah, and I've seen people using stuff under Rook from there's a Rook operator for that I mean, you know, for us it's really simply simple as a storage class, mounting a volume and then we use stateful sets actually manage everything within the pod itself so Dan, I think we're we run out of our time at the end of our and you seem to have covered off everything you took a lot of the questions that I had out of it I've done the data sovereignty stuff. I colleague of ours back in the day, Dave McCrory talked about data gravity and the need for all of our the GDP are the data sovereignty issues, latency issues, all of those things and I think as we move forward, what you're doing in the evolution of distributed systems that you've been giving us a great tour de force and history on. There's a lot of links to reference papers I'm trying to find them as you mentioned them just spurring my remembering of these points in time and when things changed and and I curious maybe to end on where you think in like 20, I don't know I don't even want to say 20 25 four years from now you know, I love that question, Diane. Yeah, I mean, I listened to the thing that is my guiding North Star for where the ecosystems going and I want cockroach to play very nicely in that ecosystem is, you know, as a developer myself, and I would like to write an application next kind of my earlier point about iterating between infrastructure applications. I want to be able to write something on my laptop. And then, when I'm done with it, it's working when I tested I want to say okay, I'm going to push this to the cloud, but I'm not going to have to deal with VMs, or, or any, any kind of infrastructure myself, whatever I've worked on my laptop is not just going to work on the cloud it's going to scale elastically. If I've got 100 million users. It's going to scale to 100 million users. If I've got users in Brazil all of a sudden, it's going to start storing their data in Brazil. And it's going to do that in a serverless fashion in terms of my, my, my relation to the whole project as a developer. And what that means is fundamentally, yes, I don't deal with VMs and, you know, the hands on infrastructure and monitoring and all that stuff, none of that, but also I only pay for what I use. Because, you know, I've done a number of startups back when I had to do exit exodus co location and build my own servers and put them there, all the way till you know more recently where I get VMs and easy to or in GCP. And, you know, the, I'm not very good at that stuff. But worse, what you end up doing when you launch your startups, you've got like 10 people poking at it, right. You don't have product market fit yet. And you're paying for these VMs to just sit there. You got a VM for your database, which is a big one right me like because you're you're planning on success. They start with these things and you just you just paying out hundreds of dollars a month and just immediately for, for, you know, what, honestly, you don't, you don't have, you should be paying cents per month, right. And so that is that's the goal like you should have consumption based billing on everything and it should scale globally to any size and you should never lose your data should just always be up, you don't have to deal with any kind of monitoring. So that's the future, it's like a truly serverless future where you can take your idea, you can develop it, and then it just becomes a global reality, like the way that Google builds right and the way that Uber has a as evolved their system so that when you go to Paris, your Uber account just works as it normally would right and it's like, you know that that is the kind of system that everyone should be building we get that 100 millisecond rule essentially for free. So, so that would be the giffy for all. Serverless, giffy for all right so. You heard it here first I think that's like if you read lots of people's marketing messaging that's already here. The reality is not and we still have to build it together. So it's been wonderful. Getting to getting to hear your story, hear your point of view and how the evolution of distributed databases came about and where you're at right now. We really look forward to having the cockroach operator working on OpenShift getting more use cases more feedback to you guys and want to hear more someday soon about cockroach cloud. And I love cockroaches in a way that you wouldn't expect is they never die and they will be here till the end of time. So, I think your naming in branding is brilliant. So, you know, keep keep going whatever the next product is I want to know what the name is you choose because you're dead on with the with your name your naming and branding as well as your technology. So thank you very much for joining us. Michael wait has got a number of other operators coming up folks coming in the next couple of weeks in stona and a few others so. Look forward to more conversations around this and learning more about what people are doing in building that runs on OpenShift so thanks for your time Spencer and your wonderful view and Jim for your your great q amp a and question and keeping the conversation rolling so thanks again guys. Thank you for having us really, really an honor. Thank you.