 I wanted to talk to you guys today about NoSQL and ACID and transactions and how these technologies play together. Is there a case for ACID and NoSQL? Hint, yes, there is. How do we get there? What are the challenges? And I'm also going to get into a little bit about what are the tools and things that have to change about how we think about and build NoSQL systems from the implementation perspective when we add and start thinking about having ACID as part of it. I'm telling you this out of experience from the past four years of building FoundationDB, which is a NoSQL database with ACID transactions. So to understand how I think about ACID, I want to get back to the motivations behind NoSQL. And I think of it simply as making it easy to build and deploy applications. You want some storage engine, some back end for your application that just makes it easy to build, easy to deploy, easy to scale. We know some of the properties. We all know these properties. Fault tolerance, have a diversity of data models to choose from, good price performance, all of that stuff. In the first generation, NoSQL systems really hit a lot of these check marks. But to me, ACID transactions are one of the key things that makes it easy to build and deploy an application, be able to reason about it and think about it and have it run well. And the first generation of NoSQL really didn't hit that. And I think one question to start with is what would happen if NoSQL had ACID? We just snapped our fingers, and every product across the hallway there had full ACID transactions. Would that make us all good for financial applications? Would it sacrifice availability in the cap theorem sense? Would there be some big performance hit associated with that? What does that look like? And I mean, I sort of think the answer to all three of these questions is, no, this really isn't what happened. These are sort of misconceptions about what would happen if you had ACID. And really, when NoSQL gets ACID, it opens up a whole different path and a different way of thinking about NoSQL. So let me make for you first the case for having ACID in NoSQL. The biggest thing about ACID to me is isolation. And the biggest thing about isolation is that bugs don't appear under concurrency. So this is a really key piece of technology that allows you to reason locally about your code rather than globally about it. Basically, when you have transactional isolation being enforced by the database layer, it means that as long as your transaction maintains the global invariance in the code that you want, then any number of clients running concurrent transactions will maintain that invariant globally. And the impact of each client is isolated. So this is sort of the really low level what happens with ACID. So what does that get us? Well, isolation gives us the ability to build strong abstractions. And I have an example of abstraction here where you can store a user in a database and you put in their name and their social. And you can also get their name and get their social. And wouldn't it be nice, as an abstraction for this interface, if we had this invariant, that if you take a name and get the social and then get the name, you get back the original name? That would be a wonderful invariant to have. So when you build this as your toy project in a weekend, you'll find that that invariant always works when there's a single client. That's because you always update the part of the data that's storing the name to social mapping and the part of the data that's storing the social, the name mapping sequentially. And so by the time the next command comes in, it'll work. But what you find is that without isolation, without ACID, when you try to run this with concurrent clients, this invariant doesn't hold. And ACID brings back the ability for this invariant to sort of survive, even with concurrent clients. So you can build these stronger abstractions. The cool thing about these abstractions is that if you build an abstraction like this on top of a scalable, fault tolerant transactional foundation, it gets to inherit those properties. Let me dig into that a little bit deeper. If you take away the scalability from this sentence, you still get to build a transaction that inherits fault tolerance. If you take away fault tolerance, you still get to build an abstraction that handles scalability. Those properties, you can add and subtract them one by one and you just get them or you don't get them. But if you take away transactions and ACID, it takes away your ability to build the abstraction itself in the whole thing evaporates. One of the cool things about building these abstractions when you have access to transactions and ACID is how easy it is. And so I wanted to give you some concrete examples of easy that we've run into from people building things on our ACID database. So the first example is that somebody built an ACID SQL database running on top of FoundationDB in a day. That's a pretty short amount of time. How did they do that? Well, it turns out that SQLite, the new version of it, SQLite 4, creates an abstraction between the query, parser, planner, and executor and the actual storage engine in SQLite. The abstraction that they chose is a transactional key value store abstraction. So because of that, we were able to actually shim in SQLite 4 to run on top of FoundationDB using the power of transactions to build that abstraction and have it running in a single day. The other example I like to give is not a relational model but a piece of the relational model, a simple indexed table. And we actually had a high school intern come in, learn Python, learn our product, and build an indexed table in three days, which would be sort of like a hard homework project, except for that because he built it on a scalable transactional fault tolerant infrastructure, it got to inherit those properties. So he didn't just build sort of a toy indexed table. He built a scalable fault tolerant transactional indexed table in three days. So there's the power. The last one you don't need to get into the code is I have a fractal spatial index in 200 lines of Python that, again, inherits the properties of scalability fault tolerance. So these are some examples of how easy it is to build these powerful things on top. With this ability to create abstractions, there's another thing that you naturally get pulled to do. And that's to remove and decouple features from the database itself. When you can build indexing as an abstraction on top of the database, build a geospatial index as an abstraction on top of the database, it allows you to remove things like efficient data structures, query languages, and indexes from even thinking of them as part of the database. And it lets you move them to user code. So what you can do with this is that a NoSQL database with ACID, it doesn't matter actually what kind of data model that NoSQL database has. If it has transactions, that gives you the power to provide polyglot data models and a variety of APIs on top of it with really strong guarantees. So keys and values, graphs, column oriented document, all of these things can be built as abstractions. And they can be built as abstractions without requiring separate physical databases, which is a huge, huge operational win. So this is the case for ACID. What I'm telling you is that if you have ACID in NoSQL, you can build all these amazing abstractions upon it, give it lots of different data models. That all sounds great, right? So why don't we have ACID in NoSQL databases? OK, two reasons. Number one, it's really hard to build this, and I'll get into that. And number two, history. And I think the history part of it's really interesting. So I wanted to talk to you about that. I want to take you back to 2008, when NoSQL doesn't really exist. This is Google Trends, and it says, nobody knows what a NoSQL is in 2008. Even the cap theorem is sort of down there. Databases in 2008 were largely sort of, we're talking about the sort of entrenched relational databases. In NoSQL, we have to remember, it was emerging to replace sharding and caching solutions that big web properties had built. They had already thrown out consistency long ago. So if you're replacing a 500 machine sharded MySQL cluster in your large web property, you gave up on consistency three years ago. And what you're actually trying to accomplish with your new NoSQL database is automating that partitioning. So things like Big Table Dynamo, Vodemol, Exandra, this is, I think, one of the reasons they didn't adopt ACID, because it wasn't a requirement of the systems they were replacing. Also in 2008, our understanding of the cap theorem was less than perfect. Eric Brewer was saying, pick two out of three. Warner Vogels, the CTO of Amazon, was saying that data consistency has to be tolerated in large-scale distributed systems. And there was wrong descriptions all over the web of the cap theorem in particular. People saying that availability in the cap theorem sense means that the system is online and up, which really isn't what it means at all. So the conclusions that you'd come to, if you were building these first generation of NoSQL databases in 2008, are that if you want to scale, it requires a distributed design. Distributed design fails. There's always a node that's going to be down, so you need some sort of high availability. And to get that availability, the cap theorem says we have to give up C, consistency. So if we want scalability, we have to give up consistency, which is a cornerstone of acid. So many things wrong in this. The C in the cap theorem is not the C in acid. Availability in the cap theorem is not high availability. And that is one of the big, I think, insights that I know a lot of you in this room understand, but people didn't in 2008. And remember, I'm taking you back to 2008. High availability, in the sense of fault tolerance, is not the same as cap availability. In fact, if we fast forward to 2013, it is not letting things say, why 2 out of 3 is misleading? It's not really picked 2 out of 3. It's when a P happens, do you do a C or an A? And he's saying that what cap prohibits is perfect availability. And perfect availability is that a database node, even if it's disconnected from the rest of the database, can continue to do rights to the database. So basically what it's saying is if two people can't communicate, then they can't agree. But it's saying nothing about the fault tolerance of the system. When one of Vogels is changing, or he's saying a little bit in 2013, he's saying, achieving strict consistency can come at a cost of latency or throughput, which I read as performance. So he's saying it's expensive, which sort of begs the question, well, how expensive is it? If you're telling me it can be done for a cost, what's the cost? So I'll answer that later. We also have this quote that I saw on somebody else's presentation already this way. And I love it, so I have it in mine as well. It's better to have application programmers deal with performance problems, again, performance being cited, due to the overuse of transactions, rather than have to code around the lack of transactions. This is what Google's saying. We invented NoSQL, we built Bigtable, and we're building a second gen NoSQL system that has transactions in our spanner product. So here's what I think is the plan for bringing Acid and NoSQL together. We want to maintain the scalability and fault tolerance of NoSQL systems. We want to leverage our modern understanding of CAP in 2013 and deliver a consistent system that has global acid transactions. That would be the ideal. This enables abstractions and lots of different data models to be built on top. And we also somehow need to do this while not sacrificing too much performance, because we know a lot of people tell us it's going to be slow so far. So I want to talk to you about a couple of the sort of really high level system architectural approaches that you can think about taking to solve this problem. The first one is what I call the both-on approach. And this is taking a traditional diagram. It's very simple. You take a traditional NoSQL database that has no transactions, and you bolt on some sort of transaction or locking mechanism on top of it. The upside of this is elegance. This sounds like a wonderful architecture if it works. The downsides, the number one I'm listing here is nerd trap. And the reason I'm saying that is I personally have spoken with several companies that have had their engineers working on a project like this for a few years to no avail. It sounds like it's a do-it-yourself project, because the architecture looks really simple, but it turns out it's really tough. And the other downside, I'll let Google speak to that, because they built a product this way called Megastore, and then they rebuilt another product called Spanner. And what they're saying the downside is performance. So let's talk about another approach. This is what I call the transactional building block approach. It's that each of these boxes is a transactional database already, and not a distributed one, just a traditional transactional database. And we're going to use those as components of a cluster and run some distributed consensus algorithm, some sort of distributed transactions to make it so that if a transaction needs to span nodes, there'll be some communication and it'll happen. So the upside to this approach is that local transactions, there's only one thing here that I'm going to say. The upside is that local transactions are fast. That is, transactions that only involve data elements that are on one machine in the cluster. The downside is that if the transactions involve data items that are on multiple machines in the cluster, they're slow and tricky. And you need timeouts as part of these distributed consensus algorithms in a network that has failures. So it ends up being a little bit of a mess. But upsides and downsides. The third approach that I'm going to cover is the approach that FoundationDB takes. And I call it the decomposition approach. It's to take the processing pipeline of a traditional ACID database and to actually go attack the individual stages of that to make each stage fault tolerant and distributed. So the stages are, number one, accepting client transactions. Number two, applying some form of concurrency control to enforce isolation. Number three, writing to transaction logs. All transactional databases do this. And number four, updating some sort of persistent data representation. So this is sort of broad brushstrokes here. I think the upside to this approach is the performance potential. By attacking each of these problems separately and applying the appropriate sort of techniques to each of the separate stages of processing data, I think you have the potential to build a higher performance system. The downside is that it's ugly. It's ugly. It's a complex architecture. And you need to solve these tough problems not just sort of once with one holistic algorithm, but you need to solve tough problems for each of these separate stages. Even if you can build a database with ACID, there's some challenges that are inherent to it. So number one, there's this disconnected operation challenge. And offline synchronization is a real application need. So what we're talking about here is I have a mobile phone and I'm updating my grocery list, which is on a database somewhere, no SQL database somewhere. And I go and do a tunnel. And I can't communicate with the server anymore. So one of the features of some no SQL databases is that I can keep updating my grocery list. And my phone will be working in a cap theorem available sense, that it's disconnected, but it will continue to pretend to write to the database. I say pretend to write to the database because it's not really writing to any central database. It's just storing up those mutations. Then when I come out of the tunnel on the other side, the idea is that it syncs back up. So that's a real application need. I think it sounds like a terrible idea to try to solve it in the database layer. I think that that's something that's application-specific logic. The second challenge with ACID is the split-brain challenge. And this is sort of the networking partition thing. This is something that intuitively a lot of engineers gravitate towards when they think about distributed databases is if there's a network partition, which means half the machines are over here and half the machines are over here, who picks up the torch, what side of that cluster is the one that actually continues working? Because they cannot both continue working and maintain consistency. So the solution to this is to use an algorithm like Paxos that something like Apache Zookeeper uses. And you have to thoughtfully choose your Paxos nodes to make sure that the majority of those Paxos nodes stay up and stay available to determine which half of the cluster can live on. So I guess what we get into in this is that there's some well-known ways to attack the split-brain challenge. But when you get down to it, it's all just a question of fault tolerance. And how many machines can fail and still have the system stay up? Big challenge in ACID is latency. And there's a few ways that this impacts the system. The most basic thing is that if you're going to guarantee every time you write data to the database that it's on the hard disk before you come back to the client and say, OK, I mean, that is going to be slow from a latency perspective. There's going to be network round trips involved. There's going to be rights to disk involved. There's going to be syncing the disk involved. Hopefully you have something like an SSD, so you're talking a couple of milliseconds rather than tens of milliseconds. The other thing is that enforcing causal consistency costs latency. So if you want to guarantee that you always see the results of all previous rights to the system, there has to be some ordering there that introduces a little bit of latency. So when you have a latency challenge, it doesn't necessarily mean there's an overhead challenge and a throughput challenge. You can bundle operations together into a single transaction to reduce overhead. And in theory, you only need these extra latency costs if you want the extra acid properties. So from some perspective, an ACID NoSQL database, you could have a client written for it that sends a transaction over to that database and then returns true and forgets about it. And in some sense, that client could have an extremely low latency, but without the ACID guarantees. There's a huge, huge correctness challenge. And of all of the ones that I've listed, this may be the biggest. And I have a thought experiment to set you up for why this is a challenge. It's called MaybeDB. This is my new NoSQL database that I don't think is a competitor to any of them out there. But MaybeDB is eventually consistent and might set the key to that value and gets a value that the key used to have at some point in the past. So MaybeDB works sort of like my memory. It sometimes puts the information out there, and I remember sometimes some of it. If you're building MaybeDB, you can make it fault tolerant, you can make it scale out to 100 nodes, you can do all these things, and you can really be able to strongly say that your implementation of MaybeDB is correct, because the contract isn't very strong, is it, about what correct means? A huge challenge when we come to try to build an ACID NoSQL database is that ACID is a global contract about the state of the data, these invariance isolation between clients. So it requires much, much more powerful tools to be able to actually test and say with certainty that we're building a correct system that adheres to the contracts laid out in its API. The third challenge, and I'm going to get into this, there's an implementation language challenge. Whenever you have a system that you want to have high throughput, but you know there's going to be some moderate latency in that system, you do a little bit of queuing theory, there's something called Little's Law, and what you figure out is there need to be a ton of things in flight in any such system. More so, in fact, than there are threads on a machine, easily. So you need some sort of language to build these systems in, maybe like Erlang, that allows some sort of single threaded concurrency model. This may be the problem and the correctness problem requires, from my experience, simulation to get right. There's nothing like simulation for actually ensuring correctness, and since this is a low level database that we're trying to build, you need raw access to the disk and really fast algorithms, and so when you think about how to build a product like this, how to build a no-SQL database with ACID, you're sort of at odds about what kind of language you might use. So the solution that we took to this was to build some new tools, and one of the tools that we're really proud of is a language that we built called Flow. Flow is a new programming language, and FoundationDB is written in Flow. Flow adds actor model concurrency to C++11, it adds a bunch of keywords to C++11, and it works by what somebody told me is called transcompilation, I didn't know the word for it. Flow code that we write is compiled by our compiler down into C++11 code, and then that's compiled with a traditional tool chain down to native code. And the question I always get asked about this is, wait, seriously, a new programming language, this seems like completely overkill, but it allows a couple of things that are incredibly powerful. Number one, it allows us to do this easier coding with these things called actors, which I'll show you in just a second. Number two, it allows us to actually test the system because the flow language allows a physical system to be simulated by shimming in simulations of every physical aspect of the system, of the network, of the disk, of the clocks, everything else, and of course it's very performant because it compiles down to native. So here's a function, a flow function, and we won't get into the code, but this is from our product, and all of those things circled in red are new keywords that are only part of the flow language. And the flow compiler takes this little class here, which is taking a bunch of incoming requests and batching them up to reduce overhead, very simple problem, and it compiles them to this big mess of code, which is four or five times longer, and if you zoom in inside this, you see that those individual bits and pieces of what was the original code, written as functions that become callbacks that are called when the events happen, the asynchronous events happen and actually trigger them. So I'm risking delving too deep into developer land here. So let me show you about the performance. Joe Armstrong, who's the author of Programming Erlang, recommends testing the performance of languages with this kind of capability by this ring benchmark, where you put a thousand things in a ring and then send the message around there a thousand times and see how long it takes. And you can see here, and okay, lots of disclaimers on this slide. These are numbers from three years ago, four years ago, when we were building this thing, and we just found them on the internet. So this is not intended to be scientific, but you'll see those orders of magnitude difference. When you try to do this with threads, it can take seconds or minutes, right? Those are all the blue languages here. What's the internet? What's that? The N and M is how many things there are in the ring, and then how many times around the ring you're gonna go. So there's a million steps in total. The green languages here are languages that directly attack this problem of concurrency, but aren't native, they don't compile to native, they're interpreted languages. And Flow does it about an order of magnitude faster than those languages. So it's an awesome primitive that we have for building high performance programs with the asynchrony that's required that you have to have in an ACID database. The last thing is I alluded to Flow enables testability of our system. And testability is one of the big things that changes when you add ACID to NoSQL. Because the contracts get so much stronger, the testing rigor has to go up an order of magnitude. And so we're actually able to do deterministic simulations of our entire database in a single thread, and we run about 50,000 of them every night. And I can tell you that I think it would be impossible to build this database that we've built without this simulation framework to be able to do correctness. I also wanted to show you a little picture of a thing we have called Quicksand, which is a cluster that we run that is a real world analog of this simulated fault tolerance testing. These are actually network controlled power switches, and they spend 24, seven, actually just turning these poor computers on and off and turning these poor networks, which is on and off in seeing what happens. And if this looks interesting to you, you can stop by our booth because we have a version of this that you can play with yourself. We have a power strip where you can turn machines on and off and we have network switches and you can plug cables in and out and we can explain to you what the database is doing as you do that. So FoundationDB is no sequel with ACID. It's what we've been working on. It's database software that is scalable, fault tolerant and fully transactional. And a lot of the word ACID is being used a lot at this conference. I see people talking about it. I see other talks about it. To us transactions aren't that useful unless they're global over the entire key space. So many no sequel systems when they talk about transactions, what they're talking about is the ability to lock and isolate the manipulations on a single data element, be it a single document or a single row or whatever, but not being able to do that and make those guarantees across data elements. So the problem that we're attacking at FoundationDB and the challenges that I'm talking about apply to systems that are trying to solve this problem globally so that you can do a transaction that actually crosses arbitrary documents, that crosses arbitrary keys and stuff like that. At its heart, because of the ability to build those abstractions, FoundationDB itself doesn't need a very rich API. It doesn't need a lot of features. It doesn't need a lot of indexing. It doesn't need to be document oriented. It just needs to be something that works. Some sort of least common denominator because a lot of that functionality can be built as abstraction. So we have an ordered key value API and then on top of it, these things that we call layers. For the layers, we're building an open source ecosystem of these abstractions. We're building all the common no sequel data models that encode and pack their data internally down into keys and values. For example, we have a graph database that implements the blueprints graph database spec and the outside world that looks like a graph database, nodes and edges and indexes and queries. Down inside the FoundationDB world, it looks like keys and values. We have a zookeeper like coordination layer. We have a distributed task layer that are one of our interns built, a bunch of really cool stuff. But one of the most exciting things and one of the reasons that the name of this conference is gonna get complicated over the next couple of years is that we have a sequel layer that's coming. And just like what I said about somebody built a sequel database in a day, that was sort of a hack project. It wasn't ready for prime time. But sequel databases internally just use these storage engines that look a lot like no sequel. They are key value stores. The big difference is they have transactions. So once you give transactions to a no sequel database, you ask yourself this question, can it be a storage engine for a sequel database? And the answer is absolutely. So we acquired a couple months ago a Boston-based database company called the Keven Technologies. And the chief technologist of that company and product is right over there. And when we met these guys, they were like, well, we build this sequel database and we're like, yeah, we're a no sequel company. And they said, it's all built on an ordered transactional key value store. And we noticed that's what you have. And so we very quickly turned from a partnership conversation to an acquisition conversation and always now leading the team to bring this sequel layer to actually run on top of our acid no sequel database. So a really cool thing. And it's not just a normal sequel database either. It has this unique concept of table groups that actually allow it to really efficiently pack related tables down into a range of keys and values that can be read with a single range read operation. And those ranges tend to represent objects in the actual logical model of your application. So that sounds awesome and complicated, so talk to Ory about it. But the architecture is interesting. It runs just as a stateless local server. So if you can imagine an app server up here running an actual application, and the app server wants to talk to a sequel database. Well, the way that it works is there's actually a sequel database, this sequel layer, running as a local process on the same physical computer. But it's not actually storing any data on the hard drive of that computer. It's just acting as a translator, like a dumb translator. And in one side come sequel requests from the local client and out back come what that translates to in terms of reads and writes to the actual keys and values down in the lower level. So really cool piece of technology. So let's get to performance, because right up front people told us it was going to be slower, it was going to be slower, it was going to be slow. Here's some performance results from our software that I think show otherwise. One of the ones that I'm most proud of is that we do a test where we read data that's fully cacheable, so it fits in memory. But FoundationDB is able to do random cached reads at about half the speed of a dedicated memcache D cluster. So this may be the only time you hear somebody bragging about being half as fast as something. But memcache D for those of you that don't know is a extremely high performance distributed cache with virtually zero guarantees. It's, I don't think you could even charitably call it a database, it's really a cache designed for extremely fast read performance. So the fact that we can deliver a distributed database that has a consistent caching layer built into it that's sort of even on the same order of magnitude sort of throws into question whether you need a separate caching tier on your database at all. Couple of other performance results. We can do random reads of 4K ranges that saturate the network bandwidth on the machines. And we have a 24 machine cluster that we use processing 100% cross node, these global transactions, not the easier local ones, but the hardest kind at a bit under a million operations per second, which represents the limits of the SSDs on those computers. But here's the big performance result. Remember the Warner Vogels said that achieving consistency can come at a cost in performance. Well, because of this pipelined architecture of FoundationDB where we do the transaction processing in sort of a certain set of stages and then we actually do the distributed storage in another set of stages, we're actually in a sort of interesting unique position to be able to answer the question of what it costs. Because we can see how much of the cluster's resources are dedicated to those transaction processing and transaction isolation roles and what percentage of the cluster is dedicated to the storage roles. And the answer is surprising for a lot of people. It turns out that only about 10% of the resources of the cluster are actually used to enforce the transactional guarantees. So if you're telling me that I can get consistency at a cost in performance, I wanna know what the cost is. If you tell me that the cost is 10%, to me that's slam dunk. The other thing that's interesting and a lot of engineers have this intuitive bottleneck that actually there's some place deep inside their day of you're telling me all this stuff but there's something about transaction isolation that is a really challenging problem. It's this problem of checking every transaction against every other transaction. So it sort of seems like this N squared kind of a problem. The interesting thing about this is within this big cluster that I'm talking about, the actual isolation checking is accomplished with less than a single CPU cores worth of resources. So it's really interesting result. A lot of people sort of feel like transactions should add 2x, 5x or whatever to the cost of a system but we think we can deliver it at a very low cost. So this sort of brings me to the conclusion and what I think is a really compelling vision for NoSQL. The next generation of NoSQL should maintain the scalability and fault tolerance that sort of the first generation was founded on, right? It should maintain the high performance because without performance this is all just an academic exercise, right? This is one of the reasons why we're coming to NoSQL while we're talking about scalability in 100 node clusters, right? We have to maintain that. But I think the vision is that we can maintain that while adding acid transactions in all of the benefits that you get from that. One of the biggest being the ability to build abstractions and have true data model flexibility. Thank you very much. I think I have a few minutes for questions and happy to take some. Everybody has their own nesting engine. Yeah. Yeah, I think that's right. Once you have the ability to do transactions it gives you the ability to act as a relatively general purpose engine. And whatever indexing or packing the data, if whether it's document oriented or column oriented you can pack that intelligently into keys and values in a way that there's not a big abstraction penalty. So that's right. Yeah. Mm-hmm. Yeah, I think there's a couple of different things there that are good points. One is that in these distributed systems keeping everything working at the same time is a big challenge, right? And that I have worked a lot on that problem and it is definitely a big problem. You get queues that fill up in one part of the system and empty out in another part of the system and then they tend to slosh over. You get these positive feedback effects. So everybody is sort of working towards this platonic idea where the hard disk, the memory, the CPU on every computer in the cluster is all working at its maximum all the time. And I think that that nobody's probably there yet. But I don't think that that's a particular challenge for ACID, for ACID databases in particular. I think that it's a challenge in all distributed systems. Yeah. Yeah. Yeah, that's an excellent question. I mean, I don't think I can answer that right now. But you should talk to Ori, who has worked on that problem for the past four years. I would think of it as the table gets pregnant. But that's basically right. I mean, the basic idea is that rows in the table get mapped down into keys and values, right? And if you have a primary key that you want the data in that table sorted by, it would tend to get mapped down into keys in the database that were also sort of the same way. So that scanning operations worked, you know, worked correctly, right? Right, once you have transactions, you have all these modeling options that open up to you, which sort of turns your NoSQL system into more of a storage engine, right? Rather than just being defined by the native API the sort of is it a key value store, is it a document? Yeah, that makes it easier. It makes it easier, actually. Then you can do it with anything, right? That's right, because if everything's packed into one value, many NoSQL systems have the ability to do, you know, compare and set on a single value or whatever. So it actually, the real challenge is when you break it out over a bunch of keys and values and then you want to do some update that spans a bunch of them and have them all be atomic. Any other questions? Peer to peer, call it, no single point of failure, but for a better explanation, go ask the guys at the booth. Yeah, thank you, everybody. Yeah.