 Hello folks. I'm Siddhartha and I'm going to be talking about trade-offs between consistency and availability in distributed data systems. I'm not audible behind. So I was saying I'm going to be talking about trade-offs between consistency and availability in distributed data systems. That was supposed to be the title of my talk but that sounded too boring. So I thought I'll use the buzzwords cap theorem which is quite popular these days. I am going to talk about cap theorem though and I'm going to actually be describing cap theorem and then arguing that it is an incomplete tool or an insufficient tool for analyzing distributed data stores. I will then present an alternative framework for analyzing such data stores and then walk through some examples. But before we get started, quick show of hands. How many of you have heard of cap theorem? That's quite a few. How many of you have used cap theorem, for example, to analyze a data store or make a technical choice? Few are still. And how many of you have actually read the paper or the proof of cap theorem, the Gilbert and Lynch proof? See a couple of hands out there. So let me start by just introducing cap theorem. A bit of history. So cap theorem was proposed as a conjecture by Dr. Eric Brewer in a keynote back in 2000 and it was proven by a couple, Gilbert and Lynch, two years later. And this is the statement of the cap theorem. I'm not going to go through that. Instead I'll demonstrate what cap theorem is saying through a simple example. And there are many kinds of distributed data stores but I'm going to take one particular one where there's a master database to which all rights happen and there's a slave or a cash from which most of the reads happen. And to walk through an example, I'm going to take the example of say a product database where there is a mapping from product to quantity. So there's a master and the slave. The blue is the master and the green is the slave. So let's take this example. Let's say there's a product which has one, of which there's one quantity in the data store. And when you check out in an e-commerce website, for example, that checkout or just your act of buying the product decrements the quantity. And then that is replicated through some sort of replication mechanism to the slave. And now when a read happens, say a product page which is displaying the remaining quantity, the inventory, or it's saying in stock or out of stock, it's actually reading from the slave. Now the problem here is within this system, if this update, while this update is still happening after the decrement has happened, if the get happens, that is while the quantity has gone to zero, but it has not been replicated to the slave. It's a read happens. So what is happening is the read is inconsistent with the actual state of the system, right? The product page is going to say in stock while there is no stock layer. Now maybe you try to buy it and you get an error saying, hey, the product has gone out of stock, right? That's an inconsistent state. How do we solve this problem? Well, one simple solution to solve this problem is to make that replication from master to safe sling synchronous. That is even before you say, okay, to the decrement quantity call that is that call, even before that call returns, you make sure that the slave or the cash is also updated with the quantity. So you end up with a consistent state. Now if a get quantity call happens, either it will get one or it will get zero, never the other thing, right? Problem solved. But we have now introduced a new problem, right? And that problem is very simply what happens if the link between the master and the slave is down, right? There's a broken link. What do we do with that decrement quantity call? Right? We have two options. Option A is we leave, we let the decrement quantity call happen and leave the database in an inconsistent state or option B. We do not allow the decrement quantity call to happen. And in essence, this is the cap theorem, which says if there is a partition, you choose availability and forego consistency, or you choose consistency and forego availability. And by the way, this is kind of how the paper proves the cap theorem. There's no math involved at all in the proof. I would highly encourage people to read that proof. It's a very simple plain English proof. They don't use pretty pictures like this, but they do prove it in a very, very similar way, right? So cap is normally stated this way. Consistency, availability, partition, pick any two. There's something about this pick any two kind of thing, right? It just catches on very quickly. So if you pick consistency and partition tolerance, you sacrifice availability. That's called a CP system that is consistent and partition tolerant, but sacrificing availability. Similarly, you have an AP system, which is choosing availability and partition tolerance, but sacrificing consistency. Or finally you have a CA system, which is consistency and availability, sacrificing partition tolerance. And I'll go through each of these and argue about their usefulness, right? And I'll start with the CA system. So when we say CA, we mean partitions are, it is not partition tolerant, right? So but what is a partition? Let's take this example database of five nodes. The N1 to N5 are the nodes of the data store and the C1 to C5 are the clients. When we say there's a partition, it means there is some sort of a network severance or some sort of a problem in the network such that a few nodes within on one side of the partition are able to reach each other, but they are not able to talk to anyone on the other side of the partition. So N1, N2 are able to speak to each other, but not to N3, 4 or 5. Similarly, 3, 4 and 5 are able to speak to each other, but not to N1 and N2. This is what is defined as a partition. And this quote and this definition of partition is straight off the proof of the cap theorem paper. So but what does it mean to say, hey, I'll sacrifice partition tolerance? Usually it means partitions can't or won't happen. When you say you're going to sacrifice partition tolerance, you're making this assertion that partitions can't happen or they won't happen. So an example case where this is true indeed is if you have only one node, of course there's no chance of a partition happening. But people do argue, place the same argument even in other cases. And their argument generally goes on to say that, you know what? Network is reliable. Maybe they say, hey, I'm using a LAN, not a VAN. This is not cross data center. Within a data center, network should be reliable. But reality is a little different. There is unfortunately no quantitative proof, but there's a lot of anecdotal data which says that, hey, networks fail all the time. Even within a data center, how many times has network, how many times has Wi-Fi broken down just here, right? All of you are on Wi-Fi. Well, Wi-Fi is a lot more complicated to do than LAN, but LAN breaks all the time as well, right? There's network equipment failure, there's power failure, power distribution units fail. There's network congestion which can seem like partitions. Moreover, there's things like garbage collection in systems like Java. And GC pauses with a nice interaction with timeouts and timing can actually look like partitions. Remember, partitions are not defined as saying that there is actually a physical problem. It just says that nodes are not able to reach each other. Garbage collection also may make it seem the same way. Or misconfigurations, such as firewall misconfigurations, can also very simply make it seem like there's a partition between data nodes. Or one of the more common reasons, bugs in software, right? There's nothing that can be, that you can do to prevent them ever, right? You can't wish away partitions, no matter how you try. You can't really, there is no meaningful CA system when you're talking about distributed databases, right? Yeah, you can say there is a single node database which is CA, but that is not of much value, right? There's no value in saying that. Let's move on to consistency and CP systems, right? So consistency in the cap proof has been defined very, very precisely, right? It's defined as linearizable consistency. So linearizable consistency, what it means is it should seem like even if a data needs to be written to multiple places, it should seem to anyone who is observing the data that it has been written at a single instance of time, right? It should seem like there is only one node, even if it has multiple nodes which are coordinating to give the answer, it should seem like there's only one node. This is in fact one of the strongest forms of consistency there is. So this is a tree of various kinds of consistencies that are possible. Linearizable consistency comes somewhere there. Most databases offer something like, say for example, MySQL's default is repeatable read. You can choose read committed or series visible in MySQL. So RDBMS is offered that part. Many systems offer sequential consistency, but rarely do systems actually do linearizable consistency. So usually things like two phase commits and synchronous replication are talked about when speaking of achieving linearizable consistency. But even those rarely accomplish that. You need a lot more. They are good for accomplishing sequential consistency, but not linearizable consistency. That's a mouthful. In fact, you know what? Memory access in CPUs is also not linearizable. Think of multi-core CPUs, right? Each core has their own instance of the L1 caches and all that. And this is like a distributed system. They communicate by writing to a common memory. That memory access in CPUs is actually not linearizable. And it is possible to make it linearizable. There's a special instruction in many processes called memory barriers which can be used to make it linearizable. By derivation, variables in most programming languages are not linearizable. They do not offer linearizability. And in Java, for example, you have to use the volatile keyword to make it access to that linearizable. So what Java does when you use the volatile keyword is it inserts the same memory barrier instructions at that point whenever you write to that variable. I've already spoken of relational databases. None of them really offer this kind of a consistency. It is very, very hard to achieve linearizability, not just because it's so strong, but also because the cost of doing that, the high latency that would be required, the coordination that you require is very, very high. But the reality is that most systems do not offer it. And we are able to build great systems, good systems, workable systems without that. You don't really need CP. You don't need that sort of a consistency. We have all been building software using relational databases which do not offer that consistency. And we are able to build good software with it. Let me move on to availability. Availability also has been defined very precisely in the paper. It's actually read something like this, every request received by a non-failing node in the system must result in a response. Pay attention to that word non-failing node. That is, if you have a distributed database and any node in that is up, it should be able to give a response to a right request or a read request. We do want availability. I mean, who says I don't want availability? Everyone wants availability. High availability is a buzzword. It's extremely important for everyone. What do we mean by availability, though? It's usually expressed as an uptime of two nines, three nines, four nines, five nines, six nines, whatnot. But when you look at it, right, cap availability has little to do with high availability. Cap availability is talking about non-failing nodes. That is, all non-failing nodes must respond. High availability is about the whole system. We care much more about this, right, that the whole system, this database as a whole, is available for reads and writes. We do not care that, hey, every node is up or not. That's not important. Also, cap availability generally does not put any bonds on the response times from the database. High availability, on the other hand, is all about real world. In real world, it's useless to not get a response within some time out, right? Otherwise, you would actually call that system unavailable. As a simple example, if all nodes, if you have a database where every node is accepting all writes and just writes it out to DevNull, and every read, for every read request, it says, okay, I don't have this key or let's say a key value store. It just returns an empty response. That's actually cap available. It's an AP system, but it's not of any use. Or if you have a system where no node ever responds, that's also an AP system. It's cap available because there is no bonds on response times. You need high availability, high availability is great, but what you do not need is cap availability. It's a very, very specific form of availability that is not all that important. So looking at all of this, you might think that I'm saying, hey, consistency is not important. Availability is not important. I'm actually not saying that. Consistency is important. Availability is important. But you need to understand what sort of consistency you need and what sort of availability you need, right? There is a big trade off between consistency and availability. I'm not even saying that there is no trade off between consistency and availability. These trade offs are real. They are very, very real. But CP and AP, as Catherine says, right, they are just two points on a very large spectrum of consistency and availability, right? Availability on one end, consistency on the other hand, cap theorem is talking about those two points on this. Whereas the whole spectrum is useful, right? You may have systems leaning towards AP or leaning towards CP or somewhere anywhere in between. And various systems do lie at most of these places. Most systems that we deal with are neither AP nor CP in the truest sense of cap theorem. But they are still very, very useful because eventual consistency and high availability is good enough, is what we need for most systems. There are some specialized cases where you need CP, like consistency systems or leader election algorithms and so on. But most systems do not need that sort of a CP or that sort of an availability at all, right? And not only that, the trade offs are not just between consistency and availability. Cap just speaks of those two things. There are trade offs of consistency with, say, latencies. This trade off consistency and availability with operational simplicity of system, right? Cap is only talking about those two points. It's ignoring the entire line. It's ignoring all the other axis here, right? Cap, therefore I would argue, is not a sufficient tool for analyzing data systems. I'm not saying cap is wrong. Cap is absolutely right. It's a theorem. It has been proven, right? But it is not a useful framework for analyzing real world systems. So what is it? What is the alternative? Is there an alternative? I will present an alternative here. PAC ELC. This is an alternative to cap theorem that's been going around for, I think, about an year now. It's just recently been becoming popular. Particularly what it does is it talks about trade offs between consistency and availability when there is a partition and trade offs between latency and consistency when there is no partition, right? So please note that consistency is actually occurring twice here, both on the x-axis and on the y-axis, on the horizontal and the vertical axis. And in fact, if you think about it, even availability is occurring twice because unavailability is nothing but very, very high latency. And infinite latency is unavailability, right? Or latency above a timer. But generally just think of this as vertical axis is talking about when there is no partition. The green is about when there is no partition, how does the system behave? And the horizontal axis, the orange, is about when there is a partition, how does the system behave? Right? So this is pronounced pass elk, by the way. It's a mouthful, very difficult to pronounce, pass elk. So I don't know why they choose such a weird acronym, but... So it states that if there is a partition, how does the system trade off availability and consistency? That is, if there is a partition right now, at any moment, if there's a partition, how does the system trade off availability and consistency? And else, otherwise, if there is no partition, that is, the system is operating normally without a partition, how is it trading off latency and consistency? This is what the pass elk is stating. Let that seep in a bit. So the first part is about when there is a partition. The second part is about when the system is operating normally, right? Most of the time systems are operating normally. I mean, we would hope that partitions are rare, if not impossible, but they are rare. So this is actually an important axis that this pass elk is adding that cap theorem doesn't even think about or talk about, right? So pass elk really is talking about all of these treatances and what you get instead of CP and AP like in cap theorem, you get four quadrants. You get systems which are consistent when there is a partition and consistent otherwise as well. That is, they choose consistency all the time. Are you true? You have systems who sacrifice consistency all the time. Both when there is a partition, they choose availability? Or when there is even when there is no partition and they choose latency and sacrifice content consistency. And you have two more options actually have systems which sacrifice consistency when there is a partition but do not sacrifice consistency and choose consistency and sacrificed latency when there's no partition. And finally you have systems that sacrifice availability say when there is a partition, but sacrifice consistency otherwise, which might seem a very strange kind of a thing. And I'll come to some examples of all of this, and hopefully that will clarify this a bit more, right? So let's actually look at some data stores and apply this to those things, right? MySQL, I hope most people here are familiar with MySQL. Can I have a quick show of hands just to wake everyone up? All right, so most people use MySQL here, right? So what is MySQL? Can I, what is MySQL? Is it from CAP Theorem? Is it AP or CP? Let's see. MySQL is AP from a CAP Theorem point of view. It's always going to choose availability, right? There's very difficult to get consistency in MySQL. And when I say MySQL, I mean a MySQL master slave setup, not a master master or any of those more complex things. Let's look at the simplest one, master slave setup, right? But from the point of view of Passelq, MySQL is actually lying in both these quadrants, sorry. Lost the word there, all right. So the normal case of async replication, which is what most of us would be using MySQL, is actually a PAEL system, that is, it's async replication, right? The kind we saw earlier. It's choosing latency. The rights go only to the master, asynchronously they get replicated to the slave. It's choosing, so in a normal case, it is just choosing latency. It's not doing synchronous right, so it's choosing latency over consistency. But when there is a partition, you can still read from the slave, you can write to the master, it's actually choosing availability over consistency, right? It won't disallow rights when there is a partition, right? MySQL also has this different mode called semi-sync replication. Now in semi-sync replication, if you set up MySQL in a semi-sync replication mode, whenever you do a write to the master, it will write to at least one slave which is in semi-sync mode, semi-sync which has been configured as a semi-sync replica synchronously. But if it is not able to reach any slaves, it falls back to being asynchronous, right? So what it is doing is in a normal mode, it is choosing consistency because it is doing synchronous replication. But when there is a partition and the master is not able to reach the slave, it actually chooses availability and foregoes consistency at that point, right? That's MySQL for you, right? Let's look at ZooKeeper. Again, how many people familiar with ZooKeeper? Wow, great. So ZooKeeper is much more straightforward. It's a CP system. But most people think of ZooKeeper as a perfectly capped CP system. Well, it's not. It's by default when you do normal reads from ZooKeeper, reads can happen from any node. So in fact, the way ZooKeeper works is clients could be connected to any node and they read from the road. All writes go through one master, but reads can happen from any node. And when you do a normal read, it may not reflect the latest data that has been written, right? Because only quorum writes are required. You don't need to write to all nodes. So quorum writes are sufficient. When you do a read, it's not necessarily consistent with the latest copy, which means it's not necessarily linearizable. It is still sequential. So it still offers sequential consistency. That is the order in which updates are read is never going to change, but it does not necessarily offer linearizable consistency, right? But there is a mode in ZooKeeper called sync plus read. So basically you do a sync call and then a read call, and that ensures that you are able to get linearizable consistency. That is a capped CP in ZooKeeper. And of course, ZooKeeper completely writes in that quadrant, right? It has nothing to do with any other quadrant. And this is necessary for ZooKeeper because the kind of things that ZooKeeper is used for, distributed locking, leader election, and other consistency algorithms, it's all about consistency, right? You need this sort of a consistency in ZooKeeper, right? Let's move on to Cassandra. How many people use Cassandra? Oh, that's surprising. More people use ZooKeeper than Cassandra. Cassandra is a lot easier to set up and use by the way than ZooKeeper. Am I still audible? All right, so Cassandra is interesting. They actually claim on their website. For example, they offer what they call as tunable consistency, right? What that means is you can actually change the settings in Cassandra. The way do you do reads and writes to get different kinds of consistency, consistency availability trade offs field, right? So if you want to operate Cassandra in the most available mode, right? You operate it by setting using right consistency of any. That means any node, any node in the system should be able to accept a write, right? You write, but to ensure that any node is actually able to accept and not just one of the replicas of the chart that you're writing to, you have to set hinted handoffs to on, right? It's a mode where what happens is you write to any node even if it is not supposed to contain that particular key, it will just store it locally and eventually just replicate it out to the right place. But it doesn't store that for long. It stores that data in hinted handoff mode only for a couple of hours and may lose data. So you do get extreme availability here, especially write availability. In fact, the writes here are cap available. That is any node will accept the write, right? It's cap available, but at the same time you lose consistency very easily. Because if the right set of nodes which where the key belongs to do not come up in time, then well, the data is lost forever, right? It can result in losing acknowledged writes, which is the worst thing that can happen in terms of consistency. Read consistency, the best or the worst you can do really in terms of consistency and best in terms of availability is you set read consistency to one. That is any one node which has that data can respond. But do note that a note that is holding data because of hinted handoffs cannot actually respond to a read in Cassandra. So you don't have really have an any mode. So Cassandra for reads is not cap available, right? It is highly available, but it's not cap available. Not every node can respond, right? On the other hand, if you want to choose to the other, go to the other extreme and choose consistency, right? Really consistent things in Cassandra. You set write persistence to all, right? Write consistency level to all and read consistency level to all. What that means is whenever you write synchronously, all replicas that should be containing, that should contain that key should be given that data. And when you read, all replicas must respond, right? This sacrifices availability like crazy. Almost no one operates Cassandra in this mode, right? So because even if one node is done effectively, any keys that should be going to that node cannot be written anymore. But it does offer extreme form of consistency. Though fair warning, Cassandra does employ something called as last write wins because of which this does not ensure by itself that you get linearizable consistency. So if you want to get linearizable consistency, it's probably not possible in Cassandra, but if you want to get sequential consistency, right? It is possible. And one way to do that is in Cassandra, set the, do not do any updates, right? As long as you do only append only writes in Cassandra, that is you'll not update any key value pair. So key, there's column and value. So you add new columns, but do not overwrite any cell, right? As long as you do that, you actually can get sequential consistency in Cassandra, but you almost never get linearizable consistency, right? That's about Cassandra. Lastly, let's look at Kafka. How many people use Kafka? Oh, I'm really surprised. Kafka is the popular thing right now. Anyways, so for those unfamiliar with Kafka, Kafka is a distributed log as they call it. It's something, it's like a message broker where you can write data and in the same order, you can read data off the other side. It's replicated, it's sharded. And Kafka is among the more interesting set of things here. It offers three different forms. It falls in three different coordinates based on what settings you use, right? If you want the extreme availability mode, you do what is called as Async producer. So in the latest version of Kafka, that's the default kind of producer you use. You set required acts to zero. That is, you don't even need an acknowledgement that a write has been received by a broker. Then you get extreme form of availability. That is, you can lose data as you care, but you get very, very low latencies here, right? Because there is no consistency requirements and you get high availability. Even if the entire broker and the entire cluster is down, this will keep accepting writes and just throw them away. On the other hand, if you want extreme consistency, as you set required acts to minus one, that is, all replicas which must contain this data should acknowledge that write before the write is acknowledged. And you can, you should also set a setting which is available only in the latest version of Kafka called min.isr, that is, how many replicas that should be live are actually live, right? That setting, you say, set it to the number of replicas. Then you get extreme form of consistency. Lastly, you can actually choose somewhere in between. If you set min.isr to one, that is, all replicas don't need to acknowledge, but just the master needs to acknowledge, you get somewhere, something like PAEC system. That is, all right. So that's that about Kafka. And actually with various settings, even in Cassandra, you can get various planes on this mode. I think what you should take away from here is data stores are not at these extremes, right? These extremes are not really useful either, right? They fall somewhere in between. And many data stores today offer ways for you, as a developer, as an application owner, to choose the kind of consistency and availability and latency trade-offs that you need to make, right? It's great. So just to recap, right? MySQL and Kafka offer that sort. So mySQL with a synchro application, Kafka with min.isr set to one, and required act set to true, they offer PAEC mode. That is consistent when there is no partition, but lose consistency and choose availability when there is a partition. Kafka and Cassandra also offer a PCEC mode. That is extreme form of consistency. Always choose consistency. Whether there's a partition or not, you choose consistency, you want consistency, right? Rarely do you want such things, right? That means, but there are cases. There are definitely cases. And then there is finally PAEL, extreme form of availability. Choose low latency and drop consistency, even when there is no partition, and choose availability when there is a partition and forego consistency. That last quadrant looks quite empty, right? It's a strange beast. So you're saying that that beast is saying that, hey, you know what, in normal cases, things are actually low latency, that the vertical line, remember, that's when there's no partition. It foregoes consistency for latency, but when there is a partition, forego availability and become more consistent. That just sounds weird. There's actually one system that I know of which offers something of that sort. It's called peanuts. It's a database that is in Yahoo. People who have been in Yahoo would probably know it as Sherpa inside Yahoo. In academic community, it is known more as peanuts. Now, so what peanuts does is, there is a master, and only the master accepts rights, and then it is replicated to all the slaves. And this is the more, any slave can, or any master or slave can serve all read requests. But when there is an availability problem, peanuts just stops rights, right? So when there is an availability problem, when the master is done, or when the master is not reachable, or the master is not able to reach all the clients, it stops rights, right? So it actually, when there is a partition, by stopping rights, by not accepting rights, it's choosing consistency. While all other times, because it is doing a sync replication and allowing reads from any slave, it is actually choosing latency, right? This points to actually one big flaw of PACKL, or PASELK, which is that PASELK is not really defining in a very precise manner what availability or consistency or latency mean, right? Unlike CAP, it is not defining them in a very precise manner. And that's the reason why that, and all of these things exist. Many people think, are speaking of this as a flaw. To me, this is actually one of the biggest advantages of PASELK, because instead of defining very precisely what consistency and availability should mean, it's leaving it up to you. It's just giving you a framework for analyzing data stores. You talk about the kind of availability you need, you talk about the kind of consistency you need, latency requirements that you have, and you actually analyze your data stores based on that, and you compare them, and you call out every time what sort of availability it is. So when you talk about what sort of availability you actually speak of, hey, this is how the availability gets lost, or this is how the consistency gets lost, right? And you actually speak to your business and tell them, hey, this is what is happening. And see if they're okay with that, right? Moving on, I'll just conclude with a few simple remarks. Availability, consistency, latency. These trade-offs are very, very real, right? Cap theorem is not wrong. Cap theorem is absolutely right. It's just not very useful for analyzing real-world databases, but these trade-offs are real, and you need to actually analyze these trade-offs if you are using any distributed data store, right? But the important aspect to remember is that it's not an engineering decision, right? Many people think, hey, you know what? The architect and the team, or the team as a whole, they can vote and decide whether they choose consistency or availability. No, that is not how it works, right? You need to ask business your hard questions. You need to ask them, hey, would you rather be down or show wrong prices? For example, in an e-commerce website, would you rather be down or show wrong prices? That is inconsistent or available. Or you ask them, would you rather be slow or show wrong prices? And you'd probably be surprised to hear the answers. Most businesses choose availability over consistency, right? And the reason is simple. Unavailable is complete loss of business, right? Inconsistent, there may be ways to work around it, and there are many ways to work around inconsistencies, right? There are, there is eventual consistency. Eventually things will become right, right? There is read repair. So dynamo kind of systems, for example, allow read repair. They let the systems become inconsistent, but later, at the time of reading, there is reconciliation, which allows the system to gain back consistency. There are also other ways of dealing with consistency problems. For example, you have compensating transactions. So ATMs, many ATMs allow inconsistencies, that is, they will allow you to withdraw more than you have balance if they are not able to reach the network. So ATMs don't shut down if they're not able to reach their central network to check for your balance. They'll just allow you to withdraw more to a limit. But then, what the banks do is they will charge you overdraft charges, right? That's a compensating transaction. They are able to work around these problems. In fact, financial institutions are good examples of places which run completely on availability. So there's a very popular article going around recently called, accountants do not use an eraser, right? The way financial institutions work, they record all information, and then they are able to do reconciliations, audits, reconciliations. This is how things work. Applications designed in this way tend to be more robust. Applications that choose availability but then are designed to get consistency and get in consistency through compensating transactions and various other means, tend to be more robust, right? Before we end this, I will show one case where at Flipkart, we actually chose consistency over availability. Our famous flash sales. So these are sales for those unfamiliar where we have maybe 10,000 products to sell. There's 500,000 people who want that product, right? And they all come to our website at that instance. So 2 p.m. Fridays is when, Tuesdays is when we usually do this. And within two to five seconds, the whole, all 10,000 products are gone, right? The traffic spike looks something like that. Quite ridiculous. It's like within just five seconds, we have 500,000 people hitting. All 10,000 products are gone. Here, say we chose availability over consistency. I mean, this looks like a case when business would hate to lose that selling those 10,000 products, right? But let's say we chose availability here. The problem could be, instead of selling 10,000 that we have in inventory, we may end up selling to 50,000 people. What do we do then? That's a big problem. This is a case where we had to choose consistency over availability because the cost of inconsistency was way higher than cost of unavailability. If we lose that window for doing the sale, we can do it an hour later. We can do it the next week. 10,000 products will lie in our warehouse for another week. Yeah, that's bad, but it's not as bad as promising 50,000 people when we have only 10,000 in stock, right? That's all I had. I have compiled a set of references for that I've used for this talk. They're available at the URL. I'll be uploading the slides as well. I'll take any questions. No questions. It was too theoretical. Hope everyone isn't sleeping yet. Thank you, Siddharth. That was a good talk. One quick question coming from the technical community. From my perspective, what should I do? I know you said that it's important to maintain simplicity. And for me, my code simplicity is important. Of course, what the customer thinks is really important, what should I do? And what sort of decisions should come from an engineer when designing a system like this, keeping in mind the trade-offs? So simplicity is actually trickier to think about. I mean, one of the reasons no framework exists is it's very qualitative, it's subjective, right? But there's many ways to think about simplicity. So the simplicity, which is incidental, there is simplicity which is inherent to the problem. So there are problems where there is complexity that is inherent to the problem, right? You cannot work around that. But then there are complexities that are brought about by the tools that you're using, accidental or incidental simplicity. What you want to do is avoid this kind of complexity, but the inherent complexity of the problem domain is never going away. There's nothing doing that. This is Ravi. I have a question on the eventual consistency. So suppose I have an application which is primarily focused on writing and I need a clone of that application for read. So I'm kind of separating the application for write and read. So how do you achieve eventual consistency when you can't define an SLA? So that's eventual, right? Eventually, so what eventual consistency and eventual consistency, there's actually good amount of literature on that. Eventual consistency is saying that eventually the system will actually become consistent. Correct, the question is more on, see you have a replication between database A to database B, but when you want to have the replication between A and B, you need to define some kind of an SLA, right? So when you make A and B replication in an async mode, we don't really have a time bound to that. It could take two seconds, it could take five seconds, it could take 10 seconds. So how do you achieve this? In a synchronized replication, the challenge will change. That is you have, you're writing to database A and synchronously it will do a two phase commit to B and A and B are in sync. But in an async replication, you write it to A but then you have, the async has to write it to B, it takes its own time to commit, I mean, write it to B. So how do you manage between A and B consistency? Even to look, so how do you manage that? So see the thing is you're not managing consistency at the time of writes at all. Consistency is being managed by the replication manager whoever is doing, whatever is doing the replication, right? Now, there are various ways to do this. So usually there is a replica log or there's a bin log which is used and the replica is reading in order. So one of the things that this sort of replication guarantees that's why is sequential consistency because updates are applied in order. And the replicas are usually keeping track of until what point in the bin log they have read and they're applying from there. And there are various ways to do this. So there's a notion of global sequential number that is kept along with each bin log. So replicas are able to keep track of how much they have run. And if it is global, if the sequential number is global, it allows you to have consistency across a larger system, just a pair. Got it, got it. Thank you. Siddharth Rahul here. Can you please go back to the slides where you plotted Cassandra into four coordinates? Sure. So in the case of partitions, right, is it not possible to still have consistency? Not really. So it depends. So if you're doing that, you have consistency. That is if you're doing write consistency all and read consistency all, you have consistency. But the system becomes unavailable in case of a particular sort of partition. No, but there's some middle ground between any and all, right? So if you have quorum consistency for write as well as reads, and if you have read repairs enabled, then you can achieve consistency in case of partition all. It's not complete consistency then. That's what I'm trying to say. So these are two extremes that I've presented and there is a lot of mode. And Cassandra in particular offers a lot of modes for write consistency. I think it has seven or eight different modes for write consistency and around five or six for read consistency. And by varying them in various ways, you get something in between. You can. But it's usually in these two coordinates that what Cassandra offers. I mean, the recommended way is to have quorum on reads. So you quorum consistency, right? It's still somewhere here. In case of a particular sort of partitions, for example, where there is no quorum on one side of the partition, you can't do writes. It becomes unavailable for writes. Well, if it's a single partition, you'll have at least quorum on one side, right? No, not necessarily, right? So if you have a five-note system and you have a quorum of three, there could be a partition where there is only one node, right? This five-note system, there are three of them need make a quorum, right? So one of these nodes is in a partition with say the other two nodes and two nodes are out. So this node is not a quorum. You cannot write to it. You cannot read from it. If you do quorum writes or quorum reads. Thank you. Okay.