 handle several terabytes of a data-based site, right, from the single standpoint. Now, how would that translate in how large applications a single minuscule instance can support, right? And if you do some math and say, well, let's not take those kind of marketing numbers, a million and a half plus, let's take something more conservative, right? Just 100,000 to 30 seconds. What would that give us? And in this case, if you say, hey, you know what, we have our application, right, which has, you know, a number of user interactions and we take maybe 10 queries for each of them and maybe we are having, that gives us so many user interactions a second, right? And looking at some, you know, medium user engagements of maybe, you know, 30 interactions a day, we can still see pretty much of a large number of daily active users, right, from the start. Now, what I mean in this case, and again, your numbers can be substantially different, right? Your transactions can be more complicated. The users may be kind of more engaged in a few in a day, right? You may have some other kind of background activities which you say are not constant here. But the point is what you can really build a pretty serious application which you do not need more than a single database out there. Now, even in, and another thing which you look at their distributed systems is what, when you look at the distributed systems, wherever they are, they tend to be more complicated to develop again, right? If I'm having a single MySQL system, I wrote the data, it's immediately available, I don't have to deal with anything like a network jitter which can, you know, introduce variable light latency or slave lag and so on and so forth, right? Distributed systems also tend to be more complicated to operate or have additional performance bottleneck or additional more complicated failure mode, right? Unlike single MySQL instance, which can be either pretty much up or down, I can have things like slave lagging or replication broke and I didn't notice that so my data is not updated, right? A lot more complicated. Now, why do I bring those two items together? Because from our side, we see folks trying to go to very complicated architectures very early and why they're doing that? Because they go to the talks just like this, right? And you would have a lot of people from famous companies like Google, Facebook, Slack, you know, talking about those amazing systems you may have built, right? And that gives us an idea. Wow, it makes sense to get those kind of massively shorted MySQL across many kind of data centers. That looks so sexy, that looks so interesting, right? That that is absolutely useless for majority of applications out there, right? Majority applications are not Google and Facebook, right in this case. So that is something which is very important to recognize, right? And that means a lot of solutions they would talk about would need to be scaled down dramatically to fit for you, right? Again, I do see some faces here. Already we talked with a large-scale application, right? And you can rightfully disagree with me, but still you have to understand what in the grand scheme of things you have a minority. Okay, now with all of those, why do we need to kind of go distributed, right? And go outside of that kind of single MySQL instance? Well, there are really few reasons here. One is high ability, right? Single instance kind of can easily crash. Skill ability, what we spoke about. And also virus needs which require our data to be distributed outside of a single location. Let's talk about the high ability of MySQL in general. And actually that really applies to a lot of other systems as well. One very kind of basic way you can provide high ability is to have a cold standby. For example, with either something like Linux-ZRBD or maybe you have some centralized storage, right? Which you can use to provide high ability for. Now, there is also a solution to failover, such as like a master slave, right? If your master doesn't feel well anymore, right? You failover to a slave promoting that to a master, right? Or there are a number of also active solutions which exist out there, such as PXE and group allocation. Now, as you are choosing what kind of high ability solution is right for you, you are good to ask yourself a few questions in this case, right, about the high ability. The first one is what really a failure mode do you consider, right? And which of them are more likely and more important for you? Because when you often think about high-ability, right? You may be thinking like, oh, you know what, it's either available or not, right? Kind of like it's black and white. In reality, different problems can happen with the systems, right? Like simple one, crash with server or hardware failure, right? You can have things like software bugs, which, for example, can cause storage corruption, which is a different animal altogether, right? Because if I have a storage level corruption, then solutions like DRBD are not going to help me, right? Because if my skill wrote some crap to the storage, it's going to be replicated with storage replication or persistent highly available storage, right? And network failure, that is a whole other story, right? Because there could be so many different network failures ranging from, you know, like a single network failure to some partitioning or some asymmetrical and more complicated failures. I remember one interesting failure case was discussed by one of the guys in Google and they have some sort of misconfigured switch, which would be killing off the jumper reference frames, right? And what I was quoting for them, what two nodes would essentially be able to ping each other, right, from a high-vability kind of thing, but then you would try to actually replicate the actual data, which would become in the large volumes they could, right? Crazy, right? I mean, there could be a lot of those very strange network failures on the tail end, right? Like rather rare but important. Now, in many cases, we have to deal with full data center failure, right? Or maybe not, right? Is it possible for the whole data center to go down and is your application expected to maintain, be available even in this case? In some cases, yes. In some cases, you know what, that happens very rarely, right? And they maybe just, you know, wait until it's come back. And then the other very important one is besides all of those kind of normal hardware and software things, there are also such nasty things as people, right? And I think you all agree, people are nasty. They take their dirty hands and they do nasty things with them, right? Like logging in and dropping the database, you know, on a production server instead of developers, right? Or writing the queries which are, you know, not really scaling, right? So those things happen and there those can be increasingly more common cause of failures, right? Then your hardware failures as those things can be getting more reliable. And then there are, of course, some of the security issues you also can be thinking about, right? Like you can get, well, of course, we can have all the set of problems, somebody stealing your data, right, and publishing that on the Internet and there is no high-ability can save you against that. But on the other hand, you can have somebody using, let's say, SQL injection, right, to trash all of your data. And that is potentially can be thought through the high-ability framework. Okay. And I have a question. What data am I allowed to use to lose, right? And of course, they thought kind of in classical desire, hey, I never ever want to lose any data. And while that is a very admirable desire, it's also very expensive. And I would say for my SQL environment, a lot of folks really develop dependent on master-slave replication for years, which is asynchronous, right? And it does not guarantee what you're not going to lose the data. But the amount of those data loss, right, would be typically just few last transactions, which couldn't be replicated, right? And in many cases, it is acceptable, right? I mean, for a lot of those kind of social media environments, right? If one of those cat posts ever get lost, I mean, I would argue that they're doing a positive thing to the world, right? I love such things to happen. Well, that's just me. Another thing, of course, you may want to desire no data loss for committed transactions, and I think that is pretty normal. I also see a number of people saying, hey, you know what? Because our developers don't understand transactions and then prefer not to handle error situations. We want to make sure... What? Oh, okay, okay, yes. I guess somebody else seen those kind of developers. That's good, that's good. Some people also do not want to make sure that transactions in flight are not lost, right? That means I have an open connection which did a couple of updates and haven't committed transactions. Database crashes, you still want that to kind of continue with no interruptions, right? And that is like a very hard thing to achieve, if you will. And finally, you have to be asking yourself how quickly do you need to recover, right? And often that has to be mapped also to a different failure mode. In many cases, you can say, hey, you know what? If I'm having an active, active cluster, then I will pretty much want like immediate recover within a second, right? If one of the nodes crashes. And of course, nothing is really immediate. And by immediate, I mean in this case is what... Even if it crashes, my users are not looking... notice any interruptions, right? For them, boom, right? The node crashes but they just maybe have some semi-invisible delay, right? For them, the web page took a little bit longer to load but they didn't really notice that as a downtime at all, right? Now, in many cases, you would have, let's say, full data center failure may take longer to recover as acceptable because in many cases for DR cases, you don't want to maintain, for example, completely full spinning environment somewhere else which is kind of expensive. Now, I think what is very important also is to understand speed of light, right? And the thing is, the things don't go instance, for example, between Europe and America, right? Even if you would want them to. And often they have to understand what they pretty much have between kind of two choices, right? For data propagation, that either kind of gets synchronous but then they say, hey, we are waiting for stuff to be replicated, for example, from US to Europe, right? And we wait for that to be there so they're sure if something happens to replace in US, it's not getting lost, right? Or we can have an asynchronous and in this case, well, data gets naturally queued, right? And if that, let's say, whole data center loss goes down then data loss can be a cure, right? And in grand scheme of things, we have to choose between those two other power power projects. Okay, now scalability. For scalability, we pretty much have three things that you have to deal with. We have to use scale reads, scale writes, scale data size and in many cases, as our application grows, we have to do all of those at the same time, right? And that is where we have to go to much more complicated systems as single instance. Now, let's look at data distribution and what I mean by that. Now, in many cases, we cannot just keep the data in one place if we are developing their international application, right? And that can come from the two reasons. One is user latency, right? You want to have a really application which goes very quickly, right? It may be expensive to go from, I don't know, say, you know, mobile network in Vietnam, right, to a data center in America, right? So we may want to get the data closer to user for latency. And then there is also legal and compliance reasons. There are a number of countries which require you to, for example, develop data related to their citizen, right, in their national border, right? Or have some other regulation which you may have to deal with, not from a kind of database design standpoint, but from a legal compliance and whatever reason. Okay, let's jump through to a MySQL and go to maybe the same things in a little bit more MySQL specific and detailed way. So when we speak about the MySQL, it is good to understand and go through those concepts. So what is a replication, right? So when you see quite a replication, they have multiple copies of data, right, which should be in updated changes, right? That is how replication is different from, for example, backup, which is also a copy of your data, or offline and it kind of not tends to be updated, right? Now replication is obviously, as I spoke, is connected to providing high-vibrility in many cases, right? And if you look at the high-vibrility from a grand scheme of things, we want for our service our ability to kind of respond queries, provide user states up even if one of the components fails. So high-vibrility generally is provided through redundancy, right? It would have more than one system, so if one of them fails, something else is going to take the load. And that works increasingly well for a stateless system, right? It's like you think all this there, I would say, like a Kubernetes concept, saying, hey, you know what, just keep maintaining five copies of my web server and everything will be fine, right? If it is stateless, then I can essentially, you know, kill them at wheel, other material at all works fine, right? But the databases can't be stateless because databases is exactly the definition of the state itself, right? And that means something else is needed and we also need not just a redundant computing resource, right, but an application server, but we also need very complicated data, right? Multiple copies of data to provide that redundancy because by definition, single copy of data, right, it's not redundant. Now, replication can happen on different levels, right? It can be on the storage level, right? On the database level and even on the application level, if you will. Now, if you speak about storage level replication, that is what we mentioned, like your son, Nas, can provide the DRBD. If you're using any kind of cloud storage, like an EVS, it also really has redundancy built in through some form of internal storage level replication. Now, a lot of storage level replication tends to provide just a cold standby, right? Because the MySQL, unlike, let's say, some other systems, it's not designed to operate on a shared storage. The system, MySQL in a DV, assumes that it has like a monopoly access to the storage, right? And if somebody else changes something underneath it, well, it won't work, right? Now, one wonderful thing about the storage level replication, of course, it's very simple and that works for pretty much any system, right? Any system which stores data on this can be managed with faith, whether you take a MySQL, Postgres, right? Or even, you know, like an NFS. You can provide, have ability through cold standby through storage level replication. Now, I also want to point out some special examples here. Some of you may have heard about their Amazon and Amazon Aurora, right? Which also does storage level replication, but that's kind of designed a very, very kind of different way. And in that stuff, I would more classify that as a database level replication, right? Because that storage is kind of pretty much a part of a database technology in their architecture. Now, if you look at a database level replication, that is pretty much the most flexible, most common replication, right? That can provide a hot spare or active environment in some cases. Now, another way is the application level replication, right? I think so many people say, hey, you know what? Database replication is just so hard to set up. Why don't I just go and connect to two nodes at the same time, right? And just build updates here and there, right? And hey, that works on my laptop, and that means it's probably going to work in real life as well. Now, the thing is, with application level environment, it is very hard to do right, right? And frankly, over the last few years, I see less of a people attempting to do that, right? So that is good. Now, in some cases, I would say it is still possible to do that, right? And it's, I would say, even reasonable to do that. For example, if you have some, you know, some complicated needs for conflict resolution, right? Or you have some other needs where you want to replicate the data across different databases, right? So for example, we see a lot of environments right now where somebody say, hey, we would use something like CASCA as our database, right? And instead of relying on the MySQL, we would make sure what all the data kind of getting written to the CASCA, right? And then we can have a MySQL, right? And maybe some other Hadoop, right? Or whatever other consumers of that data flow, right? So, you know, you can also think about that as some kind of user-created replication, right? But that is not your kind of classical stupid replication connected to the two database nodes, right? And try to write the same data at the same time. Okay. Sharding. Now, when you look at the large-scale databases, we tend to find what data is way too large to fit at the single database or even single replication tree, right? So what that means is it displays the data by certain criteria and puts that in a, you know, a separate kind of chunks called shards, right? And we put them on a separate cluster, right? Now, typically, the shard, one of those two ways, right? Is if you think about somebody like Facebook, they often would shard by user, right? That means, hey, you know, we have the map users to those shards, right? And then all the data to the user, right? One of the data by the user will be placed in that shard. In other cases, a lot of software-as-a-service companies, they shard by customer-account companies also refer to as tenants, right? Then I would often have a lot of people which belong to the same tenant, same organization. They have a data which is tightly coupled together, but different organizations are completely separate, right? Think about something like, I don't know, a Slack, right, or Salesforce. In those cases, those different organizations are pretty much separated from each other for normal operation purposes. Now, sharding is almost always used with some kind of a pre-replication. And why is that? Well, because as you scale, there is a higher and higher probability of something or a failure to happen, right? So, for example, you have just one MySQL master server and I have a mean time between failures, three years, right? It may be completely acceptable, right, not to have a lot of replication, maybe just have a backup. Well, you know, it crashes once or three years, maybe have a bit of downtime, not of a yield. Now, we have a hundred of them. That will mean the average time between their crashes, right, would be about three days. Well, which is probably less so, right? That is why when we have a massive sharding, then we tend to have some sort of a replication with that built in. Now, the next thing that I mentioned is important, is the failure management. Now, what does that mean? If you look at the many replication modes, such as MySQL classical replication, it doesn't have any mechanics to handle failure on its own. It needs help, right? What would MySQL do? You have, let's say, master and two slaves and a master crusher. MySQL will do nothing, right? Two slaves will sit out there and say, hey, you know what, it's kind of I can't connect to a master, let me just, you know, wait and sit and wait and do something. You actually need some other process to say, okay, now I want to promote one of your remaining slaves to the master, right, and do whatever environment changes needed to complete that process, right? And then two things that have to happen out there. Somebody needs to decide how to do that failure. I have two masters, two slaves, one master has to figure out, oh, this slave is kind of ahead, so I need to elect him as a master, right, or maybe have some other solutions, right, to perform. And then also the performer failover, right? Oh, now I need to re-configure that slave as a master. And I separate those two because there have been some tools existing which would say, hey, you know what, we only do automation of performer failover, but we want you as an individual to decide then to do that failover, right? Because we don't want, for example, to, you know, to perform a failover when it shouldn't happen, right? For example, then you just overload the master with some, you know, nasty queries. And guess what, if you simply failover to the slave, well, it will be overloaded with the same queries again, right? Now the nice thing is that traffic management question, right? And there are multiple functions which can be performed by those, you know, proxies, load balancers, right, you name them. Scaling large number of connections, routing traffic to the right chart, if that's where all maybe read write splitting, right, so we can actually, you know, not do it in application, but let them decide which traffic to send to the slave and which not, maybe some sort of load management, right? So if we have one of the servers overload to move traffic from it, as well as things like avoidant dead nodes, right? Maybe if I have multiple slaves, one of them crashed, I don't want to be sending traffic to that, right, even until it's kind of completely removed from configuration. So what do we have in terms of options in MySQL and what are their properties? First one is our classical MySQL replication, group replication, Galera and ProconyxDB cluster, as well as MySQL and DB cluster, right? These are the most common things which you can use. Classical MySQL replication is pretty much this. You have master, slaves, bunny logs are being written to the master and they are consumed by the slave in a synchronous fashion. Each of them has their own copy of a database, right, so it's pretty independent. So what are properties of MySQL replication? By default, MySQL replication is a synchronous. So there is absolutely no guarantee how far the slaves are compared to the master. You can also consider that to be semi-synchronous, right? But remember, semi-synchronous is actually asynchronous, right? Semi doesn't mean it's kind of just as good as synchronous, right? So what semi-synchronous does, it says, hey, that means I will commit on my master, right, here, when the data event made it to the relay log, right, on the slave. But not necessarily applied, right? So a semi-synchronous replication, you are sure now if master crashes, the data will make it to the slave eventually, right? Eventually, right? So it gives us kind of eventually conditions of the sort, right? But it's not synchronously applied to the slave. It can be parallel. And what I mean by that is, if some of you remember or know MySQL from the early days, replication was a single thread and that was very limited in its performance, that's not the case anymore since MySQL 5.7. You can also have now multiple masters for a single slave called multi-source replication again since MySQL 5.7. And two big limitations remain for MySQL replication. Is, A, there is no conflict resolution or sort of protection of any time, right? If you set up a replication and if you happen to write some conflicting data on the slave, well, you'll just have replication master and slave data inconsistent, right? And maybe replication will stop due to conflict at some point, right? But there is no specific protection out there. And then there's also no built-in failure, right? I mean, it still, by default replication will simply stop if master is unavailable. Now, MySQL replication, which has kind of its own simplicity, it allows us to create many different topologies, right? We do kind of as a few examples here. We can have a few masters which are streaming to a single slave, right? For example, to consolidate the data, right? We can have a ring replication set up. You probably don't want to, but that's kind of, in theory, possible. You can have a multi-level slave. If you want, there are some slaves connected to each of the slaves. And what is also important, you don't have to replicate everything in MySQL replication. You can say, hey, I have a two-level slave. I replicate everything from here to there and then only replicate some of the data here, right? So MySQL replication is, with its own simplicity, it's kind of a loss for good and bad to create a very flexible, weird MySQL replication hierarchy. That is the example. I want to just put a shameless plug here with our PMM for quantum engine management. We have some nice dashboards for replication among other things. Okay, moving on to MySQL group replication. MySQL group replication kind of forks a little bit different way, right? In this case, we have a group of nodes, right? We just call it replication group. And you can see from conceptual standpoint, they are all masters. They are all peers, right? And what are the properties of this replication? We can write either to any of the nodes or actually, by default, it configures itself so there is one node is elected as writer, right? And the other nodes are read-only and tend to be only read from, right? And that's done because that gives you kind of more comparability with a standard replication, if you like. This is also like asynchronous replication of flow control. What does that mean? Well, that means that the data is still asynchronous replicated but we have a flow control. That means we don't really allow slaves to be like lagging by unlimited period of time. There is like this elastic robot there, right? And if a slave becomes too far behind and say, hey, master, hold off please. I am not able to keep up. Can you please slow down a little bit, right? We also have conflicts which have been prevented in this case through synchronous certification, right? While changes are not actually being applied synchronously, they are certified, right? And that means they are guaranteed to be committed to the cluster synchronously, which allows us to avoid conflicts. It has an automated fail-over. And what does that mean? Or built-in fail-over. So what does that mean? If one of the nodes crashes, right? Then it just kind of leaves the cluster. If that node happened to be that dedicated writer when other nodes get elected and so on and so forth, right? So a lot of good stuff was there. There is still limitation, though, there is no automated provisioning. And what I mean by that is if you want to set up a node for my school group application, you need to get your data from somewhere, like maybe restore from backup, manually, same as you do for standard application. Okay. The next piece is the Perconex3DV cluster called Galera, right? In this case. Now, the idea in this case is very similar, right? If you look in this case, there is a group communication. There are multiple nodes which belong to the group, right? And reality here is what from many properties, Galera and my school group application are similar. The frankly, the reality is that what Galera and PXC was around for a long time, right? Since my school times with my school 5.5. And I think it's folks in Oracle seeing how useful this kind of application is. Implemented group application, right? And in this case, it's kind of by design. It shares many similar problems. So we have a write-to-any node. It's a certification-based application the same way. Now, in this case, it's also a synchronous kind of replication, a synchronous certification. But there is an option you can ensure there are no stale reads. You can instruct the cluster and say, please hold off selected execution until the data actually propagates completely to this node, right? So that is a mode in this case which is available. There is a build-in failure and also build-in node provision. So in this case, the connected with cluster behaves more modern, right? If you take the freshly installed node, right? Or put up the Docker container and say, hey, can you please join the cluster? It will automatically perform everything which is needed to join, like copy the data and so on and so forth, which is useful. Now, maybe I should explain, basically, how the transaction control works in PXC. And that images for PXC and users is terminology, but really the MySQL group application concepts are very similar in this regard. So if you think about the user transaction, all the updates, right, during that transaction are performed completely locally, right? There is no network communication when you are writing updates of any kind. You simply have something which is accumulated, which is called write set. This is pretty much you can think about as a set of primary key values, right, that are going to be updated. Then, when you trigger the commit, then event is sent to other nodes, right? And that is what's called the certification. And that is pretty much the only network communication which is required, like one commit per, one round trip per commit, right? And then in this case, the certification happens, right, in this case, through that group communication, and then you would have a physical commit or a back based on certification results. Now, a good thing to understand is that other nodes, which is, in this case, slaves, the slave for purpose of a transaction, right? Different transactions may have a different master and slave in this content. The certification will get that event queued and then apply threads, the threads which will apply those events locally, right? So it is possible for some glitch, right, to happen for some lag to happen in this case. Now, what is also interesting is you can see from this diagram is why it is more likely that what your master for a purpose of transaction will commit transaction first. It is also possible for slaves for purpose of a transaction to commit it first, right? So for example, if we just get a stall on a disk IOR on this kind of final commit operation, then this guy can commit the transaction faster, right? Now, you will ask, okay, but what happens if you commit the transaction, say here, and this cannot commit a transaction because, as I was saying, you had some disk error in this case. The concept here is what the node which fails to commit the certified transaction must commit suicide, right? Or pretty much crash, exit the cluster, right, whatever you wanted, but I like commit suicide. I think there is some water finality to that, right? Okay. So, we also have in the PXC a number of performance improvements we did recently and we are really very proud of the dramatically increased concurrence, which really could be up to 10 times for right-intensive benchmarks for variety of workloads. So that is good, and I should thank folks at Oracle, right, for creating a competition out there. So now we can, you know, compete in our benchmarking, right? In terms of visual solutions, it's actually faster. As you can imagine, in our Percona module management tool, we also have some nice dashboards to visualize Percona X3 cluster or actually any other Galera solution, right? Galera for MySQL, MariaDB, and so on and so forth. Okay. MySQL cluster. Okay, now let me do a poll here. Anybody is using MySQL cluster as MySQL and DB cluster here. Okay. Anybody used MySQL cluster in the past? Okay. Okay, so anybody tried using MySQL cluster in the past? Okay. Well, that is, I think, the main point, right? I feel like I do include this technology out there, but very few people actually use it because while I think it has a lot of conceptual ideas, it's really achieved only niche adoption so far. And this is a solution which has a lot of nice features such as kind of automatic sharding and distributed query execution, but it also has a lot of limitations that use different storage engine and so on and so forth. Well, okay. Now let's talk quickly about sharding in MySQL. Sharding in MySQL is typically either painful or very painful, right? So there is no such thing as, you know, like, no easy sharding in MySQL, right? Not in MySQL. There are no built-in or standard solution out there. But frankly, if you look at a lot of the early MySQL adopters like folks at Facebook, we really have some custom application operations. Let's say you have a first-class of five stars you want to update. How exactly are you going to do that? It's not easy, right? So five shards became overloaded for some reason because maybe they became over-reliable. How do we balance that shard with that traffic somewhere else? All of those things are very important, right, for sharding, but it is hard to get sharding. So what kind of solutions exist for sharding out there in the company? I would say by far the most advanced of the, right now, is the test. That is something we started by seeing, and that's now open-source and it's available from a cloud-native computing foundation. It's outside of... You think it's good, right? Because when technology starts to... it's only developed in one organization so it's designed to solve a problem for that organization, right? In fact, somebody else will be able to scale, support the MySQL protocol, and many of my solutions. Not all of them, right? But for a lot of applications, you can just... What the problem is, I did one more network call. So that is going to impact the performance of it. The other way to deploy ProxySQL is when you deploy ProxySQL on the application server side. You think about ProxySQL as part of client-library interface, right? And each of those instances, right, can route connections appropriately, perform failover, and so on. Now, in the modern ProxySQL, you can just replicate between each other so they can have the configuration changes applied to them at the same time, right? So if they know what this node goes down, right, that information will be kind of infinitely propagated to all of them. Now, this works if you don't have too many application servers, right, of course. I mean, if you have, I don't know what to say, 10,000 application servers, and that's kind of become maybe too complicated to maintain. Now, the ProxySQL, of course, again, we have a nice dashboard for that, right? Let's say... Yet another shameless plug, but that's okay because the PMM is open source, right, and let's make it okay, I guess. Now, I also want to mention a few cloud-only options. Now, we see a lot of people are deploying databases as a service in a cloud because that really allows to save a lot of rudimentary pain, like taking backups and so on and so forth. And that also provides, typically, its own built-in heavy-ability functions, right? Like, whatever, Amazon, VS, or Google Cloud, SQL, Azure Database for MySQL, all the big clouds, they do have some database-as-a-service options with some built-in heavy-ability options, right, which you can consider. None of them, at this point, as I know, have some sort of built-in shorting support, right? So shorting, if you need that, is still going to be on you. Okay. SQL is a hopefully important infrastructure for years to come, but the thing to know about MySQL is not good for everything. Solutions like Symmetric.js. We have a lot of Kafka gateways to MySQL, right, which would essentially, typically, feed MySQL binary log and put the data to Kafka, such as the Basium, right, or Yelp has open-source MySQL streamer. There are actually more of those solutions, both open-source and close-source ones. Tanzan Replicator. That is another technology which can replicate from MySQL to a variety of data source, right, or actually also from, it can replicate, for example, from Oracle to MySQL and so on and so forth. And then, we also see a number of folks have been deploying their custom MySQL binary log consumers, right? There is, for example, Tarantul and Clickhouse. They implemented their own tools, right, to consume binary log and replicate it to their technology. Like a wonderful thing in this case is a lot of that stuff is open-source, right? So if you want to write your own tools, say, hey, I want to consume binary log, there are, you know, tools in Go, Python, C, right, Java, you know, language of your choice, you would be very easy to be able to find that. Okay, so if you look at the other summary, in this case, we can see that there are a lot of options exist for building distributed architecture of MySQL. I would say perhaps there are too many, right? I mean, in many cases, I think there would be fewer options, but they would work better, right, or support better. And I think what also makes MySQL very exciting is that there are also lots of options integrating MySQL with other open-source data stores, right? So we can really build a very effective, multi-data store environment, right, where you get a different benefit for that. Now, if you are interested about open-source databases, to learn more about open-source databases, I would like to welcome us to our conference, which takes place last month. It was weird to invite somebody to last month's conference. Next month, next month. In April, in Santa Clara, it's going to be wonderful. There's a lot of talks about a lot of technologies from many famous companies. So with that, thank you, and I would be happy to answer your question. If you have a question, just raise your hand. Any questions, yeah. What would you recommend most for, like, sort of intra-continent replication, where you have fairly frequent internet general 10 to 30 seconds in your apps? Yeah, well, so if you have this intercontinent, right, in this case, then often you would be limited to the asynchronous application. That's, like, typical, a very little solution you would have. Now, I would say, as I mentioned, unfortunately, built-in mice to replication doesn't have any good conflict resolution built-in at this point, but I am hopeful what your concerns and concerns of many others would be heard, and in terms of future versions, that will be resolved. Okay. You joked earlier a little bit about the Galera nodes committing suicide, and we saw that a lot. You saw that a lot? The Galera nodes committing suicide. We had Galera across the ocean, and it would have, like, a 30-second hiccup, and then both sides are like, ah, I'm down. Yeah, yeah. Well, I would say you're right. Now, let me state something important to what you stated, right? So if you have the Galera or mice to replication of those technologies, they typically look to solve a network partitioning problem, which is solved through a quarrel, right? And that doesn't work with two nodes, because if you have two nodes, right, or two and two, right, let's say four nodes, and you break in between, then it kind of sucks, right? You don't get, you know, more than 20% votes anywhere, and both nodes crash, right? So if you have an environment like yours, you want to make sure you have either free data centers or if that's expensive, you only need two, and also put things like Galera Orbiter in a third data center, right? What's Galera Orbiter? Oh, Galera Orbiter, that is a very lightweight node which essentially does not store any data, but it is a type breaker, right? So in this case, if you have two data centers, the network between them fails, right, or let's say one of them looses connections to the rest of the world, then Galera Orbiter will decide which of those two has actually had connection, right, and to the world, and which would just become it later. Okay. Yeah, we've been to the same thing. We had, like, four of the one data centers, three of the others, and always knew which data center was going to commit suicide, and luckily, our data center was really small, so we just read straight into the data center. Yeah, so that's right. I mean, if you have a three and four, then that would work, of course. If a data center with four dies, then the other one will commit suicide, will crash as well, right, because we'll see it in isolation. Okay. And you have a question? Okay, well, no more questions. That means if it's extremely good or extremely bad, I leave you to judge that. Anyway, thank you. Thank you. Thank you so much. Okay. Okay. Okay. Go to here. Harder to get to? Oh, I struggled with that one. Yeah, and then I just put you at something like this, and then just... Got it. Okay, cool. Yeah, I learned that the hard way. Test, Mike. Test. Thank you for coming to Scale 16.