 My name is Toliver, Jal Toliver from PlanetScale. I'm a software engineer. And this is Jitain. Jitain is the CEO of PlanetScale. So if you have all any questions, you should just ask him. I'm here just to look pretty. So today we're going to talk about Vitesse, which is, and this is called, Stateful Storage on Kubernetes. Vitesse has heard of someone who uses MySQL, okay, that's good. So maybe this is for you. So anyway, okay. So what is Vitesse? Vitesse is a sharding middleware for MySQL, sharding, I think, something, and what's beautiful is that it's massively scalable and provides high availability, like five nines, availability, and it's cloud-native, so Kubernetes, et cetera. How many people use Kubernetes? How many don't use Kubernetes? No Kubernetes? Okay, everyone's Kubernetes. Oh, even better. Great. So who uses Vitesse, so this cache app is square cache. That's like we've seen the take, but for US, also YouTube, Vitesse came from YouTube originally. So they use it, not Burger King and not Instagram, yes. But those are like the top apps in the US, and so they're using it. So here are some of the key adopters. So how many have heard of JD, Jingdong? So Jingdong uses a lot of Vitesse. They're one of the biggest users in the world of Vitesse, and they have 3,000 or more databases, and they have 18,000 or more tablets or MySQL instances, so more than just one. So many. And maybe you guys don't use Slack as much, but Slack, even them, at least 1.25% have changed to Vitesse, so all the chatting is saved in Vitesse. Again, before square cache also, and Pinterest, I don't know if you know what Pinterest is, so Pinterest also uses it for a lot of their advertising. So yeah, so these are some of the many adopters. And as there are adopters, there's also lots of people contributing, changes, PRs, and so includes lots of these companies. And they all, of course, are using Vitesse in production. Am I speaking too fast, too slow? OK, I want you to think from now on, if you think Kubernetes and you want to store stateful storage, I want you to think Vitesse. Why is that? There's actually someone, some famous internet person who says, if you want to run stateful applications on Kubernetes, you're pretty stupid. It's too hard. And we agree, it's not easy. And the reason is, one does not simply move MySQL to Kubernetes. Because if you just try to do that, you're going to run into many problems. Your pods going down, whatever going down. Everything goes down, right? Who has a service that's perfect? Never goes down. Anyone go down at all? OK, people who go down, raise your hand. Only three people who have problems? Then you don't need anything. You're perfect. So just a few things about Vitesse. The test started in 2010. And that's when YouTube, who knows what YouTube is? It's like Youku for not China. OK, so YouTube had started running into scaling problems with MySQL. Like many of you, they had just one super big MySQL instance. Of course, they had some backups and some replicas. But they could have too much traffic. And probably many of websites in China, Vietnam, all you guys have lots of web traffic. So YouTube ran into some problems. And then so they needed to deal with it. And they have big apps. They don't want to rewrite. So Vitesse, since in 2010, there was Kubernetes. Kubernetes is five years old now. And Vitesse is older than five years old. But it was designed for Borg. So Borg is the precursor, the Google precursor to Kubernetes. So Vitesse was designed really for Kubernetes as well. Because it's all the same ideas. So yeah, production workloads. So besides YouTube, of course, Stitch Labs is the oldest one. So they started using in 2016, a few years ago. And then HubSpot has many key spaces, or maybe databases, as you may call them. And JD has 3,000 databases with, as I said before, 18,000 yuan, 8,000 tablet. And they have, I think they said, 67 shards now. 67 shards, on their largest. They have many databases, but that's just their largest one. And then there's another company here, Nozzle. And they migrate even between the clouds. So here, you have Al-Yun, and you have Huawei-Yun or something. And you want to migrate from one to another. So they do that. So you don't have to be tied to one cloud. So this is what I thought it was coming after. So MySQL before, it's an old software, very good. But they merely adopted the cloud, whereas Vittest was born in the cloud. It was made for Borg, which is before Kubernetes. So the whole point is, it is cloud native. It did not exist without the cloud. So here is the concrete architecture of Vittest. How many of you guys are running apps that are connecting to MySQL, everyone? Yeah? OK. I'm going to stand over here. Is that OK? OK. So the general architecture, Vittest tries not to break anything. You have big apps, they're running lots of traffic. You don't want to break anything. So before, you have your app servers here. And they used to talk to a big MySQL database. And they use MySQL binary protocol to connect. But Vittest is all of these blue-purple colors. So they're this part and these parts. So it's a proxy to all of MySQL. And under the covers for saving the data is actual MySQL. So you guys, we can actually migrate your MySQL workloads into Vittest actually pretty easily. So what's important here is between the app server and what is these VT gates. These are the proxies. They speak MySQL binary protocol right here, MySQL. And there's a custom protocol here between these blue parts. And then all of this software, the tablet, connect to actual MySQL instances under the covers. And they only talk this way. Does that make sense so far? Good. But why would you want to use Vittest? Not everyone has Jingdong-sized workloads, or YouTube-sized or Yuku workloads. So why would you want to use it? If you have a small workload, what are the benefits of using Vittest? So this was your DV right now. Works just fine. It's all good. But you can actually already put Vittest in between your main DV and the app servers. And when you do that, it provides automatic connection pooling. It does deadlines, like if you have very long-running queries. If you have an analysis, it also provides hot row protection. So if you have many apps trying to get exactly one row, it will stop and pause that while giving other people quality of service, good quality of service. It also enforces row count limits, so not two large queries. And it can also blacklist tables. If you don't want certain applications to access certain things, because it's a proxy, it can also say, oh, I don't want you to access this. And you can cut off usage. So even for smaller workloads, it's OK to use this because it provides benefits for your application developers. Sometimes they make mistakes, have bugs, and then it won't take down the database for everyone else. Does that make sense? That's for smaller workloads. But then once you've put it in, you now have some more benefits as you start growing your workload. So again, before you had only the single database, maybe you had a few backups or replicas. But with your growing workload, you had your previous main database, the master. But with the test, since it actually can parse all of the MySQL SQL, it can actually decide to be smarter. It can do replica routing. It can decide, oh, all you're doing are select statements. And decide, oh, you don't need to access everything. So you can go straight to our replicas to actually do the selects. And thus, it can also do load balancing. You have more connections and whatnot. And you can then use Orchestrator as well if you need high availability, HA, to do failover or promotion. So you can choose one replica. Oh, I want this one to become the master. And you can swap them. And to all the apps, it's invisible. So it's just all natural. So remember, MySQL protocol only happens here. And then this is only management DBA access. Does that make sense? So yes, this is why as you're growing the workloads, if you have a lot of read traffic, you can add more replicas. No one knows the difference. And now you have even more traffic or too much data to fit on one disk or many disks. So you can do a few things. And so with here, with the growing workloads, I can split. Before, I had only main DB, we'll call it DB1. And you had all of the traffic going there. And so you had to have a disk that's big enough for all of your data. But now we want, imagine you have two big tables. They're growing, growing, growing big. And then they're going to blow up in one table. So why don't we split them into two different ones? So with this, you can split table one, which is big, keep it in database one. And then you have the other big table, table two, moving into database two. So yes, you have now table one in database one and table two in database two. But to your application server, because again, it's proxying, you can have a unified view. You can see both table one and table two like they were all in one big MySQL. And again, just like before, each of the database one, you can have many replicas. And database two has different numbers of replicas. Make sense? So also you can, with this proxy, you can do key space subsetting. You can decide your application can only see table one and table two. And different applications can only see table three and table four. So you can split that way too. All right. And as you grow even bigger, remember table two was here. But table one's growing really, really fast. And maybe you're a big e-commerce website like Jingdong. And they decide, oh, we actually, we have too much write traffic. We have too many orders happening all the time. And so table one, it can be horizontally sharded. So in this example here, we see three different shards. So just table one, we can split into one third, one third, one third. And it will go to separate instances. So you can have now three times the write traffic. And you can still have many more times also the read traffic. And you can control that quality of service there. And whereas table two didn't grow very much, it's pretty constant, but still big. And it will have a different quality of service right here. But again, to the app server, it will all still look like one super big database. Makes sense so far? Good. So and then you get really, really big. So this is, again, a Jingdong style one. And they have, and not just Jingdong, but many other companies, also have before it was all in one big data center or one availability zone. And now, when you have the massive workloads, you don't want one bomb to break the data center. You want to be able to handle the failure. So we can split, even with a single shard, into different availability zones. So this gives you other high availability options. Before, it was only for the reading reason. But now, in case one goes down, we can split and have more high availability. Here's zone one, two, and three. Yes? The test will handle, sorry. Yeah, MySQL connects the actual replication. So you'll replicate the data from through the MySQL binary? Yeah, OK. Yeah. Other questions? Does that answer your question? OK. Cool. All right. Thank you to Ten for answering. So yeah, so even as you go really, really big, you're going to need extra, extra high availability. And that's the benefits here, OK? So some demo time. This demo is very simple, very stupid. But everyone can run it on their own laptops. And I'm going to demonstrate by running on my laptop. So this is a schema, marketplace schema. It doesn't really matter. It's just got four tables. My actual demo only has three tables, but whatever. Same idea. This was all one big main DB. And we have a product table customer, orders, and merchants. But in my demo, we only have less. OK. So I am going to demonstrate this. I have already brought up a show database. So this is actually my SQL. Let me make it bigger. Can you see? Is it OK? Yeah. So I just brought up a MySQL. I actually brought up three instances of MySQL and behind the tests. And if you see databases, we see only a commerce database. And inside commerce database, we actually see three tables here, something called order, called C order, but order, customer and product. OK. Makes sense so far. And I will show you there's some data in each of the tables. It just selected, showed all the tables. OK. But all right. So that's just the data, the current one, in the main DB. OK. So next, maybe again, I said table one and table two got too big and maybe wanted to split into two databases, one DB one and DB two. So here, when we shard, we can decide, oh, all of the products go into one database. And we'll make another customer database called customer in orders. So we'll move two of the tables from the original main DB into a new one we'll call customer DB. Does that make sense? All right. So how do we do that? Good question. I have no idea. OK. So we have some scripts here. And these are actually on the Vittes website tutorial. So you guys can get to the code from Vittes and actually just run these tutorials yourself. It's exactly the same. I'm just showing you. OK. So let me show you. What we're doing. So sorry if this is too small or can't read. These are actual commands from Vittes. I'm just showing you what's in the script. It creates a key space. Key space is a database like database two named commerce. And it says to route all of the traffic for it. Sorry. Sorry. Customer. I'm making a new customer database. But all of the traffic for the customer database actually goes to a place called commerce, the old database, the main DB. OK. So I'm only just going to run this command here. And it's done. And then there's a few other things. This with the next command I'm running just brings up more instances of MySQL, three more instances for the tablets. So here we go. Literally starting MySQL. I will now talk to wait for the time. So it's just bringing up more. But of course, while I'm accessing this, again, I can still see all of the, what it's called, all of the data from the original main database. Nothing's changed, as I wait for that. And the point is in these steps, we're bringing up new databases. And then we're going to slowly move the traffic from seeing all of the data in main database to a different one. But all is invisible to the app server, in this case, the MySQL client. So as you can see here, it's bringing up MySQL and also bringing up the little tablet that's attached to each MySQL. And yeah, OK, great. So it also has a master in the new database too. And so we'll see this working, hopefully. Let's see, show databases. Oh, look, we now have two databases. Before it was commerce only. And now we have customer database. And so, wait a second. OK, before I do that, so I'm going to go to the customer database and show what tables we have. We have some tables. But so all the traffic for if we wanted to select from the customer database, it will look like it came from the other database. Or it will behave like it will come from the other database. But it looks like it's also available in database too, the customer database. OK, sorry. So from there, oh, and I can still access as before all the data. Let's see, OK. So actually, now, before I was getting all the data from the database one, the main database. But now my new script just gets the data from the second database. And it shows that it's available there as well. And now, one of these scripts here is vertical split. So vertical split is the moving of tables from database one to database two. I can show you the code here. It's literally just one command. It's the rest is environment variable that is splitting off the data. So it's doing some stuff. What did it do? Who knows? Let's see. So what it actually did there is, again, we had new empty databases. And it copied all the data from one to another. And then from there, continuously replicates from, even if you write the table two in old database, it will start, you will start seeing it in table two in new database. So it sets up, it does the copy. And then replication from then on as well. So that is what this command just did. So still nothing different here. Actually, again, we haven't moved any pointers for what data we're reading. We're still reading from the old database, even if you now, as you change your app, go to read from the new database. So the next step here, is it visible? We're going to migrate where we read the data. So kind of to go back here, yeah, kind of here. Before, it was pointing originally all of the queries we're going here. But now we're going to actually say, just for the replicas, the reading traffic, hey, let's go read from here and not here anymore. So we can, there are independent pointers to go, instead of pointing from here, point to that one. So inside the shell script, it's just saying, move both the read only and the replicas over. But not the master yet. So this is all the read traffic. So migrate, select customer zero, still looks the same. Remember, if it all looks the same, that's good. If it's not the same, then something's wrong. So yeah, still selecting no problem. And then our last one is migrate the master. So migrate the master, same command only, instead of saying read only, a replica, it says master. So master is, you've decided to change all of your writing traffic over to your new place, OK? And by the way, before we even do this, right now, I have moved my read traffic from replica read only. I've moved it here, but maybe I have a problem. Actually, you can still reverse, because this is a pointer. So you can decide, oh, I want to reverse, and I can go back to reading this. So in case you're worried, you can try it, and if it has problems, just move it back. And then you can debug, try it again, no problem. But once you move the master to the new one, well, now all of your stuff is there. So that's what we're about to do right now. Migrate the master. Pretty fast. Even in production, this should be a very fast switchover, maybe a one second or less delay, just to propagate all of the configuration information. Because again, all of the replication, all the copies, have been happening all of this time. Okay? Yeah, so I've migrated the master, I can still see that. Everything still looks okay. That is the whole demo, by the way. Not very much. So we went from one big database to splitting it, in this example, splitting off tables. It can do horizontal scaling and split that way too. Just don't have time to actually show in this example. So how does all of this work under the covers? This is actually one of the keys of the tests. We call the replication or the test replication. It is filtering all of the replication. So as you see here, originally we had some tables here, and we can set up different types of composable replication, whatever. And we can have, for example, the order information propagate over this way. We can, this is a horizontal shard example right here. This is for what's called the different shard names. It's like half of the traffic go over here, or half of the traffic go over here. So because this might have two shards, split shard one, some of the row data, and split the other, the other half. So it's all doing filtered amounts of replication. So it's not doing everything. So all of this is composable. Oh, yeah. When you have multiple shards, how do you handle good question? So the question is whether you have, actually, do you mind if I answer that at the, right at the end? We're really close, if it's going too long, forgive me. Does this generally make sense? So with this D replication, these are the benefits. You can have materialized views, you can have real-time roll-up, resharding. Again, this is the before was actual examples of resharding in real-time under the cover. Backfilling of lookup indexes. So index is the test index. It's how you decide which shard to go to when you're looking up data or inserting data. And sometimes your primary index is not what you want. Like you have authentication table, you have a user ID, maybe you have user email. If you just want to look up where your given email is, you need to also figure out which shard. So the secondary lookup index, the Vindex, is what that's for. Also, it allows for a different schema deployment and data migration and yeah, change notifications in general. So the beauty of all, how all of the test works, especially with this V replication to do all the sharding and to keep everything in sync. Does that generally make sense? All right, all right, this is the end of the talk. I'm just gonna say some things and then we'll open up the questions, is that okay? All right, so yeah, we have a Slack channel that's at vites.io. Vites.io is where the open source project is hosted. We have a WeChat group. You should come to us only because WeChat has problems with more than a hundred users at a time or some other limits. So come to us, we'll add you to the WeChat group. Also, this is again, was the introduction talk and Juten over here will be giving another talk, a more advanced one on when we actually bring down for disaster recovery, bring down shards or bring down masters and et cetera, what happened. Yeah, I can edit this to show you that. And then also, we're going to have another talk about translation of documents for vites. Right now, unfortunately, all the documents are only in English, so it would be great if we had some translation stuff. And so if you would come by with that, we can talk about how we were going to be doing that. Also, just like I'm running on my laptop, we would love for you to try bringing up a cluster on your own laptops, and we can help you out with that. Read the code, and you can read some blog posts from Square Cash about their deployments of vites, all the problems. And actually, even Jingdong, they just gave a talk on other problems and how they solved them with vites, so you can, and this person here can talk about them if you're curious. I'm on Slack on the Vites Slack channel, at Oliver. And I know Twitter is not accessible, but if you can access it with VPNs or whatever, you can use these things to access. Does that make sense? All right, so open to questions now, sorry. Whoever you are, sorry. Can you repeat your question? Oh, can you? Yeah. Points when you have sharded tables, vertical and horizontal. Right, good question. So the ideal case is you don't do anything wrong, but you have single shard stuff. Like if you have a user table, authentication table, you can co-locate, like based on user ID, like all the user ID data for both tables being one shard. That's the best option. But of course, once you go across, how do you handle that? So we have, what's it called, three modes, or two modes here for cross-shard, what's it called, cross-shard transaction modes. We have a best effort where you can have each one independently transact, and of course, you have the default, or not default, the two-phase commit option, so it can two-phase commit across. That's for insert, but for joins as well, all of the, let me go back to the architecture diagram here. Here we go. Yeah, so you see actually that VT gate, the proxy, actually does talk to all of the tablets. And remember, it does parse all of the SQL, and it actually also collects all of the results from all of the different places, and can do all your joins at that place, if necessary. Does that answer your question? Yeah, so yeah, because each gate goes to all the different tablets. And remember, there's not just one tablet here, you can scale this, I think Jingdong, you guys have like a thousand VT gates or some, yeah, something like that. So it's not just one, you can do many more than one. So does that answer your question? Okay, great. Any other questions? Yes? I have a monthly master cluster replica from Amazon to Alibaba. So I want to support, I want to use VTest. So can I migrate to this without downtime? Jitendra, do you want to answer this one? Alibaba, you have a master in Alibaba and a replica in AWS. And that one is running in EC2 instance. Okay, and what is it that you want to achieve? Okay, so VTest does not support multi-master because multi-master is not a great architecture in our opinion because it can lead to issues when you have rights going to both the masters and then if they're conflicting rights. So for a given shard, we always send the rights to a single master. So even if you have a multi-master architecture, your rights would go to one or the other and we then we would allow you to fail over to start sending rights, say from AWS to the other one or vice versa. Yes, the case we use the multi-master because we have the user in China and we have the user in our China. You know, like some China user, Chinese user, so they want to upset our application. So they just insert data into the data by in China. Okay, so that's why you don't have any conflicts because you have tenant-based architecture. But we still, for the manager, so they have to outside, they need to see the data of vulnerable, so that's why we have. I mean, I will answer a more general question, which is that you can run your masters in one cloud, replicas in other cloud. You need to make sure that you have an IPsec tunnel for them so that you're not sending packets unencrypted across public network. But you can fail over, VTest will allow you to fail over from one cloud provider to another and will also allow you to spin up clusters across cloud providers. Yeah, understood. Actually, we can actually talk about the sharding, right? I mean, you can actually shard based on your tenant. You know, you are saying that you know there are some Chinese users, right? And so you can have them in one shard and others in different shards. And remember, each of the tablets, each of the masters, they can, you can individually select where they are. So again, you can have the unified view, but again, based on this one, you can actually have a custom sharding scheme. You can write your own sharding mechanism. So if you know that user one is a China user, user two, not China user, you can shard in that way. Does that make sense? OK, yeah. I want to ask about, because I see that you are moving out, moving out the table to the database. So I want to understand what needs to be changed in the application to the, first of all, we have the version one collected to the database with the table, but we have the version two collected to the table. That's the beauty of VITES, right? As long as the name of the table is identified, if the name of the table does not collide between key spaces, your application doesn't need to know. Your application can continue to write to VT gates, and VT gate will send those rights to the correct database where that table exists, as long as both key spaces don't have the same table name. If there is a same table name, then you have to specify it in the table name, correct? So if DB one had table one, table two, and you moved it to DB two, after that split happens, your app doesn't need to change at all. OK, I see. Thank you. I'm very concerned about the latency. So how VTES can help me? I'm very concerned of the transaction latency. So transaction latency at high right volume is what you're worried about, right? Yeah, so VITES itself adds a little bit of latency to every query or every transaction, because you have to do, if you go back to the architecture diagram, there are two network hops that it has to do, right? Your application is connecting to VT gate. From VT gate, it's connecting to VT tablet. And from VT tablet, it's connecting to MySQLD. So it adds about 4 to 5 milliseconds of latency to every query. But because you can horizontally charge, your write QPS on every master is a lot less. And that's why your latency ends up being a lot less than it would be if you were writing on a single large master. Does that answer your question? So yeah, it may add a little bit of latency to any single query, but it will increase your throughput by quite a lot. Is that answer your question? OK, any other questions? All right. If not, thank you very much for coming. Again, I'm Toliver, and this is Jithin. And we're from PlanetScale. We do Vitesse things. And I think we have some Vitesse here. We may have a few, but not too many, unfortunately. Anyway, if you have any other questions you'd like to talk in private, feel free to come over. Probably outside. I think the next talk is coming. OK, thank you very much.