 It's an open source project, which is part of the Cloud Native Computing Foundation. VITES was co-founded by Sugus Sugumaran, who and I worked together at YouTube back in 2010 when we started VITES. Since then, VITES has grown quite a bit and VITES is now used in production by companies like Slack, companies like Square Cash. I don't know how many of you actually know about these companies, but these are very popular companies in the United States that run really large workloads, Pinterest, YouTube, all of these companies use VITES. Here in China, JD.com is one of the biggest users of VITES. They run tens of thousands of nodes on VITES. So I'm going to spend some time talking about VITES, the VITES architecture, but the two main points that I want you all to take away with is that VITES is used for horizontal scaling of my SQL databases, one. And VITES is also used to allow you to run stateful workloads in Kubernetes. As you all know, it's easy to run stateless services in Kubernetes. It's hard to run stateful workloads in Kubernetes, but companies like HubSpot.com and JD.com have been using VITES to run databases in Kubernetes over the last two or three years. VITES is born to run in Kubernetes because VITES ran under Borg, which was Google's orchestration framework, which was the predecessor to Kubernetes. So because of that, it runs really well in Kubernetes. So what we are going to talk about first for about 10 or 15 minutes is why. And then I'm going to show you a demo of a truly globally distributed database. So everything to the left-hand side of that vertical line is the application, which is talking to everything on the right-hand side using the MySQL binary protocol. Everything to the right-hand side is VITES. Well, except the load balancer, you are running your own load balancer, but VTGate onwards is all VITES. And it gives your application the view that is talking to a single humongous database when in reality, it could be talking to a database which is very largely horizontally sharded with hundreds of replicas and so on. And not just one database like this, multiple databases or multiple key spaces with multiple schemas, right? Your application still thinks that it's talking to one humongous database, right? How does it manage that? So the very first component is VTGate. So VTGate talks MySQL binary protocol to app servers, it just basically presents itself as a MySQL server to your application. VTGate also has a full MySQL parser built into it. It parses your query, looks at the where clause and decides which of the shards the query needs to go to, okay? Then it sends the query over GRPC to the VT tablet associated with the MySQL to which it wants to send the query, okay? It does not send it directly to MySQL. It sends it first to the VT tablet. VT tablet acts as a guardian or a protector of the MySQL. Every time there was a YouTube outage because of any reason, we built in protections that would keep your databases alive in VT tablet. So what are the kinds of things that VT tablet does? VT tablet does connection pooling. VT tablet gives you hot row protection. It serializes hot rows so that you can, it serializes hot rows so that you can, your database doesn't go down if a single row is being updated. It also put timeouts on your queries and your transactions. So it also puts timeouts on your queries and transactions. In general, anything that a conscientious DBA would do to keep your MySQL healthy is done by, is done by VT tablet for you. So that's, so what is shown on the right hand side are four shards. Each shard has one master and multiple replicas. There are two types of replicas, replicas and big data replicas. You typically run your batch jobs against the big data replicas and the other replicas are candidates for becoming master. Okay. One of the coolest things about Vites is that it does not make decisions about how data is sharded for you. It allows you to make those decisions based on your domain. So just the way you have a MySQL schema, we allow you to have what is called V schema where you describe to Vites how you want your data to be sharded. So what you do is that for every table you tell Vites, hey, shard this table using this particular column and depending on the type of the column, we allow you to use a different sharding function so that the data is sharded uniformly across the whole key range. So for numeric key types, it's hash for var binary and var cal types, it's a different type. There are six different predefined sharding functions that we give you, but not only that, we also give you the ability to write your own sharding functions. And because of this, it's very easy to spin up clusters which are jurisdiction aware or GDPR compliant clusters using Vites, okay? So this is a little bit about the architecture. Now we will look at... So one thing that this diagram does not show is that for each of those shards, there is one master multiple replicas and those replicas can be distributed across multiple data centers. They don't all have to be in the same data center. And Vites has this concept called cell and each replica lives in a particular cell and we are going to... I'm going to show you a cluster where replicas for each shard are running either in US East or US West or in Asia East, okay? So I have talked through most of these points. Here is how Vites allows you to run well in Kubernetes, right? Stateless entry point to allow scaling that VT gate is a stateless proxy. You can run as many of them as you want. Some people run VT gates as a sidecar to their application servers. Some people just run a form of about 40 or 80 or 100 of them behind the load balancer, the way I had shown it in the application in the diagram before. You can VT gate, you can use it to make sure that your read traffic is only load balanced across local replicas in local cell. You can even have a group of VT gates to make sure that only certain key spaces are visible and certain are not. So VT gate is a very versatile proxy. It's stateless, perfect for Kubernetes so that your applications can connect to it. VT tablet is the MySQL minder that I talked to you about. Just your MySQL... So we run your MySQL and VT tablet in two different containers in the same pod and it just works really well. Your MySQL availability is really high. We support native backup restore health checks and observability so that Kubernetes services etc. Know when a VT gate has joined the quorum and when the VT gate has gone down. All of that is supported. And we end up pushing straight to the edges. The connection strings are in the application. So which key space, if you want read after write consistency, you just give the name of the database. So if the name of your database is userDB, if you give the name of the user database as userDB, you get writes and you get read after write consistency. If you say that the name of my database is userDB at replica, then your reads get then your reads get load balanced across the replicas. So all of that you get. And now let's jump into the demo. Let me show you what I'm going to show you today, which is that we are going to run a global stateful Kubernetes application. I showed this application to some of you. You all can go here and check it out. It's basically an application for rating little puppies. And you can all go there, 34.80.128.130 slash puppers. And you can rate these. Please make sure that you give them good ratings. They're all good dogs. So this is the application that we are going to talk about. And I'm going to show you how this is organized in Vitas. So we have created two key spaces, one called doggers and one called lookup. The doggers key space has four shards. And if you click on it, you can see these four shards. Look at how they are named. So the key space, if you take the hexadecimal key space, 00 to 40, 40 to 80, 80 to C0 and C0 to FF are the four slices of the key space. So that's how we name shards. If you click on each shard, you can see that for each shard, there is one master and multiple replicas. And if you see this second column, which is the cell, you can see that the master is in US West and the replicas are in US, some are in US East and some are in Asia one. So each shard is a truly global cluster. Now what I'm going to do is I'm going to show you how this looks like if I connect to the VT gate through a MySQL client. Take a look at the interesting version string here. We kind of lie. This is the server version, which is a Vittes 5.5.10. This is a blatant lie. We are running 5.7 underneath and you can give whatever connection string you want. But the point is that to your client just presenting itself as a MySQL server, you can do things like select count star from ratings and select count star from doggers, I think is the name of the table, puppers. So these are scatter gathers that are being sent to all four shards. The counts are being sent back to you. You are not even aware that you're talking to a sharded database. I just wanted to show you that. And now what I'm going to do is I'm going to start a script for writing into this database. So this is going to keep running while I'm running this demo. And now we are going to abuse this database. I'm going to first delete a master for one of the shards and see what happens. Then I'm going to bring a whole data center down and see what happens. So the idea is that we will see some errors, but it will self-heal without me having to do anything and the errors will go away. That's what we are going to demonstrate. So the first thing that I'm going to do is I am going to actually first, before I do that, I'm going to show you a planned reparent. So there are times when you know that you need to bring down a master. How do you achieve that? So you come here and you say, what I'm going to do is instead of this master running as US West, I'm going to choose this to be the replica, which I want to make the master, 9001. So I come here, I say planned reparent, I choose 9001, and I say reparent. And what is happening when this is running is that VT gate actually starts buffering the rights that are coming to this shard and VT tablet associated with the old master waits until all transactions in flight are finished, VT tablet associated with the new master waits until all the replication logs have been applied and then it makes that the master resets the replication for the whole cluster and then makes the entry into the topo server in which all the information about the cluster topology is written. And then VT gets once they realize that a new master is there, all the queries that VT gate has been buffering, they start sending it to the new master. The point being that your app never sees any errors, right? We continue to write these into it while we planned reparent the master. Now we will do the same thing for another, you will do the same thing for another cluster and so that we have two in US East because I am going to kill US East, okay? Again same thing, this is continuing to write correctly, no problems. Now I am going to kill one of these masters and let's see what happens. We can see the delete pod command, right? I am just going to delete that part of one of the masters that we created. What we should see is that we should start seeing some errors in a bit once that goes away. Still thinking about it, still deleting. So this has been a little flaky. This was the one that I was using before. So yeah, sometimes we have problems taking down a website. So yeah, I have had problems connecting to the Kubernetes cluster, but let's see. So what I was trying to do is basically I was going to, so while this is happening, let me show you what mechanism we use for making sure that this works correctly. So it's another open source tool called Orchestrator and that shows you, it is integrated with Wittes and it shows you the cluster topology as it thinks it is. So you can see that there are these four shards, each has nine instances and if you look at the topology, you can see that it actually shows you that here is the master and then here are replicas in two, it colors them differently because they are in different cells, right? So what I am trying to do is I'm trying to kill these green replicas from all of the clusters, I mean not at first I was just trying to kill one of them, but I don't think that this is going to happen, but what we can do is, I don't know whether it's your hotspot. Okay, awesome. So the pod, we are getting pods, so I'm going to kill one of the masters now, okay? So the VT tablet demo 701, that's the master that we saw, we had, this is the master for the 4080, okay? So I'm going to kill this now and once that pod is deleted, we should start seeing errors here. You shouldn't have to restart, we should just start seeing errors here. Oh, yeah, maybe this is, because we changed it. Sorry, only because we're using some Google Yoon, so not Alley Yoon or anything. Do you think you'll be able to control this? So what we are trying to do is basically show the errors, but I think what happened is that when we changed the Wi-Fi, my SSH to the host from which I was running this died, and so this is not being updated, okay, now just up there, no, hold on. I don't think that that is in the command history. Sorry about that. Okay, don't look at the screen now, we have all the secret information. Of course, everything was done at the last minute, just like everyone else here, right? What are you talking about? So this is our super-duper insert script, I think I can take the password out of the command line parameters. Don't copy the password, don't remember it. That of course doesn't work, remember what we needed to change last time. So I'm just going to do it like this. Sorry about these troubles, but as we, yeah, so yeah, we're trying to right now take down a particular pod, hoping that we can see some errors. The thing is that that pod has already gone away and has already been repaired, so we are not going to see any, yeah, I don't know why this is running. Our error is... Anyway, it doesn't matter, so what I'm going to do now is I'm going to just kill the whole cluster. We can show in here that Orchestrator recognized that it went down and it isolated it. And even in Vitesse, if we go here, we can see that it is now the master is 7002 and not 7001. So this master was the one that was chosen and the right traffic was being sent to here. So even though we can't continue to write to it, this is what was done. Now I'm going to delete the whole cluster. And now the whole cluster is deleted and you can see that Orchestrator will soon start complaining about it and will fix it. So just to show you this, let me show you if I do a Get Pods here. You can see that all these pods are in the process of terminating. And you can see that all these instances are dead. And you can see that some of the stuff is running, but all these pods are gone. But all of these clusters in short order have now been reorganized. And you can see that they all have fewer, so these three are dead, so it cannot get to them. But there is a new master and there are replicas that it's writing to. So all of these shards have reorganized themselves and they are serving traffic. So it's a basically resilient, globally distributed cluster that is being mediated by Vitesse. Let's give it one more try to see whether the goodest auger can actually write with the history. Oh, I need to set my SQL. That was the problem. How many minutes do we have? Six. Six minutes left. Okay. Just in case we can start having people ask questions while we're waiting. Does anyone have any questions so far? Okay. Give me a second. How do you add or remove shards and does Vitesse take care of moving over data when shard count changes? So when the shard count changes, it basically mostly either, we basically Vitesse supports what is called split sharding or merge sharding both. You have to provision new shards and then Vitesse does the job of resharding for you while continuing to serve traffic from the old ones. First it does a stale copy, then it does what we call split replication. And then it allows you to, when the replication catches up, it allows you to failover from old master to new master. Does that answer your question? Yeah. And is the service available during that process? Absolutely. Yes. That is the beauty of Vitesse. Because of all the VT gates are serving all the traffic, you can choose to migrate just the read-only traffic, the replica. And then at some point, once everything is done, you can move the master traffic over. Does that make sense? Other questions? Yes. Sorry. Yes. May I know how the storage engine to support the back end? Because when you, for example, remove a shard. So how do other customers take over the load and then continue to work with the data exists in that shard? Can you give me an example? Because for example, you just remove a cluster, a replicas. So for the existing data inside that shard, how are the cluster? So when I removed the replicas, when I removed a cluster, replicas from all shards which were in that cluster got removed, right? So there were always some replicas which had all the data, which were part of the serving quorum. So that's how it continues to. So I said, US East, I just deleted everything in US East. But there are replicas in US West and Asia One were still part of the cluster and were still operating correctly. And what we do is, yeah, we, yeah. Does that answer your question? Yeah. Great. Thank you. Any other questions? No questions? Okay. So contact us at jithinatplanetscale.com. And we have a project where we want to translate all of Witte's documentation into Chinese. And Rony He, who is one of our colleagues, is going to be giving a talk later today about that project which will allow you, which will translate all of Witte's documentation into Chinese. We would love your help. So please attend that talk. One more question. Sorry. Can you repeat that question? It's not. I see that's the Witte's, it makes a lot of amazing things in the mass square. So my question is, the Witte, do you have the Witte have the plan to support neither Poshrash, Foksim? It is on the roadmap, but not for the next, at least not for the next year. So it's not that hard for us to support it because of the architecture. But what we need, so there are two things that we need to support. One is the protocol that a Postgres client stocked with to Postgres server. That should be pretty straightforward. Then we need to extend our parser a little bit so that there are certain SQL syntax that Postgres supports, but MySQL doesn't. So we need to extend our parser to support that. That is the easy part. But the harder part would be the replication logs, the right ahead replication log that Postgres uses for doing replication. We need to start parsing that and doing the split replication that we need to do when we do resharing. So that is the hard part. There is about eight or nine months worth of work that is needed to do that. But right now our hands are full with MySQL. But we'll probably support it within a year, two year and a half as our company PlanetScale grows. But in the meantime, if anybody wants to help, it's an open source project. Thank you. Thank you. Yeah, I only have 30 seconds left, if any last questions. If not, feel free to come outside and talk with us. We'll be right outside. And we also have a booth at the hall. And at 4.45, we'll have the translation of documentation talk. In which room? In room, I think, 5.06. Give me a second. Well, we'll check and get back to you in a second. But yeah, we'd love to have you here. Any other questions? So otherwise, vitess.io is the website for vitess. There is a link for a Slack channel where you can join. There is also a Wee Sheen group. If you're interested, come talk to me and we can get you on. OK? Thank you for coming. Thank you very much. Bye-bye.