 I would like to introduce Zhatan, CEO and co-founder of PlanetScale, who will be talking to us about the test today. Take it away. Awesome, thank you Chris, and thank you all for being here this Monday morning of Veterans Day. What I'm going to talk about today is how to spin up clusters that are jurisdiction aware. More and more jurisdictions are passing data locality laws. And what we are going to look at today is how you can build clusters that are jurisdiction aware, but which you can build without changing the application logic. So your applications continues to think that it's talking to a single monolithic database, whereas depending on some marker of locality, your data can be distributed in the correct jurisdiction. So that's what we are going to talk about. A quick introduction about myself. Most of my career, I have been a backend infrastructure engineer at companies like Google, YouTube, Dropbox and the United States Digital Service. At YouTube, I manage the SRE and DBA teams. These are the groups that operated with test clusters at pre-massive scale at YouTube. Planet Scale is a company that me and Sugus Sugumaran, who is the chief engineer behind and the founder of the Vites open source project. We both founded Planet Scale in February 2018. We are a venture backed company backed by Anderson Horowitz and SignalFire. We are about 30 employees mostly in Mountain View, and of course we are hiring. And with that, I'm going to give it to Chris, who wants to talk a little bit about the upcoming KubeCon. Yes, it is the week before KubeCon and all through the house. Not a creature was stirring, not even a mouse. KubeCon is next week, everybody. There are still tickets available, I believe. There are still day zero events that you can get into right now. So grab your tickets, enjoy KubeCon and at San Diego Glory. I'm in Detroit. It's snowing. We're going to get six inches. So I cannot wait. Next slide, please. Also, CNCF is setting up Kubernetes forums. These are, think of them as like small KubeCon, but very like regionally based. So everybody gets as much knowledge as possible. There are two here that we're coming up here in December, Sydney and Seoul. Sponsorships are available still, and I believe tickets are still available for both events as of right now. Please check them out. Learn more about them, and we'll have more of those here in the near future. Yeah, and just to talk a little bit more about the forum events in Seoul and Sydney, me and another engineer from PlanetScale Deepsea would both be at both these events. So come there and get some more Vitas goodness. Yes. And before we kick off, I would also like to invite all of you at an event that PlanetScale is putting together with AWS and Vitas at on Monday, November 18, 6pm to 10pm. There are more details on our social media and on our web page and so on, so please come and attend. We would love to see you there. And now, without further ado, I would like to jump into the webinar. So here is the agenda for today. What I'll be covering in the beginning is talk a little bit about the Vitas architecture and concepts that we need for building jurisdiction of our clusters. Then I'll present the problem. We will think through and talk through the cluster design. And finally, I'll show you a live demo. So let's talk up, talk a little bit about the Vitas architecture. Those of you who know Vitas probably know this diagram fairly well. Everything on the left hand side of the green line is in our application server, which thinks that everything on the right hand side of the green line is just one humongous MySQL database. Those application servers talk to VT gates using the MySQL binary protocol. VT gates are stateless proxies. They implement the MySQL parser so that they look at your where clause and depending on the where clause, they send the queries to each shard. We are going to go into a fair amount of detail about what a shard is. But each shard is basically a traditional MySQL cluster. With masters and replicas, but each MySQL D in the cluster gets a minor process for VT tablet that does the job of protecting the MySQL D and it protects it by doing a connection pooling, adding row limits by doing adding timeouts for transactions for queries and so on. Basically, even for a single sharded database, it's impossible to write a query of depth if it's running under Vitas. All of the cluster topology is saved in what we call a lock server or topology server. This can either be an it city cluster or a zookeeper cluster or a council cluster. VT CTLD is the control plane that gives you a web UI. We will be interacting with this during the demo and it also gives you an API server where your command lines can talk to. And this is the overall architecture of Vitas. What is not shown here is the dimension of cells. So for any shard, this collection of the master and the replicas that constitute that shard can be distributed across multiple cells. So keep this in mind as we go further in the demo. Anybody has any questions about the architecture? Now we are no questions right now. Alright, let's move on to the next slide. Okay, so let's talk about what a cell is, right? So a cell is a group of servers and network infrastructure. It's a failure domain isolated from failures in other cells. And you can define a cell depending on, you know, depending on how you're running and what makes sense to you. So a full data center could be a cell. It could be a subset of a data center. It, you know, it can be a region or an availability zone. It could be a Kubernetes cluster. It could even be a rack in a data center. So a failure domain, which is basically a group of servers and network infrastructure that is isolated from failures in other cells. That's what a cell is. And if you think about this, this is the representation of physical infrastructure, which means that there is a physical location associated with it, which means that you can locate this in a jurisdiction. So the mapping between the geographical location of your data is sort of represented by a cell in a Vittes cluster. So keep this at the back of your mind. The next concept that I want to talk about is the key space. So a key space in Vittes parlance. So as you know, Vittes deals with sharded databases and a key space is a logical database. If a key space has only one sharp, then it's exactly identical to a my SQL database. But if you're using sharding, a key space is multiple SQL databases, my SQL databases, all with identical schema, right? In either case, a key space appears as a single database from the standpoint of the application. So application can think that it's talking to a single humongous database. It doesn't need to be aware that it's talking to a sharded database. So that's what a key space is. Now let's talk a little bit about what a key space ID is, right? So key space is sharded by key space ID ranges, and each row is assigned a key space ID. So key space ID is basically a street address of a row, right? And it's used to decide on which are a given row lives, right? If you are familiar with no SQL databases, which are basically key value databases, then it's an equivalent of no SQL sharding key. The reason that I have stricken out that row is because we do not save this key space ID anywhere, right? It's Internet of Hit S. The application does not need to know anything about it. And it's not stored but computed. So we'll talk about this, how this is done in the next slide. But key space ID is this each row has it. And depending on the value of the key space ID for that row, you decide which shard that particular row goes to. It's not stored but computed. Those are the things that you need to keep in mind about the key space ID. Now let's talk about another concept called V index or Vindex, right? And it's basically a way to compute the key space ID for any row in a table, right? So a Vindex for a table is defined by two things. One is a column name. And that's the column that you want to use to shard data in that table by. And the second is the name of a sharding function. So there are six or seven sharding functions that are predefined in BitS. And you can add your own. So we'll talk a little more about sharding functions in the next slide. But this equality holds where the key space ID for a row equal to the value computed using the column value for that row as an input in the sharding function. So key space ID equal to f of the value for row in that column. Okay, so for example, if you have a table called customer and a sharding column that you're using this ID and a sharding function is hash, then for a row where the ID is one to three key space ID would be hash of one to three. Right, straight forward. Now let's talk about what a shard is. So each shard represents a key space ID, an ID range. So each shard has a begin value and end value. And if begin value less than key space ID less than end value for a particular row, then that row belongs in that shard. Right, so that's what a shard is. In terms of key space ranges, each shard has one master and multiple replicas. And the thing to remember is that a shard can be located in one on one or more cells. So these replicas and masters that constitute a particular shard can live in one or more physical cells. And we talked about, we said that a Vindex is basically a combination of a column and a sharding function. And so these are the sharding functions that are predefined in Vitesse, binary md5 hash, numeric, numeric static map, unicode loose md5. So if you basically have a int or a long int as your column on which you want to shard, you would probably want to use hash as the sharding key. If it's a name or a var care or a var binary, you probably want to use the unicode loose md5. Right. There are some people who wanted reverse bits, you know, they just wanted to reverse the bits of the primary key. And so we built that, that for them. So these are the predefined sharding functions. So you can choose a column, you can choose one of these functions, and that allows Vitesse to calculate the key space ID for a given table for, for rows in the given table, and that's what decides which row that particular row, which shard that particular row is going to live in. Right. So that's what sharding functions are. So now, having understood all these concepts, let's go to the problem at hand that we are trying to solve. Now, before I jump in here, let me once again ask Chris if there are any questions. You're doing just fine. All right, since there are no questions, I'll move to the problem that we are trying to solve today, which is building a jurisdiction aware cluster, a database cluster that stores data for a given country in its jurisdiction. The client application did not be aware of jurisdictions, right. And this is important. So you need, you don't need to rearchitect your client applications, while you rearchitect your database to comply with data locality requirements. So for this demo, we are going to have four jurisdictions and eight countries. The four four jurisdictions are Americas, Asia, Africa and Europe, and the countries that are two countries that I have picked corresponding to those jurisdictions, right. So eight countries for jurisdictions. And these jurisdictions map to cells that we say are located in that jurisdictional area. So the cell that is located in the Americas is called US West. The cell that is located in Asia is called Asia one. The cell that is located in Africa is called Africa one and the cell that is located in Europe is called Europe one. And what we are going to do is we are going to create a key space with four shards, and we are going to instantiate one shard in each cell. And then we are going to use a custom geo V index. So this is a V index that I wrote. It took me about six hours to write. This is the cool thing about with us that once you index is an is an interface. And so you can write a custom V index with fairly little amount of effort that implements the shouting logic that you want to implement. So it took me about six hours to implement this. And so this is a custom V index that I call custom geo that distributes data correctly using the country field in each row. And this is how we are going to design the cluster. Our V index will use a country to jurisdiction map. That basically maps countries to jurisdictions, and you can see this map. It's a JSON. So this sample data that we would be using Philip Roth and Gary Schengart are two of my favorite authors. So I picked them out for United States and then I went to Wikipedia and picked up authors from various countries. And so we have two, two rows per country. And without further ado, I can actually show you how this looks in practice. Again, I assume there are no questions. Otherwise, Chris would I interpret me. Correct. Yes. All right. I have SSH into AWS and I have, you know, four or five posts here in which I'm going to spin up this cluster. And as I said to you, what we are going to do is we are going to start a full witness cluster. That means that first, first we will start a zookeeper quorum, which will act as the topology server. Then we will start a VT CT LD, which is the control plane. Then we will start a VT gate, which is the stateless proxy to which our client will connect. Then we will start the VT tablet my security combinations for the key spaces that we will use. And once all this is ready, I will show this to you in more detail. So I'm going to run this shell script called start cluster. And this will do everything that we had talked about. It started the three zookeeper clusters. It started the VT CT LD. And then now it's starting two different key spaces. There is one key space called lookup, which is a single shattered key space. And then we will start the actual shattered key space in which we will store the data that we want stored. And if you see the V schema object passing by here, V schema is how an operator tells Vites about how their key space is shattered. So it tells you which for each table in the schema for that particular key space. You have to tell Vites which column to use for sharding and which sharding function you want to use for sharding the data in there. So all of these, you know, this is where we started the sharded key space. All the shards. And this is the V schema that we applied to this key space called user DB. And I would like to point this out to you here that we first define a V index that we are calling GDPR VDX and it's of the type geocustom. So geocustom is hard ported in the new code that I wrote saying that that's the type of the new index. So we are creating a GDPR VDX of type geocustom here. And we are, this uses our data from this GDPR config.json for mapping the countries to jurisdictions. And then what we are doing is that there is this table called customer, which is the only table in this particular database. And we are saying that use the column named ID and the index named GDPR VDX that we defined up here as the sharding function. So this is how the V schema works, right. So now what I'm going to do is I'm going to show you all this through the VTCTLD interface. So you can see that this is the witness control panel. You can see that there is there is one look up key space with one shard, which lives in US West. And then there is one key space called user DB, which has four shards, which lives shard zero to 40 lives in Europe shard 40 to 80 lives in Asia. Shard 80 to C zero lives in US West. And shard C zero to dash lives in Africa. Right. So there are four shards. Each of them are instantiated in cells that are in different localities. Right. Now let's take a look at how the schema for lookup looks like. So there are two tables in schema. One is called customer country map. And here we are keeping a map of the ID to country. And we are also saying that this ID is a sequence that is so witness has this cool feature called so basically in my SQL in a single sharded my SQL database you can use auto increment to get a unique ID for each row. That doesn't work if you have a sharded database because each each of the shards can have the same ID if you just said auto increment. So witness has a feature called sequences, which you can use to say to create an ID which can be distributed across multiple shards. So this customer country map. So that that is the that the table that backs those IDs is also lives in the lookup piece base. Now let's change to user DB, and then there are four different shards and each each of the shards is going to have the same schema. So this is the schema for the customer table. There is the ID, there is a full name, there is a national ID, and there is the country so national ID is basically my, I just thought that we need to put some PII personally identifiable information so national ID is the current or social security number. So this is the country itself in the primary keys ID. So this is the, and if I go to any of the four shards associated with this key space you will see that they have identical identical schema, and you can see that there is a V index called GDPR VDX of type in that is applied to this particular key space. So now that you have seen this, and you can see that there are zero rows in all of these right now, right, because we haven't added any rules. So let's go back and let us let us look at what data we are going to insert. And if you look at this data, it's basically this ID will not be used, we will be generating the ID, but the name, there is the full name, there is the national ID and then there is the country. And this particular Python script for client GDPR.py is going to read this file, and it's going to insert it on the VT gate that is running on port number 15306 on this machine. So I'm going to control C, control V, and it connected to this VT gate here, and it inserted all these 12 rows. So we are going to read these rows out from the database. But before we do that, we can also see here, if I refresh this, you can see that customer country map has 16 rows in it in the lookup database. And if the user DB, each shard now has four rows. The question is whether they got into the correct rows or not, right. So, for that, what we are going to do is I'm going to actually connect to my using the MySQL client. I'm going to connect to the VT gate, and I'm going to run this SQL script called showJDPR.sql. So let's take a look at what's there in that script. So Vites allows you to either access a key space as though it's a single database, you can see here where it's saying use user DB. So if I say use user DB and select count star from customer, it's going to show me 16. But if I say use user DB dash 40, which is the dash 40 shard, and then say select star for customers, it will show me only the four rows that live in this particular shard. And if you remember, the first shard is associated with Europe. So you would expect for the first four rows to be from Europe. The second four rows to be from Asia, the third four rows to be from US and Canada, and the fourth rows to be from Africa. So that's what we are doing here. So let's run this showJDPR.sql. And as expected, you can see that France and Germany rows have gone into the shard, which is located in a cell in Europe, the cell called Europe 1. Then the Asian rows have gone into a shard that was located in a cell in Asia called Asia 1. The North American rows have gone into the one called US West, and the one in Africa have gone in Africa. And if I instead just treat it as a single database, I can see the count star as 16. So I also want to just connect here to MySQL and show you how this looks. So this is just standard MySQL client. This is a lie. The server version underneath that we are using is different. You can, there is a VTGate parameter that you can use to tell VTGate what it should return the version number as. This was the first MySQL version that we supported. So that's what a Vites server response as the version number, but you can change this. We are actually running on MySQL 5.7. So if I say use customer, and I say select star from no use, I should say use user DB. I should say select star from customer. You can see all 16 rows. At the same time, if I say use user DB-4D, select star from customer. It just shows me the four rows in that particular shard. So just wanted to show this to you. And what is interesting here, there are a few other things that I wanted to show is that the code for the Geo custom, yeah, I first wanted to show you how the client looks like. So basically we are inserting into the customer country map first, and then we get the ID that was generated and then we insert that row into the user DB customer. So this is all, it's very straightforward logic. There is no mention anywhere of the cells or there is no mention anywhere of the jurisdictions. But that mapping comes from here. Here is where we are mapping the country to jurisdictions. So given a country, we figure out what the jurisdiction is, and then there is a range that is spanned by that particular jurisdiction. So depending on the offset of the country in this list, we use the byte corresponding to the range in that particular jurisdiction. And this particular shard uses that byte as the first byte, and then the user ID, and then hashes it to create the key space ID, so that all the key space ID is generated by any country have, you know, they fall in a particular range, and that's how they end up being in the same shard. So for those of you who are interested, this is how it is done. I could have just had, you know, a byte associated with every country, but this gives you some notion of adjacency and so on. So if, for example, UK was in Europe, and if UK decides to leave Europe and get into its own jurisdiction, having it at one of the ends of the Europe's ranges would allow you to create an own jurisdiction for Great Britain. And that's why I have this, the jurisdiction ranges and so on rather than hard coding a byte for each country. And as you can see you can do similar things with zip codes, if California, as California's data locality laws go into effect, you could have a California jurisdiction and rest of the US jurisdiction and do the mapping based on, say the zip code, zip code field for your data. Any questions? Yes. Chris asks, can Bethesda be configured to support multi tiered sharding, for example, each jurisdiction may contain a set of shards defined by an alternative key space. Absolutely. So the each if so only the first byte is used to decide the jurisdiction and the rest of it depends on the user ID. So if you have hundreds of millions of user in a single jurisdiction, you can, you can split that shard and you can put both of those shards and the same jurisdiction and the rest of the logic just continues to work. Any other questions? There are none in chat. That was the only one in Q&A right now. How would you have, like you mentioned California, how would that work if like how aware does Bethesda have to be of the underlying infrastructure or how do you define that, I guess. Correct. So the way you do that, let's say that you would create a cell in Sacramento, right? And let's say that you create another cell, say either in Oregon or maybe New Jersey. Right. Right. And your application, you know, you have a table where people's addresses are and there is a zip code column. Right. Oh, OK. And then you'll write this JSON file which maps, you know, it allows you to create a shards for California zip codes which get located into the shard that you have started in the Sacramento cell. Cool. Yeah. So it's about about a day's worth of work and we are making this as general as possible. I mean, this is, you know, while I created this particular custom index. So I also wanted to show you the go code that does all this which fits into the rest of the test. And that's like 180 lines of code. Wow. And a lot of it was boilerplate, but the guts of it are, you know, so this is, this is, this is all boilerplate. But the guts of it are, you know, here I'm calculating the byte for country. So given a country, this returns one byte that I am prepending to the user ID. So here is the name that we used to define it. And if I look for, yeah. So I first convert ID to country by doing a lookup in the mapping table. I get the country. ID is the input to this map function. I get the country, then I get the byte for the country. And then what I do is that this is the hash of the ID itself. I use seven bytes out of that. And then I append the country byte with those seven bytes. Cool. And then I hash it. And that means that so similarly, if you're using, if you're using one byte for California and another byte for the rest of the country, you would do the same thing. And then just as long as you, yeah, you make sure that your shards are located. And if you remember how the shards are named is 00 to 40 so that that's basically a hex, yeah, slices of the whole key space. So, and you can keep splitting the shards as, as, as you need to. That's a cool stuff. I mean, congratulations on all the work put into, you know, push for tests over to the graduation side of the CNCF project framework to by the way, the, the, the amount of scale that I've seen people use for tests to handle is rather impressive. This, this to me is just like next level fascinating handling all the data locality rules, just with, you know, boiler pray code to kind of out of the box is awesome. Thank you. Yeah. Any questions, not right now know there are no questions 15 people on the call, besides all of us, you know, presenting and helping talking and so forth so there's anything else you want to talk about, you know, keep calm your party anything. Come party with us everybody. Yeah, I mean, seems like a cool thing. Where is it again sorry I missed the, the location. Yeah, let me go here and let me give me one second. No problem. Here's a question. What is happening. Sorry to make you change gears. What is happening if jurisdiction configuration change is it supported for example, if it's has will migrate the data in the background. So like if you added a new data locality said go here. How does that like affect the data that's already there. So you will, if you added a new jurisdiction. So there are two ways in which you can change it right you can either add a country to an existing jurisdiction, or you can add a new jurisdiction. Right. If you add a new jurisdiction you have to keep in mind that you are using bytes. So let's say that a particular jurisdiction that the way the way this has been set up is that of the 256 each jurisdiction gets a quarter right so 64 64 values for each jurisdiction that that's the way it has been set up correctly. So if currently, so let's say that a jurisdiction has 10 countries in it. When you add a new jurisdiction the beginning the start byte and end byte you have to make sure that they start after 10. So maybe out of 64 you say that the previous jurisdiction gets 32 and the new jurisdiction that you're adding wherever it is gets 32. So in that case you don't need to change anything. Similarly, Great Britain I mean the thing is that the so in that case everything works as expected. You don't need to copy data from one place to another, but let's say that Great Britain leaves European Union right and gets out of the jurisdiction. That's a tricky problem to solve unless you had the forethought to put Great Britain either at the beginning or at the end of a particular jurisdiction. Okay, so you would have to do some kind of data move of some sort to call us it maybe. Right, I mean, if you kept this in mind and you as the first country you can just split it up as a different as a different jurisdiction without moving in the data because the byte associated with that country will not change. Right. So if you split the existing shard, we just have that key range be in UK and the rest of the key range in Europe. Would you think it'd be like a good practice to if you have a data center in that country like just go ahead and set it up as a cell whether it's in a jurisdiction or not just so you could predefined it potentially as a place. I think that's it. That would be a good idea. Okay, yeah, that's, I think that's good advice. Okay, cool. Thank you for the question said. Yeah, so let me just quickly. I think I'm looking at the wrong. Yeah. So Kim just said if you're okay with it like we can open the question like open the the voice up for all participants, like if just 15 of us like I'm pretty sure we can manage that. If you're okay with it. Sounds great to me. Yeah, let's do it. All right, Kim hit it. Cool. So talk and they just have to go ahead and unmute their mic. If you are muted, and you want to talk just unmute and ask away. Again, the the Vitesse AWS planet scale Q com kickoff party is Monday, November 18, 6pm to 10pm at We don't have the address here. It looks, oh, it says meze. I can look it up. Hang on. I got it. I will pull it into the chat. Just give me a second. Yeah. So if you have a question, you know, this is the test knowledge base with you here for next, you know, 15 minutes or so please ask away. There's the website and address is. So may say Greg fusion is the name of the place. Yeah, that's pretty cool. Let's see their address is and have to click on the Google Maps link to find it. And of course, Google Maps is probably on Google infrastructure. So it's going to be raining a little slow today. Let's see. No offense to Google. All right. There we go. No problem. So, there's the address dropped in the chat. The website is gas lamp mezee.com. If you have any questions about menu or anything else had there. You know, we're live right now. So if you have a question on mute. And, you know, we could talk about nice QL I'm assuming the tests in general CNC a project graduation cube con the whole nine yards. That's right. We're here. Really excited to have graduated. Yes. A question the chat from Eric, I may have missed it. But were you able to describe how a schema change gets handled. So schema change basically gets handled independent of how the jurisdiction of clusters work. Right. It's just stand with us has commands that allow you to apply schema changes to multi sharp key space. So it it sends the same DDL to all shards in parallel. Some people in production also use tools like GHOST or PTO SC to do online schema changes. And there are plans to build support for online schema changes in the test itself using the new verification feature. Nice. Does it ask a question. Eric, does that answer your question. You can say yes by hitting on mute. Also, yes. Thank you. Awesome. Cool. Any other questions? If not, I guess we can wrap this up. Okay, cool. I think I will close out if you're okay with it. Sounds good. Yeah, thank you. Thank you so much. Great presentation. Lots of information. Lots of demo. So thank you to the demo gods for being thoughtful there. If you have any questions, you know, feel free to contact Shatana myself. And, you know, we look forward to seeing you at a future CNCF webinar and we hope you have a great day and a great week. Thank you everybody. Thank you. Bye everybody.