 All right, so hello and welcome again everybody. This is the the test charted my sequel on Kubernetes seminar and My name is Randy Abernethy. I am a CNCF ambassador and I'm a managing partner at RxM Claw native consulting and training shop, and we're basically here to hear sugu sugu moran's discussion about the test sugu was one of the co-creators of the tests at YouTube and now is the CTO of PlanetScale, and so Sugu if you would like please take it away Thanks, Randy Hi, so just to Quickly give a background about myself and with this We started Working on with this it on 2010 so it does is about nine years old as a project It was actually born out of the need that we had at YouTube where we were Managing many mysql instances and it was beginning to go out of control There were just too many outages and you were up against the wall so Mike my co-creator and I decided to think about what can we do to leap ahead of all the problems that we had We were currently facing and essentially that's how we test was born Quick background on what we test is for those of you who may not be familiar with it is Even though it started as a way to To just take care of immediate problems that is immediate problems that we are facing at YouTube What it has now become is basically a full-fledged sharding middleware for mysql What that means is this that? If you install with us on your mysql instance then you can shard it underneath and As far as the application is concerned you get the impression that you are still talking to a unified database and That's a huge win for a lot of people because Without something like the test You have to make pretty big changes to your app if you just if you want to shard your database So that is one big problem that we test solves Yeah, because of the fact that we were running it at YouTube. It is Naturally massively scalable. So any solution that we came up with had to scale into Millions of QPS and tens of thousands of nodes and also it had to be highly available available, which means that downtime Beyond a few seconds is something that was with something that Someone something like YouTube would not tolerate So we test comfortably gives you five names of Availability in most of the production workloads that it is currently running and Last but not the least with us is cloud native Cloud native is actually a term that It's somewhat of a buzzword But actually there are some very specific technical terms that you can use to evaluate whether a product is cloud native or not And so I will go into Some of the very specific things that makes with test cloud native Alright So these are some of the stats about with this like I said it was started in 2010 and has been gaining popularity so initially when When we both decided to start with this we open sourced it but It wasn't really the internet wasn't really to have other people adopted. It was mostly because we wanted to Expose the work that we had done and also not have to reinvent that wheel if you went somewhere else, right? And so it was mostly we were submitting code into the open source But just it was mostly just for YouTube's own sake We used to just import that code back into YouTube and then launch it inside but around 2015 or 16 Flipkart found us and then they said hey it looks like this is something that we can also use and So then they start they worked on adopting it and over time more and more companies came and Today actually saw the the biggest pride of with this is who are the adopters of with this? We have a pretty impressive list of technology leaders that Adopting a witness Like here is some example Slack is actually a major contributor to with this and also a major adopter Their intent is to actually fully migrate to with us And I think their goal is to do it by end of this year at this point. They are 30% migrated They actually talked about some of their stats in our last meet up just a few days back they said that they peaked at 1.2 million QPS and They even gave some number of shots. I think it was in the high hundreds Or maybe a thousand Not shards thousand instances nodes of with this nodes Square cash actually is a classic case of With this use where they had a single instance my sequel That was running on the biggest hardware they could buy that money could buy and that was running out of Capacity and what they did was they launched with us on top of that huge Database and then sharded it underneath. I believe they they are now at The last number was that they were at 32 shards and we're looking at going to 64 So the one of their Statement is that sharding is addictive Because once you shard things run better and then you say, oh, let's start some more and let's start some more so it just keeps going and you and it's mostly a lot of games with very little a little loss and Pinterest actually recently migrated to with us and one of their cool quotes is Like reshorting is even less almost boring So that's one thing that with us makes really really easy for you to do So here's another thing so cash up actually had Which is from square grew so fast that Jack Dorsey himself tweeted about it So now the top two apps in the top charts are using with this and we are going after Burger King next Cool so here is a list of Some of the big adopters that we test this slide actually needs to be updated There are a few more users that have that are beginning to come on board, but we need to Get them to list their logos logos in here. These are the people that Have agreed to have their logos listed The most important takeaway from this is that we test runs in a large number of environments because each of these companies runs has a different Production setup some are on bare metal some are or on Cloud like running EC2 instances some are running kubernetes Some are running other cloud orchestration software There's at least one we test instance I believe that runs on mesos and there's another user that's looking at running it on nomad so and there's a reason for this is because Which I will go into which which is somehow related to making with us cloud native also so Why is with test cloud native? Why does with us? claim to be so compatible With kubernetes is what we are going to talk about So this goes into the history of with us. So when we first developed with us We were it was actually developed only for bad metal but in 2013 What happened and we were Running with us on bare metal in YouTube's own data centers We were not part of Google data centers in 2013 Snowden happened and Google pretty much mandated that We are not allowed to have data Sensitive data outside of Google data centers itself that we had to migrate everything inside Google's cloud and That faced a pretty big challenge because we just was open source and we were running it on this bare metal and I don't know how many of you have worked at Google, but If you develop software that runs inside Google's ecosystem, it is pretty much Almost impossible to open source it. The reason is because Google has so many Custom APIs right there is chubby. There is stubby. There's all these logging API's monitoring API's And they are all internal to Google. So if you write if you wrote software that ran inside The Google's work cloud that is pretty much not open sourceable But when we decided to port we test into Google's blog we faced a challenge We said, okay, do we just close source it and then Relight we test to use Google's internal APIs Or do we try to keep it open source and somehow make it work inside Google's cloud? So we went for the latter decision and it was really really hard because What that meant was actually building abstraction layers for every API of Google that we were using like throttling and whatever else that That was needed right like Calling into Borgman calling into chubby calling into stubby. So we actually built abstraction layers for each of those things and and That is how we finally managed to get it working inside Borg so one cool Property of with test that even I believe not even a storage system inside Google has is with test LAN as a stateless app in Google's Borg which means that we just as far as Google's cloud was concerned with this was just an app It was not a special storage system With any privileges about What it could do with storage it was not so other storage systems in Borg where actually infrastructural elements API is that you could call into that work provided as an API for storage The test is the only system that ran as an app So why that? Story is important is because when Kubernetes came out We were essentially ready for it because because we could run as an as an app in a cloud all we had to do was Change our adapter Change our layers to call into Kubernetes and we were ready So which is the reason why you can see here Kubernetes 1.0 came out in July but we actually announced that this was ready for Kubernetes before Kubernetes 1.0 came out and And We went out and told people that you can use with us on Kubernetes and People started using it actually Stitch labs was actually the first company to adopt and they were pretty running They are a small company, but they are running. They were running pretty high QPS. I think there is a Percona talk by stitch labs where they talk about Running about hundred and fifty thousand QPS and they migrated their entire traffic into with us and in 2016 and so which means that With us as the oldest we test instance is now nearly three years old and it has been serving So they are 100% on Kubernetes on with us and on on their own storage and It's been now going for three years now. No incidents. No data loss and So they've been pretty happy Then came HubSpot Well, the big work that they did was they actually did a lot of compatibility work to make a large number of queries work then JD.com came and they actually scaled it massively also So these these companies have been running with us on Kubernetes for many years now, so it is not a theory It is now a proven thing and you can take this to the bank So nozzle is actually the last entrant that start that migrated to Kubernetes Their coolest story is that because they are fully on Kubernetes They can move from one cloud to another whenever they want. In fact, they actually started They started on Azure because Azure gave them huge credits and eventually they ran out of those credits and Within one hour, they migrated from Azure into Google Cloud GCP and I don't know if they are running GKE or their own Kubernetes most likely GKE But that is how easy it is for you to migrate from one cloud to another if you are completely on Kubernetes end to end so that's That's one One recommendation, which is if you instead use a Storage that is provided by the cloud like RDS or cloud SQL then migrating out of that is a lot harder Whereas if you are 100% on Kubernetes, it is so easy to move But then after all this Here Kelsey keeps saying don't move your storage to Kubernetes. It is problematic Run your stateless app on Kubernetes, but storage I would not recommend, right? but There is a reason why he says that is because of this Essentially what he's saying is if you just took a storage system and just moved it to Kubernetes, you're going to regret it Because there are there are things missing there are there are gaps between Kubernetes and the storage layer that need to be filled and Existing storage systems just don't run as is on Kubernetes. So that's essentially what you're trying to say And to explain why we test which is the gap. I'm going to show you the witness architecture here The way we test works is even though it can scale really really massively the witness architecture is fairly simple It's just a two-layer architecture You can see the app server here. It connects to These vt gates these vt gates are stateless app servers So it can connect to any of them through a load balancer and as far as the app server is concerned a vt gate Represents your entire cluster. It's as if it's connected to a single mysql instance But under the under the covers what vt gate does is when you send a query to it It actually has a metadata called the v schema, which is kind of a Schema but for a sharded system and it uses that schema to figure out where that query should go and then once it figures it out it sends it to a vt tablet and Each vt tablet runs its own mysql instance and what this vt tablet does is actually completely manage that mysql instance and Yeah, and also it has other benefits like connection pooling. I can also do housekeeping work like backups and restores and those kinds of things You can see here that there are many types of Vt tablets that is a master vt tablet which accepts writes and there are some replica vt tablets that accept read traffic Etc. So this is one case One architecture which you cannot Directly get if you moved your mysql to kubernetes. So let's say you want to Move you want to deploy mysql on kubernetes. So kubernetes the best The best part type to use is a stateful set right the stateful set a workload gives you the ability to have a bunch of instances But what the stateful set does not allow you to do is designate one of those instances as a master As far as kubernetes is concerned. They are all equal. So if you want to designate one pod as a master What you have to do is actually Pull that out into a separate pod type And if you did that then that part becomes a master and then the rest of the parts become replicas So then you can say okay direct traffic to the master pod direct traffic to the replica But then what happens if a master goes down, right? If a master goes down Then you cannot Failover to a replica and say now this part is designated at mass as a master So kubernetes doesn't allow you to change a pod type on the flight Which means that you have to wait for the master to be restarted So to for kubernetes to restart that part as a master and then once it comes back up Then you can send traffic to it. So this in other words if you deploy just mysql and did this type of separation then You cannot really have a highly available system because if a master goes down you're pretty much out of luck and The only way you can solve it is to and the way we test solve it is because it has this VT gate in the middle It can actually track where the master is and if a master goes down We can reparent and designate an existing replica as a master and then immediately VT gate can switch to that replica and start sending traffic and this can happen literally within seconds and Even during those seconds what VT gate does is it actually doesn't return errors to the app server It just waits for the new master to be elected. So the only thing the app server notices is a brief latency spike and The other problem is Databases like mysql perform best when their storage is local Right, so but the but the problem with Kubernetes is if you use the local storage then as soon as your Part is shut down Kubernetes wipes all your data, right and so in this case That is kind of scary for anybody running on mysql So most people that have so far moved mysql to Kubernetes have ended up using mounted Volume like EBS or a CSI based volume that gives its own persistence guarantees There is still now there is now a local PV that Kubernetes supports But it hasn't seen much production use yet and it still has a limitation that unless your Your the part gets rescheduled on the same node. You still don't have access to that The way we test solve this problem is it actually does not rely on the local data ever being present So if a part goes down what we test can be made to do is to always say well if a if a new part comes up always go to to a backup Restore from that backup and then after you restore point it to the master let it catch up and after it has caught up Then start serving traffic. So basically we test does not rely on local data being preserved for for its uptime and We there is an additional feature in mysql called semi-sync replication that guarantees that When a transaction gets committed at least one other replica has it so which means that you will never lose a transaction If a part goes down You can run with tests to also use a mounted storage and some people do run it but But you can also run it in this high performance high availability mode where You use local data and then if a part goes down just throw it away and then bring up a new pod Restore from replica and keep moving forward And How reliable is this methodology? This is how we test runs in Google Cloud in the Borg We we have never ever restored A crashed mysql we always threw it away and then brought up new new instances and this is where and this is where running a Sharded mysql makes more sense if so we don't run huge instances all our instances are like two to three hundred gigs of data Size which means that if a pod goes down a new pod comes back up and is ready to serve within like a few like five or ten minutes So that's how long it takes for a new pod to be spun up so this is what makes with us cloud cloud native and The final one is obviously the discovery mechanism where there is this topology Where VT tablet when it comes up it registers itself with the topology and these VT gets discovered them as new things come up and Last but not the least it works across multiple cells. So you can have a full worldwide deployment of with us running tens of thousands of nodes All right Moving forward You can actually Start using with us right from the beginning. It gives you benefits even if you have a single instance of mysql because You gives you connection pooling it gives you deadlines. It makes overall your mysql run better But then as you scale you start to be Using more and more features of the test The witness distributed system capabilities it can route to replicas it can do load balance it can do a master promotion and then eventually you can shard your system and As and as you shard your system as far as the app is concerned you still get a unified view and Eventually end up into a fully worldwide deployment So this is how you would roll with with us and as you can see on the left-hand side I wouldn't want to say that app servers are completely unaffected sometimes you when you distribute things you do have to rewrite things in the app because You are making trade-offs in performance and not everything will continue to work that like it used to work before But the impact is definitely minimized All right, so I am going to show a demo and if I have time at the end of the demo I will talk about the regret feature which is When you decide to shard a system You have to make some trade-offs and decisions and sometimes those decisions are not the right ones So the latest feature that we developed allows you to actually Reshard your system using a different key Which is why it's called the regret feature. So I will talk about that at the end if I have time So here's the demo. I'm going to show This is actually the use cases. Let's say we are building a marketplace Where we allow merchants to sell products to customers something like Amazon so This is a simplified schema where you have a product table, which has a list of products Customer table where all the customers are stored and a merchant table that has all the merchants and merchants list Their products the customer comes in and they say when they place an order An order row is created and this order row has foreign keys Which means that there is an order ID, which is its own ID The CID which is the customer ID points back to the customer that place the order The PID is a product that they bought And the M name which is the merchant name is the merchant from which they bought the product from So if you scale this massively you're going to have billions of orders. You're going to have Maybe a billion customers millions of merchants and thousands of thousands of products right And then when you decide to short the system You are kind of forced to split it into three databases One is the product database which contains the product One is the customer database for the customer and the merchant database Because they have different keys different root keys The orders table is a decision that you have to make about where to put it You can say that orders can be with merchant or orders can be with customer or orders can be in its own shot If that is a decision that's also an option and each one has a different tradeoff So if you say that an order can be with customer Then you then it's easy to join a customer with order. You can easily query whether You can the customer can come in and say show me all my orders or you can say Show me all the orders for a group by customer. So you can do those kinds of queries But it becomes harder to query for example What's the product description For a customer because the product description information is In this database or if a merchant comes in and says, oh, I need to find out all my orders Then it becomes a rather complicated query because you have to go to each customer shard and Find out all basically essentially do a scan of all shards So in this case, I'm going to assume that the product is unsharded because the product table is small The customer has to be sharded because they are massive and the merchant is also sharded so I'm going to accelerate and Show you some pretty cool things we can do with with us But Unlike these stunts. These are stunts that you can try at home So I'm going to bring up a so I already have a cluster that I've brought up I can show you the cluster here. So this cluster is the same cluster that I Showed in in my slide. It has one product shard It has two merchant shards and two customer shards The 80 it means it's a hexadecimal. So eight is in the middle So which means that zero to eight is one shard and eight to infinity is the other shard So in this and I have an app here This is my app My e-commerce application except it is very raw What you have to do is you have to imagine that you are a customer and when you sign up you go and Execute the query to insert yourself into this customer If you place an order you have to execute the query to Insert an order into the table. It's kind of a pretend application but the cool part of it is you can Basically, this app allows you to send random queries to this with a sharded system and as It executes it shows you what it does. So just to prime our I have this Let me show you For example, what a v schema looks like so here is a v schema a v schema Basically is a schema is a description of how a system is sharded. For example for the merchant key space, this is the v schema where it says that the merchant key space has Is sharded by merchant name and it uses in with us. We have these things called called vintexes Which are essentially a sharding key that not only define which column it is sharded by but also How to what algorithm to use to shard in this case? I'm going to use a unicode md5 sharding key, okay And I have some data here You can I have some precooked statements that insert data Into this database. I'm going to run my mysql client against vt gate and say import all this data And then if I go and refresh this page All my data is imported in here You can see that they are there are different colors every time A new row appears that row is highlighted by this app. This is just for convenience. So if I refresh this everything will go blue So here's some data. So as you saw those are just regular mysql Regular insert statements, but they went to different shards like sugu went to dash 80. Demer went to 80 dash uh, and uh, but you can see that the orders for sugu are within Uh, sugu's shard and the orders for demmer are within uh demo shard The foreign key is the c id which is customer id id one and in this case id six, which is the demo shard Uh, and now I can run uh queries here If I say select start from product. So as far as uh, the app is concerned This is one database even though it is now split into three into five different databases If I say select start from product What we test as here it shows the executed queries. It says oh product It knows that it is in the product database. It says i'm going to execute on product and i'm going to send this query If I say select start from customer that is actually a Scatter query because uh customer is a sharder database It sends it to both shards of the customer and then sends you the result If I did a join of the customer With the order table you can see that i'm saying customer join with orders On the customer id as a foreign key Uh, this query with us knows that Since all the orders are of customer are within each shard It knows that it doesn't have to do anything crazy It basically pushes on that entire join into each of the shards And then returns uh the query. So this is uh, the We test is a query routing capabilities However If I did this query So what this query is doing is is joining customer with orders and then Going to product and says Fridge the product description for each of those orders and please display that right? So that is something Of a cross shard query. So what we test will do i'm going to execute it and then explain what it did So what we test here does is actually splits that query into two parts And the first query is a join of the customer with order and as I showed before It's a within shard query. Uh, so that part of the query runs as a scatter query And then it receives three rows that came as output And for each of those rows, it has to make a round trip into the product Database to fetch the description Which is actually the three queries that you see that went into the product So this is uh, so this works in with test, but it is not optimal and Here with test gives you an option a solution Which actually no other sharded system gives that I know of Which is This product table is small What if we could Just copy this product table inside each of the shards of this customer Then we can tell with test to say a join with the local product Then this query will be executed extremely efficiently So we have this new feature that is called the materialized view. You can say materialized product Into materialize any table into any other table using certain rules, right? So I'm going to give you That materialize command So here is the command, right? So here I say I'm going to materialize uh, the product dot product table into customer dot product and It is a reference table. What that means is that I'm going to materialize all the rows Into each of the shard, which means that it is not a sharded table. It's an unshardered table With identical copies within each shard. I say I also want the table to be created because the table currently doesn't exist So I'm going to execute this As soon as I execute this and then refresh it The product table has materialized you can see that it has copied these rows The one thing you should remember is that this table can be multiple terabytes Which means that this materialization process can take a while. It can take a few hours. Maybe even days And during that time this table is not visible. So if I Continue if I reissue this query It will still go It will still behave as if this product table doesn't exist But once this materialization is done and this table is caught up I can now go say expose that materialized view to the application, right? So I say Oh, I did I do something wrong Oh not merchant It should be sorry I did a copy pasta problem. Oh, it's the other one Expose yeah, so expose this So I when I created the materialized I named this as a workflow I use that workflow to make changes to it and I say expose whatever the effects of that workflow to the application Now I'm going to re-execute the same exact query And now you can see that it is the plan has changed. It is now running it as a fully As a scatter query where it is joining with the local product itself I can still say that I want to specifically join with a product table and and that is possible And so this is so this is actually a very powerful feature because things that were extremely inefficient can be made super efficient by just through Through this materialization feature And if and more important thing is if I make changes to the source tables now, I'm going to insert a new product So as soon as I insert it Immediately that role is replicated into these target tables All right, so now the next query is what if the merchant wants to look at all their orders? That's even harder perry to implement because It's actually a full scatter on all the merchants and for each row that I received. It is a full scatter on the Customer key space, right? So that's that's a query. That's basically unimaginably expensive And again, the question is what if we materialize this orders table into the merchant table? So in the case of a product table, it was easy, right? You just copied all the rows But in the case of a merchant table, you can't just copy all the rows Because one is Order is sharded by customer ID on and on this side for this materialization to work effectively It needs to be sharded by M name, which is actually the sharding key for merchant So Vitas allows you to do that also, which is essentially saying materialize merchant the customer orders into merchant orders But when you materialize use a different sharding key And also create table, right? So now I'm going to execute it. So if you see that I refresh it here and You can see that this table is materialized But you can see that the rows are not the same way as they are in the source. You can see that Now all mono price rows are in the mono price shard and the new egg rows are in the new egg shard All right, and now if I now I'm going to As you saw I need to expose this the expose In the demo it feels like an extra step, but in production you need to actually do these as two steps Because you have to wait for the materialization to finish and then you have to expose it So now that I've exposed it, let me if I rerun that same query It now becomes a simple cross shot query a simple Yeah, a simple cross shot query in the previous case. It was a Must have multiple nested loop join very expensive All right so if you look at So If you look at the the underlying feature that helps you do this is called the vreplication feature The first case we actually copied all the rows in the second case. We applied filters to these rows and said Send some rows here send some rows there What else can you do with vreplication? So vreplication not only allows you to materialize and filter It also allows you to transform What does transform mean is that you can specify basically a select query as an expression as a transformation expression and And the select query can be actually can have aggregations and stuff which means that you can actually Create real-time rollups. So here is an example of a real-time roll up What I'm going to do is I'm going to materialize This is again the same materialize command, but here what I have done is instead of giving a target table I say It I gave a target expression saying that materialize sales as this expression. So in this case, I'm going to select product ID and the number of orders And the total price for each order and I say materialize it as a sales table And now I'm going to also expose it Expose it now if I refresh it There is a sales table that has popped up that has automatically rolled up all the orders into their counts and amounts And then I can now run queries directly into the sales table and if you actually You can actually watch this row. I'm going to change an order Am I going to yeah, let's say I'm going to add an order here If I add an order see what happens The order was added here that added order was replicated into the merchant key space And the sales amount has been updated accordingly with the total of all orders And all this happens efficiently because of how the replication works is because this happens by Where we replication subscribes to the bin logs And just basically looks at each event and applies just the incremental change to the target tables So it is very efficient and very fast. The only downside is that it is Eventually consistent, which means that if there is a network partition between customer and product Then the sales table will be delayed in its update. So that is the trade-off for this feature And just before concluding The weird application feature that we use Can also be used for the regret feature that I talked about if you looked at If you flip the story around where you say Let's say I Chose a cid for the orders and I did not like the key I could have created a workflow that migrated orders into the merchant orders and then we repeated by mono price And all I had to tell with this is instead of sending rights to customers, switch the rights to merchant And now I've resharded a table That was Sharded one way into a table that was sharded another way So going back to our Concluding remarks is the there are many use cases that we are going to deploy with the test Materialized use real-time rollouts and resharding Is what I talked about what you can use This index this weird application feature for schema deployments data migrations backfilling of tables And also change notification where you can subscribe to see What changes are going through inside my database? So that's uh, that concludes my presentation. I am now open for questions Awesome super awesome. Thanks to you. Um, the test is very cool. So we've got a bunch of um A bunch of questions in the q&a. Let me just kind of run through here So one of the first ones from gene is What versions of mysql and or other targets are supported by the test? Oh, yes a good question with us supports Mysql 5.6 All the way up to 8.0 It supports maria db 10 dot x all versions of 10 dot x all versions of perkona We test can also run on all cloud databases like rds and cloud sequel. So it's a pretty wide range of The only restrictions it has is that you need to have a row based replication and gt id turned on So any mysql or maria db that supports those two features it can run on Gotcha and any other uh tricky combinations? Um You know kind of interesting versions of mysql with other storage engines or anything like that been used at all Yes, uh, we have some proof of concepts. Uh, where we were able to run Mysql as well as maria db against rocks db uh, and And some people have actually run it in production Uh, as far as I know there was no the test limitation itself Those who ran it on production recently migrated to inno db because Rocks db itself has some limitations. So if mysql the rocks db will work for you. You can totally run with us on it Awesome, great. Thanks. Um, okay, let's see. I'm going to mix it up a little bit Just try to make sure we get as many different people represented here as possible in the time that we've got next, um Next question is what what is the benefit of using the test? Over say amazon mysql the main difference is I would say scalability is definitely the the biggest one The the amazon's mysql even aurora Beyond certain limit would not scale for you as with us. You can keep scaling forever Another big advantage is even if you can scale with amazon with things like rds and aurora You don't necessarily get to choose The instant size that you like right once you grow beyond a certain size You cannot really split your database within amazon itself in that case. You can still bring with us in and run Your database sizes at a sweeter spot that you would like to have And but then the biggest one is the cloud native nature of with us is that you can run with us within Your kubernetes instance end-to-end which gives you a huge migration Advantages So whereas if you use something like rds, you are kind of stuck within that cloud Right and you know, I would imagine that because with the test you can tailor your sharding to your actual application patterns that even on a apples to apples comparison, you know Scale wise you could potentially get some pretty fantastic performance benefits. Yes, totally totally and and also it With the connection pooling features it runs your mysql is much cooler and also The biggest nightmares of db is is the sudden Overload spikes And with us handles them really really well I don't know if i'm allowed to talk about this, but I There was a recent outage on slack and they said with us Just didn't even blink fantastic I would like to hear that. All right, so the next one And um Okay, so um, so I think the question is around replication That it says how frequently is data backed up, but I think really it's uh, you know, it's it's asking about the replication And um, if the master goes down, um, you know, is any data lost? So maybe yeah, so that's a good question to talk, but go ahead. Yeah, it depends on actually your sensitivity So you can run with us in a way that you will not lose any data Both whether you are using Mounted storage or local storage you can configure with us to not use any data So if you're using local storage Then semi sync replication is highly recommended, which means that If you configure it that way No commit will be considered complete until one other replica at least has that data Which means that if you lose that master, you are guaranteed to be able to fail over To another master and find that transaction that completed Or you could use Mounted storage, but if you use mounted storage, you definitely have to run with The conservative settings of my sequel Where every commit is flushed into the storage In that case you will not lose data and The somewhat sad part is that if you configure it that way, it does slow you down even further right So I would imagine that basically the only way to lose data is to lose every single replica In at the same time in the chart at the same time, right? Yeah in that case at the same time so the uh, so typically what I would recommend is if you are using um Like if your workload has a large number of replicas anyway because you have uh, you are running Uh, a lot of read traffic and if you are cross sell Because you you need that then local storage is by far a huge win Not only will you are you going to get huge performance benefits? The fact that you have these many replicas gives you huge survivability So it is very hard to lose the more and more replicas you have and the more replicas you have cross sell Uh, it is almost impossible to lose all your data Uh, whereas if you are smaller Uh, where you need only one master and nothing else you send all your queries to one master In that case, it is recommended that you go with mounted storage. That is uh, a much more cost effective We are running with this right, right, you know, I um I was really intrigued by the whole kind of materialized sort of master data distribution approach that's um That's that's super slick and especially the fact that you can have different You know partition keys on the the data set, you know to make it You know fit with the uh with the model of the the parent that's that's pretty cool stuff And I'm guessing that in this case that all of the replicated data there I mean even if you had the the exact partition same partition key that these these uh materialized guys They they're not eligible for election, right? They're just they're just copies of the data Correct. Correct. They are they are they have to be treated as copies But you could um You could go through a uh, so the there are two workflows that we plan to support. One is this materialization On an almost identical workflow, which is actually uh migration Right. So instead of saying materialize if you say migrate the exact same thing happens In but in that case what allows you to do is actually cut over your rights to the other table, right? So materialize comes the master and vice versa Yeah, so it is a it is a failover in that case, but it is not a it is not meant for unplanned Uh, it is more of a planned thing that it's not really efficient for me to write data In this format. I it's more efficient to do it that way. So it's more for a planned uh scenario Yeah, it Wonderful ability for you to explore like what is going to be more efficient and then just sort of like, you know, learn as you go Exactly. Exactly. Uh, like a lot of arguments that people have about how to short things. Sometimes they get very heated So this at least lets you uh At least says well, we'll do it this way and if things break we can go we can always change our mind You know, it's it's a lot like microservices, right? You want to be able to iteratively discover the right architecture. Yeah, exactly Cool stuff. Okay, let's see. Um, so how about a general kind of commentary on latency? in big deployments Oh, yes, that's a good one. So the way uh, so with us adds a static overhead Because it is uh, which is actually basically the number of hops that you have to go through Before you reach my sequel It is typically about two milliseconds so if So if your query, um, some people Do are sensitive to it those that are used to directly sending to my sequel and my sequel responds in a millisecond Then this suddenly is three milliseconds But most people find it absorbable because typical Uh, my sequel latency runs in the five to ten millisecond range per query And in that case, this is something that uh, you won't even notice. Yeah, so you're seven to twelve or something. Yeah And I would imagine if you get into a massive scatter gather operation the test crushes my sequel Yes, totally. That's because uh, when you do scatter gather, uh, now it's all parallel, right? Yeah, you can run 8 10 30 30 x faster than you run before. Yeah. Yeah, that's awesome Um, there's a question about joins across shards, which I think maybe that happened before you demoed all that um, how about, um Let's see So there's a question here on are the demo scripts available somewhere so that people could you know, maybe retry your examples and stuff Yes, I uh, I uh, just pushed my latest branch into ssv demo 6 Which is on the planet scale. Uh, let me show you I'll show it on the screen and maybe you can post it later So branches So this so the entire demo that I showed is in in In this branch of Of the test So if you can see my screen That's where it is and uh, maybe I can send it to you if you send me an email. I can send you the link to this Great branch. Yeah, I think there's like a summary email that comes up after and so maybe we can drop it in there Did it out to everybody that way? Um, maybe we've we've got time for maybe one more question. There's a lot of people interested in Now that you've got them so excited about the test, they're like, hey, can I use this with other database engines? Like wins postgres support coming and you know other databases. So can you comment on that? So, yes with us architecturally is not married to my sequel It is mostly an engine that looks at sql and decides where to send it and what to do with the results So there is nothing architectural that prevents with us from being ported to my sequel from with us to be ported to postgres Uh, it's just that we have had our hands full With so much my sequel people asking for features that we haven't had time to spend To port it to postgres, but it's definitely and as more and more people are asking We are beginning to think that maybe we should do this sooner than later If a community member, um got really excited about this. How many man hours do you think they'd have to put in to build the my sequel shim? And is that easy is it is the test built to be able to be pluggable there? I mean, I know you mentioned a lot of the other kind of facades that you could drop in but is that one of them? Yeah, yeah, I think I would estimate something like six Six to 12 person months. So if people on it, maybe in six months they can have with us on postgres Would be my guess Very rough very very back of the envelope, but that would be my guess All right. So, uh, a four people working hardcore could get it done in a quarter Uh, yes, but then they uh, there is the ramp up time Right, so christmas we expect it by christmas. Yes Okay, cool. Well, it is uh top of the hour again. So thanks to go super awesome You know the tests great stuff and thanks for the fantastic presentation. Hope everybody enjoyed Sorry, we didn't get to all the questions, but I feel like we you know covered a good swath of them. So Um, great show and um, hope to see you doing another talk on the test soon Thank you very much. Yeah, take care Bye everybody