 So before we begin I kind of want to define what I do here and I want to start by this very detailed drawing of How planet scale works? This is a very you know in our infrastructure. We have obviously at the core of things. We have my sequel There's some other glue in here that makes everything work And then there is this layer that I call edge and I want to call this out because edge is kind of different for everybody and in different contexts, but for us and for me specifically edge relates to the Outermost part of our network. So this is the edge of our network It is if you are connecting to a planet scale database If you are running sequel doing anything against the actual my sequel database You go through edge So this is you this is all of you Some of you are very close to us as in you might have your servers in the same data center And ideally you would provision say you run your stuff in AWS US East You also provision a database in US East and you're kind of next door to to us And that's kind of the ideal situation for most of our customers But there's also a good amount of you believe it or not that are far away from our databases and I am by far. I mean geographically far So this might be for example, you running something on your laptop and you're doing some data analysis or something like that Or which is a little bit more common. You might be using a serverless thing where you don't actually know where your stuff is You've just kind of thrown it out into the internet and the internet is just you know do doing its computation And all of that is now phoning back home to to us So these are kind of two very distinct different problems that we need to deal with And we don't want to really be harmful to either of those So if we do optimizations for the people that are far away We obviously don't want to degrade the experience for the people that are very close to us This specific layer So this is wrapping all of our MySQL. This is Edge. Edge is responsible for Kind of our API, I guess we can call it This API is for directly interfacing with our database Typically, this is something that you use with SQL. It is responsible for authentication So when you have credentials you talk to us We we're not just exposing MySQL directly to the internet that would cause a lot of problems So we have to have this shim layer of kind of our planet scale Sauce that you know doesn't just just here's a MySQL socket on the internet So me personally I care a lot about performance. I care about your experience. I care about fast applications For the most part we can kind of say that everyone here probably cares about running queries against the database you care about things being quickly you probably run some sort of web application Hit or miss they all kind of at the same at the end of the day You want things to be fast that's kind of the sales pitch of our product is that we can we can we can scale really You know large and we can also maintain being fast But as a software engineer working on this layer There's only so many things I could do and if you can tell that is a very sad face of mine that I would like to do So much more But within the scope of my responsibilities You can tell me rewrite it in rusts. You can tell me you know make this algorithm more efficient You can make the computers faster but at the end of the day There's different orders of magnitudes of when it comes to performance a Lot of what I do and a lot of my optimizations are in the nanoseconds to microseconds world I Have a lot of control when it comes that if someone tells me hey make this algorithm not on That is usually for the most part typically in like a microseconds or nanoseconds layer But the real issue is the milliseconds and that's generally for the most part what we can't control my entire stack of software Executes well under a single millisecond so any kind of incremental improvements we make Generally are in single digit microseconds Nanoseconds these are important for us because when you when you think about doing database access doing a select one is Very fast So adding on a millisecond of latency to a select one is very noticeable It's less noticeable when you are talking From Europe talking to a US database that millisecond isn't going to be noticeable But it is going to be noticeable if you're in the data center Or in the same data center or data center next door But I can't control that path. That's that's either clients. That's your internet connectivity. That is your ISP That is your data provider That is out of my hands so where this kind of comes together is our database has an API and I kind of want to Separate the concern that this is a an API for The database layer not for managing our Managing the product so it's a little bit different line here. So this is how I'm going to get access in and out of our database So we have two of these APIs The first one is a pretty obvious You you interface with my sequel and by that I mean explicitly The my sequel protocol Not necessarily my sequel the database as an API. We run my sequel database, but the protocol is its own separate thing This is what you get if you are using let's let's say Python for example And you you install that you know a my sequel client package use the my sequel CLI tool You're gonna be communicating through the my sequel protocol So we all know this you might not know it to a level of of how I actually write my sequel protocol or what the packets look like But it's really an RPC API that like there's nothing really too fancy going on. It has a Honestly a series of commands like for example the common one is comm query and that is its own structured binary packet You can send comm query and that is how you send along sequel It's a really easy thing especially for to implement You just open up a TCP socket and you just put some bytes on it and you expect to get some bytes back It's its own unique thing The one thing I want to point out it is it is a stateful protocol So a good majority of you are probably aware that You open up many connections and you need to you know typically manage some sort of connection pooler and a lot of the state of Your connections or transactions session variables things like that are tied to that open connection So if at any point if that connection is severed The state that is maintained with that connection is lost So if you were executing 10 queries it got through and you ran through five of them and then you have a network interruption Depending on what you're doing if you have or within a transaction things like that You've lost your session. Maybe some of it's half committed It it's really kind of you know for the most part in deterministic from my point of view here So the other API that we have is HTTP This is the language of the internet whether we like it or not. It is pretty ubiquitous just in existence It's everywhere But you probably didn't know that we have this API It's not a restful API, so it's not something that you're you know what you would traditionally find it's it's very different from Our API to manage our product. It is specifically designed for interacting with the database And One nice thing about it is it is stateless. So in this case your state is not tied to the actual connection So similar to how you interact with a browser session on a website you're logged in you might bounce to you know Your connections gonna go away, but you're not logged out So HTTP kind of is inherently stateless in that case People also really like this thing called JSON or JSON. However, you want to pronounce it I like to say JSON, but I think it's wrong But it's a thing people like to do it's really easy to interrupt with you can write JSON by hand It doesn't require you to implement parsers. It's just a very known structure Which means it interrupts with other languages very very well It works with curl so it's really nice in our case. It's kind of convenient. You can just Command line just run something with curl and work with JSON if you needed to Where things get interesting is that it also supports gRPC and if you're not very keenly aware gRPC is transported over HTTP. So it's kind of a it's not its own thing It's kind of another layer of a format on HTTP So we kind of get that I don't want to say for free, but it's it it works with us and it's very complimentary This is all undocumented because this is all pretty new just FYI I Fully plan on getting this documented very very soon, but it's it's complicated, but we're getting there So right now though, everyone is using my sequel. I Hope that's gonna change I have selfish reasons as an infrastructure provider Why why this would be nice, but for the most part like it's not gonna go away No matter what we can't we're never gonna convince people like hey You're gonna adopt this bleeding new, you know vendor lock-in tech. It's just not gonna happen. We have to provide Compelling reasons so at some point we might be able to convince people to do it if we can sell them enough reasons to say This is objectively better. Here's what you're getting but for now It's never gonna go away. We have to support the my sequel protocol But maybe someday that's something that we can we can pull off So here's where I'm gonna rant a little bit So as a service provider My sequel is not a very good thing for us to support So starting with stateful connections as a service provider Your your your all of your session information is tied to that individual connection you have Which means we have no capability of Disconnecting a connection that you have open if we do that is very disruptive to a majority of people's applications depending on what clients they use in Some cases it's not that bad, but for the most cases it's kind of It's disruptive So what that means for us is like we it makes it hard for us to deploy software We can't easily Restart things like you you you typically could all of this can be summarized, but it's not HTTP And I kind of mean that I don't mean that HTTP itself is superior The ubiquity that came with HTTP just kind of gives the ecosystem a Lot of these tools and it's meant to handle Very lossy scenarios Things going up and down if we want to add new servers remove servers deploying code. It's Relatively non-trivial with my sequel My favorite personal part is clients once they have a my sequel connection because everything is stateful They literally never disconnect Some do some connection poolers are a little bit more kind and say you know every hour Maybe it'll recycle connections But in a vast majority of cases Once a client has opened up a TCP connection it kind of assumes it's never gonna go away and a lot of these assumptions are because you are Running your own my sequel server and the only reason why you would disconnect is like I've restarted the server or done something Disruptive there's not typically some layer in between like us where we need to deploy software and do more of like continuous deployment on But clients never disconnect which is which is tricky for us Because all of these stateful connections. There's a lot of them those of you that do any large-scale web application are pretty aware that you kind of need a Unique connection per parallel Request that's happening. So you tend to have applications that have hundreds of connections Dozens of connections at minimum just to be able to handle the throughput throughput because each individual connection is its own State nothing can be multiplexed over it It's its own thing And the last one is a little bit unique in that the TLS support for my sequel is kind of weird in that It's something that we have to support But it uses this thing called start TLS which is a Basically what that means is different things. There's a few different things that implement start TLS But it's a way for my sequel to have port 3 306 Standardized and serve both encrypted and unencrypted traffic in our case as a service provider We don't want to do unencrypted traffic. We want to only give TLS But the way that start TLS works is there's an initial handshake that's unencrypted and Then after those few bits we say hey we require encryption And it's up to the client to also say I agree with this and Then you negotiate the TLS handshake But what that means is because of that behavior? It mitigates Eliminates the ability for a lot of other software to help us. We can't offload TLS We can't do anything like we have to own all of this because we also have to own the my sequel handshake So we didn't invent the HTTP stuff just for fun We didn't do HTTP necessarily to solve this problem Supporting HTTP by itself doesn't necessarily get us more customers It doesn't they're just difficult software engineering problems for us to solve but Some of our product demands were wanting to extend the capabilities of my sequel So the two that I'm gonna call out are serverless so whether you use them or not it is a decent amount of our customers and Platforms such as like versell netlify cloudfire workers, whatever all of those like edge compute things they tend to have a lot more Restrictive environments in which they run and they don't just want you to open up. You can't just run a TCP So you can't just open up a TCP socket and do whatever you want to anything on the internet they limit the functionality to basically HTTP and When we want to augment new features It's not very trivial for us to just add on to the my sequel protocol while technically We can add new of those like basically RFPC commands and add on new packets and add all this stuff We're kind of at the same problem in that we now have to ship a Modified my sequel client for you to view and then like how do you access it? You know, it's not necessarily sequel so This kind of leads to Being fetch API compatible if you're not very familiar with the web fetch API has kind of become the browser standard for a JavaScript API for making HTTP requests so basically it's this this just interface a lot of things implement it Beat for us being able to support a the fetch API opens up the door to a lot of different stuff like serverless or even from the browser directly and We chose for adding new features We want to be able to support gRPC because gRPC is a pretty common thing if you're using a Lot of like Google services because gRPC came from Google all are all kind of just natively gRPC And you're not particularly aware of it. It's just what what they happen to use for transport so So why is gRPC interesting to us and where does that fall in here? so gRPC combines the thing called protobufs or protocol buffers and HTTP 2 This is all nicely on top of TCP and HTTP gRPC is just kind of a specification for How to you know similar to like a rest API or something like that It's just some structure of how to make calls But for the most part if you're if you just wanted to interact with gRPC You don't have to have like a gRPC client. You can just use HTTP and shape or request to match protobuf is where things get a little bit interesting with that in which the data that is transported Can be sent encoded via protobuf and protobuf is a format for It's a definition for how to pack Arbitrary structures into bytes so similar to how the mysql protocol has a calm query You know whatever in theory those could be Defined as protobufs and have kind of a uniform way to encode and decode those structures And that's kind of what protobuf gives us as just a generic API So similar to like JSON we can use protobuf, but we get a binary format instead of this plain text JSON so Is the only slide I promise that has Hex and and raw binary But I want to make the point of what this kind of looks like on the wire because there's a lot of preconceived notions of Doing it this way is is worse so if we just kind of look at a calm query packet in mysql and compare it to what the protobuf encoded version of What we have is a execute request which accomplishes the same goal we can actually see that the protobuf one is shorter and But between both of them what dominates it is the actual payload and this could be the same for Basically all the packets. It's it's both on a response You're usually getting back many many rows and the padding and overhead of the actual protocol is is for the most part minimal compared to The actual payload or even just queries so The only thing that makes this not a one-to-one comparison is Because on the protobuf side we are ignoring The HTTP 2 and the gRPC part of it and there's kind of a reason for that It's a lot more complicated to just kind of give an example of what it is because of the way the hdb2 and 3 work It's not a one-to-one like here is the payload here is what you're gonna get out hdb2 and 3 kind of optimize and improve Headers as it goes along so the longer you have a connection open it will it kind of like compresses and Optimizes a lot of stuff away So it's it's it's a really interesting thing and it's not worth getting into the details there, but it kind of just goes away A lot of cases so hdb1 was tech space and that's kind of where A lot of these these preconceived notions came into play is that you know, hdb Is very slow and bloated and and just very quickly. This is why you know when we think of hdb This is what we see. This is kind of what your browser shows even if you're using hdb2 But for the most part you got the you know, you got the You got key value stuff. They're separated by colons. You got a bunch of metadata It's not an easy format to parse. It's got a lot of bloat None of this exists in hdb2 or three hdb2 is binary and not a plain text thing But we get a lot of really cool stuff because of this so it's not just to do binary format They've kind of superseded a lot of use cases of things like web sockets And that we've introduced bi-directional streaming so you can have one one connection And both client and server could be sending rpc streams back and forth We have multiplexing so we can handle multiple concurrent streams over one actual physical connection hpack is the uh, the binary encoding of headers Which is it's it's genuinely pretty wild A lot of header compression happens and it can get down in a lot of cases to one or two bytes Per header pair as opposed to parsing and splitting, you know ascii text Uh hdb2 also brings along tls 1.2 and this is kind of important because web technology tends to push A lot of the boundaries with security and and that kind of tech and when we're talking the mysql protocol A lot of that is Used on internal traffic So it's not as battle tested with with staying up to date with the leading, you know technology for tls You can do it There's nothing that prevents it just in practice using tls with mysql is hard because it's not the common case Whereas browsers these days especially is default tls everywhere So we we just recently got hdb2 But now we got this really interesting thing called hdb3 and it's not super common. It's This is excessively bleeding edge Um and a lot of it's excessively bleeding edge because It took basically a lot of what it learned from hdb2 So a lot of the header compression and all of that the binary all of that is pretty similar But they decided to tackle the worst part about hdb and that's tcp And to be clear tcp is also what mysql uses So the same issues that hdb2 have or hdtp before three Have the same issues over the mysql protocol. So such things as like packet retransmitting Head of line blocking those kind of things are just inherently issues with tcp So for example, if you drop a packet All the other packets on the network have to wait or on that socket are waiting until you know, the one is Retransmitted and if if you're on high latency is when these get really really bad So hdp kind of tack you'll want to tackle this and They decided to not use tcp And they kind of built this new thing on top of udp And there's a gut reaction to udp if you're not, you know, real familiar is that udp is lossy. So how does this work? And so instead of just saying it's on top of udp, they've created another protocol called quick QQIC and quick is intended to replace tcp. So If you think about tcp, tcp is built on top of an unreliable network. It has physical things underneath it It has a physical ethernet wire, you know, down the layers Network packets can be lost but tcp provides that reliability on top of Raw, you know ethernet packets and stuff like that QQIC is is the analogous and it's it accomplishes that in a very different way To more accommodate modern internet tcp was created in the 70s A lot of things have changed in 50 years With networks network performance network reliability. We use them very differently. So quick is attempted to solve that QQIC also integrates with now a newer version of tls called tls 1.3 And following the trend of web kind of pushing bleeding edge tech tls 1.3 Entirely shakes up how to do a tls handshake to a server So typically if you think about the, you know, there's like this multiple handshake of like, hey, I'm doing this and I'm doing this And with tcp, you got the sin sin ac, you know that whole handshake When you're very in a latent network All of that takes a long time if you're 20 milliseconds away It's going to take you a minimum 60 milliseconds to establish a connection through the handshakes So tls and quick are a unit As a part of establishing a new quick connection the tls handshake goes with it. It is not a separate Do tcp connection do the tls negotiation? They are they are one it is inseparable With tls 1.3 It also has this really cool new feature called zero rtt, which is zero round trip Meaning it can re-establish a connection to an existing something it has seen before Without doing any handshake. You just you connect and you're there So this kind of so kind of thinking about the broader picture is that now that we Fundamentally have an http api Can we start mixing these things together and like http 3 kind of goes once you have http We can kind of just use this stuff for free. It doesn't require a lot of effort for us to Just use a newer version And we got to get all those nice things So I started experimenting what if we combine The niceness that we got from grpc and requiring protobuf as the as the wire format and pairing that with http 3 So now from here, we basically have a Very light efficient protobuf encoded version of our of our request and responses and now over A lighter weight actual transport protocol And this is something that my sequel cannot do because it is based on tcp So I kind of started theory crafting here and this is genuinely was just like I wonder if this can be better at this point like you doing with when everything was based on tcp It didn't seem as appealing to kind of dig in and see if we can do better Given that I know the downside is tcp I know that we can't you know, if you are an unreliable network, there's only so much we can do But now with http 3 I started thinking can we actually if we were to build a driver on top of On top of http 3 Could we be comparable? and it's It it's a low bar I didn't want to be worse um So kind of my thought process here was that the protocol overhead is negligible at this point We got super fast networks, you know, especially you're in aws. You got some 10 gigabit pipe that just basically is plugging into us Send it, you know being even though http 2 Is pretty efficient over the network. It does have some extra bloat It's it's negligible though on this scale if we're adding an extra even 100 bytes or a kilobyte per payload back and forth It's dwarfed by the amount of data and an extra 100 bytes over A super fast network is It's it's not a thing that is a concern these days network latency Dwarfs all of this stuff If you have a network hiccup if you are not in the data center next door You have a five millisecond ping to us. You have a hundred millisecond ping to us All of this is substantially Larger when it comes to uh performance than Uh adding an extra few bytes on the wire tls is slow Using a mysql client. It's hard to get the modern tls It's hard to get tls 1.3. You can do it But again kind of leading the trend of using web technology That kind of leads the The race here, especially if we can adopt quick that mandates the tls 1.3 And like it or not even if http isn't the most efficient protocol It is so scrutinized Http Is so efficient to parse even the text-based protocol is so much faster than a lot of binary protocols Just because it's so scrutinized and so heavily used. I think of http like json People are always trying to squeak out performance out of json Just because json is so ubiquitous not because json is a better format It's just so heavily used so it's scrutinized So to me now that I was kind of looking at this. I was like, what What if what if I did write a driver? What if I played with this? What what would I claim as success? And to me I like to set low bars on things. I like that, you know It's hard and disingenuous to think that I can do Better in a lot of cases than my sql like you you run a select one It happens in microseconds There's not a lot of room to make that more efficient You got to send the string select one you got a Like there's a minimum amount of work that needs to be done without a lot of bloat So to me I just didn't like if I can be comparable Was pretty cool and that to me seemed like a pretty pretty good result because at that point There's not a reason not to do it There just might not be a reason to do it So I didn't want to go into a ton of the All the individual benchmarks. I so the blog post that we have that accompanies this went into A bunch of them in a lot more detail But I wanted to call out Two of the test results that I thought were the most interesting and I say that because To me they were actually pretty interesting in the other results that were the same And that's interesting because we just matched the mysql protocol in my testing The ones that didn't match are not necessarily what I expected. So I had hunches so the connect and select one test case was one that I would have had a very big hunch that Uh, because that's where the new tls comes in because you're worse. So this test was establishing a new connection And running a select one and disconnecting It's a very extreme edge case. It's not something that is very practical in your application Most people fire up a connection pool So it's kind of easy to counteract this and be like, hey, I I don't really care This isn't this isn't really doing much for me But it was an extreme case to kind of test That part of the protocol And while it might not be important for you or a lot of people This does become relevant in serverless stuff or if you're running on like aws lambda You have the the boot up cost time of of getting started. So these do become pretty relevant in those cases And as we can see We're pretty drastically improved on both htp2 and three And that's only because The mysql client is doing tls 1.2 And the other two are doing on tls 1.3 We don't see much of a gain Between htp3 and htp2. For the most part, I would say that those are not scientifically Different Where it kind of came interesting was a large select. So this was me kind of stress testing the A large, you know, I think it was like 50 megabytes or something of data that's basically being downloaded I did a similar upload test which at that point I started also stressed the mysql database and the disks and stuff like that which It was a pretty good result as well in that the protocol wasn't getting in the way What I find interesting that I wanted to call out was that on the high latency network Is where htp3 Kind of shines. Oh god. I missed the slide So htp kind of excel or quick specifically is what excels on the High latency large amount of data Networks which makes sense Because now at this point we're trying to send 50 megabytes over the wire And yes to answer that question it is it's using protobuf. So this was trying to be the most Refined version obviously if we're using json and things like that there's going to be a little bit more compromises But this was trying to be the you know as as efficient as we can So htp3 in this case is where this starts shining when you're sending 50 megabytes Over a network over a high latency network. And this is a large geographic region There's a lot of network switches. There's a lot of things involved if it drops a packet You can pretty quickly see that things get slower Pretty pretty quickly. So having htp3 there was a was a big success What was kind of this is the only test the only test in anything i've done Where htp3 was universally worse Than both htp2 and the mysql protocol So this is on a low latency network. This is Literally in the same aws data center as where the database was So we tried to remove everything, you know, that's removing All network hops as much as we can And what i my theory here htp3 and quick specifically are still very immature They're not you think about tcp tcp is built into the kernel. It's been around since the 70s We're really at this point stress testing the implementation of quick And I find it very Hard to believe that this won't improve over time I would imagine and I didn't get into actually running like profiles and things like that But my guess is we're actually losing CPU clock cycles to actually parsing quick Um But that'll improve Rather than that we can still see that htp2 which is something that we can use today is still, uh That was shockingly better So my summary of this was that in every scenario I tested htp2 With htp3 with the caveat of that last example It was comparable and better in a lot of cases And what this kind of meant to me was that like If we were to write any a driver today that was you know the planet scale mysql driver And you were able to plug it in with like an orm or use it in your existing application And you just use it and you don't have to know that it was Uh not using the mysql protocol You're not going to degrade in performance in a lot of cases. We can give you an improvement But why emphasize being comparable? Is that that makes it a drop-in thing even if you are on a low latency And you don't actually have the problem set that it is improved If you open up a connection and never disconnect it like if these aren't problems to you Using something based on htp2 wouldn't be a downgrade. It would just be a lateral movement But if you are in those scenarios, we can say we improve As well as not being worse in anything else If it was worse now you'd have to make a decision because you're making a compromise But it's not and that was kind of what was a little bit more shocking to me in that There's basically no compromise at least getting into htp2 So it's not really left with you know, what do we do now? We've we've done these experiments Uh That's cool Matt. But what do I do with this? And honestly, there's not a lot so outside of our own product development Um We're we're definitely going to build new features that use the htp2 protocol because we need g rpc We're going to be experimenting with writing drivers based on htp2 or three We're going to see where that goes But it's not something that you're just going to roll in today and be like, hey, let me just Let me just fire this up and and get you know, double the performance. That's that's not going to happen The coolest side effect that I find from this Was within our product, we have a mysql console This mysql console and this isn't explicitly because of htp3 But just in general because of having an htp compatible api We directly communicate from the browser to the database using our api And because you're using it in a browser Modern browsers will support htp3 and you kind of get it for free In this case, it's not really going to buy you anything because you're typically not doing Large data processing and things like that But it's kind of a way for us to experiment and and see what our downsides are before we Committed to other stuff, but it's something that we kind of get for free So the downside to this is that htp3 because it comes with a whole new effectively a whole new networking stack of quick and udp and all of that It is very rare to Find implementations of this it is very new And the demand really isn't there on server stuff So outside of browsers It's really hard to find honestly. It's also kind of hard to find htp2 So this is kind of an upward hill if we wanted to actually lean into htp3 But I kind of hope this is like a chicken and egg thing If we can provide a compelling reason and say hey if uh Vercel if you want to support communicating via htp3 I'm willing to help you make this happen, but it's going to make your customers also a lot faster because especially in that case It's going to have a little bit higher higher latent Connection because they're distributed all over But fortunately the conclusion here htp2 was also really good and htp2 is quite a bit ubiquitous So being g rpc Compatible kind of opens up the door that If you want to use it or if we wanted to write a g rpc client That is very prevalent and very available and the fact that htp2 Gives us a good amount of the benefits um, that's something that we can explore today without needing to Go too far in drinking the Kool-Aid of of getting all into htp3 But keeping htp3 in mind for future adoption is kind of what What I hope most So to me the future is interesting. There's We we haven't really tapped into all of the Things that we can do with this. We haven't you know written drivers. We haven't gotten out there and said hey uh, I want to work with you to Get cloudflare workers to use htp3 or whatever it would be We we haven't had a compelling reason So part of these experiments were to kind of get in front of that chicken and egg thing and then say hey like We we can measurably improve this situation with no downsides Or at least for the most part udp comes with its own Stuff but for the most part there's no real downsides if you can if your infrastructure can handle it So we kind of need to see where this can go. We're we're not getting rid of it We're planning on documenting the apis And when I say g rpc compatible, I'm I'm doing very intentional on that that we support more than just g rpc For example using our web console does not use g rpc from the browser. It is just using json So the api is going to be fully compatible with using json Which does make it really easy if you're not in a performance Critical situation you could just use some json like if I want to curl from command line I can do that because I don't need to write a protobuf or binary in code, you know through Something to be able to send that payload. So we're going to keep Exploring and seeing what we can do. I would fully expect at some point for us to publish some experimental libraries that directly speak over the Our new htp apis whether that is htp 2 or 3 It's probably going to be language dependent like if you're using a ruby application for example Good luck getting an htp3 client like it's and i'm not writing it So it's kind of interesting to see see where this is going to go. Um, it's been fun So yeah, so i'm i'm i'm down to Answer anything i'm pretty At least try i'm not the most knowledgeable, but you know, I've written a lot of this stuff written a lot of this code I've played around with it a lot It's been fun Yeah, so I guess we'll start with the first thing. So I want to address one person asks about packet loss with with udp So I think I I touched on that um That that has been the probably the number one concern that someone has brought up and been like, hey You know, I hear that udp is going to lose lose my data. Um, and that's true but so does The wire that you plug into your computer There's always data loss on the protocol The whole point of tcp Is to make it reliable. So tcp is its own wrapper around raw network bytes And that wrapper is what gives sequence numbers and all these other things so that the other side knows and like, hey I missed a thing Re-give me the thing that I missed or it is the structure for the unreliable data So if we think about the same analogy udp is that primitive in in this case udp is the is the lossy primitive udp is just sending some stuff. You don't know if they got it. There's no acknowledgement. There's no nothing So in that sense udp is is lossy That's where quick comes in. So quick is that structured protocol That uses udp as a primitive and quick is what gives similar semantics to tcp in that With the same goal of reliability So if you're going to send a request You will know that the requests got there in very similar semantics to tcp So in the case of using it as a service or something with us We're not just losing data. There's not things that are We're not just losing queries. We're not receiving everything works as you would expect Okay, so So the api used in the plan to scale console So yes So it's kind of clever how this works I plan on writing a blog post a little bit more in detail about how our web console works Because it was something that we we did very silently because it was experimental for us It solved a lot of Problems that we had so the old way the console worked was When you when you fired it up it would talk to Our api server so you'd basically make a json api request to our api server Our api server is in us east. I believe And then it would talk from our servers It would get the credentials it needs and talks to your database Wherever it is. So if you're in Let's Europe. That's an easy one if it's in eu west You are physically in eu west you already have a lot of latency talking to our api because you're going from us Or eu west talking to us east And then the us east server is talking to your eu database And then back all the way to you so That's obviously a lot of latency if you're not in the us So we wanted to solve the fact that that it and if you're in like asia it was entirely Unusable So this became appealing because I was like we could just let you talk directly to the database and not go through us Because now we can talk to the database over xgdp So the way that it works today um You can kind of see what it's going if you if you were looking at the network inspector and see what happens On connect we issue Some short-lived ephemeral credentials So part of all this is that the credentials can be stamped out for a minute five minutes Whatever a short lived that is bound to your browser session So you have to talk to our api to get credentials The credentials say in your browser and then they are refreshed just like an oauth token Uh in the background. So while you're still on the page, it'll continuously refresh if you leave the page they expire And then that is direct communication between Uh the database and and you which is kind of nice because now it also doesn't go through Our api servers, which is just less liability for us. You know, we're just knowing that you're not sending Hopefully, you're not just querying credit card numbers or something in the console But you know, if if you are it's not going through another intermediate hop It's directly talking to the database I do plan on writing a bit about more about how that works because I think it's neat and we've used that as kind of this, uh Something's, you know a test bed for some of this stuff Anyways, so Benefits for highly concurrent workloads. I'm gonna go down these lists Benefits of high concurrent workloads in principle avoiding headline due to tcp should reduce long tail latency for multiplex streams In simple terms, yes, that is the um So the benefit of being able to have one connection for everything Kind of eliminates the mysql like connection pool issue This isn't necessarily an issue for everybody because some people just open up a connection pool of 100 connections And then you just use those 100 connections That's not the reality for everyone. Some people are creating a lot of connections starting them up tearing them down But in the case of multiplexing, that's where quick comes in And solve some of the headline issues, uh, as opposed to tcp And that's just something that you're really going to notice on latent networks. You It's relatively rare if you're just in aws talking to us next door But yeah, having quick and stuff does solve that. It's a little bit harder to measure and quantify that because it does rely on the network being unreliable, so uh It does just objectively improve things. Um Are there are there places in vates to use this for internal traffic? So that's also an interesting question. So vates uses g rpc And that is correct. We communicate with vates Internally everything g rpc we we for many reasons We don't use mysql connections in our internal infrastructure For a lot of the same reasons of what we're solving for public connectivity. It's just a pain We don't want to manage connection poolers. We don't want to have to deal with Severing connections and all that so we use the g rpc interface internally The differ whenever we communicate with vates is Also within the same availability zone explicitly So even in in a us east one or whatever data center in aws we explicitly keep everything within a us east one a and Follow that chain all the way through the stack so the cases of Saying let's use htdp3 for this It like it genuinely would not do anything because we're it's not solving any of the problem Like we don't have high latency networks. It wouldn't necessarily be worse But it's also just not a problem on very reliable networks We keep hold open connections for a long time. So it's just not a It's not a problem. We maintain we can do tls 1.3 internally Like there's just a lot of things that we can do because we control that environment That it's just not that practical The one thing I will point out g rpc is very explicitly Also bound to htdp2 We cannot just use htdp3 With g rpc nobody can it's just not a It's not a thing the libraries do which is why I was calling out that we're g rpc compatible When you use htdp3 We kind of use a little deviation and which is why I kind of called out It's just protobuf over htdp3 Because we can't actually just use g rpc You'd be kind of swimming upstream. There's nothing that prevents you from doing that Just none of the ecosystem Really does it So it's definitely not something we could just bold into a test and say like hey, let's use htdp3 now Anything else for me? You're you're pretty good on time. I nailed that So the last thing is my only real marketing pitch In that if you are not a customer You should you should definitely check it out. You should sign up We're continuing to try to make this stuff better um I like seeing how we can push the envelope. I like challenging the assumptions of of you know preconceived notions Uh our serverless driver, which is what you would run in versell use our htdp API It doesn't necessarily use htdp3 There's nothing that prevents it from using htdp3 that goes back to The support of providers when you run this on versell You're kind of at the mercy of what versell gives you as the fetch API So maybe someday we can put enough pressure on The providers to be like hey, can you make can we get this to work on htdp3? And it's going to be faster But anyways for the most part the serverless driver just it'll use htdp1 it'll use htdp2 It just uses whatever that is htdp compatible Uh planet scale connect is the first feature That demanded an extension to our apis or extension to the mysql protocol This is if you're not familiar this feature is um It supports like etl So if you wanted to change Or if you wanted to change capture in real-time rows that are added mutated all of that Shove that into something else that was not a great It's similar to like if you were writing your own thing that parsed and tailed bin logs and things like that But we kind of want to make it a little more user-friendly and that we can just hook into existing systems That api is based on g rpc We can have g rpc clients that interact with that api And then I talked quite a bit g in browser mysql console outside of these things. There's not a lot of um Direct application today, but now that we have these primitives in place and we have things to play with We're definitely planning on building new features and newer tech that's going to come out and leverages But for the most part these are all these are pretty cool things And it's all production ready. It's stable. Um It's ready to use Last question here Late test, did you run comparison against aws or raw serverless with the hp driver? I did not because I don't even really know what aws aurora is And I don't even I'm not even entirely sure is that still available. I thought that was one. They shut down But either way, I haven't done a lot of comparison with Different providers because I think that introduces a lot of variables Um a lot of variables that are not just testing the protocol and that gets into are we stress testing aurora or we stress testing You know, whatever else it's not as controlled Environment and it's hard to get tangible Results out of that unless like I'm an aurora expert and I'm not Um testing the protocol is kind of what I was trying to really focus on here Which is why like when I got into stress testing mysql I didn't want to just add capacity to mysql or do you know planet scale shard things up and do things like that because it didn't It didn't make sense. I'm really stress testing the protocol if we're bottlenecked on the database That's good means the protocol is not getting in the way. It's not slowing things down So comparing other things to me just feels kind of moot because it's really comparing planet scale as a service being fast to aws aurora um Or measuring network latency like it could be aurora has a closer region or a closer hop Then then where we are it's just not a it's it's hard to get a one-to-one comparison. All right. Well, I think uh I think it's good to go. Thanks. Thanks everyone for Wasting some time with me. It's been it's been fun Thanks a lot