 What is the hotspot and foundation to be a hotspot in any database? For that matter, high volume of rights or reads to a small set of blocks, a small number of partitions, in this case on a storage node, or a set of storage nodes, a team, as it depends on how much, as based on your application. A couple of metrics, just to sort of give you context, everybody's mentioned, Steve mentioned the storage node keeps five seconds of data in a version tree that are non-durable. This can grow and shrink in size. The storage queue is the size of these mutations. The, and another metric called non-durable versions is the number of versions that have not been persisted to the SQLite. And the only thing you really need to know is that when storage queue goes up, ratekeeper goes down. When NDV goes up, bad things happen when it goes too high. So there's a simple talk, there's just a couple of takeaways, and maybe I can give some more time back. If you want to detect two kinds of things, if you have a right hotspot, you will find that both the storage queue and the NDV go up simultaneously on a specific process. If you have a read hotspot, you'll find that only the NDV goes up and not so much the storage queue. And the reason for this is FoundationDB tries to prioritize reads over writes. If you have a large amount of writes, then they get de-prioritized and the storage node is busy supplying all the reads. If there's not a number of reads, then the storage non-durable versions will go up as a result. So how do we solve these problems? We made a code change, which changes the priority of the right tasks. You will see this in the previous slide. That solid line up top is in the lower chart is we said, oh, writes are not getting pushed at all. Let's increase the priority of writes over reads. That helps us a lot. We applied application level throttling that can throttle reads, writes, finer grain throttling for, oh, these are certain applications that are doing worse. But most importantly, this is the second takeaway from this talk is you need to understand your data patterns. And this is something that being a database company, we should be really good at, but it turns out we weren't as good as we should be. And we've been sort of knocked on the head by various people, including some sitting in this room about that. So how would you do this? Foundation DB keeps track of partition stats, but doesn't have any application context. The application of the layers itself have both application context and API that they can use to get partition stats from Foundation DB. I mean, we may have made some changes to it to give you both the storage and the port information, but it's a trivial change that can be ported to the open source quite quickly. And so now if you take your application context and merge it along with the partition stats, then you kind of know where your data accesses are going, which storage nodes are they going, where are your writes really flowing to. And then we keep stats for every single transaction. We just buffer this up in memory, and when it's time, we flush it out to disk. If you're running very high throughput, you can switch over to sampling. This is a sample stat. All I really want to show is that we have partition rank and we call it partition rank, but it's really a shard in FTB terms. And we see, aha, we fed so many bytes from here, we fed so many bytes there. And then you take this information. In our case, we push it back into Snowflake because we can create easily. You can push it anywhere else you want to. I've heard the FTB team has considered pushing it back into FTB itself and running queries on top of it. But it's very common to see zipping distributions for your database. There's a large amount of certain kinds of data and very little of the others. This in itself is not a problem. However, skews in data access is a problem. And here what we've done is we've tried to figure out how much data is read or written per minute for, like, top partitions. And that's the graph on the left, which is per partition, what are the reads? And then the graph on the right is by the entity. So this is, the graph on the left is the view from the FTB side. The graph on the right is the view from the application side. But they've both been sort of merged in and sort of looking at them together. So what do you do? You address this in various ways, like the first two or three sort of data access patterns here are clearly egregious and need fixing. You can cache, you can fix your accesses. But, you know, if you're going to access close to a gigabyte a second, or, sorry, a gigabyte a minute, then at least in our system you will cause problems. And that's it. Thank you.