 Today, we're going to talk about Cortex. Specifically, we're going to talk about what happens since the beginning of this year, and then what do we have in plan for Cortex. So here's the agenda. Actually, let me introduce myself first. So my name's Alvin. I'm one of the software development manager at AWS. I joined as a Cortex mentor about a year ago. I started to look into the query path a lot. So I have a little bit of technical knowledge there. But nowadays, I mainly do releases, doing some show work, like upgrading go-round time to 1.19.2 and all that, or circuitry patching and all that. And here with me is Alan, who is the actual brain of Cortex, and I'll let him introduce himself. Hey, my name is Alan. I also work for AWS on all these teams. Been working with Cortex for the last two years. Working in scalability, availability, became maintainer in the last year. Yeah, trying to make Cortex a better place. Cool, thanks, Alan. So Alan does most of the work. I essentially do nothing, and just tell Alan to do work. And here with me is the FedRex. Of course, he's not here with me physically, but he's very unfortunate I couldn't make it. So FedRex is actually a very long-time Cortex user. He deployed Cortex clusters for Adobe, and he runs, I think, the number he told me it's hundreds of clusters. And if you were ever in the CNSafe Cortex channel, then you ask a question, then Cortex is the person who will answer your question. So he is really active in the channel, answering any questions regarding configuration, optimization, and all that. Let's but not the least, we also have maintainer from Germany, and then he said it's a Niklas. I didn't want to go into linking and searching why it feels like stalking, so I just grabbed his GitHub profile picture, and that's why his GitHub profile picture looks like. And he's the maintainer for the HelmChart, and he's very active if you feel that, hey, HelmChart's missing some configuration added, added, and then Niklas will always be there to merge the PR. Cool, so to this agenda. So I would imagine not a lot of people know what Cortex is and how does it work, so we will do a quick introduction to talk about Cortex, and then we'll do a little bit of the architectural dive deep for Cortex. Then I want to introduce three exciting features that's coming to Cortex in the next release, which will be 1.14. And I'll give one operational tip, so if you're running Cortex right now, then you should do this when you get back to work. It's very useful. We run a lot of Cortex cluster back in AWS, and that's one of the tips that save us a lot of memories. Then we'll just look back to see what happened for the Cortex release 1.13, then we'll go through the voice of community or we call it the roadmap, then we will talk about the call to actions, we need help, Cortex need help, and then I would love more contribution to Cortex. Then we'll do our favorite part, which is the Q&A. All right, so what is Cortex? Cortex is a horizontally scalable, highly available, multi-tenant, long-term storage for Permethias. So what does that mean? When we think about Permethias, initially when it was designed, it's designed to be installed on a single machine, scrap a cluster of metrics, and then it's stored onto local drive. The problem is your local drive cannot be as big as you would like to, so you usually have a very fairly short-term retention period. So Cortex trying to solve that problem. By in order to be able to store a lot of metrics, millions, billions of metrics, you need scale. So essentially what Cortex is doing is to take bits and pieces of Cortex and then make it into microservices. So you have a microservice occurring, a microservice or ingestion, a microservice for the moving data to the long-term storage, a microservice for reading data from the long-term storage. So like saying Cortex is a long-term storage is a little bit misleading because it's not just storage. It is actually a system that allow querying. And Cortex is a CNCF project that's incubating and Apache 2.0 license, so you can do anything with it. Bunch of contributors, about 250 of them, lots of watches, 5K star, and so far we have about 5,000 commits. And then Alan here is making the commits going up every single day. So it is actually a very active project. This is a very high-level view, a bird's-eye view, at a very high level of Cortex. So essentially what it is, a typical use case of Cortex. You have a bunch of Prometheus, right? So you have one premise for cluster A, one premise for cluster B, so on and so forth. What you can do is you can configure Prometheus to do a remote write into Cortex. All of them can just do a remote write to Cortex. If you want a differential between clusters you can do add a label during remote write to differential different clusters. They're just saying to Cortex, then you attach your dashboard tooling like a Govanna or whatever dashboard that you're feel like in, then you get a global view of your matrix, right? You don't have to go to like Prometheus A to look at cluster A, Prometheus B to look at cluster B's matrix. So because remote write, it is the protocol that it's fairly stable from Prometheus. So you don't have to just use Prometheus. You can also use the tools like an open telemetry to send metrics to Cortex. Or if you're a little bit more adventurous and then you like writing code, you can actually write your own code in Goliang, Java, in C, C++, whatever you want and just make sure the message is in the remote write format then you should be able to send it to Cortex. And you can send a lot of metrics to Cortex. You will be able to handle it. So next I wanna dive a little bit deeper into that Cortex icon to see how it actually work under the hood. Oh God, this looks complicated, Alan. I think I need help for this one. Well, I will try my best here. Yeah, as you can see, like the community is saying that Cortex can scale a lot. It can send a bunch of metrics as high available but it's complex to set up. So this is what uses to be Cortex in the past year. So we are trying to, we are deprecating some dead code and some deprecated storage. So we're moving even from the code base and don't worry, this is not what Cortex looks like anymore. It looks more like this. This is a typical Cortex deployment that you can find. You can see like in the yellow there, that's the write path. So remote write come from from ethos. And in green there, that's the read path. So we still have some components. I will try to explain what are each one of those components and hopefully to make more sense at the end of the disk. I will start with the write path. So what happens when Prometheus send a write request to Cortex, right? The first component that's rich, that is called distributor and what is the distributor? Distributor is basically a gateway, but that will forward the request to the writing gestures. In gestures are the storage nodes, distributor will just forward the request for them. But why we need the distributor? So distributor, it will do sharding and replicator data and do like pertinent sharding and replicator data. And optionally, you can set up things like shuffle sharding that will improve your tenant isolation or zone awareness replication. So distributor is the guy that will make sure that he's sending one copy of our data to each AZ. It also does things like rate limiting and AJ data. So if you have a Prometheus server that is sending is deployed in a AJ mode, distributor is the guy that will receive one sample and throw on the floor the second one. After the distributor, we have the ingestors. And ingestors are basically a multi-tenant TSDB. So remember, Alvin said that what Cortex does is take Prometheus and put in different microservice. Prometheus is basically a TSDB and a query engine. The ingestors is the house of the TSDB. So the first time that I receive a sample to one tenant, ingestors receive a sample for one tenant, it will create the TSDB block, the TSDB instance. We'll keep appending that. And the after is configured, but typically after two hours, you'll send those TSDB blocks to the block storage. And the block storage can be Google Cloud storage, S3, any block storage that you want. We have support for Azure as well. But now, we can see that as I was sending data, so the data that were on this was replicated. I sent to one replica for HSE. And now I have all these data, duplicated data on S3. So there's where the compactor comes in. The compactor will get all those blocks. It will compact and compress those blocks and make sure that this data is in the optimal way to be queried. It also does things like housekeeping. So if you configure your potential period for one year, compactor is the guy that will start to delete blocks that's older than one year and things like that. Again, all those components can be shuffle-sharded and deployed with zone awareness. So you have easy, tolerant, and tenant isolation. This is basically the right path. But then you have to query your data, right? So the read path, the first component is called query front-end, which you do. This is a similar thing that distributor does but for the query. It will shuffle-shard and make sure that query is for a given tenant or spread across this. But it does more than that, right? Like it does QoS, for instance. So it makes sure that one tenant is not starving other tenants or do cache of cache. So imagine that you have your dashboard there that's refreshing every minute. And instead of recomputing or re-executing the whole query, the query front-end will just fetch the delta from the last refresh to the refresh right now. And do things like vertical and horizontal-sharding, query-sharding that I think Alvin will talk a little bit more about that. But it's basically trying to split one query in multiple smaller queries so I can run that in parallel in multiple queries. After the query front-end, we have the query-er. The query-er is the house for the promo-ql engine now. So we run the Prometheus promo-ql engine in that component and basically receives the query request, fetching data from ingestors for recent data or from historical data, merge all of them, evaluate the query, return the result back to the query front-end and to the customer. Do things like rate limiting as well, like to prevent out-of-members and things like that. Now we have the store-gator. What is the store-gator? Well, it's the gateway for the store. Basically what this guy is doing is it's keeping an up-to-date view of the block storage. So every time that I receive a new block or I compact a new block, store-gator discovered that, advertised that to the query-er so this block starts to be query-able. And also download parts of the index, the block index to make sure that we can have a faster time series look up when you are running queries. So basically this is like a normal Cortex deployment. Officially, you can also run rulers and alert manager and those components are basically a multi-tenant version of the Prometheus ruler and alert manager. Again, zone awareness, again, shuffle sharded. Rulers will basically evaluate, record and alert in rules. We will send the alerts to the alert manager. Alert manager, you, the dub group and send the alert for the right explanation like slack or page dirty, you name it. Basically this is what Cortex is right now. Those are the components. Hopefully it makes more sense after that. And now it's back to Alvin. Yeah, definitely, I think it makes more sense than the diagram we showed at the beginning. All right, cool. So this is the list of the company they are currently using Cortex, a running Cortex cluster. Cool, so now I want to introduce the three features that I was talking about. And the first feature is the open telemetry bridge for tracing. If you are the operator of a Cortex cluster, you will like this feature. So this feature essentially allows you to send traces to different destination. In the graph here, we have the example of sending it to AWS X-Ray. So the story behind this feature is because one day I was just writing the normal status report that manager should write every day, Alan came into my office and he said, Alvin, Alvin. I think there's a bottleneck between query and query from end. I don't know how to do it. I told Alan, there's the anchor, Cortex supports the anchor. So the person open trace, just, no, set up the anchor. Then Alan said, no, I don't know. I don't know how to set up the anchor. I said, well, spend some time, set it up. And Alan threw up his hand in the air and said, hey, you're not helping. Then he went out of my office. So I get back to my work. Few hours later, Alan come in with this screenshot, exactly. I said, hey, Alvin, look what I got it working. I got it working with X-Ray. Now I see there's a bottleneck between the query from end and the query. It's because the queue is overloaded. There's a queue between them and it's overloaded. Oh, cool. This is awesome. How did you do it? And I said, oh, yeah. I integrated with the open telemetry. And I said, okay, cool. Awesome. Alan asked me, should we open source this? I said, of course, why not? So this is hours of work from Alan. It is awesome. If you're ever run cluster, you have a problem. Use this feature. It will help you troubleshoot a lot of issue. Alan here actually use the trace multiple times. So I define multiple issue and do optimization. Even for the query vertical sharding, Alan used that to analyze the hay. It is actually boosting performance. And then, which I'll talk about a little bit later. So yeah, so with the open telemetry support, you can send to multiple destination. That's the major selling points, like to Jagger, to Zipkin, to Kafka, and to AWS X-Ray. That's a lot more. You just have to configure the exporter. And the reason why we say the open telemetry is good is because open telemetry is a CNCF project. And it is the protocol. It's a specification. So you don't have to worry about vendor locking. All right, cool. The next feature I want to talk about is a partitioning compactor. So I haven't finalized a name for this feature. So I promise I'll work with the creator of this feature to come up with a better name. But for now, what's the partitioning compactor? Permitius has a limitation. Each block, so Permitius GSDB is essentially bunch of block, right? And each block has a limitation of 64 gigabytes of index size. And the reason is because they have a reference that it's only able to address up to 64 gigabytes. Sure, we can fix that problem, but that problem is a little bit hard to fix and might take a little bit long time. Imagine switching from 32-bit Windows to 64-bit. It will take a while. We don't want to wait. We can wait, but we don't want to wait. So what we're doing here is that, hey, currently in Cortex, if you try to merge two blocks, whose index size is close to 64 gigabytes, you merge them together, and then the result is 100 gigabytes. Then Cortex will choke. What you have to do is you have to upload this no-compact marker to the source block of the blue one and they tell you, hey, just don't compact this. That's a workaround that's not fixed, because if you look at the size over there, 63 plus 63, it's bigger than 100, right? But then if you go through a compaction process, you guys should do the symbol table, index the duplication, which will reduce the index size by quite a bit. So you save about like 26 gigabytes over there. If you still go through with the compaction process, right, 26 gigabytes is a lot. You can save a lot of HD videos with 26 gigabytes. So what we're doing is then, hey, the new compact will say, okay, I'll partition the matrix in such a smart way so that I will still end up with two blocks, but each of them will have a smaller index such that it doesn't hit the limit of the permissive. So we'll figure out like, hey, how much partition we need? We might end up partition maybe four blocks to three or two to two, like in this example. So with this, another possibility to begin to show up, right? You can actually do compaction in parallel because your resource is two blocks. Each compaction can be run in each individual compactor versus before with the blue boxes, you need to do that in one box, right? So the result of the new partitioning compactor is that we actually, it's observing about 50% compaction time reduction in our lab results for a single tenant with 200 million times series, right? We're still doing testing. This is still work in progress. It's almost done. The implementation is there. We are just doing the testing and finalization and then we'll merge the PR. Yeah, so with this feature, you don't have the issue for the bigger than six or four gigabytes issue and then your block will be optimized. And another side effect of the design is that the algorithm is able to figure out who's the source block of a distinction block, right? So you don't have to download all the blocks from S3 into compactor when you do compaction. There's a potential speed up of the compaction process as well. Okay. The last feature I wanna go into is query sharding. This is a very cool feature in my opinion. And before I go into that, I wanna shout out to the Athernos community over there for making this possible because we actually use the Athernos call to do some query analyzer and spit out the outputs and say, hey, this query is shardable, not shardable. And then we saw, hey, we have Quotex user, why not we bring it to Quotex as well, right? Collaboration is always beautiful. So what's vertical sharding? What does vertical mean? When we talk about horizontal sharding, just imagine you have query, you wanna query from day one to day two. So you have two-day query. Horizontal sharding means that you shard by timing interval. So you can actually split the query into day one to just beginning of day two and beginning of day two to end of day two. And then you can split them, round them concurrently and then get the result. They all works. But what if today your query is actually instant query? Where you wanna know? The instant query means that I wanna know the data right now. There's no timing interval. Then how do you shard? You cannot do horizontal sharding because there's no timing interval to shard. That's where vertical query sharding comes in. So I'll do a little bit of deep dive into the vertical query sharding just because I think it's a very cool concept. And then it's the first step to a more optimized, bronchial engine. So this feature is already available right now. You can use it. And I forgot to mention that the open telemetry support is actually in the mainline branch. So if you are the type of person who don't mind using the mainline branch, please do, you can start using the feature. And then cortex thing, we try to make the mainline stable. So because internally at AWS, we actually use the mainline as well. So we test the stuff before we push to mainline. So yeah, so the vertical query sharding is available in mainline branch. And then the speed improvement can be up to, it can be a 30% plus and simple flag to enable the dosato to flag and the documentation. So I just wanna touch a little bit on the documentation. It is under the V1 guarantee just because it is an experimental feature. So it's not in the configuration list. So just be aware that all you can use the slide. So let's do a little bit deep dive under how the vertical sharding works. So consider this matrix. You don't have to steer too high, it's fine. It's fairly simple. You have matrix to calculate how many users you have per region. So I'm using the North America and Europe as example. And then when you collect the matrix, usually you have multiple permissive instance for scalability or redundancy. You wouldn't just have one, right? So that's why you have part one, part two, part three. Redundancy of three is always beautiful and cost effective. And then now imagine I wanna run this query. Don't worry about starting this query. This is like, hey, I wanna know the user per region, right? So, and I wanna get the result. Hey, I have 100 users in North America and I have 95 in Europe. Without vertical sharding, what's done is that the whole query is sent to one query and the query will fetch all the data and just merge them, send it back to query from end, back to user. But if you kind of think about it, hey, I can actually do the query or the aggregation for North America and for Europe differently and color-coded with, I guess it's blue and purple. Yeah, blue and purple, I'll stick with that. So, this is how, without query sharding, how it works. Query from end, query go to the store and the store has its color coded just to show you that, hey, the store doesn't store the same region in the same store. They are just, you know, interleaving everywhere. Query go to the store, do everything, aggregate table, send it back to query from end. Custom is happy. Whoa, when it's a little bit slower, they will not be happy, but, you know. This is what happened to query sharding. It is a little bit more complicated, but what's happening right now is that the query from end will actually do a splitting. Notice that, hey, query one, please, do the European aggregation. Query two, do the North America aggregation. And the query one actually talked to the store to say, hey, please just give me the European one. Don't give me the North American data. What this allows is it reduce the network traffic, right? So right now, store one, store two, does not have to return all the data like before. Overall, the data, the data send overwatch doesn't send, each part will have less data to receive. So query one will do the aggregation for the Europe and query two will do the aggregation for North America. And right now, query from end will have the job to merge those together, but the merging is simple. You simply merge two table. Like you have two roles, merge and two roles, easy peasy. So it's not too much of an overhead for the query from end. And query from end is designed for this kind of stuff. Yeah, and you return to customer, customer again is happy. I couldn't find a happier emoji, so that's the one I have. All right, cool. So that's a feature. And we added this to streaming support belief in 1.13, so it's already available, right? So if you wanna take away anything from this talk, take this away. Enable streaming between query and jester. It will save you a lot of memory issues. So we're actually adding the query in jester made out of streaming. I think it's recently, or is it already released? It's not released. It's not released. Okay, we'll make sure it's released soon. Cause a little happened to Cortex. Cortex has a lot of limits, right? And then we're finding different limits as we go and we try to combine aggregate limits to come. So why this safe memory? It's because if you were to get all the response from the ingester in a query here, it's actually big, right? You can query like the gigs of data and those have to be cached in memory before you start processing. It's gonna increase your memory footprint. You'll see a huge spike of your memory footprint and then you will not be able to be garbage collected as long as you're processing the query. With the streaming, you process it, you dump it, you process it, dump it. For some query, you can process it and dump it, right? So those process data can be garbage collected. So you will see a smaller spike, but it will still be spike. So yeah, so streams are beautiful. Please enable the streaming. It will save you a lot of headache. All right, so looking back. We released a 1.13. 1.13, one of the major feature of 1.13 was parallel compaction. And that was the first set to speed up for compactor. And it's actually complementary to the partition compactor. So partition compactor actually make the parallelization a little bit better. But there will be complementary to making compaction faster. And we also forked the repository from Govana Lab that's Cortex Tool to Cortex, because it just makes sense. We wanna continue to support Cortex Tool as the Cortex maintainers. There's no reason to leave it there, leave it there and say, hey, we don't support it anymore. It's related to Cortex. We want to support it. And I wanna shout out to Ireland and Fredrish and even Nikolas for stepping up to become a maintainer of Cortex when needed the most. Cortex went through, went through some rough time this year, but then it's not back in good hands. Cool. So this is the long-time customer, it's Fredrish, because I wrote this slide before I got his permission to use his name, but he gave me the permission so I can use his name. So Fredrish, he run a lot of cluster at Adobe for his internal customers. And it's fast, right? The ingestion time is less than one second, but Fredrish has been very conservative. In AWS, our ingestion time usually is less than 10 milliseconds. And also, 99.9% of the request is less than 1.6 seconds. 1.6 seconds might seem big, but then if you query a long range of query that's fairly long, that it's actually not too bad of an output of time. And usually I think Fredrish told me that his query usually spent across one to two months. If I remember correctly, I have to double verify. And Cortex will usually take the backward compatibility very seriously. So when Fredrish tried to upgrade from 0.61, which is a very ancient version, to 1.13, it was easy, right? He got some configuration that was removed, but then just removed those configuration from your YAML, then you're good to go. And then he upgrade to 1.13.0 and then was able to support 150 million times series instead of 32. And the BlackHound like was actually just a compaction. The parallel compaction actually enabled a lot of times series. 150 million might, it's not the maximum. Cortex can handle a lot bigger. But there's a few factors we have to consider which can be a whole other talk. All right, so we also have the voice of the community which are the feature that customer users of how is that one. First one, what first of them is the out of samples, of all the older samples for backfilling data. This is already available merged into Premiere CS, I think not long ago. So we will enable that in Cortex soon. Sensitive for down-samples. Down-sampling, there's some discussion about, hey, what exactly problem we're trying to solve for down-sampling? Is it because faster querying or saving the storage? We don't know yet. So that one's I still been discussed. Pretender encryption. It is more for like the people who cares about the, hey, sometimes I might accidentally, I might have like a PII data, personal identifiable information, PII data in my matrix. Then I would like to encrypt it on the rest. Even though I sent to the cloud, I sent to a cluster, I wanna encrypt it at rest. And of course you don't wanna decide one key for all the tendons, they will be, you know, what if like your external customer or your internal customer are two different teams? They would like to be encrypted separately by different different keys, right? Which goes in hand with the delete time series. Quite a feature is important is, imagine that you suspect that, hey, one of my classes compromised. And you go ahead and rebook the access to a key, which resulting in, hey, your matrix are not available anymore for that specific tender. So once you identify, oh, actually, I only have like the two time series. They are affected. There has like a maybe critical number for some reason by some silly mistakes. Then you can say, okay, I'm gonna delete those two time series and then re-enable the key. Then you're back to business. So that's one of the use case for using the time series and pretending encryption. There's other use cases as well. There's a lot of reason driving behind these two asks. And there's a lot more in our backlog. So please do tell us what you want. Join the Slack channel, talk to us, check with Ellen and I, federal, who are all there, who are all friendly people. You say hi to me, I'll say hi back. Oh, I was an emoji wave back to you. Oh, you should use beer, right? I like beer. And it goes through a backlog to upvote the issue that you want with a thumbs up. Or you can use the smiley face, anything. Then to let us know that, hey, this is important to you. If we don't understand what is important to you, we'll maybe hopefully have a conversation on the GitHub or Slack. So if you want to contribute, please just go find out this issue tag with a good first-time issue. Those are something like fixing a typo, fixing this, fixing small things. And then if those are something you're interested to do, please go ahead. All right, and call to actions. Cortex currently have three maintainers. And then I would like more, because there's so much more I want to do, but then we're spreading a little thin here. So if more people can contribute, more people can become maintainers, it would be nice. And help maintain HMTRA, maintain the Cortex.io website and the documentation. So I'm the person who's trying to maintain the Cortex.io now, but I don't have any artistic sense, right? I need someone who really know the web type, web type in order to help me to pre-defy it, organize the information a little bit better and all that. Same thing for documentation. Cortex documentation is not bad right now, but it can have a lot of room for improvement. So if you find any typo, if you find any information can be reorganized on the website, it will be awesome. For example, I wanted to update the architecture diagram on the Cortex website that, oh, I showed previously. And the last one is the automatic benchmarking framework, which is available in the Premier CS repo, but they're not Cortex and I really wanted it because it will be so nice if a peer comes in, I do a slash benchmark, then you can automatically see the performance difference or that, so you don't have to do yourself. Eventually I might get to that, but then if you cannot wait, please tell me that. Oh, I forgot to mention, please engage with what CERNOS community is trying to do. They are trying to create a Procure Engine that's scalable, right? So please let's join force with them. Let's create a more scalable Procure Engine that supports sharding, beta sharding and all that, right? It's going to the arena of like how do you optimize like Procure as in the SQL in the old days. What is this? So thank you, that's everything. Sorry, I'm a little bit over time, but I guess we have some time for questions. I can maybe kick it off. I noticed that the architecture of the sort of query and storage looks very similar to Druid. Was there any exchange of ideas there or have you guys looked at Druid at all? To Druid, the database. I guess not me, because I have no idea what Druid is also. You might check it out. It's a very interesting, similar structure, yeah. Cool, cool, yeah, that's what I was checking out. With the object storage on the back end, yeah, anyway. Any other questions? I have maybe a weird question. Sure. So in the new architecture diagram, you have many components. I wanted to know which one were stateless and I could run on say on the spot instance versus some that are stateful that I don't want to lose. This one, right? I'm imagining the Compactor, for example, if it's interrupted, not a big deal, can restart. Yeah. Stuff like that. So. Weird question. So some, so distribute or query front end queries, they are always stateless. Some components we can choose to run stateful or not. Like, ingestors should be stateful, we should not never lose your data. But components like storage gateway, you can choose. Mainly because like as we are downloading parts of the blocks, it makes sense to be stateful because open restart, you don't have to download everything again. Compactor, usually we run stateful, but just because we want a more disk. It doesn't need to, but like as I wanna put a PVC there so I can put more disk, if I have a huge tenant that I have to compact. But if you are noted, has the disk enough for the size of your tenants, it can be stateless. Okay. That was my question. Thank you. Thank you. Hi, thanks for the talk. I have a question regarding diagram. Why a ruler talks to the querier and not to query front ends? It's a good question. We are thinking and changing this. And the reason because we want to, right now this is not 100% correct. What happens is like the rulers run a query embedded. So it's the same code, but running the same process. Logically it's like that, but right now it runs. The ruler doesn't really talk to the query, talk directly to ingestors and install gators. But we wanna change that. We need to do lots of benchmarks to see if makes sense, especially because our rulers are running instant queries like getting recent data mostly. But we want to change that so we can get some of the optimizations that we do in the query front end. Okay, thank you. And one more question. How distributor does the duplication? So usually you have like a console or a TCD. So it gets one of the pairs and say like, this is the main one, this is the one that I'll drop the data. Okay, and those prometeuses, they need to talk to the same distributor all the time or you can have multiple distributors and they will. Everything like that. That first arrow, there is a load balance. Okay, thank you. Very good observation that Buck will be from and PR will come by the way. No, we actually start working on data already. We disappear to enable that, but it's just a first step. Feel free to chime in in that PR or the issue. Hey, so quick question. You mentioned, is it a feature in progress to a sample, or sorry, to accept out of order sample ingestion and how are you looking to approach that? It's a great idea. Very helpful. So you wanna take that, Heather? How? Yeah, so I think the how is mostly prometeuses, right? Yeah, so like prometeuses, TSTB already have support for it. Basically it's creating a new head chunk to accept the out of samples. There is some overhead there. In our case, it's basically making it available for the customers to enable it. Yeah, just before we enable, they wanna be very careful about not to have users set up their own tools. So when I do our own benchmarking, wanna see, hey, wanna give them a set of recommendations, if you enable this, maybe you don't wanna backfill like one year ago, simple, they might cause some issue like that. Cool, so I guess that's everything. We run out of time. Thank you so much for joining. Hopefully you enjoyed the talk. Thank you.