 I hope to see you here next year. Hopefully, the pandemic will be more managed by then. So first of all, a quick recap of what the remote storage systems in Prometheus are. Prometheus has the ability to send its data to an external system for long-term storage and analysis. This is usually done to analyze performance of your systems over longer periods of time or for auditing or similar reasons. Right now, on the website, there are 27 different remote storage systems listed. And looking at this list, you might say, ah, how do I choose which one is right for me, right? So this talk is for you. It's all about the tools that exist in the ecosystem to help you make this decision. So a little bit about me. I am the developer of one of those 27 systems, Prometheus scale. The system is able to store Prometheus data for the long-term. And today, we announced that we are also able to store open telemetry traces within the system. But here, Prometheus scale, we also believe in contributing back to the Prometheus ecosystem. So some, not all, but some of the tools I talk about were developed in-house by Prometheus scale, but are applicable across the remote storage systems. So today, I'll talk about three different criteria that you should use for picking remote storage, correctness, performance, and the ability to migrate data both in and out. There are a lot of other topics you should consider. This is not a decision you should take lightly. You should consider cost. You should consider the availability of different query languages and operational concerns like high availability, security, et cetera. But unfortunately, time is limited, and we don't have tools to verify those things anyway. So I'll just talk about these three. And I started with correctness, and I started with it for a reason. I think often people go straight to performance and forget to ask about how correct the system is. And having a system that's very fast but not correct is like having a fast car with a really fast steering. You'll be fine until the first turn. So the first tool I discussed was developed by Julius Wolz. He's now from labs. He's one of the founders of Prometheus. He wrote this great tool that goes across these remote storage systems and tests how accurate the query results are for different systems. And the gold standard here is Prometheus itself. So the question is, given the query over these other systems, are the results the same as what Prometheus would return if you have stored the system there? And you can see a whole bunch of systems, Chronosphere, M3, Sanos, Cortex, Prometheus all have 100% compatibility scores. The others, well, they all have varying degrees of problems. And what I would say is a Prometheus would really believe in the standards, but you might not necessarily believe in it or you want to trade off some performance to get some sort of non-compliance. And that's fine, but you should really think about the trade off you're making deeply before you make it. The other tool here is a tool written by Tom Wolke at Grafana. This is a bit different. This doesn't test the remote storage systems, but it tests systems that write the remote storage systems. So Prometheus is not the only writer, if you will. There's the Grafana agent. There are others. And this tests how compliant those systems are when they're writing data. And when you're designing your system holistically, you should think about this as well. The other correctness considerations you should think about, whether the compression used by the system is lossy or not, how crash-safe the system is, and the latency of seeing your data in the system. But there are no tools per se to check this, but these are other things you should think about. The second thing you should consider is performance. And the first tool to measure performance is a tool used by Prometheus itself. It's called Prombench. It deploys an entire simulated environment for you and generates load on the environment. And then it measures how well Prometheus performs within the environment. And this is great if you don't have your own data. This could simulate data for you, and you can easily configure, in this case, Prometheus to do a remote write to whatever system you want. However, if you do have Prometheus data, promscale wrote a tool called the remote-to-write benchmarker, which can take your Prometheus data, read it out, and play back into any remote storage system that accepts remote to write. So critically, due to the exact same logic as Prometheus itself would, so you can be pretty sure that the tuning that you perform in trying out the remote-to-write benchmarker will apply when using Prometheus itself. When playing back this data into the remote storage system, you could do several transformations. You could change the timestamp to be the current timestamp instead of whatever was previously recorded in your data. You could kind of multiply your series, this kind of simulates the blowing more instances of the same servers. You could speed up or slow down playback, and this simulates changing the scrape loop interval. You could also play around with the configuration of the remote-to-write parameters in order to tune the system. So these systems are kind of different. A Prometheus works off of simulated data, but they can test both ingest and queries. The remote-to-write benchmarker is great if you have your own data already, but they can only right now test ingest. And by the way, I should stop and say, at the end of the talk, I will have URLs to all of these tools. So, OK. Last part is migration, because once you pick a system, sometimes you will need to change your mind. Things happen. The first tool to do migration comes with Prometheus itself. It's called PromTool, and it applies to any remote storage system that writes data in the exact same format as Prometheus itself does. And those are Cortex using block storage and Thanos, and it's supposed backfill. So given current data, you can migrate data from a previous time period, which is important when you are starting up a new system, because you need to start up your system and at the same time migrate all data into it. The other system is PromMigrater. This is another system. Within PromScale, it can read data from any remote storage system that supports remote read and write data into any remote storage that supports remote write. And it kind of does the obvious thing by just piping the remote read into the remote read. But it has some nice properties. It limits the memory usage, so even when migrating terabytes of data, you can tell it to use 500 gigs or whatever. It will show you progress as you're doing the migration. If the process crashes or you want to pause it, it is able to resume with left off, and it is completely stateless. So you don't need to slow your state there. And again, it's universal. So any system that supports remote read to any system that supports remote read, unfortunately, not all systems support both remote read and the remote read. And furthermore, there are some things that you have to watch out for, because writing to a fresh database is different than backfitting data to a database that already has data. And so when choosing a system, you might want to think about, hey, can I get data out? Can I get data in? So these are all the URLs to the various tools that I just discussed. And yeah, we're hiring as well. So come talk to me if you have any questions about any of the tools I just mentioned. I know it was kind of a whirlwind tour. So yeah.