 All right, we're going to jump right into this lightning talk. My name is Ken Arons, and we're going to talk about scaling time series databases. So you might be wondering, why should I listen to this guy? So I bet that at all the speakers that you have, I'll probably be one of the only ones that has graphs on their LinkedIn profile. I really like working with time series data. And of course, I'm here at KubeCon. That's my third KubeCon. I'm excited to be here. I love working with Kubernetes as well. You might wonder, what is time series data? If you ever look at a stock chart or maybe crypto, you want to know what the price is today versus yesterday. It's time series data. If you deal with measuring the CPU on your server over time, that's time series data. Any data where the thing on the bottom of your graph is time, that's time series data. So it turns out that if you stick this in any old database and query it back out, your performance will be measured in minutes. So you might not want to wait that long to get the answer back. So if you want to get nice, pretty-looking graphs like this with all the data filled in, you need a time series database. So at speed scale, where I'm one of the co-founders of speed scale, we selected time scale DB. And a couple of the reasons why we really like it, one is it's open source. And it's built on top of Postgres. So you don't need to learn a new programming language or a new way to query it. So you can just write a SQL query. I didn't make this up. I prefer to use a copy pasta. So I took this directly off their GitHub repo. But it's as simple as this to make a query that will get you a graph of data. So so far, we haven't done a whole ton of work yet. We picked existing SQL-based technology. And sure, you pump some data in. And then you can query it out and get a graph. So the next part is to deploy this out into your cluster. So if you're super lucky and you pick time scale DB, they have a Helm chart. So you can just deploy the Helm chart. And actually, you're done, right? Everybody knows that. So the whole reason why there is a data on Kubernetes days is because that's not the end of the project. That's the beginning of the freaking project, right? Pardon me. So is the data going to get loaded and stored the way that you want? Are you actually going to be able to have your data queryable from the endpoint that you want to query it from? And I exported this into PowerPoint. So I could make sure it looked right. And it said 1, 2, 3 in my original one. So we'll go through these three challenges, obviously, after you install the Helm chart. So the first thing is I get all of my technical questions answered by Stack Overflow. So the first thing is, and I don't know where the AWS people went, I've stored the data in a volume and I want to read it from another availability zone. You can't. So as soon as you load your data into time scale and you have some disruption of your node of any kind, the most common kind for us is when we do the little node rebalancing. What happens is your time scale pod goes down and then the new pod is spinning up. And the autoscaler will say, I'm going to give you a new node in the wrong availability zone. And so it waits. And then a couple minutes later, it's still pending. So it makes another node in the wrong availability zone. So you can actually have extensive downtime because of this PV problem, if you've ever dealt with that. We shifted to a product called Carpenter, which is open source, the AWS folks who are here. We're talking about that as well. This is a link. We'll share our slides. This is a link to our blog about how we moved to Carpenter. You also happen to save money by using Carpenter because it can right size your nodes so that you fit everything in a little bit better. So challenge number one solved. Challenge number two, so your time scales now, it's not running an RDS or something like that. It's not really hooked up to your VPC. It's running inside your cluster. If you're as lucky as the other guys, you got 120,000 IP addresses. You need to make sure that whatever is trying to call it can query it. So in our case, we had two workflows. One is a data ingest workflow. How fast can we write time series data into the system? And then the second is querying it back out. They both need to be able to address it. And actually the problem was being able to have our indexer. We followed the AWS recommendation running it as a lambda that's running on the outside. And we had a nightmare getting the networking to work. So we moved the indexer into the cluster. Turns out there was a side benefit. It ran 10 times faster. So if you rely on things like ingress and exposing the IP address, you're going to be working on that problem for forever. So you've got to think about putting your compute close to where your data is. And then the third challenge I talked about when you want to go and tune your database. So in our case, we have a lot of variability in our data. We can get storms. So a lot of data coming in all at once. And you need to make sure that you replicate these conditions. We use a system called traffic replay, which our product can help with. And you can see at the bottom our connections are super smooth. So that was the end of weeks of testing. But this is the kind of thing you need to create realistic workloads so you can find these problems. Rapidly say, change this, change that, change my code. And we did a ton of different cycles on this, basically in two sprints. And the result was for us, we were able to handle enormously high scale of ingest of traffic. We typically have about 150 millisecond as our most our queries are that faster or better. And it's actually pretty cheap considering how much that we're running inside our cluster. So this was a quick lightning talk. Here's my QR code. If you want to scan this and give me feedback. And that's it. Thank you very much.