 Hello, I'm Brian Borum, and I'm going to talk to you about Cortex. First of all, a little introduction. I am nowadays a distinguished engineer at Grafana Labs. I moved from Waveworks, if you've followed Cortex for a while, you may have seen me there. At Grafana Labs, as well as Cortex, I work on Loki and Tempo, which are kind of projects that were created out of Cortex for logging and tracing, respectively. Hi, I'm Alvin. I am a software engineer at AWS. I work on a service called Amazon Managed Service for Prometheus, or AMP for short. It is an offering based on Cortex, and I recently become a Cortex maintainer. Thanks for that, Alvin, and welcome to the team. Now, to look at what we're going to be talking about in this session, I'm going to give an overview of Cortex, first of all, just run through how it works, because I know a lot of people come to these sessions without that background. I'm going to go through some news, what's been happening recently on the project, and then Alvin is going to run through how AWS are working with the Cortex project. I'm going to touch on some future work and then flip over to live Q&A. So, first of all, what is Cortex? Well, it's a large scalable system for handling metrics, and by metrics we mean some kind of number that varies over time, sometimes called a time series. Cortex is built to be a centralized service, highly available, very scalable, and also durable. It will hang on to your data for a long time. It can be used for many teams or users. It was originally designed to create a Prometheus as a service, but you can use it within an organization and keep multiple people's data separate. So, who uses Cortex? Well, a couple of the big Prometheus as a user names there, Grafana Labs, Weaveworks where Cortex was created, companies like Electronic Arts and Etsy use it internally, and one of the newer announcements, Amazon Managed Service for Prometheus uses Cortex to run that service. So, what does Cortex do? Well, I've drawn a very big picture diagram here. Over on the left you have Prometheus, which is doing its thing of gathering metrics, what we call sometimes scraping, from exporters, from programs, from machines, wherever your metrics are coming from. And then sending those, this is Cortex, the Cortex logo in the middle, which gathers, collects the metrics, stores them, and then you have some kind of dashboard over on the right where you're querying the data and viewing it. Let's drill into that a little. We have Prometheus on the left, and we have a store on the right. So this is going to be something like an AWS S3 or a Google Cloud storage, some kind of blob store. You can also get this kind of thing on-premise, your storage vendor, EMC, Dell, someone like that, will have usually described as an S3-compatible store that you can use the same way. And that's what makes the data durable and pretty low cost to store thousands and millions of metrics over time. So when the data comes in, Cortex has a process called a distributor, and you can run as many of those as you like. You can scale out horizontally. You could run a couple for resilience, or you could run 100 to handle a high load. And distributors pass the data on to ingester processes. And what they do is they collect up the data and they make what we call a block. A block is about two hours of data, which might be, I think in terms of a gigabyte, send that data up to the store at that two hour mark. Distributors and the ingesters use a key value store, so that's something like at CD or a console, to decide which ingester is holding, handling which data. And if you add more to scale up, then they automatically figure out how the re, what we call sharding, shards of data, different slices of data go into different machines, is handled automatically by Cortex. And one more thing on the ingest path, once we've created these two hour blocks, this process called the compactor will download a set of blocks and create a 12 hour block and even a 24 hour block. So we crunch the data down, we remove any duplicates from the data and we create these bigger blocks which are a bit more efficient to work with over time. On the query path, here on the left I've got Grafana. Everybody uses Grafana to draw dashboards, right? And it will send in queries in the Prometheus promql language and they're going to be served ultimately from data in the store. So the way that works, first of all we have a query front end which queues the queries and also splits them. So if you send in say a seven day query it'll get split into seven one day queries which are more efficient to process. Those are passed to querier processes to do the actual work of fetching the data and interpreting the promql and again you might want to scale that out, you might want to have 100 of those to handle your load. And there's another set of processes called the store gateway. Those are the ones that fetch the data from the store and again they have one of these key value stores to decide who's doing what and the purpose of that is so we don't have like 100 different programs all downloading the same gigabyte block. They will again shard that divide up the work into slices automatically using the key value store to persist who's doing what and it's more efficient that way. Now lastly we have a lot of caching on the query path. We cache the data that comes from the store, we cache indexing and metadata on that and we even cache partial query results and this is done again to make things more efficient to make your response time on your dashboard blazing fast. Okay let's move to the project update. Since the last time we did this talk there's been four Cortex releases with 293 contributions from 59 authors working at 22 different companies so that's great. Thank you to all our contributors. The maintainer team Alvin Lin who's our newest maintainer works at Amazon Web Services myself Gotham, Marco, Peter and Tom Wilkie the original creator of Cortex all work at Grafana Labs and Jacob Lisey has now moved to Google. The releases that we've done just quickly going through the highlights 1.8 gave us per tenant retention limits. 1.9 gave us shuffled sharding which is a feature to as you get into hundreds of processes to limit the extent to which failures will impact anyone customer or user. 1.10 got us exemplars which we're showing an example of here these little blobs that if they are collected by your program your application that you're monitoring that you can set it up so if you click on one of these it will take you to a distributed tracing view and you can see all the low level detail of what went on for that particular operation and overlaying those onto the metric view is really powerful way to navigate between metrics and traces and get a great insight into your system. In 1.11 we got some new metrics and also an SNS feature I'm going to hand over now to Alvin and he's going to tell you about that feature amongst other things. So today I'm here to talk a little bit about Cortex and AMP we're choosing the foundation of AMP we're looking to many alternatives including some internal AWS services other open source projects we even considered writing everything from ground up ourselves but we ended up going with Cortex because it's rich Prometheus compatible feature set it's supported remote read, remote write from QL, alert manager and rules it was really easy to set up Cortex to start working with Prometheus Cortex micro service architecture is scalable we can scale individual parts individually as needed by using Cortex AMP actually saves a lot of time and energy to offer the service that we wanted to offer our customers we have been running Cortex for more than a year now we run many Cortex clusters to handle the scale that AMP needs during the journey of operating Cortex we have implemented and proposed sign improvements the first example alert manager now supports zone alert replication running software in different zones for higher availability is in AWS DNA whenever we are designing a system or a feature we ask ourselves what happens if a zone goes down can we survive we ask the exact same question when we look at Cortex alert manager and we found that zone alert replication was not available so we submitted PR to implement it we encourage everyone to enable the feature for higher availability second example many customers wanted to have SNS as the receiver for alert manager so we work with both Cortex and Prometheus to fulfill customers need we submit the PRs to both Prometheus and Cortex repositories to implement the feature now customers can send their alerts to SNS and enjoy the benefit of integrating with more types of endpoint like text message email and SQS through SNS aside from providing new features we were also keeping an eye on the operation aspect of Cortex during one of our weekly overview we noticed a metrics from Cortex related to block storage operation failure was spiking up but we looked deeper and found that the metrics was unintentionally incremented for expected HTTP 400 error we quickly submit the PR to have it fixed and runs Cortex using block storage because it is cost efficient but Cortex does not support series deletion for block storage we heard many feedbacks from many customers that they wanted to be able to delete series for example they might want to decrease their cardinality so we proposed a design and submitted a PR for the block storage series deletion feature since Mslunch we received a lot of good feedbacks from the customer and I have to give Cortex a lot of credit for it we are not treating Cortex as yet another open source software and expect others to provide new features and fix bugs we are committed to increase our investment of active Cortex contribution my recent award of Cortex maintenance it is a proof of our commitment we will contribute more and more and hopefully we will have more maintenance of course Cortex being an open source project only gets better with involvement from all of you so please if you are interested, contribute to Cortex any contribution is welcome it can be as simple as improving an error message thank you thanks for that Alvin it is great to have more people joining the community I want to mention we had two CVEs we had two security vulnerabilities reported and we do have a process whereby we ask people to disclose those privately to the maintainer team and then we work on the fix in private and get that out ahead of describing the vulnerability both a little bit similar people finding ways to trick Cortex into sending files off the local machine the first one by alert manager templates which are generally available feature the second one only if you exposed Cortex to let users kind of forge this header this HTTP header and we always recommend you run Cortex behind an authorization proxy that will use some proper security credentials to figure out who is calling and then create this header internal to your cluster let's take a look to the future now on the Cortex project I am going to hand over to Alvin again to tell you about a project underway we wanted Cortex Compactor to be able to compact time series a lot faster than you currently can this will allow Cortex to store most time series so we propose a design to increase the concurrency of Compactor the design is becoming a collaboration between Cortex and Thanos thanks Alvin another proposal that's underway is for down sampling meaning if you originally recorded at one rate say every 15 seconds then you might drop that after a while to every 5 minutes to have fewer samples to deal with makes the system go faster so that's underway now and I wanted to mention something I've been working on the ingestors when they restart either after a crash or because you're rolling out a new version it kind of takes several minutes to get started load the data from where they were last time so that work is actually going on upstream in Prometheus and when we get that settled we'll then bring it into Cortex and benefit from that and everyone else in the Prometheus ecosystem will benefit from that too now question that always gets asked so I thought I'd put it in the video how would I compare Cortex and Thanos they're both CNCF projects that are based on Prometheus they're both solving certainly one problem which is if you have a lot of Prometheus data how can you handle that and we share a lot of code the blocks that I talked about a lot of the code for that is Thanos and for instance the caching, the query front end Thanos has taken that from Cortex so in some respects it's easier to talk about the similarities than the differences but I know people have that question and they want to pick one to use certainly I would recommend picking one and not using both so the main thing I would stress is that Cortex is designed to build a large centralized system it has for instance this automatic sharding and shuffle sharding facilities to distribute the work across many, many machines it also built for multi-tenancy Cortex has had multi-tenancy added but it's not as mature so if that kind of sounds like you that you want certainly a very large system because if you don't want a very large system you'll probably find Prometheus will handle the load just fine but if you're going beyond what a large Prometheus can do and you want to build one of these centralized services that's what Cortex is designed for Thanos I would say shines when first of all you're starting out you're adding it to your existing system it kind of goes a bit easier in that process because Cortex being designed as a large system can take a bit of configuring and tuning deciding how to set various parameters there is a bit of work there with Cortex because the view that it has on the world okay let's flip over to Q&A see you in a minute