 Hello, everyone. We are here on the maintenance track learning about Thanos. And yeah, some kind of introduction about Thanos. And we are super excited to be here in person finally. And it's our first really CubeCon after COVID. So it's amazing to be here. And hopefully there will be some interactive time where you can ask some questions because this is, yeah, we can have this versus whatever we have in virtually that wasn't so possible. So today, we'll learn about Thanos. But we also learn about other projects in the ecosystem, open source projects, that really are fulfilling the whole observability story. Because as we know, metrics are kind of the cheapest and the most reliable first observability signal where you have your instrumentation, you have your monitoring, maybe you have Prometheus, maybe you have Thanos. And then you end up having a kind of reliable and quick to use alerting and debugging platform. But it's not everything, right? There are amazing tools that you can combine with. And we'll discuss what are the challenges there and what are the ways to mitigate those problems. But before that, short introduction. My name is Bartek Plotka. And I'm principal software engineer at Red Hat. I maintain many, many projects in open source. And the biggest one are Prometheus and Thanos. Really mainly about Golang. And my passion towards Golang and open source kind of made me also create a book. And this book is coming in December. It's called Efficient Go. And it's really about, you know, performance and efficiency of Golang in a very pragmatic way. And there's lots of, yeah, I couldn't stop myself. There's lots of observability there as well. And I have Kamal with me. Hello, everyone. The name is Kamal. I'm also maintainer of couple of open source project, not as many as Bartek, I guess. And recently I joined Polar Signals. We are an observability company and we are building continuous profiling products. And we have an open source offering called Parker. We're gonna actually demo on it. Also, we recently open sourced another part of our company, which is an Arctic DB. It's an open source database columnar storage, specifically for the observability signals. And we are looking for database engineers. So if you can go to our website, you can find the details. But today we're gonna start talking about Prometheus, which everything actually start with. Prometheus is a monitoring system inspired by Borgman, which is a tool created in Google. With the Prometheus we are providing highly reliable, easy to operate monitoring system. A lot of components just included in a single binary, like alerting, like querying. Prometheus also has a query language, which is very flexible for you to just like monitor and create alerts. So, and that kind of brings us to Thanos, which we may just say distributed Prometheus++. Prometheus has certain problems like it's just a single binary and you can only just vertically scale it. And when you need highly available system, you need some additional facilities. That's where Thanos comes in. With Thanos we are providing highly available, horizontal scalable data storage for Prometheus, basically. So Thanos is nearly five years old now and we've been an incubating project nearly three years, if I'm not mistaken. We recently crossed like 400 contributors and we have a lot of Slack users, so we are a really vibrant community. Like we always active and we answer your questions. So if you wanna reach out, like just join our Slack channel. And we just recently crossed the 10k stars on GitHub, which is the most important metric of all, as you all know. We have many, many adapters, many enterprise adapters, like the people are using this production and like with really, really highly scalable environments. Okay, now let's see how we actually get from Prometheus to Thanos. In Prometheus we have different components and all these components kind of like living in the same binary. We do this because like with a single binary, with a single process, we can actually provide more reliable solutions and it's easy to operate. Like of course there are some drawbacks, like you need to install Prometheus closer to your workload. So for that, like for each component with Thanos, we created another dedicated component, like for the query pieces, we have Thanos query here. For scraping pieces, we have a sidecar where you can actually deploy near Prometheus and for the rule engine, we have another thing called Thanos ruler. So Thanos has different deployment models. One of them is as described here, it's a sidecar deployment model. You can just put the sidecar module near your Prometheus's and you can have a global view of all the Prometheus's that you connect with that sidecar. Another model is to have an object storage for like long-term retention. Then with the help of the sidecar, you can ship all the blocks to the object store, which is like cheap to store and then you can just query using another component called Thanos store, API store, which becomes the API part. One last thing to mention, we also have another component called Thanos receiver, which gives you the ability to actually push your metrics directly to another storage. It depends on, depending on your network topologies, you may find this useful rather than having a pool-based model. So how we actually do this, thanks to the store API, which is a flexible API for us, each component kind of like exposes this API and from the Thanos query perspective, this is seamless, right? So when we check the store API, like Thanos ruler is a store API, receiver store itself, even sidecar itself. When we say the store API, like it's not just a metrics, but we have all other meta information that are available with that. We can get all the exemplars, for example. We can get the alerts, we can get the recording rules, like everything. So with the flexibility of the store API, we can just like interchange all these components. So what we've been recent to working on is a couple of improvements, mostly around like performance improvements or kind of helping to community to use different solutions for their like caching backends. For instance, if you prefer to use Redis, now you can actually add Redis as your store cache backend. One other cache backend we add is group cache, which is easier to operate. It's not an external dependency. You can just embed this into your Thanos components and you don't need anything else. Also, recently we improved our like TLS support of 1.2 by default, more security improvements. We've been, as I told you, working on a lot of performance improvements. One of them is GRPC query API. It's not just for the performance. We also has a new model of querying, which is like push down querying, which is like we are getting the query and pushing down to all the store APIs and then collecting them back, kind of a may produce model and it really helps us to improve stuff. Somewhat similar to that we recently introduced. This is not merged yet, but it's about to merge. This will help us to get the data from store API a little bit faster. And as we talk about push down queries, we just added some aggregation support for the query push down. The PR is already there. Like we keep working on that, but if you want to know more about query push down, we have another talk coming. I guess it's in two hours, right? From Philip, he will be going through all the recent changes about the query push down API. And the big news, okay, we finally met in person and we had our dev summit and we decided that our APIs are pretty stable. We haven't changed them for the couple of even years, right? So it's time to actually bump the Thanos to version one. For that, we have couple of agenda items that we need to take care of, but we targeted the next KubeCon for the Thanos 1.0, basically. This doesn't change anything for you, I guess. Like the APIs are pretty stable. We just need to clean couple of flags maybe so that we won't be supporting some experiments. Anywhere, anything else to add in here? Yeah, it's great. Cool. And it's demo time, actually. Let's see Thanos in action. Yeah, let's do that. So we're working on this very, very hard and like literally before the talk as well. So let's see what happens. I will start by just starting the demo setup and I will then explain that demo setup to you. So it's written in kind of Golang unit test and I will talk about that maybe later, but I will start in meantime. It starts pretty quickly, but then metrics have to propagate. So it takes some time. I will discuss the actual setup. So the setup is following. I wanted to kind of mimic like, you know, the proper production setup that you usually would have with Thanos. This is one of many options and as Camel mentioned, there are kind of, ooh, you see, it's starting. There are kind of three major deployment models of Thanos. So this is the most complex one that you probably should do at the end of your journey with Thanos, like the ultimate SAS kind of metrics. But at the end, how it works. We have, I kind of, let's imagine we have like customer cluster. So a separate cluster under, you know, different network where we run our applications. So I have demo application. It's like simple pink server. It's like a simple, you know, slash pink HTTP. And I have finger, so kind of clients that are spamming those things within the pink pink web server. And I want to observe my pink and make sure it's healthy and I can debug that and I have full observability. So I kind of install lots of goodies into that. So first I instrumented up with metrics and I use Grafana agent, but I could use Prometheus as well. And Grafana agent or Prometheus in this case as well, actually Grafana agent is embedding Prometheus. You know, we scrape metrics from this pink application and then remote ride. So kind of export to a separate cluster. And we call it observatorium, because why not? It's kind of like, you know, imagine you have like observability centralized cluster. And it's very simple setup at the end. Maybe, you know, it's the most complex deployment, but it's still kind of simple. You only have three microservices, courier, receiver and ruler from Thanos project. All of them are one binary with like different flags. So you run that as your backend and as your UI for querying as well, the alerting. Now you, so we have metrics propagated into our observatory. That's great. But we want more. Let's grab, oh, I kind of missing the line, but essentially Grafana agent is then tailing the logs from the file from the Spring server and then sending again to Grafana Locky, which is another backend, but for logs. They call it Prometheus, but for logs essentially. And we store logs there again in our centralized cluster, imagine. And then we do similar with tracing and we use kind of push API from OTLP, so open telemetry protocol, and that pushes to Grafana agent. And then agent is able to push that using some Yager format to Yager, which is right now the simplest Yager in-memory storage. And again, Yager allows us to store traces and query them. And all of those three systems has some UI to use. But we want to make it even more nicer. So we want to add continuous profiling, right? So what it does is that we installed some agent on the customer side. And this is Parker developed by Polar Signals. It's open source as well. All of this is open source. And it scrapes a special Pprof endpoint, so instrumentation for profile. GoLang has it, Java has it. But there are also other ways how you can grab those continuous profiles. So with this, we have really, really nice overview of what's happening. We have all the magic observability signals, metrics, logs, tracing, and continuous profiling. But as you can imagine, this gets a little bit complex to use, not only operate. And hopefully, I mean, industry is moving to making it easy to operate and scale and use. But what about usage? What about user? Imagine I have to go to Thanos UI, Rafaana UI, Yager UI, and then Parker UI. It just kind of like scary, and you have to learn three different kind of search for different search engines and APIs. It's kind of crazy. So I was thinking, it's actually not that hard to implement a microservice that will understand those APIs for you and will expose it in a very simple correlation mechanism. So we wrote a very complex correlator AI that will essentially correlate and hopefully will help us in our journey. And I'm joking, of course, I wrote that two days ago. It's very simple service and it's very yellow. But the point of this demo is to show you how easy is to implement with 100 line of codes, like amazingly useful correlation mechanism that ideally we can bring up to all those UIs some day, maybe we embed it in deep browser. So maybe it's easy to have, maybe in some point we can work with those projects like Thanos, Loki, and Yager, and any other to really embed this kind of service in our browser so we can kind of show you way around. Okay, but enough talking, let's see if this will work. So the situation is following, right? We have some, let me quickly rearrange things. So we start with Thanos UI and Thanos Allerging UI. So you imagine we have our ping service alert, we are happy, la, la, la, la, nice. We can go to conference and just enjoy our time. Unfortunately we are on call and imagine you get paged and you get paged on your phone, you check and there is a link to maybe Thanos UI that contains an alert. And you can see there are too many errors. Well, this is what I can read from it, right? And there is some complex query to make that, to express that. But what I do next, hopefully there is some runbook, there is some links that maybe to dashboard or something. But what if you don't have those? What if you are in the middle of the night suddenly and you really want to fix it very, very quickly? So maybe you go to Thanos UI and what do you do? What do I query? Maybe I can just search something. Okay, there are some requests HTTP. Sure, but there might be more services that I'm observing. How do I get to the information I really want, right? I have to understand PromQL, I have to really understand what is happening. So maybe there is things we can improve here. Now let's go to our logging signal. And I open just main Grafana UI. And if you are familiar with this, you can do a lot here. But you have to think and like in the moment on the on-call kind of page and it's incident situation, it's really hard to kind of find the correct data source and really try to maybe, I don't know, find something. It's kind of hard. And the same with tracing. You go to tracing and say, oh, I want to, what should I do? What tag I should use? Like how do I find my not only service that is instrumented with those traces but also the exact situation, exact error that is happening, right? And again, because we have four things, there's also PARCA and how do you use this kind of query filtering mechanism of PARCA? So at the end it's kind of crazy. So this is where we thought it's kind of simple to write automation that will understand those things. And again, take it as an example, it's just I'm not front-end developer so I didn't build a nice UI for that. But think about it. What I have is like a simple UI or like simple input. So I copied the other name, pick service too many errors. I just put into that. So being service too many errors, I want to use exemplars and I will tell you what that later on. And I click correlate and voila. It actually gave me a lot of useful links I can use. So let's go through this page and imagine it is embedded in your browser so it's easy to use. But correlation this AI found that there was an alert. It's firing for a certain service with certain instance with certain job. And what's more amazing is that it found out that this alert is actually using HTTP request total counter that is also instrumented with exemplars. And exemplars is essentially a way to provide an example request ID that contributed to those errors. So we know that there is a certain request that goes among many that goes this alert to fire and we put that into exemplar, right? It might be thousand, but we only want one example. So you can see that my service queried actually tunnels and make like two different API calls and to gather that there is a request ID that is correlated to those errors that cause alert to fire. So it constructed useful links for us. So for example, we can go and check exactly the query behind the alert here. So we can see confirm error rate is above 65%, right? And even like what are the actual error rate requests per second? It's slower than it's less than one per second. But in this actual case, it's not very useful. So maybe let's try a different correlation, maybe as I think it was amazing talk on Prometheus Day from Bjorn around anomaly detection. And he mentioned that it's useful to have those theories that you want to check, but only few of those. So we click from theories that what could go wrong and where is the best place to look for the reason why things are failing. So let's go to the logging. Maybe logging is useful. You can see I didn't need to craft this query. I have, and you don't need to learn to look well. I have already kind of automation crafted me this query and found that for this request, we have only two log lines, unfortunately, for starting the ping and finishing the ping. Is there anything useful we can get from that? Unfortunately, there is not even any, is there error? Yeah, I don't think it's error. Well, we are unlucky here. At this time, logging was not very helpful. Let's try tracing. So with tracing, let's go that correlation, where kind of directed us to tracing. You can see that immediately they gave us, this correlator gave us the trace view, right? Which exactly is a response for this request that failed and contributed to this error. So it's very, very useful information. You can see it's an error and the whole request took some time, but there is this evaluate ping function which took over one second. And we can see it turned error because decided to not return success. That was not helpful, but at least we know a function. So we already knew something, right? So we are unlucky this time, but who knows? Maybe on next alert, that information would be helpful. So what we can do, let's go to the last link, which is parka. And let's check if continuous profiling will tell us anything. And you can see our correlator crafted a query that checks the exact job and profiling in the exact moment when this kind of exemplar was happening. And already told us a lot. So we are actually spending a lot of CPU time, let's see. So apparently we spent five minutes in total across all the pings on this nasty back function, right? And if you go to my code and search for this, actually let's me copy, what happened here? So yeah, this is my ping code, let's search. Oh yeah, I have this nasty ping that just does nothing and burns CPU, exactly this. So yeah, I found error, just delete it and now we should work, right? So this is kind of the story we wanted to show. Should we show experimental feature if this is actually working? So let's try. So we have another link here. And this is like, so what you have seen is totally doable right now with master or main of all those projects, like latest releases. But Parker is going crazy with features and one thing which is amazing is that we can use exemplar. So trace ID, so usually you have profiling that is across all the things that were happening in the binary. But trace ID, like we can label, you can filter out things by certain labels and we could filter out essentially only CPU time spent for this request. And it's experimental and it was not working before. Whoa, it works. All right, so, yes, oh my God, yeah. And you can see that Corollator crafted a query which directly asked for request ID and you can see that just this request took exactly one second of CPU time. This is incredible because if you have a larger binary that has, do lots of things and maybe it does a lot of other CPU time work, you wouldn't see the exact problem, performance problem in this request. So this is, yeah, it's amazing. Congratulations to Polar Signal for really hacking profiles. It's amazing. So that's for the demo side. I want to show you one more important thing. So how we even constructed this exemplar and those kind of few, all those observability signals together, the key of how we were able to correlate those stuff is in dysfunction, it's like a middleware. So it's not like very visible in the code because it's hidden in this helper that wraps your HTTP handler for pink. And in the go, but it will look very similar in other languages. So as you can see, we create a registry, from it use registry from our official client Golang library. We create a metric for HTTP request total. Does this work, yeah? Then we grab exemplar if there is any. Well, grab trace ID really, if from tracing library that I wrote that wraps open telemetry library by the way. And I wrote myself because I wrote another one because kind of like open telemetry library was too complex to kind of use for normal humans. I wrote like opinionated smaller packages, but anyway. So if it's sampled, if it's have trace ID, it will return trace ID that is then consumed into official instrumentation. So this trace ID is connected to our HTTP request total metric, which is kind of like logically and semantically making sense, right? And of course, we have to wrap with some tracing library to have this trace ID and all of it, right? So I will not explain everything, but you can grab my slides and you can reproduce everything. It's like in our BWBlocker later repository, you can run a unit test and it will spin all those things. And to achieve this, I use this one library you can see here, efficient go end to end. And if you want to learn more, join our talk today on, you know, 4 p.m. 50-50 and there are Jessica Maci here who will be discussing, you know, just this end to end framework because it's running containers on your local laptop. So you can totally test on production many, many things and we use that to test TANOS on every PR. So we spin up the whole TANOS deployment complex systems and it's ultra fast. So I recommend checking that. And we also want to mention that we do mentorship. So if you have any new joiners into, you know, open source, we are there to help you. And you can also watch a video from Lucas to learn more. And with that, I would like to thank you and yeah. I guess before the questions, one disclaimer, like you seen that we added exemplars to the alerts, which is a feature that we are going to add in client goal and pretty soon, next version. Same goes for Parker as well. The trace ID queries will be included in the next release. So this is very building edge demo. And we have, we have time for questions actually. I think five minutes or more. So any questions that you want to shut out, I guess, because, or maybe you want to help? Yeah. Thank you. Thank you. Let's just wait, yeah. Hello. Okay. Yeah. So you had to, well, you, you built this on, on what appears to be not very deep tweaks to the entire ecosystem, right? Changing a few things on each client and maybe how far are we from this being the default? Every, every one of these products out of the box, having being configured so that you could just plug in this correlator. And is this an initiative, something that is, that is happening in, in, is it a conversation that's happening? There are definitely conversations, but it's really like organically growing. I think we are there because we already, you know, built stable metric packages that are metric libraries for different languages with exemplars. Now we have to find out, you know, how to work with other like, you know, tracing libraries and logging libraries to, to create those, let's say auto instrumentations for maybe your framework of preference. And, you know, I think you can discuss about even, you know, more magic auto instrumentation in this space. Yeah, so the most important thing, I guess here is the labels, right? All these tools are based on the label models. That's why we, how we actually correlate those. There are a couple of initiatives to just like, ease up the correlation, like open telemetry, which, but you need to do that into instrumentation size, right? So it's kind of like tightly coupled. We want to do this in a more loosely coupled way, which by just using labels. So Bartek and his team is like working on a project called Observatorium, which is also an open source project. One aspect of that we are trying to correlate in that project as well, like without any instrumentation. So you have like one package you install and you have all this, you know, right side thing, you know, by just one click of the button. That's the kind of idea. So we have this part done. And then, you know, we have to correlate all those kind of efforts within different libraries as well to have instrumentation part. But I think what you, I wanted you to say. I will, I will come to that. For this, still with the people of labels, you need a little bit instrumentation, right? You need to grab that tracing idea and put that as in label. What we are also working on with the parka, there's an agent, which is an EBPF based agent and collect those metrics by just using EBPF, which is zero instrumentation. What we plan to add on top of that, some type of like a labeler, you may say, which will just discover the metadata and just attach those to those auto instrumented labels. So it will be somewhat auto-magical. At least that's what we plan. Let's see. Even without labels, I think it can work. But the idea is that there is EBPF, you know, kind of observability direction going on as well to kind of have all logging metrics and tracing and profiling just by EBPF. So hopefully we'll see more improvements on this side. But really, there is nothing better than instrumenting your own application. Only, I trust you. I mean, trust me because we have amazing observability because we kind of added those statements and this is like called open box of observability. There is nothing better than that. But of course, if you don't have this power, you have to, we have to work on those auto instrumentation tooling. Thank you. Okay, maybe last question. Oh yeah, very close by. That's good. Hello. Great talk. Thanks. I was at the open telemetry community meeting and they were talking about adding profiling into one of the signals of open telemetry. Has anyone getting in touch with you to discuss this? Because I mean, that would be awesome because then we can have the trace ID and spend ID into everything and then we have logs metrics and profiling all together. Yeah, it's kind of the holy grail. Everybody wants to go there, right? But those discussions are very early stage. Obviously, we'll be part of that discussion. We are building this open source tool. So there's tag observability, there's open telemetry subgroup. So yeah, just be part of those discussions. If you want to see any future that you want to be added, but yeah, we are just like scratching the surface of this profiling era. Awesome, thank you. I'd like to mention that still, even with open telemetry having all the signals in the world, you have to run multiple agents because at more scalable solution, like if normal production, you wouldn't survive one binary having all the signals. You have to have different, maybe open telemetry, for example, just different instances of open telemetry, but still you have to scale those across metrics and traces and profiles and probably logging at some point, but those are like different characteristics of performance of collecting those. So you have to manage that in practice. You have to manage that in separation. So maybe it's one binary is helping. Open telemetry is doing an amazing job on kind of getting APIs to work together. That's amazing. But the agent side, you have to still probably run in the separate agents. So that's kind of the unfortunate reality. Yeah, thank you. Okay, so last question, maybe. Anyone? I will check questions on virtual side. I have that opened. We have question. No, it was around like a kind of audio, right? So thank you very much for joining. Thanks.