 Hello, everybody. Thanks for being there. I know it's quite a long day. That's the last session of the day I'm Simon. I'm software engineer. That's where that Hello, my name is Brian Borum. I'm also a software engineer. I work for Grafana Labs We're gonna do the wave So we wanted to have a show of hands just so who who's heard of Prometheus Yeah, wow, wow, okay. Is there anyone is like totally new to to the community? We won't bite with nobody nobody's owning up to that. Okay. That's cool. Interesting Okay, any Let's yeah, we wanted to make just in case are you gonna do this? Yeah Just in case if somebody was shy and didn't know about Prometheus before entering the room Just a quick like TDR Prometheus is like a complete monitoring metric base solution. That means we offer solutions to instrument your applications via client libraries. We offer exporters to collect system metrics and we have the Prometheus server itself Which would collect and store your metrics. It will also expose query and alerting APIs And you got all that in a single binary of course The key point regarding that confluence coupon is that Prometheus is perfectly well suited for cloud native environments, especially one where Resources are very dynamic. So pods are moving very fast. And this is where Brian is going to explain a little bit Oh Yeah, I wanted to put up an architecture diagram over on the Left we start with where the metrics come from in Prometheus world because we call all of those things exporters That can be something talking to your infrastructure. You can get a node exporter a windows exporter Postgres exporter something like that or something or your application itself can export metrics We collect the metrics we talk to something like Kubernetes the service discovery we find out where everything is running so you configure that and And then we store all the metrics in the tsdb the time series database. That's the the big kind of resource hog of Prometheus I was thinking actually it's a good thing everyone's into LLMs now because they make Prometheus look small right We should call it a large metric model We could make billions right Anyway, so continuing across the picture we have our query language promql And you can set up rules to fire alerts if something goes wrong Prometheus will Has does have a UI it will draw Graphs for you, but a lot of people use other products Who knows what those might be which you're not allowed to name So just a bit of history because I find it interesting to know more about the inception of the project so it started 12 years ago, so quite a long time It has joined CNTF in 2016 Became graduated in two years later right after Kubernetes If you want to know more about like the the project itself how it grew from the the start Recommend you you go you find on YouTube that Prometheus the documentary video very Like very interesting to watch. It's only 30 minutes. So you'll find the link on the presentation The numbers are at the bottom. I've been provided by Grafana labs. You're present this basically the numbers represent the number of Grafana Installations that have at least one Prometheus data source. So we see quite a Nice growth in the numbers and hopefully in a near future. We should be able to celebrate like our first million Prometheus installations Okay, let's just go through what happened in the last 12 months in Prometheus Prometheus, that's that's the the central data store We made eight releases So I think that's pretty good for a 12 year old project still releasing fairly steadily every six weeks We also have the LTS long-term support version of Prometheus and For that that's for for people who are a little bit more scared to take the latest version We're just releasing patch releases to that We also made You know Prometheus project is not just Prometheus the the metric store the alert manager the Java client declared 1.0 with a pretty big rearchitecture The go client had very many releases and all the exporters that I mentioned earlier also got released many times in the last year More specifically Native histograms I got a slide about that in a second We have an experimental open telemetry ingestion endpoint Who who was at the top previous to this about open telemetry and okay a couple of people so you need to tell me what they said because I Didn't go to that one The The next one as staleness that one sounds sounds kind of niche right But have you ever noticed when your container stops the metrics carry on for five minutes One person has noticed that okay two three four people. Okay, trust me. It's annoying Anyway, this is the fix for that. You need to turn it on But go look go check that out you need to turn on the stale list for metrics that have their own timestamps I Won't go through everything we put a bunch of things in over the last 12 months Native histograms there've been a lot of talks about this if you not noticed the basic idea is is that we move from Very low resolution kind of that blocky picture at the top to a very high resolution picture and they're more efficient So They are still marked experimental in the code, but we we have been working all year on native histograms and I would say try them out turn on that experimental feature flag and Make sure your clients are exporting them and Yeah, give it a go see what you think What's big? Well, Prometheus is pretty big as I as I noticed but Has got smaller I gave a talk at the observability day six months ago Went into a lot of detail about that And there is another Thing that's supposed to knock another sort of 15% off the memory called Djupe labels that is a very experimental Separate docker tag So give that a go that was that was released Yesterday. Yeah, well, it's not released. It's experimental, but the the the version was stamped Yesterday so 2.51 Plus Djupe labels and so the feature because you've got lots of new things, but we we want to go further You may have heard about it But as we said promise is 12 years old and we like the team said, okay It might be the right time to cut a version tree of Prometheus Like version two was seven more than seven years ago So here it will come but the the plan is to get it out by the end of the year Ideally in September or October. We don't we are not yet sure if you want to know more about the details We've got like a project on GitHub where you can track the different issues to know exactly what's going to be in the gist of it is We like like Brian said we have lots of experimental features that are that have been added over time And we feel like it's a right time to make that available for everybody Because we've gained confidence that those things will not break or Have been tested enough so that we can offer that for all users It will also be the occasion for us to clean up a little bit Some of our technical depth Having said that for people that that went through the V1 to V2 migration The idea is to do something a bit different which would like we anticipate for for the vast majority of our users That's going to be seamless like it Hopefully it will be as simple as upgrading from a minor version to another I cover the last year with like I think we have a pretty good record of not breaking people Hopefully if not you tell us at the end Now if we go a little bit into more into the details of what that entails We've seen over years open telemetry getting traction obviously And we want prometheus to become like the Back end of choice for open telemetry if we you want to use an open source solution The the main pain point right now for users is that when they ingest open telemetry metrics The metric names and the level names might differ from what they configure or what they instrumented into open telemetry Because prometheus has some restrictions on the character sets We want to drop that and there is work in progress for supporting any utf-8 strings For metric names and level names We've put an example in the slide out that would look like from the promql point of view But yeah, but that's the plan so that It's going to be more natural for users to to go from open telemetry to prometheus The other thing we mentioned is that we have another otlp ingestion endpoint right now Which is again behind the feature flag. We want to make that available for everybody You you don't you shouldn't have to configure it to to have it available There are more to come if you are interested and I'm sure many of you are I'll cause you to go to our block Prometheus blog there is a post from last week by Gotham and Carrie from graph and alabs Where they would go into all the details of what has what is being worked on right now in the same like kind of Area we have native histograms and again the idea is that today you need a feature flag to get access to that Prometheus v3 will be you can ingest from native histograms by default and We also like want to make the migration from what we call classic histograms as use know them right now Into native histogram, which are which are much more efficient kind of again seamless for users and Same thing we have like a github milestone where you can track if you're interested in that all the progress and The another big chunk of work that is happening right now is remote right so V2 which is going to be confusing because we say Prometheus v3, but anyway Remotrite is a protocol that you can use to export metrics from your Prometheus Typically, you would use that to send metrics to solutions like Cortex, Thanos, Mimir Or even to manage Prometheus services that all cloud providers now offers offer to you I won't go into the details, but like the main aspect which is probably worth noting are We have been network bandwidth and CPU optimizations that would greatly reduce like the cost of sending metrics over the wire for you and last but not least We have like a refreshed UI in progress that's been working on by Julius one of the Prometheus creators This page on the screen is the four people that know the existing UI. That's the alert page So you you will see that you will notice that it's not quite different from the current page But it looks cleaner and nicer. So again, it's the idea that We are not going to change everything upside down But we'll want to like make it even more efficient and nicer Okay back to me Yeah, I want to speak a little bit about the community Which includes you. Thank you very much for for coming along so late in the evening Since it started Prometheus has hit 12,000 in a bit commits 872 Contribute whose hands up if you if you made a contribution of PR or anything at all quite a few in the room and I want to we have some of the maintainers here Joe Can we get the maintainers to stand up who are here? Augustin And and Simon and me are maintainers if you hadn't figured that out Yeah, Arthur Arthur is one of our newest joining the team one of the maintainers of the client go Library so thanks very much for for joining up and At this point, let me point out. We this is a 35 minute session We're trying to get through the slides quite quickly to give you time to ask questions. I don't want you to ask them right now Stay but You know, if you're thinking of a question or you have something in your mind get get that fully formed so we can get through those questions Generally the best questions are ones that don't involve me knowing your config but Aside from that We'll take questions on any any aspect of the Prometheus ecosystem When we finished the slides. I yeah, so this topic is about Something that's but someone was talking to me at the Prometheus booth yesterday, and I thought this would be a good thing to highlight So you can come along to the Prometheus repo or any one of our Few dozen repos around the project and you can make your request like you know, I would like Prometheus to do this I would like client go to do that When you do that you are the product manager Right because there's no company behind Prometheus there's no Prometheus Inc. I don't have shares And we did not hire any product managers So I think just think about it that way you you need to advocate for the feature. Maybe you need to find someone to implement it But we are all volunteers working on the Prometheus project so That said we we would love more volunteers we'd love more contributors We are working on a governance refresh To make it easier to have To people to sort of move up through the ranks of the project make that clearer So do Do come along Yeah, and the best way to start is tomorrow in fact So we've got four experienced Prometheus developers that offer you to to attend a contrib fest They will be there to help you there is a topic like they have some ideas of what they want to do But I'm sure that they will be also delighted if you have like you come with an issue and you say hey Can I work on that? Can I submit a per request? Can you help me show me how it works? They will be yeah Very happy to to help you Obviously we have like events Regular events like we have Prometheus dev summits which are open to everybody again It's not only for the maintainers you can find on the website or calendar with all the different events that we organize virtually and Hopefully in September will add another prom come again probably in Berlin We hope and we hope to see many of you there That's a bit different from this type of conference because they are it's really Smaller it's kind of a we like to call it like a family gathering in a way. Yeah So again, don't be shy. I think tomorrow is a great way to like start contributing to the projects And I think we are done for our part now your questions, please So we Who's are you gonna? Hello, you said you Supporting a native histogram format. Will that require a new exposition format from things that published metrics? Yes, so if you want to expose native histogram right now You have to basically upgrade to a recent version of the the Prometheus client library that would offer that and The exposition format is a bit different because instead of being text format. That's going to be protobuf Which would works better for that kind of representation of histograms There is an idea floating that you should be able to Still expose the kind of classic histograms, but have them stored and I'm speaking under the control of Bartek obviously that You should be able to like the primitive server should be able to say that in the format of the native histograms so that you can still get the Performance improvements that come with native histograms. But yes, the I guess right now the client going supports native histograms Java client 2 Yeah, and there should be a way also from the open telemetry SDK to convert that to and Their exponential is histograms into Prometheus native histograms too Hey, thank you. Great work by the way in great presentation You mentioned that the utf 8's Charles will be available Supported and metric names and label names So open telemetry Matrix names will be fully supported. However, there's still a difference. Will we still be required to add the unit? the suffix and mean metric names I Think we don't know I Think there is a I'm think I've commented on an issue about that in I mean, that's strictly speaking an open telemetry issue I because Everyone wants a standard, right? They want to sign up to the standard and and not have people like me mess around with the code But the the standard gets it lays down Something that is kind of inconvenient like you have to you have to change all the dots into underscores or something like that. So So we say, okay, we're gonna change Prometheus So you don't have to because Prometheus used to ban the dot the full-stop character. That's why that was done and So we take away that restriction, but now somebody has to go change the standard, right? Has to go through governance of the standard and so basically the same question for units and it's a little bit more complicated because You there there are corner cases where you can have the same metric to two metrics. They're identical except for the unit I mean, that's not a great idea, but but apparently that's supported by the software and So if we stopped adding the unit to the name it would break I do your question is a very good one and I there is work in progress. I just don't know what the outcome of the work is Yeah, it's still open question But the other point is that the promql system like if you look on recording crawl you want to see the metric unit immediately There is no like system, especially if you look in your the ammo to like expand it to exactly what unit it is so there are lots of benefits to actually keep the unit and There are really no good reason to remove it other than maybe some data the duplication So it's literally yeah discussion among maintainers and yeah, please bring your arguments as well, but let's see Thank you. Any other question? I'm sure there are a lot Thank you for the presentation. I have a fun topic open metrics and exemplars and Exposition formats so what's the status of open metrics and I'm mostly interested in exemplars So should we just I don't know if the protobuf exposition format for supports it does okay Yeah, so just open metrics. What's the status of it and whether we should care about it or not and so So yeah, in fact, there are like three Standards in a way that probably to support or will be supporting its Prometheus from it is format open matrix format, which is some particularities Which are a bit different from the primitive format and they are open telemetry obviously The idea is that yeah, we would we want to support exemplars for everything for in all cases but there are challenges Right now for native histograms is how many exemplars are you going to store in the database Given that you have much more buckets than before for instance Now back to open metrics The like the idea is that the open metrics itself is going to be to land into the Prometheus organization and We that way will be more flexible and make it evolve so that it's going to be Like not stuck on a previous version of the of the form of the standard, but we'll adapt to all the New new things that we are envisioning. Yeah Yeah, so future is bright You did not mention it, but I have heard whispers about remote read. Yes or no, is it going to be better than remote, right? Or robot read you want to take it Bartek? better in what sense Oh, it's just to be On it just what thoughts have you been putting into it? No, it's a good question. So I think it's well used and when well-matched like we added like a streaming to it Like very very long time ago like talking I came to two two two years ago for Thanos purposes another kind of Projects and I don't think there are like any plans for that other than just making sure it's stable and Very well versioned. There are lots of plans for remote right by the way like we are evolving and there will be a remote right to zero version as well and Yeah, so stay tuned. Yeah. Yeah, but yeah, really much with is going to stay It's not because we have a new version of point twos that it's going away But yeah, I don't think they are like unless you have specific use cases like where you would see improvements I think everything is workable. Yeah Okay, we have a question from very very far away Hello, thanks. Hello. I wondered if you have done any thinking or work related to large scale and sharding in particular, maybe sharding of data that's being scraped and whether there's going to be anything client side to make that kind of standard or Or and another kind of service discovery related Issues related to you know scraping large amounts of data without a single Prometheus failing So If in terms of splitting up the data across multiple Prometheus The we have the hash mod operator, which will let you Automatically Shard targets across a set of fixed number of Prometheus If your question is about one single target, which is too big for the Prometheus to handle nope nobody I believe has worked on that and Yeah, I mean We have only two cases when we see this CubeSit metrics and they essentially create their own Instrumentation that that solves it like you can shard it manual You can say CubeSit metrics give me essentially free replicas and that will be sharded for you And I know Istio might have some kind of like Problems like that, but then they have like a very customizable a telemetry you can configure So we didn't see any need for like something standard rights Maybe some instrumentation or even Prometheus or open metric specific But please if you have this problem like literally put issue on Prometheus and we want to highlight that there are solutions Just it wasn't pressing enough Yeah on target sharding we had Arthur and Nicola did the talk on Tuesday Which is going to be available soon about using the Prometheus about or to shard targets automatically so I recommend that you watch it if you did we are not present Question any question left going once twice Just in time good I mean, I wasn't sure if I wanted to ask you but giving you a alright Let's say Prometheus treat a dough hits. How is the Prometheus operator project handled in is like the timelines Are they aligned if treat a dough hits dense when Prometheus operators starts running? So I can probably take it because I'm Prometheus about our maintainer Right now we have like we follow the same schedule than Prometheus itself So we have six weeks between each release. We just have a little offset so that we can just Wait for Prometheus release to be available to publish the apart or Again the philosophy general philosophy of Prometheus victory is that we are not going to break lots of things so that should work again like Mostly the same hopefully We don't have all the details how the release will happen like how many release candidates will need to have etc But I'm sure we'll work out something like maybe we need some branch in the apart or itself Just to test in advance that it's going to work fine, but yeah, I don't see big blockers there Yeah, thank you All right last question still room for that All right, thank you so much. Thank you for yeah, thank you