 And with that, we are going to move into some project updates. First off, we'll be pictured to talk about Prometheus. Thank you. We are five minutes ahead of schedule. So for the AV people, should we just keep going, or should we actually? OK, good. Good. So hi again. You'll see all three of us once more for the project updates. So let's talk updates. First, one thing on growth. And again, if you're in the room and in the audience, please don't talk, because it's really, really loud up here. It's not your fault that it's loud up here, but please keep it quiet. So yeah, this is the growth over the years of Prometheus. And the important part here is this is the best numbers, which we as a Prometheus project have. Grafana makes them public once a year during PROMCON. So these are pretty fresh there from last month. The thing is, we only see data sources. We don't see actual installations. So this is the absolute rock bottom baseline of active installations. It's definitely more. From guesses, which CNCF and NINUS Foundation gave us, it's more than a million to tens of millions. We are by now the 91st most start GitHub organization worldwide. Kubernetes is 27th. And the 161th most start repository on GitHub. Kubernetes being 40th. Second, basically, for all of CNCF and pretty much all of the observability things, except Grafana that's higher. So let's go into updates on Prometheus. So first and foremost, we now really have native histograms. They really actually work. And while previously histograms already worked nicely with in Prometheus, you were kind of forced to know what you're doing to be really able to leverage them really effectively. And the thing is, this often meant the cycle of figuring out stuff, like what is the actual behavior, and then only choosing proper cut-offs or proper buckets for your histograms, and then getting the maximum value out of your thing, which works, but it's not the perfect experience which you would be looking for. With native histograms, the short version is everything just happens as if by magic. Just enable them, and you'll see actual high resolution things. High enough resolution that you can do things like this, if you so choose, or obviously also this. Other things, memory usage. Brian is here. We'll also give a talk about this later today. If you are running anything older than 2.44, you basically want to upgrade right now or after this talk. Of course, you will literally have your memory usage in particular if you have any large Prometheus instances. The substantial. Again, as to the details, Brian is going to talk about this later. We have a quite nice quality of life improvement for you. There's something called key firing four, which you can put into your alerts. It's not yet a full hysteresis, but what you can do is when your alert is basically at the edge of your alerting condition, you don't stop the alert and then refire it 30 seconds later or whatever. You can actually maintain this for some time. Also, in case you have a fluctuation, it's a little bit more convenient. Things don't go away while you're trying looking at them. So if you haven't played with this, I highly encourage you to put this to basically a few minutes and see how you like it. We now allow you to actually build your configuration from an abundance of independent files. I'm probably not the only person who has done this basically through shell scripts for basically 2015, I think, in my case, like forever. But the thing is now this is actually a first-class feature. So you can actually start building your configuration from distinct blocks, which just makes a lot of automation easier. It also allows you to have a little bit of a common theme in your different instances, and then maybe have a special one on your own laptop course. You need that one other thing or whatever. It's just much more malleable and more of a building block thing for your config files. On Open Metrics, Open Metrics will officially join Prometheus. Some of you will be absolutely thrilled. Some of you don't even know what that means, that's fine. In CNCF speak, the Open Metrics project will be deprecating itself, and then the assets of it will be joining the Prometheus project similar to what Open Tracing and Open Census did to form Open Telemetry. Speaking of Open Telemetry, there is now an experimental OTLP receiver in Prometheus, while push is still not the recommended way to get stuff into your Prometheus, and it never will for reasons which we can discuss and defend over a beer. The thing is still, the fact is that a lot of people want to directly send OTLP data into their Prometheus instances, and they now can. On the Alert Manager, I apparently didn't copy things properly. So for the Alert Manager, we will have a completely new React-based UI. We are also the same as with everything else looking for maintainers all of the time. Of course, while we have the second largest installation in all of CNCF, we have 12 active maintainers. So more hands is always better. And if you want to get involved, and if you want to maybe at some point become a maintainer or a team member, please reach out. And if you speak React, that would be a nice well-scoped thing to get started on. For SNMP expiritor, we have a breaking change, which is a good thing in this case. Of course, you can now split off your auth block. Again, this is one of those things. If you don't know what this means, you don't care, and you don't have to care. If you know what this means, this is massively going to improve your life. It's one of those things. The short version is basically you're now able to, again, have a more malleable configuration of your complete Prometheus installation and be able to have building blocks to actually make a more specific configuration out of more generic bits and pieces, which just, again, reduces the total amount of config lines by sometimes orders of magnitude. The Windows exporter has become an official exporter. This took ages for licensing reasons, and we're pretty much there by now. This is one of those things which happen silently in the background, and lawyers get involved, and other people get involved, and it's just thankless work. And once you're done, it's basically as if it had never existed, but it was super important. Now we can actually have a proper first-class Windows exporter as part of the Prometheus project. My SQLD exporter now supports multiple targets. So if you have been using, I mean, pretty much everyone is using Prometheus will probably be using Blackbox exporter, but also things like S&P exporter, Modbus exporter, IPMI exporter, Prusa exporter, all of those are also basically reverse proxies. I mean, this is what exporters are general, but they're not tied to a specific instance of MySQL anymore. You can now just have one instance, and query 10 dozen different MySQL servers if you so choose, which also obviously means you can use this for SQL, for SQL if you want, because it's compatible with any SQL server basically. The Java client is now stable. It's 1.0. It has a very nice extra feature, where it allows you to send if you so choose completely native OTLP, but you still, you get to use the APIs of the Prometheus client library, which basically means you have to write a lot less code. So that's nice. And also it supports native histograms. So what's coming? Oh, there's the alert manager. OK, sorry, I forgot to copy this over properly. We also will have a bunch of metadata improvements. You will finally be able to deduce what the metric type is this might also, or this will allow us to do more magic on. This is a counter, and we can actually prove that this account or not just deduce it from the name. So you can automatically put a rate on this, and all of those synthetic servers and quality of life improvements are just easier to do. The created timestamp will be redone also as a metadata, because underscore created is nice, and it's quite useful for a few use cases. But in the worst case, it doubles your metric load, or metric cardinality. And that's not awesome. So we are putting this into metadata, and they're actually going to be persisted in the TSTB. So it's not the case anymore that you basically have all of this in RAM. And once you restart your instance, this is gone. Some backends already could do this, but now Prometheus is also going to be able to do this. Exemplar improvements, in case you don't know what exemplars are, those are basically just IDs, which you can attach to metrics or to logs, and they point directly to a trace, which means this is a high latency bucket. I know this is a high latency execution, and I jump directly into a trace and know that this is high latency and don't have to self on board or even have this needle and haystack problem of trying to find the relevant traces. And this is also going to be persisted in TSTB and retained in recording rules. And we are going to overhaul the remote write V2, basically making it much more efficient. It will become stateful, but it will reduce spend with method massively. And if you still want to have stateless, you can still use the old style. We are not removing this. One final thing, we are working toward a Prometheus 3.0. We are going to be very, very careful with the breaking changes. There will probably be a few, but maybe there won't. But it will be big enough to actually be a major version release. So if you have any big audacious goals which you always wanted to have in Prometheus and ideally are willing to also invest some of your time into this, now is a really good time to get involved because this is going to be a substantial jump. And with this, over to Eduardo, who there is. Good morning again. How are you? Good. It's Monday. Well, my name is Eduardo Silva. One of the FluentD maintainers, created a FluentBit and founder and actually within a CEO hat. But the most important thing here is like we're going to deliver the Fluent updates. The Project FluentD was created around 2011. And in 2011, the vision was around how to solve logging at a massive scale where you have this common problem of multiple data sources with multiple formats and where, as a side effect, it gets really hard to understand and analyze data. And now if you look forward, after more than 12, 13 years, you will find that the problem is bigger. Now we have distributed systems. We are deploying microservices. And this is really messy. FluentD was donated to the CNCF around 2016. And now it's a graduated project, same as FluentBit, which was a small brother. But I would like to call it today the Next Generation Project. Because there are many reasons around performance and how they were envisioned. Everything that is telemetry data, not just logs today. We're looking at logs, metrics, and traces, and how to do also processing when creating these data pipelines inside this solution. What FluentDats pretty much connects multiple sources, multiple destinations. But in the middle, you can do a lot of processing. Processing means if you're in metrics, add labels, remove labels, if you're in logs, did enrichment, and all things that you can think are possible. The way that we see Fluent, I know that you were thinking this, is not a drop in replacement for anything. One of the biggest bad assumptions when using technology is that we are always looking for the new thing. Oh, this is going to replace this. But when you go to production and you go to the companies, nobody's looking forward to replace everything. Because I'm sure most of you are operators, you don't want to replace everything in one day to the other. Things take time. What you need to do is to integrate. If you are using syslog and you want to move to open telemetry, you're not going to rewrite your applications that you wrote maybe 10 years ago, 20 years ago. We're going to try to find this channel for conversion. In Fluent ecosystem, we integrate with primitives with open telemetry, we do open metrics, and with everything that you can imagine from a telemetry perspective. Where is Fluent? Fluent is used by all the major cloud providers. Oracle just jumped into the boat. But today, Amazon, Google, Microsoft, and Oracle deploys Fluent bit heavily. If you go to GKE and you just apply, I don't know, Kubernetes cluster in Google, you will find that Kubernetes is running on every single node. And this is just a small and maybe outdated photo of who used this technology. And today, we are achieving 10.9 billion downloads. I didn't want to put 11, but this is crazy. This is crazy, because that means that more companies are moving their workloads to containers. I'm not talking about Fluent. First, you deploy the applications. Once you deploy, you need to monitor. And then you go with the default stack. And Fluent bit has been the de facto in the latest month. I think that, I would say, 80% of this number has been in the last 18 months. It's crazy. And so I appreciate this effort from the developers, maintainers, and also thank you the community for this accomplishment. And what we are announcing today. Today, we are tagging. We are tagging, means like in a couple of hours. The next version of Fluent Bit that is called Fluent Bit 2.2. And I want to deliver the updates about this. Fluent Bit, from an architecture perspective, what allows you is to create a data pipeline or a telemetry pipeline, where you can handle logs, metrics, and traces. And as I said at the beginning, you have all the capabilities to connect to different sources, different destinations, and with the ability to filter out data, meaning enrich it, transform it. For example, if you have a log, you want to sometimes remove some keys or add some new keys, right? But here you can say that step one, you have the input plugins, the collectors. You can, we have the new concept of processors that can be attached to the input plugins. We're going to dive more into that into the next technical session this week in KubeCon. But as you can see, you can connect the data from one input to multiple outputs and segment different type of logics for processing. Yeah, and you might be asking, hey, why do I need to do processing in the output connectors? And they are different, really interesting use case. Everybody's trying to, for example, optimize in cost, right? But who's a Splank user? Please raise your hand here. Oh, it's like, yeah, I am. No, be proud of your technology. Just fix how you're using it, right? So the thing is that one of the major complexities with these technologies is like, there's a dependency to send all the data to these plugins. But in reality, you're not using, you're not taking the best of that data in that way. Actually, there are different ways to optimize. For example, everybody's used a more cheaper storage for your data, like S3. You send everything to the S3. You don't process the data, but if you're going to send the same data to Splank, maybe you're just going to remove a couple of keys that you are not interested in, you're going to get all the value from Splank, all the analytics, but with the most cheapest cost. And now, continue with updates. We just shipped a new filter and processor, which is called sysinfo. We found that all users when collected logs, they like to attach the host name, the pretense system, the Linux kernel version, and so on. So we just wrote a simple filter and processor that automatically attached this information for you when the data gets processed through the pipeline. Also, I mentioned that Fluentbit is able to collect metrics from different places, right? One of them is from Docker containers. So if you're using Docker technology and Docker stored relevant information in the file system, right, the sys file system, and the way that this information is stored, for example, for metrics, is by using the C-groups V1 API or V2. So in the past, well, as of a couple of months, we only supported V1 if you were using Docker. If you use Potman, yeah, in Potman, we support both. So this is kind of just bringing in parity to the development of these connectors. So if you want to extract Docker container metrics today in a Linux box, yeah, you can use it with Fluentbit and will auto-handle everything if it's running C-groups V1 versus V2 behind the scenes. What you have in the left that is called classic format is the configuration format that Fluentbit used that we started, hey, let's create our own configuration format, right? And we ended up with something that it doesn't look like anything and sometimes adds a lot of complexity, right, because it's stricken indentation. If you want to have two levels of indentation for different configurations to make it more readable, it gets really complex. So this year, we launched a new YAML configuration format that runs at the same time as the classic format. So you can use both if you want, even mix it, right, in different files. And with a new YAML format, we're trying to have the concept of a pipeline, the inputs, the outputs, and now it gets more easy. Easy, for example, if you're deploying Fluentbit at scale and you have your own tooling to manage configuration with YAML, so now you can take advantage of your tool and have a better experience with Fluent. Now, talking about metrics, and I always say this story a couple of years ago, users were asking, hey, I have this problem. I have multiple agents on my machine. I have Fluent for logs. I have Prometheus not exported for metrics. And I have, I don't know, 10 more agents. And this is called the agents fatigue. And now we got the idea, if Fluentbit is so versatile, they have a pluggable architecture, why you don't provide a plugin to collect metrics same as Prometheus not exported, right? So the Fluent team, what we did, we took Prometheus not exported and did a copy paste from Golan to C, to be very honest. And in C, we replicate the same metrics, the same dimensions, and everything. So now you can get rid of one agent in your environment. This is not Fluent, it gets Prometheus. Actually, we have a really good relationship with the Prometheus team. And the goal is to provide a more unified experience for the end user. And now we just ship a processor metrics. So for all processors that you have in your machine, if you want, you can scrape all the metrics for it. We just ship also a new spotter inside the same agent for Mac OS. You might imagine why we want to monitor Mac OS, but there are a couple of users who are shipping, how to call it, tools for the employees that they can monitor the metrics of the laptop. So they collect CPU, the stats, memory, and so on. And our love it operating system, Windows, right? For Windows also, we just ship another collector. And this collector collects CPU, thermal zone, logical disk, and everything that you need from this type of environments. And some time ago, we integrated with Grafana Loki, but the Grafana team used to contribute back to Fluembed with a goal and connector for Loki, right? One of the downsides of having an external plugin as a connector is that it's really hard to assemble a final image for your own environment because you have what is Fluembed, then you have two layers of, if you're using containers, right? You have to bring the other plugin and sometimes really hard to be updated. But at the same time, the Fluem team started creating a native Loki connector. And now in Grafana Loki, the goal and plugin is being deprecated in favor of the new one because we just bring this parity which took a couple of, like, a year. So if you're a Loki user, you can still use the goal and plugin, but looking forward, we want to use the native integration. And in order to ship what we have in Fluembed, we just ship the connector to retrieve Kubernetes events. Kubernetes events is all events that happen in the Kubernetes cluster that you can use for security purposes or to do some audits or to understand what's happening inside your cluster, right? So Fluembed also has native connector for that. And the biggest one is the performance improvement. We always like to talk about performance because one of the problems is like when you deal with telemetry data, if your agent, your tool, or your solution is not fast enough, easily that tool can consume a lot of resources in your node. And one thing that you don't want is that that tool consume a lot of CPU and memory. And Fluembed is very lightweight. So when we did an improvement, so when you collect data and then you send the data and you do filtering in the middle, we come up with a solution for a six times improvement in performance. Let's CPU this memory and all of this, you will understand that this is a huge win for any type of environment. People don't use one filter. Sometimes they use 10, 15 because they have different business logic and this is the biggest improvement. And a community update. We have one community member, Bill Wilkins, who wrote a Fluembed book. And if I'm not wrong, he was for Oracle, external, not affiliated with Calypia or with Fluembed directly. And he started writing the Fluembed book and now he's writing the Fluembed book. So I was asked to have this slide with a QR code. So if you want to get access to the book for free, now it's your time to take your phone, scan the QR code and if you wanna fill the form and you will get a copy of the early access to this book. The way that Manning works is like you get access to the first chapters and they will notify you after a couple of one months, every two months. Hey, there's a new chapter available so you can consume the book over time. And the final release date is for the next year. And we are trying to align the stars here with Fluembed 3.0 that will be out the next year around KubeCon in Paris. So a lot of things are happening in the Fluent ecosystem around adoption, increasing integration, performance and now I think that the next part is content. Well, and with that I'm going to hand over to Austin to talk about open telemetry updates. Thank you. Hello again. Yeah, so Austin Parker, Director of Open Source at Honeycomb and recently elected member of Open Telemetry Governance. If you voted for me, thanks. So open telemetry has had again a very busy year, a very exciting year and I wanna kinda go through some highlights of what we're really talking about this week at KubeCon. It's important to remember that open telemetry is a huge project, right? Like it is gone from being quite frankly, a dream that about 10, 20 people had five, six years ago and expanded into being the second highest velocity project in the CNCF and measured by the CNCF. It's almost as big as Kubernetes in terms of activity, contributions, committers, PRs, all that. It spans dozens of languages, hundreds of frameworks and libraries and integrations and we're starting to see huge uptake of open telemetry outside of the project where many, many third-party projects now, commercial products are adopting open telemetry as their choice for a standard for observability telemetry data. So what's new? Logging is now generally available in the spec. Logging is done, it is stable, we are not changing anymore. It's in Java and .NET, it's GA, other languages very soon, it's experimental in JavaScript, a few others, Python is pretty close, yeah. So with logging being done, that's all three things, right? That's metrics, tracing and logging, how exciting. Earlier this year, we announced that we are aligning our semantic conventions with the elastic common schema to further reduce the number of standards and to further establish open telemetry as the single best way to express observability telemetry. On this note, we just announced this morning, I just saw the PR go through, HTTP semantic conventions are now stable. Yeah, if you know, you know. Again, we're reducing the number of duplicative standards, we're just trying to make this easier, right? For too long, every single observability stack looks very vertically integrated and it winds up with a lot of duplicative effort, right? Like I love my other observability friends in the CNCF but we just talked about how Fluentbit is gonna let you do stuff that Prometheus does, that Open Telemetry does, that all this stuff does. It's having that standard means less work for everyone and it also means that we can all pile in on sort of that unified base layer, right? And this alignment with elastic common schema does that and also helps move Open Telemetry towards supporting security use cases. Another big thing from earlier this year is OTLP is 1.0. This has been why we're seeing more and more uptake of Open Telemetry as a protocol for transmitting this data back and forth. We're seeing this in the hardware sector, right? In telecom, in IoT, GPUs, there's plugins to get this stuff out of NVIDIA GPUs so that if you're doing wild ML model training, you can emit those metrics in Open Telemetry format. And again, this unlocks a ton of value for you as both a developer, as an operator, wherever you sit by not having to kind of rely on these highly specialized, vertically integrated tool sets anymore. And as I mentioned, it's the second highest velocity project for the second or third year in a row now. And it shows no signs of slowing down. We have an all-time high in contributors and contributing organizations. We have, on a quarterly basis, almost a thousand contributors. We have about 2,000 organizations represented for the year. Another important thing that happened this year is that Open Census was deprecated. Now, I want people to think back to a couple of years ago, several years ago, before the pandemic. There was a KubeCon in San Diego. Anyone, was anyone there? Couple y'all? Who had heard of Open Telemetry in 2019? Did you hear about Open Telemetry through a blog post that had this section in it? This was our initial plan that we published on the Open Tracing and Open Census blogs. And these were our major milestones. In 2019. I'm pretty sure it's not 2019 anymore, everyone. But if you go through this and you look at what we've accomplished, right, we have done our reference candidates. We have formed teams. We have formed so many teams. We have, again, thousands of contributors every month from dozens of companies, both people that are competitors in the observability space, end users that are building, at all sizes of organization and all levels of sophistication, people that are actually using this in anger every day. And God bless you for it. And they are contributing back. They are making this better for all of us. We did officially launch the project at KubeCon Barcelona. We do have these major languages having parity with Open Tracing and Open Census. We have officially sunset Open Tracing and Open Census now and deprecated them, which means we're a couple years late, but it's time to officially celebrate that we have achieved our original goals as a project. We have merged Open Tracing and Open Census successfully. We have replicated its functionality. Y'all did it. Give yourself a hand. This is honestly, like, really, really exciting, right? Like, who's heard of Open Climbing now? Most of you, I would hope. We've gone from 20 people sitting on a Zoom call. A lot of people aren't where they started anymore. Some people have left the project. Some people have gone on to it, you know? And if you had asked me in 2018 when I was getting into Twitter fights with Open Census maintainers about, like, what people should pick to use as the telemetry standard for their project, I could never have guessed that I would be up here today, you know, announcing this achievement. So if you are a contributor or a maintainer, actually, yeah, if you're a contributor or maintainer, can you stand up? Let everyone... Come on, I see y'all. Hotel, get up. Give them a round of applause. We would not be here without all of you. And we wouldn't be out here without everyone that has used the project, everyone that has made a PR, everyone that has filed an issue, everyone that has adopted this. Again, thank you from the bottom of my heart. Thank you so much. We'll talk more this week about the future at our various project meetings and at our various things on the show floor. So this year, we have what's called the Open Telemetry Observatory. This is gonna be on the show floor. It's a little lounge area, thanks to our friends at Splunk for sponsoring that. We'll also be at the booth in our project pavilion. So please come by, meet with your six, meet the maintainers, get to know each other. We would really, really, really love more contributors, more maintainers in the project. There is room for everything and anything. If you do front-end work, you do back-end work, go, Elixir, Rust, whatever. There is a home for you in Open Telemetry. Please come find someone that is a maintainer or find me and I will plug you in to those conversations. We would love to have you. And with that, and come to ContribFest. Yes, when's ContribFest again? Wednesday? So Wednesday will be ContribFest, that's Python and JavaScript? Collector and JavaScript, so go and JS. You can come and actually make some contributions and learn how to get plugged in. So please see Jirasi for more information. And Alex, right there. And Anthony, right there. Find any Open Telemetry person and they will tell you about it. All right.