 All right, hopefully everybody can hear me and can see my screen. It's time for the Prometheus Functional Update. All right, so Prometheus team, who are we? My name is Ben Kochi, I'm the Prometheus team lead. We also have Joshua, Jose, Mike, and Powell on the team. But soon, we will no longer be doing Prometheus Functional Updates, we will be doing monitoring team functional updates. And our new mission will be not just Prometheus, but we're going to be doing metrics with Prometheus and we'll be adding open tracing and logging features to GitLab. The big news from our team this month is we've finally fixed and stabilized the Unicorn Gem integration, which gives us really great metrics inside the Rails app. This has been quite a lot of work and I want to thank Powell for all of the hard work that he's done to stabilize and improve the performance of the Gem. It's now super fast, super efficient, and something we're very proud of and we'll be contributing this work upstream as soon as we have some really good confidence in the long-term stability of the code. And coming soon, we'll be enabling this in the gitlab.com production so that we can get direct instrumentation out of the Rails application. We've also seen some very good steady growth for the Prometheus project use. So basically this is a graph of the usage that as users see the Prometheus button in their projects, in their GitLab installs, they've been enabling this and allowing Prometheus to monitor their apps on their GitLab installs. And we've seen pretty good growth of this over the last six months. We also have, in 10.3, we have a few features. Basically, we've been enabling more metrics and more parts of the code base. We've now got some integrated browser testing. It's a really neat little feature that allows you to get direct, quick feedback for applications based on a utility called SiteSpeed and we'll be hopefully doing more of this in the future. We've also got other new features. I won't continue to read the slides. We also have a nice new Grafana dashboard that we're gonna be shipping in 10.4. This is just a very basic starting point for helping users monitor their own GitLab installs. We're also gonna be shipping a nice new feature called Custom Metrics that will allow merge requests to show not just the default CPU and memory usage metrics, but also any kind of metric that an end user would want to view. In production, forgetlab.com, we're now using Prometheus 2.0 as the default instance. So this has been very good since it has a much higher capability for ingesting metrics and letting production get the most out of Prometheus. And that's it. Let's go to questions at window, anything. Hello, chat. Yes, the presentation, I will update the slides with the link to the presentation. Sorry, I'll update the calendar entry with the link. I had a question. Can you maybe give a better, not a better, but a more real-life explanation of how I would use the option to, you said I can start tracking different events or different points all over my app or my service. Now you gave an example with Shop and Car. Can you give an example of why I would use it for and how I can benefit from it? You're talking about the Custom Metrics feature. Yes. So, say, for example, you have a login screen and you might wanna track how many new logins per second are happening. So that you could see if there are, you could see if you make a change to your login screen that the new logins per second drops off, which would be extremely valuable quick feedback that you've made a mistake to your login screen and none of your users are able to log in. Another one is error rates. So we don't actually automatically know what your error metrics are. So you could add your application error metric as a custom metric to the input so that you could get an immediate feedback that your application is throwing errors. All right. And obviously there is overlap with other tools that people are doing this already with, right? Yes. Yeah, so sometimes if you're using, for example, this part of the reason why we're gonna start integrating open tracing is that we can start getting those kind of, those like exception handlers and other tracing events into the interface. Yeah, so it's a little more color here. Basically right now in the Prometheus monitoring system that we have today in GitLab, we have support for the kind of well-known exporters. So these are the exporters that are published and been developed and kind of linked in the Prometheus page. So this would be things like Nginx and HAProxy and like Kubernetes CPU memory support. But if your application uses it, that's great, right? So for example, you can pick up the Nginx metrics out of say the Kubernetes ingress and get instant request rate, error rates and things like that out of your system, which is fantastic. But a couple of things that Ben's mentioning here is that if you actually have instrumented your own code for Prometheus, there's no way for us to know what those metrics are. And so you need a way to actually enter those and so we can begin to track them. And so this is a way for folks who are using a system that admits Prometheus metrics that isn't kind of one of the current library. And this is a way for folks to go ahead and get those entered and then get tracked along with all the rest of the Prometheus metrics that we track as well. And this way it would show up on their dashboard and the environment. And eventually it would also make its way into the MR widget which right now only shows memory but we're also looking to enhance that soon here to show other metrics that perhaps have changed as well more significantly than memory. Makes sense, thanks. So one of the questions in the chat was is the custom metrics configurable at the project or the merger quest level? It's at the project level but you can have multiple charts so you can see different things that are affected. Yeah, and that's just because James, the metrics that are being emitted by the application code probably aren't changing MR to MR. And so if you're emitting new ones, for example, you can just add the new one to the metrics list and that way going forward it'll be picked up the monitor, does that make sense? Yeah, cool. All right, any other questions? Going once, going twice. All right, thanks everybody. Have a nice day and I'll see you in the next meetings.