 And I'm here to talk about Grafana, some history, some current state, and a little bit about the future. For those of you who don't know Grafana, I think it is somewhere... Why is that? Okay, nice. Grafana is a dashboarding tool for visualizing time series data. It's not a database, and it's not a collector's, so it only visualizes data. And in the latest release, we also alert on data. I'm not sure if the lighting is... Okay, this is going to be fun. So yeah, the most common use cases for Grafana are visualizing home automation in... Visualizing metrics in home automation, application metrics as what is users doing, not what is the service doing, Internet of Things, industrial sensors, and also infrastructure monitoring. The intention of Grafana was not infrastructure monitoring, but that's the most common use case by far. A small disclaimer, I was not a part of the project from day one, but I worked with Torkel Ödergård, who is the creator of the project at the same company when he made this, and I joined about a year after he decided to leave that company. So I can't take credit for everything. The reason he created Grafana was that we didn't have any good metrics dashboard at our former company. The former company's infrastructure started to look something like this. We had a process of splitting a big monolith into smaller services. I don't want to say microservices because this was like five years ago, so it wasn't made up yet. And also, there wasn't super, super tiny. We had around 65 or 70 deployable units on a bunch of different servers, but there's nothing compared to the madness of today. Anyway, to get an overview of the system once we have split it up so much was very, very hard, and logs just didn't cut it anymore. We started using the Elkstack for logs, and we liked that for logs, but still it's not possible to get the complete overview of what your users are doing right now. So we started using graphite for user data, not anything regarding servers. We still had our proprietary monitoring system doing checks on all the servers. I don't want to mention the name of the service, but that wasn't the main use case. User metrics was the main use case. So that's why we developed Grafana in a certain way from the beginning. And the initial version of Grafana was a front-end only application. It didn't have any backend. It was just a web application asking for JSON dashboards from Elasticsearch. And when you save them, you just send the dashboard as a JSON blob to Elasticsearch, and that's it. Kind of a simple start, which was good because it got a starting rather than doing everything perfect. Eventually, we got support for InfluxDB and OpenTSDB in Grafana 1.x. And we started having more panels. We got the single start panel and, of course, the graph panel. That was the main part of 1.x. In 2.x, Grafana got its own backend. Everything in Go, of course, because that's what you do. You write stuff in Go. And we decided to use our relational model database to store all the dashboards instead. And the only reason for that was to make it really simple for people to get started. So by default, Grafana uses SQLite, but you can switch it to MySQL or Postgres or something like that. We really wanted an easy start experience for new users. Grafana also got an API at this age for automation and so on. We introduced users, permissions and so on. And this was also about when I decided to quit my former company and start working full-time with Grafana. In Grafana 2.x, we have three new data sources. Prometheus, Elastic and CloudWatch. And we also had a new panel called TablePanel. And around this point, we started thinking more and more about the plugin system. Because we had a rough plugin system in 1.x for data sources. But now we kind of realize that we also needed plugin system for panels. And we also know that everyone was creating time series databases. Every start-up strategy included time series databases at this point. And we also added the coloring, amazing. Added support for mixed panels so you can mix different data sources in one panel. And we found this really useful when one of you infrastructure data with user data and even business data, because they tended to be stored in different databases. So we had some information in Elastic, some information in Graphite, and some information in, no, that was it, in other system that we had custom-built data sources for. And mixing them was really important for us, because it gives much better overview when you have them in the same graph rather than two panels side-to-side. And I will catch stuff that, you can't just catch stuff without, it's so much easier to see stuff and find a problem rather than trying to find a problem by scrolling and comparing panels side-by-side. Graphana 3x. Yeah, but we need them for the camera. So, Graphana 3x. So anyway, Graphana 3x. We rewrote a lot of the UX to make it much, much, much more prettier. Graphana 2x was kind of rough, we didn't really like the design, and UX is important for us, because having people adopt Graphana is one of the key strengths we think at least. Compared to other dashboard tools, we think this is the most beautiful one, and it's the easiest one to get started with. We know it's kind of rough in the edge at some time, but compared to others, it's still much easier and inclusive. We created a... Nice! So, we also created a companion site for Graphana called Graphana.net, where you can host plugins and dashboards, so you can upload your data sources and you can upload panels and so on. And this was a major rewrite from our part, because we had to stabilize the API for a long period to enable the plugin system, but there was no big feature release for the end user per se. But plugins should enable end users to do more stuff. We also created a CLI for managing the plugins. And that was kind of enough history, I think. It was the major changes, and this is the current state of Graphana. So these numbers are public on PlayGraphana.org. I don't think you can read that URL, but in the slides later. And every day, every Graphana server sends some statistics about user count, dashboards, and so on. There's no sensitive information in that home feature. It's very important for us that people can feel safe about using it. But it gives us so much nice data, like versions and so on, so we can see new features taking off or don't take off, or stuff like that. So it's very important for us. The Graph panel is like the bread and butter of Graphana. I'm not going to go too deep into it, but I'm going to give you some important highlights. So the Graph panel is by default displayed with lines, but you can also display it with dots instead of lines. If you find that useful, I usually find this way too intrusive, but in some cases it's much better, because it's easier to locate the time stamp rather than with the lines. You can also use bars. If you use bars, you might want to stack them, because otherwise the big bars will block the lower bars in some cases, depending on the serial name override. We also have a new feature for displaying the series on the x-axis instead of time. So you can do bars on aggregate. So you can calculate the total of each area and display like this. And whenever you create your dashboard, I think it's important that you try to fine tune it to make it beautiful, because that will make adoption much, much likely. And two of my favorite ways of doing this is the series override. So in this example, which shows percentiles, I've changed the draw mode for the 95 percentile to use dots, because the 95 percentile usually is very volatile, while the other percentiles are kind of steady. So using dots for the 95 percentile makes it less inclusive, but it's still there. And the other one is if you have an average of something, you could also draw the mean and max and fill the gap between them. So you will see what the average is calculated by, because sometimes the average will hide information for you, depending on how big the gap is. And this is a small and simple way of making sure you would catch that information as well. And you can do that by displaying series overriding the display tab, and then you specify a name of a series or a reggit, and then you can apply overrides from the settings you currently have in the panel. This is something I love to do, because nice panels is so much, much, much better. Another caution about the graph panel is when people use legends. If you have mean, max, and average, and a query containing aggregation, like mean, then the mean will not be the raw mean, it will be the mean of the averages. And this is very common and confusing because people are expecting to see the real mean value, but that's never the case, because Grafana will only see the average value, because I don't think there's any time series database that returns the average or max value in the query for the series as well. That would be awesome, but I don't think anyone does that. The other panel that's very common people use is Stingerstat, and you can use a few different settings for it. The default one, you can have a sparkline to get some kind of sense about the graph in the background, you can have the gauge, and you can also map a number range to a string, and that might make it very easy to get a feeling for what the status is. So in this case, you can map it to unicodes. So of course, you should have a smile for your service status. A bit caution about the support for unicodes in your database. Grafana support it, but your database might not, so check it before. The third popular panel is the table panel. So this is a way to display time series as table data, and you can either display them with a timestamp per row, or you can do aggregation per series, and you can do coloring with thresholds and so on. I think this also serves a good purpose in overview dashboards where you might want to get a last status of some series, and is it up, is it down, or something like that. A very powerful feature in Grafana is the templating feature, which allows you to add template variables, which you can control from the dashboard, and by modifying them, you will modify all the panel queries at the same time. You can also use all and that kind of stuff to get all the queries. Templating has a few different types. First, there's the custom one, which allows you to pre-define values that you can then later replace in the queries, and you can also have queries that will send queries, in this case, to Graphite asking for server variables that you will find here, and you can have template variables that depends on other template variables. You can also have data sources as template variables, in which case you have to specify which type of database, and Grafana will suggest the data sources you have. This makes it easy to have one Grafana for different environments, so you can have production and test and staging in from one Grafana and reuse to the dashboard. Annotation is a way of highlighting events in the time series, and by default they are added to all the graphs in the dashboard. You can give them tags, and you can also have links in the details. So if you want to link to some systems with more information about the ploys or something happening to the system, it's really important to get that in the timeline as well, I think. A cool feature, which was one of my first big features in Grafana, is the playlist one. So you can create new playlists, and then you can add the dashboard to them, either by name, or you can use tags, and then you say them, and when you start a playlist, you can either press one of these buttons, or you can go to the URL like this. And since you can start a playlist from the URL, it's very easy to set up a smaller computer like Raspberry or something back on a TV, put it in the lobby, and then you can control whatever dashboards are shown in the lobby from Grafana itself. Because every full cycle, it will load the playlist from the data source. This is a bunch of shortcuts. I don't think you see any one of them. The most important one is if you stand over a panel and press E, you go to edit mode directly instead of having to click the title, which is super nice for speed. You can also share the dashboard, and if you have internal Grafana, you might want to share it publicly. And public Grafana instances means most of the case that anyone who can access the Grafana instance can access your time series database. And that's not a good idea because it's kind of easy to overload a time series database. So instead of hosting Grafana publicly, you should create snapshots with the data you want to show. GitLab had a maintenance problem last week, and they did a really honorable way of publishing a lot of data, queued us to them. It was kind of sad to see that Grafana servers went bonkers in the same second they published it. Using shell snapshots instead would have been no problem because when you publish them, you can publish them to the Grafana instance you're currently running, or you can publish them to our Grafana, and we will publish them behind a CDN. In the latest version of Grafana, we also added support for multi-tooltip. So if you enable it in Hovra, Syria, you get a tooltip for every Syria. And this is also very nice when you want to debug stuff, you want to get a lot of data in the same dashboard and drill down. And whenever you zoom or changes, it will just follow. It's a small performance impact, so you might not want to have it on by default, but it's only in your browser anyway. So we also recently added alerting, and that was a... It's been a really, really, really long-standing issue in Grafana. I think this is our third iteration on alerting. And one of the first problems is that Grafana's data sources is in the front end. So we created queries in JavaScript. With alerting, we can't do that because alerting has to run in the background. So we had a few different strategies for solving this, and the first we tried was to execute JavaScript in the good back end. And we just said that we don't want to maintain this code. It's possible, but we don't want to maintain it, and it's complex, and it's hairy, and we don't want our alerting code to be in any way complex. So simplicity was really important for us, which meant that we have to reimplement all data sources in the back end, which limits plugins, so plugins cannot have alerting right now. We have plans for adding plugins support in the back end in the future, because Ghoulang is going to release that soon, but it's not ready yet. But if you're in the graph panel, you've got a new option for alerting, and if you decide to create an alert, you get all these options, otherwise it's just a button asking you to create an alert. And the way it works is that you refer to a metric query in the Metrics tab. You give it a time range, five minutes to now, and you give it a reducer, and a reducer is a way to get a final number from a time series. So in this case, we would calculate the average of each time series returned by this query, and then we would compare it to this number, and then you can slide the threshold up and down and all that kind of nifty stuff. When everyone alert is triggered, there's a few different alert notification channels, Slack, of course, which will take a screenshot of the graph where it's alerting. It will give you all the series names that violated the threshold, and some alerting descriptions and so on. And this is the model for the alert. So alert contains conditions, and this can be an array of conditions, and whenever one of these conditions are met, well, you can have all statements or end statements. It depends on how you create your query. But if the result of these somehow triggers, we will send you an alert. It might be simple compared to other solutions, but we want it to be really simple. Our aim is not for 100%, 100% of all business cases. We really want to satisfy the beginner-to-medium complex users. Grafana.net, as I mentioned earlier, is our companion site. We're going to rename it Grafana.com eventually. The reasoning why we pick Grafana.net is strange even for us in hindsight, where you can find plugins, both data sources and panels, and you can also find dashboards. And a few plugins that I really like are the first one is the World Map, which allows you to map time series data to your point of location or countercodes, which can be kind of nice to see if you have multi-data centers or you can see where customers are coming from or stuff like that. The second favorite one is the diagram panel, which allows you to draw diagrams like this and connect the boxes to a time series. So it's not a weather map, but it's something more simple. And you create these diagrams by Markdown, not Markdown, but DSL called Mermid.js. And the last one is the custom JSON data source, which allows you to integrate any old legacy system to Grafana. It's mainly a way for Grafana to send away requests and expect your custom web app to respond with time series. And the custom web app can also suggest queries and so on. Because there's always going to be some old system that you want to graph, and you're never going to just have one system for monitoring. So this is a possibility to always keep moving and get everything into Grafana. Dashboard was also introduced in 3.0, but it's been growing steadily. So if you work on a project like Prometheus Exporter or some framework work that exposes metrics, you can create a dashboard, upload it to Grafana.net, and others will see that and can download the dashboard from Grafana.net. And this makes it much, much, much easier to get started with your product. And it's a way of sharing and contributing to open source without having to code in Grafana. If you want to import a dashboard, you just have to click the import button. You can enter a URL to Grafana.net and it will import it for you. And then you have to select some properties like data sources and all. And as I said earlier, sharing dashboard is a good, good way of contributing back to the community and making sure everyone can consume and visualize metrics. This is our first poster we created for Grafana. And I think the dashboard thing we added very much aligns with this, making everyone able to view the dashboard metrics. We also recently introduced host to Grafana. It's still in beta, but if you're interested, come poke me afterwards and I will hook you up. You just give it a name and the cloud will take care of the rest. So that's enough about the present state. Let's talk about the future. So the future contains four things that I want to mention. I don't want to give a clear roadmap because so far we have been changing our goals a lot and that's still our goal to do because we all want to work on the thing that currently is the most powerful. So the first thing I know we're going to do in the upcoming year is iterate on alerting. We're going to add silencing. We're going to add alert groups. We don't support high availability right now or there's no clear way of doing that. That's something we want to solve. The second one is dashboard folders. So many people started using organizations for teams. That was never the intention of organizations. Organizations what meant for hosting provider basically, having different company using the same Grafana. By introducing dashboard folders, we hope to make it easier for people to have one Grafana organization with different teams connected to one folder. Folders will also include permissions and so on, which all that kind of stuff that goes along with arranging and structuring dashboard. The third one is backend plugins. Right now we see a few different use cases. First is alert notifications. We currently support 8, I think, but there's no reason why we shouldn't be able to support endless amounts. Databases would also be great to have in the backend so people can add the custom databases and have alerting on them really simply in Grafana. Authentication would also be a good fit for plugins. The fourth one is automation. That includes dashboard revisioning, some way of making sure that when this Grafana instance boots, it has these data sources, it has these dashboards, it has these users, et cetera, et cetera. We created the API for that use case, but that's a stateful solution. We want to add a configurable solution instead. Not sure how to do that yet, but that's something we really, really want to address without sacrificing startup speed. That's about it for me. I also have a bunch of Grafana t-shirts that I can't tag back with me because I don't check language. And if you contributed to Grafana, I will give you a special one. So any questions? So I have one question. If I want to implement a simple Grafana solution, say at my company, we want to graph how long a certain process runs every day, and I'm currently drowning in options with regards to time series database, backends and so on, what would I need? What would be a quick start to implement such a demo? So the first thing I would suggest is to just get started with one time series database. If you can't decide, just pick one and get started, and once you see the value, you can pick the right one for you. Because I do agree, there's so many right now, it's hard to choose. Exactly, but just choose one and you'll get started, and then you will see the value, and then you can spend time and money deciding which one suits your use case. We support a lot of different data sources, and all will return time series numbers, so we will graph them all. My question is a little bit strange. I'm trying to integrate Grafana with basic authentication, basically hiding behind an internal server, and somehow the basic authentication is not picked up. We can look at that afterwards instead. That's our kind of specific question. If that's okay for you. Any other question? I'm leaving tomorrow. I'll stick around. Very simple question. We've seen that we can now import dashboards. Can you hear me? Can you hear me now? We've seen that we can import dashboards as JSONs. Is it possible now to export them to JSONs without uploading them to... Either you can view your JSON, or you can export them to the format for importing. You can export them, and you get a variable selection for data sources and so on. Is there a plan to allow full-text search? Is there a plan to allow full-text search, like I would say in Kibana, so to support elastic search, full-text search engine? In Kibana, there is a free text field, and it would be really awesome to have that in Grafana. Is there a plan to have that? That's something we've been considering, but since panels are supposed to act by themselves in Grafana, it's kind of tricky to introduce, but it would be possible. You could use template variables for that. You can just type whatever in the template variable. You can just type whatever. But we should add a template variable just for that. Thank you. Next question. On the roadmap, you didn't mention templating. Do you hear me? On the roadmap, you didn't mention templating, like arrays as template variables and stuff. Is that something you're looking at? Sorry, I can't catch it. Arrays as template variables? I'm not sure, but we've been thinking about adding tuples. You can have hidden value and display value. Is that what you're asking for? Not yet. Not yet. Now the roadmap is too wide for that feature. Any other questions? Regarding the last question, I can show you how to do custom JSON for the custom JSON data source for that. Any other questions? Pope me via email, Slack, Twitter, some way. IRC, Grafana channel. When we scale Grafana to multiple servers, how would the alerting work with that? Alerting on multiple servers. It happens in the backend and it runs on one server. If we would run Grafana multiple servers behind proxy, would it... Oops, sorry. It would cross multiple servers redundancy. If Grafana runs on multiple servers at the same time, would it generate alerts from those multiple servers or how do you keep them in sync? Every server would generate alerts. You would have to disable alerting on... You should only enable alerting on one server, and that's the HR issue that you want to address somehow. Any more questions? I wanted to use the organizations for teams, and the issue I went into is that data sources are not global, but per organization. So I made a patch to support global organizations, and I was actually working on getting it proper so I could do progress, but you're probably not going to merge it because of the folders which you just spoke. Yeah, we don't intend to do... make organization sharing in any way. So I should not have spent time in it. I'm sorry. No, it's good stuff. Thanks. We have another question over here. So does your data need to reside always somewhere in a time series database, or can I build, for example, a dashboard where a value comes from a REST API call at that moment to watch like a website or a Twitter account or whatever? The way you would do that is to implement our custom application that returns time series. That's the easiest one. You could also create a plugin in Grafana that reads from any API. Any other questions? We have time. This is really kind of nipping, but is there any way to get the playlist feature to wait with displaying the new dashboard until it's fully rendered in the browser? No, not that known. Okay. Please file a feature request. Any other questions? That's a good feature request. Going once. There's one on the end. For annotations, you said that by default are visible on all panels. Can we turn it off? You can only enable it for a per dashboard. We added support in the code to have it per panel, but that's not configurable or ready for merge yet. In alerting, it would only be per panel. See, if you have alert for a graph panel and the alert triggers, we will create an annotation and that would only be rendered in that panel. But so far, we haven't added the UX support for working with annotations that way. Any more questions? Going once. Going twice. Colin, thank you, Carl. The next talk is in 25 minutes.