 Good morning. I'm Aditi and today I will be talking about getting started with Prometheus instrumentation from the ground up. If you're a beginner looking to instrument your application to tap into Prometheus, this one's for you. So a little bit about me. I was an LFX mentee last fall at the Thanos project and I'm currently an intern at CouchPace. I'm from Bangalore, India and my interests include distributed systems and databases. You can find me on those links. Broadly, this talk is meant to be a practical introduction to instrumenting applications, ideal for beginners. That means quite a bit of code and a focus on implementation. I will be talking with respect to specifics of client Golang but the broad principles should be language-agnostic. So let's start setting the stage with a very brief look into the Prometheus model. Prometheus uses a pull approach where it actively scrapes or extracts metrics from an endpoint, exposing them in a compatible format at regular scrape intervals. To harness Prometheus for metric observability for our applications, one of the ways is to expose metrics over an endpoint accessible to prom from where they are then pulled. In an application developed in Go, this can be done in client Golang as you'll know. So today I will be discussing client Golang with an example application and then drawing on my mentorship experience from last fall. So instrumentation involves thinking about how to integrate Prometheus in your code base ground up. This is different from an exporter which you may have which I'm sure you have heard of where existing exported metrics are converted to a compatible format and exposed over an endpoint. So an exporter is like an intermediary of sorts whereas instrumentation walks at the code level. Briefly, instrumenting code could be walked through a four stage process defined by the questions. What should I instrument? Also covering the why? How do I get the data for it? How do I organize it? That is what types of metrics represent this data the best? And finally the implementation, the nuts and bolts. To set the context for one of the examples, Thanos is a metric data store extending the long term storage capabilities of Prometheus. So taking the example of some common Thanos processes compaction down sampling and retention. These are each multi-stage processes which can take varying amounts of time based on the data to be processed. These lack observability and hence these were the perfect candidates to instrument to gather observable metrics. That was the focus of my project during my mentorship. So looking at some terms, metrics, this is a fairly common term but just reiterating. So each time series has a unique identifier ideally providing some idea about its function called a metric name. Labels are key value pairs along with the metric name which uniquely identify a time series. These show metrics in another dimension that is the help identify metrics by the attributes represented by those very labels. So quickly the four metric types are counters. These are used for continually increasing values most commonly used with the rate to find the rate of increase per second. Gages, in contrast to counters these are used when the values can go in either direction increasing or decreasing and these can be used to compress snapshots of the metrics at certain intervals. Counters can be said to answer the question how often since they are mostly used in getting the rate of increase and gauges can also be reset to zero on demand unlike counters where that happens only on restart. Histograms, these sample observations based on frequency and slot these into specified buckets each of these buckets is a unique time series and holds a counter which contains the count of attributes in that bucket plus all of the preceding buckets. Summaries, these two track distributions but in contrast to histograms these expose quantile values directly and each of these four can also be organized into vectors. On the top left is a counter and on the bottom right is a gauge. This is another default gometric a histogram and a summary. So let's apply what we've just looked at to a practical example. How did I get the metrics data for my Thanos project? An estimate about the number of data blocks to be compacted or down sampled or processed in any form would be a pretty good proxy for the estimate for each processes estimate. Since the metrics chosen for these process that is the number of runs or the blocks to be processed can be computed when supplied with the data without actually running the process simulating them provides the needed metric data. For example, this is the down sampling process to simulate this and get the number of blocks to be down sampled all I needed to do was follow the course and filter out any steps not involved in the decision making. I segregated blocks based on the resolution time or resolution level indicating how sparse the final down sample data was to be. Then part two to get the number of blocks to be down sampled from that I proceeded to further filter them based on whether there are already existing source blocks with the same resolution and if the time difference of the block data was above a certain threshold. In the Thanos processes the data was segregated into groups based on the minimum time and each of these were computed per group. So coming to my thought process and on how I picked the metrics metric types to represent these since these values could increase or decrease and would need to be reset like I discussed earlier I went with gauges and since each group required a separate identifiable value I used gauge vectors labeled with the group IDs. The number of blocks to be down sampled was divided based on group keys which are unique to each group. I have an intermediate map group blocks which you can see in the snippet where I store the results for each group and this is how its gauges updated in client Golang and after all the blocks have been processed I reset it to zero the first line of the snippet and each group key becomes a label. For each group I then increased the value by the number of blocks for that group. So this was a quick peek into how I gathered the data and implemented instrumentation for one of the processes. For compaction and retention of data I followed a similar thought process. So the first example of seeing exposition to prom via HTTP is exposing the default metrics on an endpoint. So I'm using the prom HTTP package here and these are accessed via the specified port 3112 and the handler function here is the default way of exposing these via an endpoint. The default prom config YAML supports this and the scrape config scrapes port 9090 every 15 seconds as I'm sure you'll know. The code on the top right serves these metrics on the custom port on the custom endpoint and I have added a job to the config just below it and having all of these default metrics in the default registry is actually an anti-pattern and should be and will be removed in the next version of the client. This can be extended to support exposition from custom applications in a similar manner. So I will start off with a quick example using basic weather data for cities and then move on to a more involved example. In this example every two seconds I get temperature data for a city and that's a gauge vector with labels for the city which I update based on values only for that city. Ops processed is simply the total number of such updates which I increment each time an update takes place. So now let's look at some more examples of client goaling. So I first create the endpoint with this with a handler on the same port and here I have used a custom handler with a specific registry unlike the previous example. With the default metrics I use the default handler which came with the default registry and hence the default metrics were already registered. So point to note using a custom handler allows me greater control over the registry contents and only the metrics which I will register will be part of this new registry and let's discuss the more detailed example. In this example I will be discussing a small application involving Apache Kafka and event streaming platform which can also be used as a message queue. I will be using their go client to create and delete topics, produce and consume messages and instrumenters to expose some basic metrics. When creating a topic I have a struct topics created with metrics for both the possible cases successfully and unsuccessfully created. So the successful metric represents topics the net number of topics created at the end of the run categorized by labels and hence it is a gauge. So when creating this I use prom auto to register these without an explicit registration step. In the Prometheus package creating and registering metrics are separate with creation preceding registration. So since this is a very simple case I have gone ahead and used prom auto but one disadvantage is that this has a higher chance of failing and a not so clean way of handling errors which has a higher chance of occurring in larger code bases where multiple packages may try to register variables whereas explicit registration which returns an error can be handled cleaner and allows for more fine-grained control of the metrics and neater ways of handling registration errors. Since this is the first metric type we are discussing let's look at what the options mean just in brief. Name this is the unique identifier for the metric and it's a mandatory option when querying we use this. Help is a longer more verbose description of the purpose. Namespace and subsystem are used to create the fully qualified domain name when combined with name and the last slice of strings represents labels and there can be multiple such labels. So since the stock is about a quick practical introduction to Prometheus instrumentations a few things to keep in mind while naming metrics. The namespace should specify the domain or the application of the metric ideally. Here I have used Kafka. The suffix should have the unit or the word total for accumulating counts like counter I have used total here and avoid adding the label name into the metric name since that's a redundancy. An example of that would be naming the message count metric for a topic messages total. The preferred way to name it would be message total and have a label topic. So now on to the create topic function. When a topic is created I increment the create the topics created successfully gauge vector for that category using the increment function. So I categorize topics by this really simple method based on their suffixes either is order even. So I add to the gauge with the label for that topic and specifying that label makes sure that only that time series gets incremented. So the recommended practices for code instrumentation make a good point about adding a metric for anything interesting that's worth a log. Here since I think tracking the number of errors while creating topics is worthwhile I have added a separate counter for that which I increment on an error. I initially considered using a gauge vector for this function topics categorized in a separate counter without labels. The options for that would be similar to the previous slide and here I thought I'd use type to categorize topics. So I'm mentioning this because this is a deliberate anti pattern I'm bringing up instead of a separate gauge for just categorizing topics the create topic metric could have had labels specifying types. Naming a label type is also considered bad form since it's so generic. While we are on labels a point to note that adding a label is equivalent to adding a new time series and cardinality is the number of possible values for a label. Assuming multiple labels on a metric let's say x, y and z each with three possible values multiplying that could could take us to 27 unique time series and it can have an impact on performance after a point since each resulting time series is one that might need to be processed index stored after every scrape interval and although each combination is unlikely to occur it's something to keep in mind while deciding on metrics and labels. Outstanding message count in a topic is the count of messages yet to be processed and this will be incremented on producing new messages and decremented on consuming them. It's just a very simple trivial counter. So let's write some messages to the new topic. Here we are looking at a struct message produced with counter vectors and this too has its error equivalent in message produced failed. These are incremented as obviously on successfully and unsuccessfully producing a message. So the histogram buckets here indicate message consumption latencies which are less than the specified buckets and the buckets before it. So there are multiple ways that this can be specified. The first is the method I have shown explicit demarcations and the second when they are evenly spaced and the count is known is using linear buckets and finally if they are not linear one can also use exponential buckets. The message latency here is the duration between writing a message and returning from the write function. So the histograms have observations added using the observe method. This is the timer in the client Golang library and the duration is calculated using timer which can be used with histograms or gauges or summaries. The observer method gets called by the deferred observed duration and that updates the histogram. If there is an error while writing the message again that's worth a log and so the error metric for that topic is incremented. On successfully sending a message the successful message counters incremented along with the message count gauge. This is how the counter looks consuming messages. So the message consumed struct is pretty similar to the previous one and each time a message is successfully consumed the gauge message count for that topic is incremented using the decrement function. If not this error too has a corresponding metric. So another example of decrementing gauges is provided by the delete topic function where each time a topic is successfully deleted a gauge is decremented. Here too there is a corresponding metric for the errors during deletion which is incremented on an error. So I have organized my test suite using table testing. Each of the elements in this list is a test case which will run through this block for each of the cases only these few name input in the expected parameters change whereas the rest is just boilerplate code common to each case. The table here refers to the struct which holds the name input and output for each case and these run independently of each other meaning a failed case won't stop the others from running and each is called a subtest. To check for equalities I use assertions from assert. Let's start by testing the create topic function. The input test struct has the topic name and the key and the output has the expected count of successfully and unsuccessfully created topics. I'm creating a new topic create each time to reset this to 0 and to test the gauge vector values for this key use get metrics with label values and compare it with the expected output that's the second those are the last two lines. To test the send message function to I do something similar I have specific structs for the subtests. I run the functions to be tested again and get the metric values for the labels so you can see I have my custom registry I reset it all and I get metrics with label values and I check for assertions. A lot of this talk was meant to provide a beginner's perspective since I learned of this during my mentorship and I was hoping some others would benefit from the perspective I had during the project. So this brings me to the end. I hope my talk helps any novices getting started with instrumentation to leverage Prometheus observability. Thank you. Any questions? What was the most difficult thing that you had to learn when learning the how to instrument your Prometheus applications? So in the client I went through the source code and understanding collectors in metrics all those interfaces and how those could be applied to the Thanos processes was challenging. All right we have a question from Slack is should there be separation between the business logic like creating the Kafka topic and the Prometheus instrumentation? I think so from my limited experience with this I feel instrumentation is harnessing it from the code level so I don't think there is too distinct a separation possible like not too thick a line. Any more questions? Be bold. Do you want to add something? Um no I'm sorry. Okay last call for questions three two one thank you very much.