 All right. Well, I would like to begin this session and the goal here is to do a deep dive into the open telemetry metric system. I was asked to give this talk by the organizers and they suggested that it would be a good idea to have a conversation. A year ago we did this earlier on in our open telemetry project and had a similar presentation here where Liz from Honeycomb spoke with me for an hour about the Metrics API and it was really helpful to have outsiders and insiders talking with me about it as we went. So we're going to do that again and I have with me Sheldie and Justin. Sheldie is at Honeycomb and has agreed to be an eager participant in this session and then Justin is at New Relic and has been one of the contributors, especially working on our semantic conventions for the open telemetry system. Okay, I probably should introduce myself. I'm an engineer at LightStep. I've been working in observability for I think about 15 years before LightStep I was at Google and have been involved in like tracing metrics and logging for a long time. Okay, so the outline is there are three parts here. So we're going to start with what we were trying to achieve and what we're after in this project because we're not done. It's underway. And we'll talk about the sort of timeline we think we have. The bulk of this sort of slide deck is talking about data model and the things we discovered when we tried to combine a Prometheus system with a Stasti system with a tracing system, what happened. I'm going to talk about some of the ways you can configure this thing that we've built the open telemetry metrics, which includes both SDKs and the collector and all the various protocols that we have at our disposal. Okay. So this is for anybody visually impaired I'm better at drawing slides and I ended making them on the computer so I've scanned a bunch of hand drawings for this deck and hopefully you can follow along and read my writing. I'm trying to sort of start by saying who this is for. I want to attract like an audience to talk about metrics now so I think we can basically agree that there are many different participants in the in the in this type of community. We have people who are trying to configure telemetry and diagnostics for their platforms for their for their for their companies or where they work. There are engineers who are actually writing instrumentation and software at those companies trying to make more observability and better diagnostics for themselves. And then there are actually users who are trying to understand why is my system broken why can't I, you know, log into my, my machine or whatever. So these three different groups have kind of different perspectives on the problem space. And we end up thinking about what each of them wants as we talk through this problem space. Okay. Ultimately, the goal is to get some dashboards or some other form of alerting or monitoring on your data and that looks something like this drawing that I made here on the right. This next diagram comes straight out of the open telemetry library guidelines this was put together in the very early days of the project basically saying what we're after as far as the project as a whole. One of the things that we wanted to do is build a vendor neutral system so that you could as a as a developer decide to use open telemetry without locking yourself into somebody's SDK or some some vendors system. So what that means is that we've created an API separation from the SDK. That means our interfaces are decoupled from our implementations and this means that you can swap in another SDK, or some alternate implementation later. So the diagram has here your your application code running sorry. And then it goes into this sort of green box which combines both the API. That's the spec that we've put together for how you interact with metrics, as well as the SDK which is the default implementation that all the open telemetry libraries are going to include. And together this should provide you a high performance pipeline and you will then configure an exporter for the protocol that you want so that you can expose your data in such to the system that you want to. So how much is how much how much more effort is it to use something to use the telemetry setup versus some of the vendor neutral integration or the vendor specific integrations that people may be used to. I'm not exactly sure which vendor integrations you might be thinking of. So some say SDKs come in with a bunch of built in metrics for say your platform so you start using this library and you get host metrics by out of the box you get Kubernetes metrics out of the box. This is something that the open telemetry system will include. So automatic metrics as much as possible are going to be included for you. So we really structured this talk to leave sort of the details that you as a programmer might want to know to write your own metrics. It's really not the most important part of this topic, this talk because what we're really trying to do is help you set up an ecosystem help you set up a collector and an export pipeline, very few engineers actually write custom metrics and so that's sort of the least important part of the top talk here. Gotcha, thank you. Alright, so what's next here. One of the major requirements here is that we are an open source project and this lease, at least from my perspective was one of the biggest challenges of joining an open source project is now you don't work for a company anymore you kind of have your own. We have the community to think about first and so this has been a project that that moves at the pace of the community because we're getting what the community wants so I've drawn a picture here of a kind of a system that basically is a collection of a pipeline for both tracing and metrics data and we're trying to show that this collection infrastructure is going to work with all the open source systems that you're already using because that was one of our priorities. Okay, one of the sources of our sort of most sophisticated requirements came from open census the open census system is really what gave us the open telemetry project, combined with open tracing. And the open tracing side of that combination gave us that requirement about SDK separation from the API. What open census gave us was the requirement for a very high performance SDK, and the requirement to have that SDK be configurable. And in metrics what that meant was the ability to choose which metrics are going to be exported to choose which dimensions are going to be exported, and they called that the views API. So all those requirements were given to us sort of at the starting point we need to have a better way to configure which what happens when you use a metrics API. Next is not just about how do you record numbers. We need to make sure that the entire problem is being solved so in addition to you creating your own application metrics, you have plugins for your host metrics you have plugins for your these metrics, you have instrumentation for tracing is shared has shared resource attributes for your metrics and so we want to have specifications that tell us exactly how you should label data in a standard way. So that users will be able to find it and understand what they're looking at. So this is really a task that bridges technical stuff with kind of language stuff and understanding and questions about how we talk and what names we use. So this is an interesting sort of corner of the specification which is really about language and terminology and and understanding. Before I talk about data model and what you can do with this I want to be honest with you about the current timeline we are right now the open telemetry project is trying to finish the tracing spec is and freeze that and make sure that we have a stable tracing environment soon. Because of that attention being given to trace the metrics project has been slowed down a little bit. So we're kind of like putting our attention and tracing while we are waiting to finish metrics, but there is also just a large number of moving parts in this problem space, we have specifications which we've, we've definitely finished the, for the most part we finished our API spec, but the SDK specification is probably going to be the last thing to finish because we have to actually implement this SDK and a bunch of different languages and figure out what else needs to be answered. But as far as sort of the collector support that's been sort of ahead of schedule because we had the open census collector already and a number of the integrations and the receivers and exporters that you've heard about today, we're already in place for metrics so that's that's a little bit ahead, and then there's a few kind of lingering questions about data model and the protocol that are things that we may want to solve in the distant future we may not need to at all those are sort of open questions, things, questions about sort of the obscure corners of the world where you might want to use the protocol for something different. But for the most part we're getting close and we think that by the second first half of next year you'll be able to use this for real. Okay, I'm going to start talking about data model now. Before you do, we have a couple questions in the chat that I wanted to share so Raj has a question. Does the SDK capture raw data at a time stamp or is there data aggregation processing in the SDK. Right, so I do plan to talk a little bit about that later there is definitely that's one of those performance requirements that we got from open census. We need to have a high performance metrics library and we know that the sort of gold standard at the point when we started was Prometheus libraries. So Prometheus libraries are organized in such a way that your updates are very fast, mainly because you're actually pinning memory to keep, you know, a variable in place that you can update quickly. So we're going to talk later on this talk about something called aggregation temporality which lets us move that memory in our system, but we're still going to be able to aggregate in the process over short windows of time and output data that's been aggregated before it leaves the process. And yeah, we'll go into that later in your presentation. And then the other question that's been coming up a lot today. I'm not sure if you're planning to talk about this a little bit more but the relationship between open metrics and hotel or Yes, that is absolutely going to be part of this presentation. It's one of the bigger questions that I have to answer and I hope that I will be able to answer that by the end of this presentation. Great, then I will get out of your way. Well, alright, so I just want to start with data model like this is a, like what are we doing this is about metrics so I kind of want to bring us all back to the top level which is, we're trying to look at some numbers probably in a visual We may also be alerting and monitoring on them but the classic application for metrics is to create some charts and put them on a dashboard. So I want to talk through the different kinds of charts and dashboarding facilities that are commonly available through metrics, because there are different data types here. So the next few slides are going to talk about your sort of typical charts that you might get out of a metric system. That's what we're after. So first year as a count time series what I'm calling a count time series is just that there's some counter and it's, it's cumulative in the sense that I've begin this counter at the start of my process, and I may add to it, I may subtract to it, we haven't talked about monotonicity yet, but over time that total is kept as a sum, and whatever value I'm looking at as a function of time is the total cumulative count for that metric. This is going to come up a lot throughout this talk I use the, we use the terms cumulative and delta. I'm going to use the Greek letter sigma to refer to cumulative since it's a mathematical association we all have. And then you can imagine looking at the same data as a rate. Now this is one of the big deals here in metrics is that you can often represent count or some data is either a rate or a total and that's this comes. This is one of the complications that we're going to have to deal with, because they have different properties, but they're roughly speaking equivalent. If we talk about showing a rate, it is going to be associated with what we call deltas your reporting the change in some quantity and then you're going to plot the change. So this change could be can is a number that can rise and fall it and if the number, the count is not monotonic this change could actually be negative. So, this is another way to talk about reporting some data. We also have this notion of a gauge in the kind of traditional metrics interfaces and the data model here is a little different from sort of. What I'm trying to show with this visualization diagram here is that you may set a gauge many times during interval, but what we commonly report when we're talking about gauges is the last value that was set. So you have many points visualized in this in this graph, only the blue colored dots are the ones that are going to be actually reported. So this is called last value reporting we call this a gauge. And it's interesting because we don't actually evaluate or keep information about every point in some sense the number of points is irrelevant here. All we want to know is that there's a signal, we can evaluate it at a point in time, and we often record just one time. This is a relatively inexpensive type of aggregation. Now we also have this thing we call histograms or sometimes we call them distributions. The idea is that here you have these individual measurements, instead of just recording say the last value of one of those measurements. We're going to somehow capture more, we're going to capture both account and the value so that we can speak independently about the count and about the values. And here now is a distribution. On the top I've got red, purple and blue showing you quantiles so this might be P99, P50, P10 telling you where the distribution of latency or some other value in your system is. And then on the bottom, I have a rate plot which is showing for the same data set how many points were there per unit of time. So a histogram is sort of the most expensive type of metric data that we report because it includes two independent pieces of information. And we often use it to summarize a whole distribution. So these tend to be a little bit more expensive. The goal of those last four pages there was to show you that there's something familiar and common that we are after even though this data model may feel confusing after we start talking about it for a bit. Now, as far as what's different in open metrics and open telemetry, this is the, this is where it starts. There's two pieces of the data model that are not present in an open telemetry system. And I just want to say it's not because they're missing, it's because when you're pulling data in a metric system, you don't need these features. So the first thing is resources. Resources are a concept for these key value attributes that we attach to our diagnostics or instrument or observability data. It's not just metrics. This is also for scans and blogs. So I use the term attributes for this key value association. And the difference between what we've gotten open telemetry and what you have and say open metrics is that this concept or resource is in the protocol so that when you report a batch of metrics data, you're going to have one section for resources, and then you're going to have a lot of metrics data inside that encapsulation. And so this allows us to begin creating common coordinates and telemetry between traces and metrics because we have standard resources, and we can configure these export pipelines to attach standard resources to both our traces and our metrics as they pass through the export pipeline. So having resources is a first class concept. And then of course we have resource conventions for how you should name your resources. I've shown one example here, which is for the service attribute namespace. You've got service name, service namespace, service instance ID and service version. We are suggesting that these sorts of attributes may be included with metric data. And then some point later you may have to export them into a system that doesn't have this concept of resources and we're going to have to talk about whether those resources are or are not attached as metric attributes or sometimes these are called labels and tags, we have a terminology problem there and try to avoid it for this talk. So, so the first thing that we're adding that's different from open metrics is the concept of resources. And they say that you don't need these in a pull based system, because if you think about how Prometheus works, you scrape the target, the target gives you all of its own metric data, including its own metric labels, but it doesn't know its own resources. The whole point of scraping and a pull based system is that you discover the targets, you scrape them, and you know what their resources are so that's the person that pulls that data, who's responsible for attaching resources, and that's why you didn't need it in open metrics. The other thing that's different in open telemetry compared with open metrics is this notion that we have support for temporality. The word temporality is one that we sort of made up it is a real word, but we're using it to refer to this distinction between delta and cumulative reporting. So, and I want to make a distinction between because we use this word in two senses here it's pretty important to the design of the whole system. So bear with me. I think that there's something called temporality is trying to describe a relationship with time. And when we report metrics data, they may be cumulative, or they may be delta. Generally speaking, deltas mean that you're reporting differences since the last report, and every report is independent and does not build on the previous we need to you. So we're reporting changes one after the other in cumulative reporting we're going to repeat report some total since the beginning of time, and we're going to keep re reporting that total since the beginning of time. Okay, why are we talking about this. Well, there's something here called instrument temporality which is what type of numbers does this instrument deal with, and that's an API question, and there's something called aggregation temporality which is what is in the protocol that I'm putting through my pipeline. Okay, this is confusing. I have no slide on it. So why do we care. Well, the, this choices entirely about keeping state and stateful interactions between out of the SDK and the API for open telemetry. So, if you think about reporting say a total, your current memory usage, you don't want to report your current memory usage as a delta because that would require you to remember the last value that you reported for your current memory usage. So we find it's better if you're reporting something like a current memory usage to report a current total. And then the system that we're going to have if you're reporting information from a callback, you're generally going to report totals. If you're reporting something in a mainline code as part of a transaction or request, you're going to report deltas. From the start we have this built into our instruments, and it's because we think it is more convenient to talk about deltas in some contexts, and more convenient to talk about totals in other contexts, it keeps state out. Now, the point of creating this concept in aggregation this instrument, sorry, the point of creating instrument temporality is that we can adjust aggregation temporality. So what goes in is not necessarily what comes out. It turns out that if we have deltas coming in, and we want cumulative coming out, we're going to put memory. So this is why we have these questions that there's a trade off trade off is between reliability and memory costs. There's going to be more on this topic we will keep talking about this. So, I need to. So we're sort of, we're still talking about data model but we're trying now to integrate the concepts from Prometheus and the concepts from stats D, because one of those systems uses cumulative and one of those systems uses deltas. So, however, in the API for both of those systems counters use deltas so both Prometheus and stats the expect when you're using a counter that you will tell it a delta. Or when Prometheus writes data to its right ahead log or exports data through its remote right is sending you cumulative. And so, this is why Prometheus has trouble with cardinality requires you to keep memory inside the client library for every counter you've ever used because it has to track the cumulative value. In open geometry, we are giving you these instruments that have delta instrument temporality counters, and this not this instrument that we call up down counter are for for inputting changes or deltas to a sum. So these are just like the Prometheus concept these are just like the stats D concept, except that we've given you two ones for monotonic counters, and ones for non monotonic counters. And this is actually the simple case. These are deltas. Now when we talk about gauges. I mentioned earlier in some of this breakout sessions that there's a little bit of a terminology problem. And the, that I don't want to go too close to the details here because I lose you and you'll get bored but basically Prometheus and stats to use the term gauge in slightly different ways. And I've copied a definition out of Wikipedia for the word gauge itself, turns out that the actual word has two meanings in our actual engineering practice. This is a confusing term to begin with. And it turns out because there's it's used differently in Prometheus and stats D. It's even more confusing. We're trying not to use it. So the thing that you were using a gauge for, however, is a real application in Prometheus and stats D gauge got used whenever we wanted an individual measurement, but not the cost of a histogram. So open geometry created a new distinction to cover this gauge use case. And it's when you're recording an individual measurement. So I'm going to talk more about what an individual measurement means, but the point is that if you take a measurement, and there's something. So you say suppose you measure a latency, you would never add that measurement of latency to another measurement of latency, just for the sake of adding those two numbers together. So that is an individual measurement and you're interested in knowing individual latencies. So there's some. So, when you when you use this value recorder instrument, you're going to be computing a distribution as opposed to computing a sum which is a much simpler operation and much cheaper operation. I mean, why gauge doesn't exist and I think I'm getting a little confused or I'm not sure that people are following you at me at this time, because gauges are very confusing concept. But what I'm trying to say here is that we've replaced the concept of a gauge instrument with two new instruments. One is going to be called value recorder, and one is called value observer. We use value recorder to record individual measurements. We use value observer to witness an individual measurement such as a callback observing a total. I'm afraid this slide's not going off, but I'm going to keep going. The problem I'm trying to explain with gauge is that sometimes they got used to record sums that were cumulative or sums that were not monotonic. And sometimes they were used to record sort of other measurements such as a temperature or a latency which is not something you ordinarily add together. So the point is gauges got used for several different things in each of these systems and we are trying to provide you instruments that get used for exactly one thing. If you're observing a sum or you're observing a non-monotonic sum, instead of using a gauge, you're going to use one of these new instruments called sum observer or up-down sum observer. And these are instruments that have so-called cumulative instrument temporality. We're going to keep talking about this. Can I pause you? Yes. Please, Shelby. Can you go back one slide? So when you say monotonic, just because I might have missed that meaning at one point, that's like something that's unrelated to the previous value, right? I may have confused you and I would love to just completely reiterate this. So I've used term deltas and cumulatives like you just described. This is about the relationship between a prior interval and the current interval as far as reporting some values. The question about monotonicity is really about whether the function is always rising or always falling, for example. So the distinction is between sums that rise and fall versus sums that only rise. And when we have a sum that only rises, more often we're interested in showing that as a rate. But when we have a sum that rises and falls, more often we are interested in showing that as a count. And so you're probably familiar with metrics interfaces having the ability to choose whether you're talking about a rate or a count. This is something that we're actually encoding in these instruments for open telemetry. So I can imagine like for in a very lay example, right? Age is always in relation to the previous year. You can always just say like age plus one, right? And that's my birthday. It's related to the previous year. But for like temperature, it's completely unrelated to the past yesterday's temperature. You might say like you might your Google alerts is like, oh, it's going to be seven degrees colder today. But usually you just want the individual like today's temperature. Yeah, that's a good observation. You don't ordinarily monitor changes in temperature. You monitor absolute temperatures, but you might monitor something like that is being counted as a rate. And then you might monitor that as a change. Gotcha. Yeah. I like using the very human examples, but it's also helpful to have them. I let your other examples of things like CPU usage, right? Where you always want that at the particular moment. You don't care how you don't usually care how much that relates to like five minutes ago. Yeah, I like your example of using age, though it's a great one. So you could imagine two ways of monitoring you an age of a person. If you're if you use a counter, then every year you increment it by one. If you're using a sun observer, then every year you output your current age, and that's always one more than the last year. So and we could talk more about the differences. We will talk more about the differences is one of the key aspects of this design. Gotcha. That's super helpful. Justin, please. Would you explain a little bit about why an instrumentation author shouldn't just always use the up down some observer. When you could use it to record age as well right is there a good reason to only use the sum observer in that case. Yes, and I am, I promise I tried to put a slide about this actually it's the next one. So, there has been this debate and I promise you we're going to spend more time updating our specs to try and clarify this issue. So, I am now regretting that I didn't include more detail already on the instruments that we've designed we've got six instruments and some of this is what we're talking about right now. But when we talk about semantic kind there really are two to my knowledge and in the technical committee we are in the SIG we've been talking about the words adding versus grouping, what we're trying to distinguish is between things that you add or things that you average. And things that you add are interesting. Let's suppose you're sampling. So if I have a bunch of observations of things that I add. Well, the larger numbers are more interesting because they contribute more to a sum. So when I'm sampling numbers and I know that there are things I care about the sum. Well, then I put weight on those, the higher numbers get more weight. When I'm sampling or down or, or any kind of aggregation that I'm doing, where where some is the property I'm after, then I want to know that it's a sum. Whereas if I'm doing something like down sampling or reducing dimensionality of one of these other types like a, where traditionally we have used a gauge or where you use now a value recorder or a value observer, then there's there's no just there's no difference in importance between a small number and a big number. You have a zero latency that's a significant measurement is no more or less significant than 100 second latency. So when you're talking about these individual values or individual measurements, you want to think of that as a different semantic type than it is then say an increment where we contribute to a sum because bigger increments matter more. This is a little, we're sort of straying into the theory of measurement and I should have put I could have put some slides in here about that but we do in sort of in statistics or in math we talk about scale for the numbers. So you can talk about ratio scale and interval scale and logarithm scale. I answered the question in those terms as well but I'd rather do it with the way I just did. So, I just discussed roughly what was on this slide is to say that you, it, we are interested in keeping the semantic type when we have when we have that information, and this choice Justin's question the choice between value recorder and or sorry value observer and and up down some observer. So they're very close but conceptually we want them to be different. And I promise you will keep writing to try and clarify this point. Okay. Well, I finished my section on data model. So that's good, we made it through and you asked the same question, right, as I was reaching my last slide. Okay, so for the rest of this talk we're going to go through some of the ways that you can configure export for your metrics data, especially talking about how we control costs. Okay, so in not particularly great order here we have a few features that are sort of new and interesting for open telemetry. One of the things that's been, I think, holding back the industry. Generally speaking is that we need variable bounds histograms or we need histograms that can support high resolution and be relatively compressed various ways of saying the same thing sometimes we call these sketches histograms by however you formulate them are approximate representations of a distribution and we want to get better at this. So, I've named some algorithms that are that you that are used for this. We are currently negotiating and working out some of the protocol decisions that we want here. And we are looking to standardize on one of these algorithms as a sort of recommended approach. Open telemetry is going to offer you essentially better histograms then then you've been getting out of the box for some of the metric systems that you're interested that you've been using. DD sketches the leading contender. A lot of the discussion that we've had about temporality in the past 15 minutes, ultimately comes down to cost and cardinality. So, what we have a situation where we expect that users or developers are going to be writing instrumentation and putting metric labels together that they're sort of that they may not actually know what are the useful labels or what the cost tolerances of the people running that code are going to be that you may end up in a situation where you're generating more labels than you actually need. So what do you do with that situation. Two things that we know how to do that we're including in part as as part of the open telemetry metric system. One is that we have the ability to do in process aggregation so these these SDKs are relatively sophisticated they include the ability to co less events that happen over a short interval of time. And then if we want to do label label reduction we can actually erase those labels and just aggregate those values together so that we get fewer time series with more aggregation happening. So this is a facility that the SDKs include and it's something that we have to configure so later on we'll talk about how to configure that but for now the SDKs include this mechanism. The other thing that we can do to control cost is the reason why I've been talking about temporality so much. So in a stats the system you may have been used to cardinality being okay because every at every flush interval, you can completely forget what you've been reporting and just begin accumulating a new state, whereas in a Prometheus system. If you use high cardinality those labels are stuck in memory for a very long time. So a stateless export pipeline is one where you force the use of delta export of delta aggregation temporality in order to allow yourself to flush out memory. So, look at some diagrams. Here's a diagram of a standalone SDK running open telemetry metrics. So you've got your application code running. You have a runtime metrics instrumentation package running that's telling you garbage question statistics maybe you have a host metrics instrumentation package that's running that maybe telling you CPU usage and memory usage and so on. You have the open telemetry API. Beneath that you have the SDK. The SDK has a frontline component we call the accumulator which builds up short term state. So over one interval whether that's a second or a minute or 10 minutes could be anything. You build up a record of everything that's happened during that period of time. And then every once in a while the SDK flushes its state. So this orange box here with a diagram saying here is where we do delta to cumulative conversion. So this orange box represents a long term memory commitment. In order to implement cumulative export for your metrics, you have to put memory somewhere and in this configuration, this standalone configuration, you just put that directly in your SDK. This is a configuration where you don't want to use a lot of cardinality because it's going to sit in memory for a long time. But this is a this is a configuration that's very compatible with downstream systems. You can blindly send this to an endpoint say a collector and it can export to Prometheus because you put the data into cumulative form before you sent it. Now the problem is you have to put a lot of memory in your application for this and you have to hold on to that memory for a long time, and it prevents you from using high cardinality metrics. So we have this other option to configure an export pipeline that is stateless. And the way we do this is by making sure that the aggregation temporality matches instrument temporality. If you're putting deltas in, you give deltas out. If you're putting cumulus in, you put cumulus out and this way you have no memory requirements. So in this diagram, it's the same as the diagram before, but now there's no long term memory commitment and there's no orange box. There's nothing happening in that box. It's just passing straight through. So this is going to cost you less. And if you have an OTP endpoint that supports deltas natively, this is actually a good configuration. And I want to say if anyone's listening here, if you're the author of an open source backend for metrics data, this is an opportunity, a big opportunity to accept native OTP data because it will allow the clients to begin configuring themselves statelessly. And this is going to be open the door to high cardinality metrics. So that was a sort of complicated configuration here because it is standalone. You've got several things running inside your process and all those things running inside your process add up to risk and cost. So there's another way you can configure this. And this is using the collector as an agent. So I might want to export cumulative metrics because that makes the downstream system happy. I can, for example, write to Prometheus, but somewhere in my pipeline, I have to configure this point where I convert deltas to cumulative, which leads to a long term memory requirement. So in this configuration, I have an application running inside of a host, but it is a stateless exporter. So it is, it is outputting OTP with aggregation temporality to require no memory. The hotel collector is running on the same host. It is able to implement that delta to cumulative conversion. And then you can output OTP converted to cumulative from your host, but your application is still running statelessly. So this is a way that we can reduce risk on the application itself. If you were accidentally to put high cardinality metrics in this configuration, it'll crash the collector, not your application. Actually, the collector is probably written to handle that better than your application is. So that's one configuration. Of course, there is still a long term state requirement here. We've just moved it from the application to the collector. So there's another configuration. This is where we talk about it just a standalone Prometheus exporter. Suppose you have an existing Prometheus setup, you just want to add one new target. So in this situation, it's roughly the same as that configuration that I showed you before. I'm running my runtime metrics plugin that tells me GC stats. I'm running my host metrics, which means I don't need to run some other Prometheus tool to export node statistics. I've got my API and my SDK. I'm still doing that delta to cumulative conversion because I must for Prometheus. And then I can either pull that data using open metrics, or I can push that data using a Prometheus remote write exporter. In both of these cases, I'm writing to Prometheus. In both of these cases, I have to do a delta to cumulative conversion because, again, Prometheus requires it. But I could also do that statelessly. Again, almost the same diagram as before. I've got a stateless application. I'm sending through a stateless export pipeline. I'm receiving that at the collector agent. I then can do delta to cumulative conversion. But now I'm going to use the Prometheus remote write because there's some impedance mismatch using open metrics to pull from a collector. We're not going to do that right away. So this is a way you can export to Prometheus through a stateless in process exporter and a local agent. Not the only way you can configure export for open telemetry metrics. I want to do a little review on some of the stuff that we just talked about. There's sort of two ways that you might, two ends of a spectrum that you might configure your metrics export. You can do either push and pull or you can do stateless or cumulative. But we generally pair the pull model with cumulative, which is the Prometheus model, and that's the default for open telemetry. And that has properties are good for Prometheus. You're automatically going to work with Prometheus, even if you have a collector in the loop. You get easier reliability because when you drop a record or a data point in Prometheus, when you drop a cumulative data point, it gets smoothed out by the next data point that arise so loss loss of data is pretty okay in a Prometheus system. But high cardinality is pretty much not allowed and it will blow you up if you do it by accident. So, in this other side here we've got this Prometheus this push model and this stateless export model so one thing that's going for is it's exactly the same as we're doing already for traces if you're going to set up an export pipeline for your span data. You might want to set up the same exact topology for your metrics data. So the idea that you're going to attach resources as you collect data through your infrastructure. It fits very well it matches trace collection. It means you're using delta can delta aggregation temporality. It means that you have to be a little bit, you have to work a lot harder for reliability. So you have to avoid drop packets means loss of data and replayed packets means double counted data so we have to be careful to avoid dropping data but we also have to avoid replay. But the upside of this configuration is that that high cardinality is no longer a cost inside the process and high cardinality if the downstream system supports it is going to be okay and actually could be good for the user. I didn't cover this combat this sort of worst and best of both cases which would be a push cumulative model that is definitely a valid configuration and it's exactly what you want if you're going to send Prometheus data through a collector. So, these are choices. Currently, we're, we're the default for open telemetry is cumulative because it's the most compatible but it does mean that basically we're giving you Prometheus status quo you're going to be keeping memory for your cardinality until we reconfigure your system to use this push stateless export strategy. Yeah, question came in in our chat about what things exactly do we need to configure to get that stateless model you're describing. Um, good question. So we need some more collector development there is not an actual cumulative delta to cumulative processing stage today. It's imaginary. The reference implementation for the hotel metrics SDK which is the one that I've been developing and go does have this ability to select the stateless versus cumulative mode today so the SDK support has been specced out the reference implementation does exist it works, but as far as a collector support it's not there yet. If you had a vendor that was supporting both forms of otlp. Today, you could get that from police from the reference implementation. I hope that answers the question. I'm very excited about this because as a user from a stats D background, I was used to having a little bit more cardinality than I'm able to get from a Prometheus configuration so I would like to push more high more cardinality on the metrics world until it's a real problem and I'm hoping I think there's a good opportunity for data scientists to come help us with actual cardinality control mechanisms like we can down sample to control cardinality. For example, I think that's actually really exciting topic, but I won't say no more. I think we're almost at the end here of my slide deck. I have a few more diagrams just to kind of give you a greater picture of all the ways you might configure open telemetry metrics. This is an example of a sort of Kubernetes deployment where you've got a node, you've got a daemon set collector running on every node. You've configured your receivers for open metrics for stats D for otlp. You've got the Kubernetes receiver built in so you're getting the Kubernetes state metrics, you've got a host metrics running on the collector so you don't have to run those host metrics on every machine running on the pod, or on the node for example. So these are just a number of sort of this is the scale of these installations is now going we're going to be able to talk about one collector per node and a bunch of different targets per node and so on. And this is just part of this the plan for open telemetry collector. And part of that resource model is that we're going to implement hierarchical collection so that you have a node collector that's collecting all the resources, all the metrics locally and it's going to pass that to a regional collector that's going to attach all of your regional resource attributes, and it might pass to a global or other levels of hierarchy that you can use to organize your metrics data. So that you might be able to bypass certain Prometheus features with if you're using Prometheus you might be using recording rules to get some more functionality, but we can just talk about an expert pipeline that aggregates everything together into one place without that type of recording rule functionality, which is like a right time aggregation. All right, there's one more stated for last, I've discovered that there are a lot of Prometheus users out there who have invested a tremendous amount in their configuration so Prometheus does a lot of things and it's going to be difficult to replace all of that functionality. So we were looking for ways to help Prometheus users get on board with open telemetry and begin using otlp. So I found this thing that the Stackdriver group had done a couple years ago. It is a Stackdriver Prometheus sidecar. I haven't talked about this much, but today it's open source now we opened up the repository publicly. This is a sidecar for Prometheus, basically lets you read Prometheus data and send otlp. This I see as a short term solution because ultimately the Prometheus project will add metadata to its remote right protocol and then it will be able to replace this sidecar. So today you cannot send Prometheus data directly out of Prometheus into otlp without a tool like this, but hopefully in the future that restriction will go away. So I've got a few links to this code base as well as a couple of the Prometheus issues that are currently trying to address this shortcoming, so that hopefully we can retire this code in the future. But meanwhile, Prometheus users should be able to try their open telemetry metrics system, and this should also help us migrate from Prometheus onto open telemetry in a gradual way. Now I think I've actually reached the end of the talk and I'd love to have questions and discussion from the group. Are you expecting questions on the Zoom chat or we can ask them just like. Since I've been presenting I wasn't reading Zoom chat but I'm going to un-present. And then we can all talk and I'll read the chat. Have a quick one please. So this morning we were talking about traces and context and context propagation and this afternoon we are talking about metrics and labels and tags. So do we lose the context of traces when we handle open telemetry metrics or is there a way when we can link or correlate the traces context with the metrics labels? First I just want to point out there's been a bit of terminology debate and we haven't actually settled it. So the term attribute, the term label and term tag are almost synonymous and used interchangeably here. I hope that's not the question you're actually asking. I believe you're asking about how we talk about distributed context and getting those attributes that come from, say, distributed actors in your system onto your metrics. This is something that I mentioned in one of the breakout sessions earlier today was the sort of driving, I think a driving motivator for the census design and part of the sort of the heritage of open telemetry at this point is that we were based off of open census metrics. So open census metrics did include a way to, open census did include a way to get your distributed context attributes into your metrics. It's something that you have to configure. We're talking about the terminology for this. We've got something called an enhancer at this point, which is a hook that you can use to pull attributes out of your context and put them into your metric system. I hope that answers the question. We're still talking terminology. Okay. I think maybe there are a few pieces there. One of them is that we can pull key values from your baggage from that correlation context and put them onto metrics. Another one I think is just built into the semantic conventions of our metrics. We are trying to design the semantic conventions so that you can make correlations between those metrics and the traces that might be representing the same timed operations. And then the final thing that is maybe the most exciting in my opinion is the concept of exemplars to go on metrics. And what this is is like each metric will include just like a list of a few data points a few spans that are examples of that metric, which then there's like a very strong correlation between those spans and then that specific metric. Yeah, so to make that clear, we have ways, at least planned out or specced out in protocol to get data from your context, including both those attributes as well as your span ID and your trace ID, and to get that associated with metrics. Fantastic. Cool. Thank you very much. So we were planning to take a five minute break before the next session. But if there are any other questions, you can take them into the hotel community day Slack channel. Thank you so much, Josh, for presenting and sharing your expertise. I shared my notes in the chat. I'll share them in the Slack channel as well. I got a lot out of this. Thank you, Shelby. I'd be happy to stay on for a little bit. I do see the questions now when I was presenting I couldn't see them. I'm not sure how I could have fixed that but there's one here, for example, I can see about hotel policy on discovery when using open metrics approach. That is really one of the greatest questions and it's not one that we're really answering here open metrics or the Prometheus system has a great big piece of code that does service discovery. And right now there is an ongoing discussion about how to either emulate or simplify or replace such functionality in the hotel collector world. And the current state of affairs is actually not so good. We've linked in a huge dependency on the Prometheus system and are actually using that service discovery configuration, which has caused some friction and some dependency bloat. We are working on that. Bowdoin, would you like to speak? Anybody want to speak on that? Something that needs to happen. Yeah, I actually have a dream here that somebody will take the Prometheus service discovery code, factor it out of Prometheus and create a first class service called service discovery service. And then we can have an hotel collector plug in which will reach out to the service discovery service with some sort of sharding information and get a list of sharded targets for it to scrape so that we can then rather than linking in huge dependency on Prometheus just actually call out to a service that does the same stuff. And then the Prometheus service discovery will become its own thing. Awesome answer, thank you. Also, it was mentioned on the slack that this is a break of five minutes before the next session. So everyone who is willing to chat more, feel free to but otherwise, it's considered a break. Service discovery service discovery service. Oh geez. All right, everyone. Thank you. I'll be in the next session. I look forward to it.