 Hello. Hello. Hello. Hello. We have only six people. So let's do it like this. We give people a few more minutes to join, but when we start, we tell them next time we'll start on time and then we actually start on time, because I'm honestly somewhat tired of always waiting. We only have 15 minutes anyway. If we always wait five minutes, we're basically at 45 minutes. And all of this is way too interesting. So here's time. Yes, it is. It is. But that was the time all alone. Actually, I kept the Mohawk and the colors longer and the dyeing at the second time into different colors was also not planned. And coinciding with the lockdown ending in Germany or being lessened to really nice levels. I went completely bald. And it's super interesting for anyone who like all of the people on this call have hair. So you all don't know this and I didn't know either. Your ears get so cold because all of the wind, which is all of a sudden not catching in your hair, it's absolutely fascinating. And your head gets hot in random bits and pieces and spots. Of course, your head is trying to heat the brain and not have you die of cold, which is all irrelevant. But I'm trying to fill those two minutes until the five after and then we start and then going forward and I'll also send email. Actually, let me take an action item on this one. We will simply start and not wait on people. I've been looking through mail and things. I'm back from vacation now for two whole weeks. So I've excavated my inbox down to before I left. It seems we still haven't heard back from the TOC. We did. Order on the agenda. Did you hear anything? Well, the thing which I heard, so okay, it's far after. So let's start. So first things first, next time we'll start precisely on time and I'll send this out for future meetings too. So update on the TOC. There was this TOC sick call last week and we have a sponsor for both Thanos and Cortex incubation. She is called Katie Gamanji. I hope I pronounced it correctly. And she took both of those due diligence items for TOC. I poked her today and didn't get any update yet on what she thinks about the due diligence documents. But I also don't have any timeline or anything. So for now we'll just wait and see what happens. But we have a TOC person who is looking at both Cortex and Thanos, which is already a lot more than we had a week ago. So that's nice. So this was this. And the other thing is the open source analytic platform thing. Bartek, do you want to start on this? Yeah, definitely. I can essentially give some quick, can you hear me? Yes. I can give some quick update what was done from the last meeting because at the very end of the last meeting two weeks ago, we started this topic. And I actually kind of announced that we are definitely looking that as kind of prom to use community and kind of announced what are the communication channels and what we want to achieve. Let me reiterate. Essentially, if you click on this link, there is an email thread and also kind of GitHub issue where everyone is welcome to give feedback on what analytic system you are using in kind of, to be honest, analytic space using different signals to be honest, not only kind of observability, but signals from any data source. And what we want to achieve here is to kind of integrate better the monitoring world where, especially as part of the kind of prom to use project, we want to give more ways of using those monitoring data, for example, metrics in analytic use cases. And so we are looking for kind of voice of the community on what systems and what APIs are the most needed and what would kind of solve the most use cases. So we are super happy to kind of create those APIs as maintainers, but we need to have some kind of feedback where to go because, you know, we are not experts on this field. So this is the topic that we want to kind of move on. So that two weeks ago, we agreed that seek observability is definitely a good spot for discussing those things because, well, there is no better seek related to this topic. Furthermore, we are touching observability data, which in those days, we are, you know, kind of retrieving and gathering for long term times, like, you know, years and months of the data. So we want to make use of it. So I guess we'll be having some conversation in those meetings and on our communication channels. So I would love you to speak up even now if you have any feedback. So there will be time for that. I would also maybe give some status of what happened outside of our meetings very quickly. First of all, we have Prometheus Community Meeting, where we touched this very topic. I will kind of copy the link where you can find the video and so you can watch it fully. But essentially, as part of this meeting, we talked about this very topic. It's kind of in a tweet, but it's available. And we spent some time and what we had here in this meeting, we had like Rob from M3DB. So kind of long term storage metrics system. They shared what they are looking for in terms of kind of spiking this topic, how to integrate their metrics with analytics system. And he mentioned that they are planning to address some kind of Spark integration and Presto. So this is very kind of specific things that we can already try to integrate with. So you're welcome to check this video to kind of know the details. And we also have lots of feedback on the kind of GitHub issue. I tried to summarize this topic here briefly so we can kind of check this out. It's like a summary of what we know so far, what projects were mentioned and how we can kind of solve integration from metrics to those projects. Apart from Spark and Presto, Clickhouse once mentioned, Druid, all sorts of timescale DB, so all sorts of systems. And we are looking for the direction that will be the most useful for the community. On top of that, one more thing, Ozan, who I think is on this call, was super active and we found a couple of cool, maybe not very popular projects that are already integrating Prometheus kind of storage format and kind of couple it with, I think, some kind of analytic API that you can connect with Spark. Ozan, do you want to maybe give some quick DLDR summary of what's the latest of your findings? Hello. Do you hear me? Yes, perfectly. Yes. I am recently a little bit in a research mode about the topic and focused off metrics and maybe doing learning on that metrics data, but which every common using ML algorithms are in general in Python. There are lots of frameworks. And we are planning to use Spark jobs on our site because of the every analytic team has built-in and they are currently using them. If we can reuse this platform, it will be very good. So I am focused to them. Spark integrations uses some kind of internal Java and also Scala internals of the libraries to connect that. And there should be extra effort to maintain every each Scala and Spark release. Some elastic search, Cassandra, Cachebase, drivers are common in this sense, I can say. But a little dream or very crazy idea, but I see lots of every Python analytic or I think we lost all of them. Yeah, seems like I can see. But in the meantime, once it's back, like that, yeah. Yeah, we lost your last minute or if you could. Yeah, we cannot hear you. So while we wait for him to return, would it be useful for us to sort of brainstorm a list of scenarios that would be enabled by having these systems? I don't want to lead any witnesses, so I will speak last or later. But this is an active topic for at least my day job. We have exactly this squarely on our sites for a variety of scenarios. But if we don't already have that curated, no wrong answers, just brainstorming starting place, that might be worth a couple of minutes. What do you offer? Definitely. So in this kind of issue, we can gather the use cases maybe from our point of view, like from newbies who are not maybe using those analysis systems, so analytic systems, so obviously we can be wrong. So definitely it's worth to have maybe a shared document, Google work that we can offline collaborate with. So maybe let's have that an action item, right? So I will share some document work and focus on requirements, not necessarily about exact solution already, right? That makes sense to me. I think you need to understand the use cases of where you want to end up to go beyond the immediately obvious approaches. Like on a very high level, you can obviously say we want to enable deeper analysis, but to answer what do we mean, what are the boundaries, runtime, blah, blah, blah, blah, we probably need to agree on, but not even agree, but at least stake out a few common users and what actual needs to resolve. I can see the pot. Do we want to talk about this now? We can start. Sure. So one of the more high value scenarios for my team would be to be able to build data sets and curate them and expose them such that they are suitable for model training for anomaly detection, particularly in our case with periodicity on the month or multi-month or even quarterly or yearly scale. Many of the commercial offerings today, like Datadog, for example, do analysis of periodicity for anomaly detection on the one to two-week maximum time horizon. We run a lot of machine learning and we serve a lot of models for real-time auctions and other fairly real-time things being in ad tech. So we'd like to apply some of that same methodology to the vast fire hose of data we have coming from not just infrastructure and various layers like Kubernetes and the various cloud hosted services we use, but also leveraging custom metrics that are exposed as Prometheus or open metrics formatted data. Today, we're dumping that to Cortex, but we've explored everything from timescale to influx, to basically anything that can take a remote write input stream. And so we haven't solved this challenge yet, but that is one of our questionnaires so that we can, for example, in the insurance market, health insurance in the US is an open enrollment thing, right? And it's usually in the fourth quarter of the year, so we will see huge spikes there. But in other domains like financial services or other places, you have fairly predictable ebbs and flows of traffic that dramatically change particularly cloud-native systems. So being able to have a learning that's not threshold-based and is not anomaly detection based on what happened last Wednesday, but it's more about what happened last year at this time or last quarter at this time, those are some questionnaires. Amazing. That totally makes sense. So let me put here also the thing that we gathered so far in our kind of, let's say, in our, let's say, organization. So first of all, and also from the perspective of kind of from cues, what is missing or in what is actually difficult to obtain from it. So we were talking about kind of producing multidimensional reports from years of data. So very often you want to have a nice visualization that is not real-time and not for alerting, but actually to analyze that by humans, for example, or to show some characteristic and just print it right now for your managers, for your senior management. And this is something that is very painful in current kind of state of frontiers because all the queries are very heavy in terms of cardinality and currently print use is focused on real-time kind of response with low latency. This means that it's not easy. There's no kind of API that allows it maybe for like a streamed slower API that will allow producing those reports. So this is kind of use case that we found problematic and that we want to solve with this analytic forum. Second thing is good discoverability of what dimensions we have available. So one thing is that we want to be able to fetch this data for machine learning for other use cases, but first we need to know what data we have. And in prompt use it's not that easy because the best knowledge you have if you are producing this data, right? If you know what your application actually allows you exposes, what metrics are exposed, right? If you want to kind of discover those, then you are putting a heavy load and there's no very good APIs on that. So something that this is trying to solve is discoverability of your data. And allow joining data from multiple sources. Yes, there is definitely use case for having those reports or anomaly detection as you said that will combine not only metrics but also logs and maybe also the data that you got from segment that, you know, from website and what users are accessing actually. So those are further use cases that we could see, but definitely, yeah, let's gather most of them. I could speak to two other scenarios, but again there's a lot of people on the call and I don't want to hog the airtime. So is there any help? I also just want to make one note. So everyone is explicitly aware of this that we already have different use cases and as much as Matt was talking about shipping data off and doing analysis somewhere else whereas Bartek was talking about doing it within the observability stack, which is already a huge difference. And with that I'm going to show up and require someone who's not Bartek, Matt or me to pipe up and voice an opinion, please. We only bite if you don't talk. If you talk, we don't bite. Your cup goes away in the magic background thingy. No one? No one has any opinion on this? I mean on some level I like to keep things small and composable. So shoving a whole bunch of data analytics stuff into an observability stack kind of feels like creating a massive monolith instead of a bunch of little tools that compose to do something nice. But that doesn't mean you can't have a marketing umbrella that's got a bunch of smaller projects underneath it that go and fit together nicely. Just as a thought, you can also have something in the middle where you basically enable it. Like enable composability for example something, but we again very deep in Prometheus land, if Prometheus had a batch API versus an interactive API where you can put requests which basically they can return in an hour or in a day, I don't care. I just care about this happening at some point. And this basically what all mainframes grew large with this distinction between interactive database access and batch database access. And this simple split enabled insanely powerful use cases. So for example, this is something which an observability engine could offer and then try and just interface with other things nicely. Like for example whatever. And I think that's I'm 100% behind that. I think that the really interesting story is the API's that enabled the external use cases. So instead of going and shoving a Jupyter notebook into Prometheus, Lord help us building an awesome API that goes and enables those new types of workloads is definitely 100% the Unix philosophy composability community play. Yeah, I think at least for time series metrics, I think remote read and remote write are fairly brilliant. I think there's a need for some backfilling. And I've talked with some of the folks here already about this in the past, but we're looking acutely at maybe even writing it ourselves or doing it in the context of the Prometheus community, but a mirroring proxy for remote write would allow us to have our existing observability stack in place where from all sorts of places, both Kubernetes and not, we have metrics heading to a centralized backend. And if we could tee those off very easily and just replicate all of that to go to other systems like perhaps timescale or click house or whatever, that would let us kind of not muddy the observability stack with this analytic stuff, but would still make it very simple to move. And in the same spirit, one of the things that I have the least amount of like concrete plans about how we're going to achieve it is the same with trace data. In particular, we have a lot of our services that were universally instrumenting with open telemetry, the linker D mesh we're using also can export headers as well. And in those spans, many of our development teams are finding it useful to put additional metadata like log messages or stack trace failure, things unique identifier. So we have this whole bunch of data that's not Prometheus, or which I don't have the same set of APIs today to handle across different trace backend. So we're sending everything to Yeager, right? But how do I get it all that in a good way? And how do I use other backends or whatever to bring that trace data into scope for analysis? I really love the mirroring idea because going back to the segment comments from earlier, like that's what I use segment for is literally just mirroring and splitting out all my analytics data into a bunch of different backends, whether it's Google Analytics or BigQuery or whatever. And it's my favorite tool on earth specifically for that reason, because there's a whole bunch of different tools that do a bunch of different things. I want a single stream that comes in and then splits it out into other backends that I can slice and dice different ways. Is this a SaaS hosted thing? Yeah, I think there's a segment knockoff that's open source. There's a lot of competitors to segment. I've definitely used it. It's a pretty cool product. Yeah, I think it's only touched on it before. And again, I should be less afraid about talking, I suppose, to set an example. But you know, one of the reasons why we want to find a way to do this with open source tooling is primarily so we can run it ourselves. We handle a lot of PII and EPHI. And so many SaaS tools we can use, but it requires a fairly FedRAMP certifications of just a lot of SOC2 compliance, auditing. It's a heavy lift operationally to use some of these SaaS tools because the data needs to stay within our own VPCs and networks. So from a requirements perspective, I think at least for our business, self-hosted is a pretty important component. And when we get into the volume, we did some tests with Hay Linkerdee, a service match, giving me everything you can. We actually have a whole data platform team and a data analytics team and just get very quickly dwarfs to give them that in terms of tooling. Is there anything in the CNCF umbrella around reasoning on trace data that folks have found useful? Or do I have a knowledge gap about what I could just be using? Or is there an actual gap in projects? Well, if you have traces, you can actually send them to your last six search, for instance, and do the add six there. But it's very, it's not obvious out of the box that you actually need to take care of that in the back end. We had exactly the same question internally. We generated a lot of traces, a lot of metrics, and how do we do analysis on that? Right now, it's not easily discoverable, basically. Yeah, I think we've identified, at least for our roadmap, two buckets in this general topic. One, correlation of traces, logs, and metrics for operational use cases, what's going wrong, one-one-splat, or iterative development use cases. And for that, we've been using Grafana to stitch together logs, traces, and metrics, and we expect that to be a nice fit. But on the analytics side, to dive deeper in, I think there's a lot of opportunity. Does anyone know of projects that might help fill this gap? Or should we at least take an action to report up to the TOC that this is one domain that is right for new projects to join the CNSCAP? I think Yeager is probably the project that does most from the open source side. I think the rest is all in the commercial part, project space, where people are really doing the analytics on top. And projects are open to them, which we also try to focus really on the data collection, but not the processing and not the analytics part. But I think that the biggest part of analytics is definitely Yeager, especially with the Yeager project has done recently. I mean, there are other questions that obviously come up there, because do we have even like a uniform data model how you store things? Like you mentioned that people use log metrics as part of traces. This is what we started telling people like 10 years ago not to do, because it's highly inefficient from a processing part later on on the back end. And you want to split up these two data sources and just link them together via IDs. Because usually a trace processing needs to be even higher performance than you are in a log processing. So I'm not sure what I'm helping here and I'm following the discussion that's why I was like listening in here for most of the time. But if you want to combine these different data sources, I think that's really what you need. You need to agree on how data is labeled that you can query at the cross multiple data sources. And on the other topic, I mentioned before about processing and input output. I personally like one source of data or like one stream of data processing into different tools. I just see this getting harder and harder the more data you're consuming. And if you have like massively large systems just reading the data again after you're stored, it becomes more and more of an issue. This is also like a lot of the work that I'm focusing on right now. How do I not have to read or how can I reduce the amount of data I have to read as much again as possible? So if you say one day produce 200 or like 20 terabytes of data and between it to five tools and storing it in five tools is 20 terabytes became 100 terabytes. And if you even add more tools, we're going to also store data again and not just the results of the processing. It just becomes more and more. So that's why if you wanted to look into something, you would look more into a stream data processing model and integrate data can easily be processed in a stream data processing type of approach. Obviously a lot of people are using Kafka for these kind of scenarios. But this I'm just not sure when I'm off topic right now or I'm part of this discussion. I just wanted to. I personally think it's on topic if we're talking about, you know, traces being part of this observability data that's cogent directly. What we've been looking at in our group is, yes, we'll be using likely Kafka to actually shovel things around. But in particular, tail based sampling with the open telemetry collector and seeing what we can do to much like fluent bit moves processing of logs and tokenization and some of the compute cost out to the edge near where the logs are being transmitted so that, you know, you don't have all that stuff on the wire necessarily or you can reduce the raw volume. We're looking at how can you even considering rather, how can we, you know, do as much as possible where we're collecting all traces but, you know, at the point where we're collecting and we're calling the things we don't care about. So programming models to do stream processing at that edge locations where the services are in cluster or on a virtual machine, whatever before we store all that is, we think might be the best approach. Yeah, I think we are touching exactly this topic on the two ways either you move everything convert everything to the tools that target format or we have some way of aggregating and stream those data and just fetch what you need from the various data sources. And this is, yeah, just just nicer because you don't reproduce data. But I think one actionable thing that we could that like anyone on this call could start working on, maybe we just make a ticket for a task for it and see who who's interested is. But, you know, on this very topic, like I have active questions now that I don't see easy answers to without just trying it around Yeager's scalability. Like what happens if I take blank amount of data and throw it into a Yeager back end, and I just keep putting that at what point does it fall over and not become usable or should I run lots of little ones, you know, so once like we're psyched about Cortex because it provides the centralized fairly cheap way to store lots of time series data, you know, and make it queryable, you know, all the things the Cortex does we I kind of want something similar for Yeager for trace data and I don't I don't right now know what that is. So in scope for the city to be like making some case studies and just taking some base measurements so that I'd say like in the year 2020, you know, in July, were I to put this much data and this is what the performance would be. Yeah, I don't know what the right approach to it is, but there are definitely good discussions in the Yeager team how to scale it more. There was recent spike on budger, budger, I think database, and there are much, much more what's going on. But I think this doesn't solve like analytics kind of problem right now is just another maybe more card in card high cardinality kind of database and and the question can we scale it but but still we just another data right and I think we should have this topic how to scale Yeager and and and but I think that might be yeah something we can pull, you know, Yeager team as well here and definitely talk about that and help them scale because they they need some help. But I don't mean how to make Yeager scale, I mean, what is a suitable durable store for trace data that could be used for analytics purposes. Yeah, so it's more it's less of a Yeager thing and more of a like we have remote read remote write write and I can have different backends what that that to me seems like a gap but that sounds like a valid use case right we would like to have a tool that maybe compose the data from both from to use and Yeager and something that we are some system that we allow to do that it's actually a good verification for for for this topic right for this spike we are doing. Yeah, good point, Matt. And on the previous topic, if you're sitting here on the stream processing, are you we saying this is something that you're considering or my audio got garbled when I when you were talking is there projects or are there APIs already that could be suitable here where that could be applied to gain stream processing on? Well, technically everything you do with trace data really is sweet processing because you get in the stream you just have to merge a lot of data together that's how you ideally do it at scale. And also when we we did it before when we exported data it was also stream based because you need high performance output otherwise just the amount of data drives you crazy. But it is trace data processing to be able to make sure if you did and maybe that makes it so hard to obviously to tackle because the mixture of street data processing if you for example need to calculate metrics from that stream of phrases that you get in there like service response time you should want to collect them on a per trace level which you also don't have to do if you have metrics for response times but if you want to slice and dice them and then it's like really this high cardinality database type of queries that you run against these data. But even there we found that stream data processing solves a lot of issues like we analyze data for example for problem patterns as we are collecting the data and not later on. I mean my personal opinion is if you do good data analysis on trace data you reduce the amount of use cases where you have to do a lot of real-time queries on that data because that the bigger work right now goes into aggregation of traces. It's kind of interesting to watch and we have Jonah here who's working in space for a while. We started always Cindy traces until to the point where we realized that looking at Cindy traces is nice for a developer to understand what the code does or for debugging intermittent errors but reasonable and large systems just have too many traces and similarity and outlier analysis of traces which eventually graph-based data structures is something that they're usually looking for. The individual parts you rarely look at because it's just too many. So one of one of the things that Yeager is missing is like the ability to create metrics from traces in that in that real-time way which is definitely something we're looking to contribute back into the project and then the idea would be to expose those to be scraped by something like Prometheus where you could then start to do trending and other types of analysis of the trace data that's being collected and ultimately you could actually detect a lot of problems just based on metrics versus having to actually analyze all the traces because with Yeager it just hammers the back end and it becomes really difficult with elastic search to do these types of things at scale. This is exactly what we're dealing with in our system as our customers continue to send more trace data to our back end and it's definitely kind of a disconnect between traces and metrics that I'm seeing in the community. So the same approach is one of the reasons why we've chosen Loki for some of our log aggregation scenarios in particular Promtail which is a demon set that runs in a Kubernetes context anyway and tails all the logs. It can look for things and have log derived metrics that are exposed as a Prometheus endpoint that can then be scraped by the same in cluster Prometheus to get exactly the same thing to get like frequencies and various custom processing. It also does it on the server side in a slightly different way which is which is a differentiator actually because you can do log derived metrics for things that you didn't set up ahead of time to make metrics for but different topic for a different day. I mean I think it's a different argument. I mean I believe that logs still have a lot of value if they're indexed and Loki doesn't do indexing so unless you think of all this yeah sorry go ahead. No no I I'm sorry you're absolutely right. I didn't mean to say it was holistically everything it's one prong of an approach but we're also doing some indexing on targeted workloads as well. But my point was that the notion of at the point of collection deriving time series metrics because it's simpler sometimes and more expedient to reason on the time series than it is to plow through the full trace data or plow through the full log data. It's a common approach. And you also have to do it sometimes like if you want to calculate metrics across different instances you have to aggregate them somewhere again. You could argue you're also doing it based on metrics data. I think this discussion we're having right now for say what to use for what has is one that has been going on for over 10 years in the monitoring community. And just telling people what to do is it is a good idea like what to use metrics for what to do with traces what to not do with traces. Like if the only thing I want is a response time for a service sending trace data that I then use to calculate the metric to then do alerting might not be the best move at all. Might not be reliable essentially. Yeah I guess that's why I wanted to focus on scenarios. I've been holding off but I could just really quickly just bounce through two additional scenarios that we might want to add to the list. I'm curious if other people have these same ones. One is correlation of infrastructure metrics with cost. So you know most cloud providers that we use give you costing information and being able to correlate those two things. There's two others that we had when looking at horizontal versus vertical pod autoscaling. That's a place where we would like to run experiments capture the impact particularly for legacy workloads that were inherited shall we say that aren't completely understood but have you know good test collateral or we can filter some traffic to run experiments. You know when does it make more sense to scale vertically versus horizontally and just capturing the metrics for all of that and then correlating it with what the autoscaler did can provide you know the ability to create models to game out how we should better utilize hardware and then correlating all that back to cost. Those are those are two scenarios at least that are ways we internally at every quote might want to use analytics on and then the third I'll mention is that we've launched internally to brag a little. It's actually quite simple just a deploy tracker service so some of our CINCD when it makes deploys or when a lot of hand deploys that aren't automated yet are made. We've got that going to a back-end service which is then making in our case Grafana annotations and also surfacing that via a Prometheus aggregation push gateway time series metrics out of our deploys so we can correlate like deploys to anomalies or errors or whatever happening good or bad in our infrastructure and that's another thing where there's so much data and so many different points of measurement and so many different services that applying a more systemic analytical approach that that can make it less you know needle and haystack kind of traditional debugging uh is this scenario we'd like this is exactly it's exactly something we've considered uh basically when when we're doing a deploy you want to look at the trace data to see exactly what changed if we can generate metrics out of there then we have any insights into I don't know maybe this span took 10 percent longer suddenly because we did this deploy events and we can correlate that also to response time on a metric and maybe look at some logging at the same time so aggregating all that is it's kind of really hard right now because you have to integrate with lots of different systems and yeah with traces you don't have really good tools right now to do that that can handle the load and scale well but yeah one of the things you said was you wanted if others were kind of heading in the same direction definitely I mean we with exactly those use case right now yeah I think I want to kind of bomb this thread because we are actually thinking about that in in kind of from Qs community because with the Loki and and the hacker we are kind of collaborating together and we are already having some poc of solutions that for example um are having a strong links between this data so you are still pushing you know gathering metrics you are still pushing blocks like you used to you're still pushing traces however you there is a kind of way where you only index once because this kind of you have actually the same resources uh in both of those signals right and so you can totally have the same index that will give you you know the the the certain application for example so that's already reducing a lot of space and complexity and and resources that you have to use for for those signals right so for example you have Prometheus are giving exemplars that can hold trace IDs for the for example slower um some interesting thing that happened during observation of latency of this request for example so then you can navigate to trace um and and and have traces and the same you can share the trace ID and the request ID for the logging so you can jump from that between logging and tracing and if you would be and kind of work together in a way that those databases will share some kind of information for example indexes um which are very heavy there is a huge win here so um we're trying to to go through this because you really need to gather metrics and push metrics right as we agreed for the for the different purposes like for alerting you have to have reliable real-time metrics um for traces you are fine with the kind of slower latency for um for for kind of trace availability um so there yeah it's hard to have just one solution and just use traces and forget about anything however we want to make sure uh with this observability group as well that we can collaborate together and have uniform solution yeah and then that's what we care for you it's also interesting it's you know you mentioned coordinate everything and if you use the elastic set completely and i'm in no way like trying to sell it or anything but you can actually already do that but you have to be completely locked in in that system it would be nice to be able to use now the cncf projects in the same way that you can correlate all those things all together uh and still keep you know the cloud native kind of way of working exactly yeah this is this has been awesome i'm just looking at the clock at we have i think we have two minutes left um uh richard what do you think next steps are i mean i don't want to i think i was just moving forward and and become actionable things that we can fan out to so i think one uh one thing which we should be doing is is start collating use cases um start collating use cases um in a shared document ideally if you want i can send them on around without any restrictions so everyone can just toss stuff in and ideally we maybe let this sit for a week or so to to have a brainstorming session and then from their try and distill more and more um generalized use cases out of this um the other thing um cncf to see asked us to to uh select a third chair with a little bit more view on diversity so if anyone of you have suggestions um highly appreciated because initially we did look around it didn't really find a lot but we'll we'll restart that effort so if you have anyone i think those are the two anyone is anyone interested in working on sort of a case a set of case studies maybe not on what tools to use but just like either working with either reaching out to projects like Yeager and open telemetry and seeing what they have in terms of again like i'm i'm tactically looking i'm going to deploy Yeager for example or i'm going to deploy whatever um just raw sizing it seems like a you know figure it out as you go and i'd like to just pre provision what's needed so there might be some little hanging fruit there for new new contributors and new members of the SIG i would i would encourage us all to make suggestions like in Slack or in GitHub like i would i would like to period a backlog much larger than the folks on this call can deal with so that we can say to new potential SIG members hey come join us this is all shovel ready we just if you're interested in this stuff you can contribute and here's how so um perhaps that's a separate brainstorm and i'll follow up with something on the list i think we can we can do that collaborate the kind of together right so i'm happy to set up this document because kind of started this this topic and kind of announce it so everyone can reach that the close projects they are working on and and and give an input on the case studies and requirements right this is our goal for now and use cases or didn't i catch you saying use cases because i think from for the SIG like i think case studies are valuable but somewhat orthogonal in as much as either someone has something working which they're able and willing to to talk about but this is largely disconnected from us coming to an agreement of what use case is to cover or what use cases are considered within yes they're totally orthogonal i just we had mentioned them in line and i just wanted to call them out as potential next step they're not as much more going to Bartek who seemed to mix the two but uh yeah okay i think that's it and see you in two weeks yeah ideally we should really and i i know i keep saying this but we should try and move more work into into text and use this more for open discussion but i liked the open discussion today that was pretty good cool thank you maybe i'm gonna try to think of a joke involving observability for next time yes please i'll put the challenge out because i'm not a very funny person uh let's add action item i'm doing thanks