 For joining us, I hope everyone had a great lunch. To our virtual audience, thank you for returning. So I am very pleased to moderate this panel about the present and future of observability. We have gathered together some really excellent minds in this space, some open telemetry maintainers from these contributors and various other people to, you know, I'm going to ask them all a question about the future. But we also would love to hear from you. So, you know, part of this is we have these wonderful people lined up. So there'll be plenty of time for questions from the audience. To that end, I will be able to bring a microphone around to people that want to have questions and I'll indicate when it is time for that. So without further ado, let's get into introductions, starting from my left to the right. All right. Hello, everyone. My name is Rula Si. I'm a software engineer at Grafana Labs. I'm a member of Open Telemetry Governance Board. And I'm mostly active on Open Telemetry Collector. I've created a couple of Open Telemetry tools like Open Telemetry Collector Builder and Operator. And I'm really glad to be here. Hi. I'm Anthony Marabella. I'm a software engineer at AWS. Also work on Open Telemetry Collector. I'm a maintainer of the Go SDK and also work on the Lambda layers that we provide. Hi, everyone. Very happy to be here today. I'm Alulita Sharma. I am a member of the Open Telemetry Governance Board, Governance Committee, as well as co-chair for the Observability Tag in the CNCF. And I also lead Apple's the IML Observability Teams. So very happy to be here. And again, look forward to some of your questions later on. So please do feel free to get your thoughts organized and do ask. Hi, everyone. I'm Alex. I'm an engineer at LightStup and I'm a collector maintainer. I also wrote a book on Open Telemetry. And yeah, I'm glad to be here. Hi, I'm Lily. I'm not an Open Telemetry maintainer. I think I'm the only one. But I've been working in the Observability Space for five years or so. I was a CubeSafe Metrics maintainer, part of SIG instrumentation, and I worked on OpenShift monitoring for three or four years. And I'm an advisor to Polar Signals. So thank you for that. So real quick, just to kind of get the conversation started here, this is a panel on the present and future of observability and we have quite a selection of people who are involved in both the present and the future of observability sitting here. So what do you think? Like where are we at? Where are we going? And how are we gonna get there? Just to kind of start the conversation and then we'll throw it to you all in the audience. I have the microphone, so I guess I start. I think we were kind of having a discussion the other day on this topic. And I think the way that I see it, observability is still in its infancy. So we have this, the collection part is kind of close to solved, not entirely, but very close to that, for some signals at least. And we see observability a lot in operational parts. So we want to monitor things because we want to operate them better. But I think the real revolution comes in when the business folks realize that the same telemetry data can be used for business analytics. So they can enhance or they can improve their businesses by using the same telemetry data. And I think that's gonna be a game changer in the future. Yeah, I would definitely echo that sentiment. Once you get business metrics and visibility into the state of the business and business processes, things get a lot easier for you. I also think that with open telemetry in particular, we're kind of at an inflection point where we've really stabilized the core signals of traces, metrics and logs. And now we're moving into a phase where there's gonna be a lot more work going on, not on building the APIs and SDKs, but on building instrumentation and getting that instrumentation pushed as far down the stack as possible. Right now a lot of instrumentation is application authors doing their own instrumentation. But in the future, hopefully we will see more library and framework authors instrumenting their libraries and frameworks so that as you build your application, it comes effectively for free. And I hope that we're really moving into that stage where that becomes the focus of our efforts. I think that's a very good point to Anthony. I think, again, there is a lot happening in observability as many of the tools that all of us are working on, the projects within the CNCF as well as other open source projects. And there's a tremendous amount of innovation that's happening in the open source space, which is really addressing not only cloud native observability, but also at the edges. So a couple of areas that I really feel that are an open telemetry is very representative of that is that when it comes to the amount of data that is being produced for telemetry, the different signals are in one sense. Every signal is generating a tremendous amount of data, whether that's traces, metrics, or logs. One of the shifts that I see, which is happening, is that today metrics are used a lot. And logging is not as pervasive as it used to be from a telemetry standpoint. And then we have traces coming in. But the key area in this data space, and as more data such as profiling as well as other types of error handling comes in into telemetry data, I think there is a need for continuing to evolve the common specifications, such as OTLP, as well as there is a need for a common query language, which is an initiative that originally started from the open telemetry discussions that we had in the last year. But then this discussion has moved to the CNCF tag for observability, where we are in the process of setting up a work group for defining, or at least working together, as a group of end users as well as developers and vendors towards definition of a common query language that addresses all telemetry data and reduces the fragmentation that an end user feels. And the other area that really is another unification point, if you will, is visualization, where we'll continue to see a lot of evolution towards smart visualizations, especially with the technologies. And I won't mention the buzzword here, but different technologies coming in into being able to drive a lot more correlation of data, and then, hence, being able to actually provide even smarter visualizations. So I really see a lot of convergence happening in the next couple of years and evolution of new products. Yeah, I agree with you, Elida. I think the correlation between the signals is really the key thing that I see moving forward in the next few years. As we saw this morning, people still think of metrics and traces and logs as different signals as the forsaken pillars. But I think as people start adopting open telemetry and start seeing the ability to correlate across those signals, hopefully, the advances in tools will prevent people from having to think of them as separate. It's just a data stream at that point. And the other thing that I also wanted to touch on is what Anthony mentioned, which is I think we're going to see a lot more adoption of open telemetry built into a lot of tools. I was just playing with renovate bot last week, and I found out they allow you to export data and using OTLP, which is exciting, but you just enable it with environment variables. So I think that's all really exciting stuff. And regarding open telemetry specifically, I think we've done a lot of work to build the signals. I think there is some work on the way right now to make it easier for end users to get started with open telemetry. I think the barrier of entry is still a little bit high. So I'm looking forward to seeing that evolve as well. I think everyone summed it up really well. But I think that the missing thing is the user experience. So improving the dashboarding and the querying and those kind of things is really important because we can send as many signals and we can come up with as many signals as we want. But if the users don't know how to actually get the value out of this, and I think that that will definitely change, like we've been working on signals. We've been working on the various common libraries and various ways to export. And I think that the next step is empowering the user, essentially, with user experience or with the AI predictions and things like that. So there I mentioned it. So yeah, I think that that will be, like, in my opinion, the future. All right. So what about, what are the audience? We have any questions? Anyone wanted to ask a question? So please decide it all yourself and then. And please do introduce yourself. Oh, yeah. I'm Friedrich, software engineer at Adobe. Also cortex maintainer. You shouldn't have asked me to do that. But anyway. Anyway, so I wanted to ask around the, and I'm part of the system, right? So I'm part of open, adult, and everything that we're doing. I don't want this question to come across as an attack. But as the elephant in the room that I see for telemetry, I see an ever-increasing high amount of signals, which translates into high cloud bills, and sometimes high vendor bills. I see auto as a great offering. But at the same time, it comes to the user as a way. So how are you going to handle this? How are we going to sell this to customers in a way that they buy in that it's not just going to be more expensive, that it's actually value-provided? Well, I can take that question because I actually am an end user. And I can say that if all the vendors stop charging for data costs from one service to the other, then it does definitely help in the world of observability data, right? With that said, Jurassic. Sure, why not? I do believe that there might be an initial incentive on getting more data in, of course. But I think there is also a market pressure that if I don't do things in an optimized way, if I don't charge in the best way that I can charge, someone else is going to do. And eventually, things might get on the same level. And then turn the price down or the cost down. I'm perhaps a little bit on a different position because Grafana offers both a cloud solution and open source, meaning that we only get people on the cloud if we can offer a differentiation between what they get with open source. So people can use open source to host themselves. And if they need extra help, they can pay Grafana some money for that. That's a different kind of incentive to the pricing. So we have to charge in a way that makes sense for folks that are self-hosting that data so they can choose what to do. It's perhaps different for cloud vendors that don't provide this path, I suppose. But I don't know, perhaps. So one thing I think that's going to be important in trying to tackle the cost problem is giving users tools to figure out how they want to process their data and to be able to process it as early as possible to minimize those transit costs or whatever other costs. So if you're trying to do span metrics processing rather than having to send spans to an open telemetry collector and do span metrics conversion in the collector, can you do it in the SDK? And if you shift that all the way left, then you're only emitting metrics. And you can sample all of your traces fairly heavily then and send a lot less trace data downstream. So I think it would be really good to help understand what are those use cases that users are trying to solve, and how can we, as the open telemetry community, help provide the tools to push that transformation left so that you only have to send as much data downstream as you absolutely need? Yeah, I think it's really important for vendors to provide the tools to provide cost controls for their end users, ultimately. And I think the open telemetry has, using the collector, it allows you to control those costs. It's not quite as straightforward as just configuring your sampling rate or something like that. So I think there's going to be some work to be done on the collection side to ensure that the way that you would configure your collector is smart enough to capture the data that you do want, but without going over a certain amount of cost that you're willing to pay for it. I think that until we come up with something like that that is smart enough, I think users need to eventually know what they need to drop, what kind of metrics, what kind of logs, whatever, that are not useful for them, and just essentially drop them to save cost, until we're in a place where we know and can predict what's actually been used, what's actually useful. All right. Thank you. Great discussion. Who's got next? All right. Back there, I'll come around the site. Come on over and introduce yourself and ask your question. Howdy. I'm Sean from Atlassian. Am I allowed two questions? The questions that I have is, what expected signals do we have that we are currently missing that will come to Hotel? And the other question that I have is, will Hotel ever consider data residency as a problem that should also help solve for ROM? Hopefully, that makes sense. You got two questions. No guarantee they'll get two answers. Sorry, can you repeat the second question? Was it data residency? Questions about data residency for real users of monitoring? I'm actually not sure how much of that hotel can do. The best question was that with the different types of data signals that are being supported by Hotel today, do you see other signals coming in? Was that your question? Well, I think that Hotel today is definitely committed from its initial charter, as well as mandate as a project, to be supporting the primary types of data that exist in majority today with metrics, traces, and logs. Now, I would say that the order of stability in terms of the signals and the implementation stunning stable has been kind of in reverse because tracing in one sense is a smaller segment of the instrumentation of applications today, whereas metrics and logs is much more pervasive. That said, again, I think profiling is very important as the fourth area of data and the type of data that's coming in. And I think that there is also other types of data that intersect from a larger analysis of applications end-to-end, which haven't been supported in general as well, where you see a convergence of, for example, a lot of the debugging information or data that is at an application level that is not typically considered telemetry. So as some of those areas converge, and that's inevitable, it will because it really is an unsolved problem still. And there's a tremendous amount of fragmentation as you go up the stack because there is a whole different layer of application tools, profiling tools or debugging tools in the application space, in the BI space, which are not considered observability data. So as you go up the stack and that converges end-to-end, you're bound to see some of that convergence happening and also landing into a hotel as a collection mechanism. Also, I hope that we can actually provide as a project the support for all the other projects that actually are experts in their areas of functionality that has been built and converging interoperably into supporting hotel also. So I think I'll let others speak before we go into your second question. Yeah, I really want to kind of pick up on that. And what Alex was saying earlier about correlation, open telemetry doesn't, I think, need to solve all of the problems for all of the signals. But the work we're doing, merging with the elastic common schema to provide a set of common semantic conventions that can be used to describe data and the relationships between data, I think, will be hugely important for giving the tools to the people who are working with other types of data that are not strictly under the hotel umbrella, but are still related to process that data and to describe the correlations between them and how they relate. And I guess perhaps one interesting part of that question, Sean, is that the next signal is going to be the one that the community wants. Because we are working very hard on logs, metrics, and traces right now. But some folks have stepped into the community and proposed a note for profiling. And now there is a group of new people to the open telemetry community, getting the profiling signal advanced. So some of them are here, but really folks from different companies, with different backgrounds and expertise, are trying to build a standard within open telemetry. And I think the same is going to happen for other signals. So there's going to be something for mobile, perhaps. Perhaps not a new signal, but perhaps expertise from people in that part of the industry that we are lacking right now within open telemetry. And perhaps a browser as well, so real user monitoring that you also mentioned. So we are thinking about RAM, but not as part of the core open telemetry right now. Because we are too busy with the three signals that we have at the moment. So I guess the message really is that the next signal is going to be the one that the community contributes to. Can I ask a question? What signals would you like to see in open telemetry? Profiling? What was the second one? Analytics. Analytics? Anyone else? Shout them out. Data lineage. I hope someone's writing these down. I think it's being recorded. Did the panel want to talk about data residency requirements? Did the panel want to try to talk about data residency requirements? I'm sorry. Huh? Data residency? Not right now, I think, Shun. But if there is an interest, I would welcome folks to just form an OTAP and put a request together. Because that's typically the first step in really also gathering a community opinion, as well as interest in being able to support this and also identifying the use cases. As in all good open source projects, pull requests. Welcome. Not as an epithet. Who's next? Who has a question or discussion point? All right, I see you back there. And then you, I think. Or and then you. Well, I saw this guy first. Sorry. Thank you a lot. I present myself. Hugo Trotignon. I'm working as a solution architect for Suprastaia, France. Speaking of data residency, a thought came to my mind. We are in Europe. So GDPR is a concern. And like I say, I'm quite new to observability concepts. But what can we do by addressing metrics, traces, logs on the collector or processors, maybe with the community to address the kind of GDPR concerns without having a negative side on what knowledge we can gather from this information? So to rephrase that slightly, is there a way that we can provide a European safe preset for open telemetry? Can we as maintainers, as project leaders, make the decision about what is safe to kind of automatically collect, take it away? Yeah, I mean it's the first thing that comes to mind is it seems like we could use best practices or some form of documentation around what the best way to accomplish this is. We often say that open telemetry is not a back end. So I guess this would apply to the data in transit. But I don't know if anything particular to open telemetry would apply here. I think it's best practices that ultimately will apply to the vendors who are providing you the services. Yeah, totally. And I think that as Alex was pointing, the data typically is, there are several ways of, again, implementing GDPR compliance as well as other data compliance in terms of privacy and security. And that's typically supported or implemented at the business level. By the time it hits open telemetry, it should not even have those concerns. But as you said, encryption and transit as well as other schemas can be implemented around in order to guarantee further requirements. Did you want to address anything else? Yeah, I think again this is a place where having a good semantic convention or common schema that can provide guidance and metadata about how this telemetry data needs to be processed can be useful both for data residency and for GDPR concerns or anything like that. So from that perspective, I think the goal of the open telemetry community needs to be to give application authors and operators the tools that they need to be able to create the data from the beginning in a way that it can then carry with it the information that needs to be processed safely and effectively. I guess only one question that I do have for you. You mentioned without the bad side effects. Which side effects do you have in mind? Latency or ability to restore the data or what kind of? Yes, indeed. I thought of latency due to maybe ciphering and ciphering of data and so on. Yeah, so one way of achieving that without the extra latency or without causing extra latency would be to not collect this data at all at the source. And I guess that's also part of the suggestions here that providing you guidance on how to configure the SDK so that you don't collect that kind of data would be the first step, would be the most logical solution, at least in my view. Thank you. It sounds like a great place to follow up more with conversations later, so be sure to use our networking breaks. You wanted to, yeah. Next question. Hi, I'm Mario, working as a platform engineer for Grand Centrics, and I think one of you mentioned dashboarding at the start, so my question is what can we expect in the future in regards to dashboarding? Because I think right now it's really separated from the whole OOTP spec, right? Yeah. Yeah, dashboards, not part of an open telemetry, but... Not really, no. But I know one of the things that we did in developing the Metrics API and data model was try to ensure that all of the instruments that can be created through the Metrics API have a logical default presentation in a dashboard. That's why we have a difference between a counter and an up-down counter, so that hopefully you can just instrument your application and the dashboard system can then derive from what types of metrics you've sent it, how it should present those in a default manner. I don't know that that's Grafana, you may have a better insight into whether there's good implementations of that at this point, but that was our goal at least. I guess the question was directed to someone who explicitly mentioned dashboards. Was it Alex? I think several of you mentioned dashboards for the future, right? So I can give my take. I mean, our idea is indeed going into the direction that Anthony mentioned, that have you instrument your application and then have a backhand determine or understand what kind of instrumentation that is. So if you're using HTTP semantics, then we should have an automatic dashboard built for you for HTTP-based services. So that's the idea, right? So this is where we are going for the future. Now, there are of course several semantic conventions, and some of them are more important than others, and there is the whole problematic of correlation of signals and so on. So it's not that simple, but I guess the idea is going into this direction indeed. But your question was directed to someone explicitly, I think. No, okay. It's open, I think Lily has, Lily, do you want to? Yeah, I think that auto dashboarding is a really good thing in the future because personally, when I have an incident or something, I don't like predefined dashboards, as sometimes they just point me to the wrong root, like cause of root, basically. And I like the idea of saying these are roughly the things you have around this alert and have a dashboard that basically opens, but I don't know where we are with that. I do know that there are some open source projects around core dash, I think it's the org, within CNCF that's working on some dashboarding. I think Persees, was it Persees? Yeah, so maybe you can check that out for the open source side of things. Okay, I'd like to see all the walls of dashboards go away, though. I was gonna say, the future, when are we gonna get to auto SLOs? Focus on what matters. I think someone back here, or you? Okay. If you're raising your hand and I'm missing it, then you should raise it higher. I'm Aaron, software engineer at CalCASA, and my question is more about the collection side of things in user controlled scenarios, so the browser and desktop. It feels a bit scary right now to try to do client server spans because then you have an open endpoint where everyone can just submit anything. Is there any way or path for it to authenticate data, basically when you end up with it in your dashboard? Did someone fuzz the endpoint or is it real? So to make sure I understand your question correctly, you're not asking about authenticating to like the collector service, but the information that's sent to the collector service. Because if the browser can submit, then the user can submit anything in the browser. Yeah, I don't know that we have any good solutions for that at this time. I know publicly exposing telemetry collection endpoints is a common risk and question that we get, but I don't think that there have been any solid solutions because if you want to collect it from your end users, you have to have them be able to provide it. And for the most part, the tools to authenticate to that have to be in their hands as well. Yeah, so the threat model is quite different. So on a backend, all of your services are trusted, at least at some level, so you can trust to store that information. And you can trust to a certain level that you are not gonna get DDoS'd by your own services. Now, even though the telemetry data is very similar between the browser and backend services, the threat model is quite different. So you do have to have a separate collection pipeline for the data that is coming from the antrusset sources. So you need to do authentication, you need to do a WAF, a firewall in before your collector. You need to do all sorts of bot protection and DDoS protection and so on and so forth in there. And the other side of the problem is you don't want to collect everything that is running on the browser right now, otherwise you slow down your web application. So you need to create the traces or create telemetry data and send them to a collector that perhaps are gonna be stored for a few seconds only or a few minutes and then sent to the backend if this data is necessary at all. So it's a totally different type of problem to deal with. And I think those are the concerns that the folks behind RAM, the real user monitoring a part of open telemetry are starting to think about. Okay, we have time for one more quick one if there's anyone that has a quick one, no. Well, in that case, you have a question? You can use your own mic. All right, I have a question for all of you. And again, I'd like you to be bold, to be, you know, take risks and give an answer. What are some of the features that are your pain points or some of the areas you'd like to see addressed? You've seen, you've brought up security as an area, you've brought up data, you know, the diversity of data as an area, please, you know, in three words. Okay, lightning round. Simpler metrics SDK. No breaking changes in the APIs. Ha! A simpler configuration? Good one, who else, who else? Oh, back here. More performant Java agent. All right, that very scientific survey, just, oh, we got another one? Morgan has a question. Morgan? You're eating. How's the cookie? Oh, that he's eating. It's good, I clearly started to eat at the wrong time. I was gonna say automatic configuration for the collector, like, zero config. And for those of you who need a little few more minutes to think about this, there is an open telemetry project meeting later today at four. Four o'clock. So please, please join us there and bring your questions and all kinds of hard problems to solve. That very unscientific survey, that just, yeah, you failed the roadmap for the rest of the year, so good job. Thank you so much. Y'all can open source governance on your resumes now. So I'd like to thank everyone on the panel for coming. Please give them all a big round of applause. And thank you for all your wonderful questions. And we will now let the next speaker come up.