 I'm just popping the agenda doc into the chat. Please register yourself as attending, and if there's anything you'd like to pop onto the agenda, please do it. Also, it'd be good to do a little open tracing review for the first half. So I'll try to edit up the notes based on what we're discussing. So feel free to add notes and comments to that section as well. Ctuk V4Q2 plus CR, that's an interesting handle. Okay, let's kick it off. I suggested that it sort of end of summer, fall is coming. It might be a good chance to do a sort of open tracing project review, kind of where we think the project's at, what we think's improved, what we think we could do better on, and where do we think we should be focusing our time and energy for the project coming forward in the fall. So kicking it off with good stuff, sort of like what's been going well, and I didn't go grab a bunch of metrics and make pretty graphs for this. So this is all anecdotal feelings about the project, that people do have data that is interesting. But yeah, what's been going well? For my perception, there's a lot more open tracing users and awareness of the project and awareness of distributed tracing in general seems to be up from where it was last year. I don't know if other people have a similar experience on that front. But that's been my experience from going to conferences and other events. Does anyone else have sort of comments on open tracing user base or just distributed tracing in general and where that's going? Definitely distributed tracing. So we gave an open tracing tutorial at Velocity this week. And when I asked how many people used tracing, half of the room raised hands. This is like the biggest I've ever seen. And there's like about, I think, 70 people in the room. Great. Yeah. We're starting to see people ask us about integration with different systems. So that's kind of cool that we're in the past. They didn't actually know too much about what we're doing. Yeah, it definitely seems to be a growing concept. And I know when people come to use LightStep as a product, I'm told we get more people who have already integrated open tracing or some other tracing system, as opposed to just coming to it from scratch. Yeah, I would say one of the most real things on this front that I've seen is the New Relic earnings call, I think for two, three or something on it, remember, but like, you know, the founding CEO of New Relic is talking about distributed tracing on their earnings call, which is kind of bizarre to me. And it's like 10 years ago is literally unknown. And it still feels to me like it should be kind of an implementation detail, but it's become a marketing buzzword that is interesting to, you know, institutional public investors. And that's the first, for sure. Yeah. One shift I feel I see is the move from tracing being, yeah, this sort of limited tool that did a lot of sampling, and you were only using it to get certain kinds of like latency averages, like a very specific tool to this world where people just want all of their logs, they want all of their event data contextualized, and their systems are distributed. So they have to propagate some context in order to contextualize the data. And so tracing via context propagation is becoming more and more central to just observability in general. That's, that's my impression of this shift from tracing being this sort of niche thing you did on the side to more and more something that's in the middle of your observability due to context propagation. I have an opinion on that. Something I'm really interested in is features and analysis that's on the tracing data. So currently Yeager's primarily trace focused, and we have some dependency graphs and everything, but I think there's probably a wealth of exploration that we can do on trace data analysis. During the Amadeus talk, the guy mentioned it would be interesting to have specs around the data structures, not just the collection, but maybe something that supports post analysis. So by the data structures here, which which data structures are you thinking about? Well, I don't really know, but I'm sort of like, I think right now open tracing is seems to me and I'm not familiar with it, but primarily focused on collection instrumentation. So if we were to also sort of facilitate and support structure around post analysis, so maybe trace transforms that sort of thing, the common data model, etc. I mean, what like open tracing tries to be sort of, let's say, agnostic to the implementation, but I do, I agree that like all implementations will have the issue of what to do with or what is useful about traces, right? So yeah, I think I mean, if we can define interfaces, and then people can write code against those interfaces, then it's not like installation specific. So yeah, I can work on light step data or Yeager data or anything else. Yeah. I think so the part of that, when it comes to analyzing this data, I think there's, there's like two parts, there's like you're propagating this context, and you're doing all the analysis internal to the system that's sort of underneath the tracing API. So that's like one world where your tracing system is just hoovering in all of this data and doing some analysis somewhere somehow. But then there's another world where you're propagating context, and that context is being propagated for the availability of other systems to do some analysis or have some access to it. And I feel like that's a place where we by design don't have an interface right now, like we sort of do with baggage, right? Like you can set baggage in one place and get it somewhere else. And that's our context propagation method. But if you want to say propagate something for the duration of a single process or even just setting a tag on a span, and then using that tag later, like right now, we don't, that's not like an explicit part of the open tracing interface. And people have written some middleware things on top of open tracing to do this stuff. But it's not like a thing we're waving our hands about. I'm wondering if that relates to the sort of data structure analysis you were talking about? I don't know. It sounds like that might be like adaptive tracing, so to speak, or something along those lines, where you can respond to the trace data as it's kind of in flight, being collected. Yeah, what was more like post analysis. So we have the eager query, which you can send it a trace ID and get back a trace. Something like that, like where you can, basically you have an interface that allows you to work with spans and traces in a way that's agnostic to the underlying, I guess, installation. So yeah, your light step. So then people can build analysis on top of the tracing data, regardless of which subsystem it is, or which vendor, that type of thing. So you're talking actually about a data format, like an export format. We have, I guess, protocol buffers and soon to have JSON and stuff like that. If we aligned on that, then people could write analysis against open tracings, that would be eager for instance. You're thinking about the use case where people ask to get a JSON dump or something of the data, and then if there were a standard way to read or to expect what types are getting back, then people could do more across the board. Yeah, like, if there was a definition of the schema that was in OT for every like language might want like scholar Java and go and JavaScript and Python, and then people could write analysis and aggregations and derivative products against open tracing data instead of against the eager data or light step data. It also probably helped developing tracers because there'd be a standard representation for the data structures that then you could shove into whatever your tracer wanted. Just like the kind of idea is collecting all the data is awesome and so far so good. And I think there's like a wealth of value in what we do. So we can maybe put some energy into taking a look at that area, that domain. Yeah, go ahead. Kind of a side thing, but I've been talking to Simon Stewart on Twitter, who's the web driver Selenium guy, and he's been looking at open tracing for observability of Selenium 4, which is still sort of early days, but I think there's a lot of interesting stuff there with kind of using open tracing integrating open tracing into testing frameworks or testing tools like Selenium to not only let you observe, not only let people observe, okay, what's the software doing, but also to think about like, you know, tests and performance as traceable and observable events tree, you know, as traces, right. So there's cool stuff happening outside of just sort of the, you know, context passing RPC, yada, yada, yada world. Yeah. Yeah, to that note, I saw a talk from the guy who, I guess, started finagle and maybe started Zipkin, Marius, something. He wrote a like a data processing framework, and they're using Zipkin format to collect traces and then pump it into X-ray, and then they're doing a bunch of analysis on that data to determine where they should focus their developments on the framework, like where the, I guess, low hanging fruits are, and also high impact efforts are. So it's not really, like it's all based on aggregations. Yeah. That's just another example. I can give some color on where this thinking has come up before as well. One place where we'd like to see more standardization both inside open tracing and like across the industry is not just the data format in terms of like what are the shape of the protocol buffers, but what we refer to as sort of semantic conventions, the actual keys and values, and that's more the part that people end up seeing in the past have seen as useful. Like it's easy to write a parser for like one format to another, but if all the data is different, that's where it gets really nasty. If you can't, if you're having to like within each span, maybe there's something called HTTP code or something called HTTP status code or this or that, both having more standards in that field than we currently have in our semantic conventions, and then also having like more structured data in our actual APIs for entering that data into open tracing. Right now you sort of have to look these keys up as constants and kind of compose all of these values, but people have asked for things in the past like could I just have like if I'm going to report on an HTTP request from the client side, can there just be an object with all the fields I should fill out to report on that and shove that into open tracing? To sort of make it easier to do the right thing? Yeah, so both come having more standard keys and values and like getting everyone kind of on board with that whether using open tracing or not, and then in the open tracing API like pushing those standards more explicitly. Yeah, we have structures. We see kind of a lot of variety across types used in across across keys used for given data and also across types for the data for a given key. It's kind of yeah all sorts of stuff. So yeah, and as an aside just some more color that the people who are least interested in this are vendors it seems because they're like well I already have a tracing format so this seems like work and also it's not like too much trouble for us to just pay an employee to like do some translation from one format to another. So they don't see it as super important for them, but where people do see this as important and helpful are organizations that have tracing data that they might want to export or give people but they're not a tracing vendor. So some infrastructure company you know Twilio has given us an example of this, but anyone who has customers who are using them as a service as part of their backend is going to start caring about potentially handing out you know the portion of the trace that goes into their system. So the sort of like standard tracing trace context headers that we're trying to standardize in W3C this data format thing comes up there in the sense of like people who aren't vendors would be more comfortable if there was some standard format like if you could just tell them here give us put your data in this format and people are guaranteed to consume it they might be more inclined to actually go through the efforts producing that for their customers if that makes sense. Vendors themselves actually are sort of like we'll just we'll just import every format. Do we have tickets for these for this issue? I don't think we have a ticket for this in open tracing. I think one part is just sort of a story arc of like should we you know should we add more conventions and if so like what should they be when we just start making up our own or should we be either like other you know this we could partner with. Yeah there is a I think it's a ticket in spec for specifically the HTTP API error object thing. Right there's some IETF standard that showed up right is that what you're referring to? Yes is that IETF? Yeah there's IETF. Yeah you were on that. I mean sure. Well why not right? I guess this one feels like one of those things that's going to be more stuff that goes into the conventions and you know you don't have to use it. Well the API can also export sort of like kind of helper utilities that aren't part of the core API that you can opt into and then that's what factories for tags and stuff well now they can be like statically typed on like error you know is Boolean and then you can just send that into something that pulls all the tags out of it that type of thing. Yeah that was kind of my thought is that it would seems like more like an extension over here to the side rather than like core API stuff but yeah yeah. I do think whatever conventions we pick we we could do more to make it easier for users of open tracing to adopt them. Well the simplest simplest way is to extend set tag interface to take a sort of a factory object and then in the country we can create special factors like HTTP tags thing and then it will have like with status code with URL blah blah and then boom so tracers don't need to change but they can just ask that factory okay explode this to me to a map of tags automatically. And we can also have custom stuff for different frameworks like express we could pass as like an express middleware could generate that and then edit so that there would be I think pretty extensible and pretty friendly. Yeah it's basically we have tag types right now but they're all simple value data structures and we want to have something tags that are more like complex structures. Well potentially. To your point we don't necessarily if I understood what you're saying we can maybe change the way tags are set rather than their representation. So is that right so it's sort of like an object that is aesthetically typed helper factory that we you know use to generate tags instead of modifying the representation. Well yes because we like our API is already export the constants for tags right saying like HTTP status code that's like a constant in every API. So instead of doing constants let's just create an object saying like this is what you want to set if you're doing HTTP with like typed fields and then pass that object to the tracer and tracer will know how to do it. Then we can have some built-in stuff for the semantic they've condone semantic conventions and then people can extend it. Cool yeah that is a thing that has come up kind of repeatedly in other areas and yeah I think it would be useful. It actually would be better if it's not in the country but in the actual proper API because again like why define constants instead of just defining a structure. I think right. We can have both but the constants become implementation details so that did like you don't actually encourage people to use constants constants in this way right. They can but you would rather say just use that struct populated as much as you can and then we'll do the constants ourselves. Yeah I'm saying like if we can handle the structs kind of in a general fashion other people can write the same sort of utility for different contexts or the built-in structs can be for the kind of blessed conventions. Well we already have conventions for like messaging for SQL for STD. This is an API support right so I'm saying if we handle in a general way other people can kind of create API support for other contexts. Yeah they can yeah but this is like I think like it's one thing to have custom for specific framework. It's another thing to say oh there's like a generic HTTP. I suddenly need to import country for that. That's kind of nasty. Yeah right it's better to have an API directly. I agree cool. All right did anyone feel like being a champion for this issue? I can write up a ticket for it. Yeah that's a start. All in the spec right the ticket for it. Yeah in the spec in the spec repo. All right Joe I'm going to put you down. Okay okay that that seems definitely useful kind of next step thing. Getting back to going through like what do we think is going well. I also feel like issue PR staleness responsibility. I feel like that's gotten better with the project. We have more process around things so the part where you know there's like live debate and responses happening feels better to me. What doesn't feel great is the actually like closing out and shipping new releases like there's a lag between oh we should do this and then debating it and then it seems like there's a lag at the last step of like actually getting the APIs at the door and sometimes that's a good thing because we want to create room for debate and stuff like that. Also I think it's just we could like tighten it up there. Yeah there was an era you know a year plus ago and we moved very quickly on stuff like that and I think that ruffled people's feathers and now we don't do that and I think the result is just an unnecessary hit to velocity for the project. I mean I think it's great to have a time for open debate but once that's over I think that we've got to just move forward. I mean this I think the thing you were talking about with you know a type struck for HTTP calls I mean I think that's been discussed for years and no one's ever thought it was a bad idea. It definitely is a little bit of a hassle to get it over the line but it's not that big of a deal and we should just move these things forward. We've got to do that. I mean in terms of the I just wanted to go back a minute as well to the you know what's happening with the project kind of thing like I would note that in the beginning of the project I think when CNCF took open tracing on it was a very early project for CNCF and we got a lot of attention. I think the attention open tracing was getting was actually ahead of the puck in terms of where the project was. Then there's a period where I think it was about it was like an parody and now I think it's actually other way. I mean it's like open tracing because it's sort of a standards project and we've intentionally not been making a lot of changes. There's not that much to announce about the core API which is sort of a feature in a way but the I think the experience of velocity that I've heard from folks who are there for instance and also and other just sort of mixing with the industry type situations it's very widely deployed. It's much more widely deployed than I think that is documented frankly and so there's a there's now an opposite gap where I think the sort of like perception of open tracing I'm not really sure what it is but the actual deployment tracing is quite broad right now and I think that it's not perceived it's not it's not documented. I think I don't know how it's perceived but it's not documented and I think that's a shift from how things were like a year ago or something like that for the project. I'm not sure if it's good or bad it's just I'm also curious to hear if people hear of a different experience of it. I just am surprised why I see it coming up like this. So we did once on the Yeager project we opened a ticket called a survey and we pinged people who we knew were like active on questions etc to ask asking them to write some short summaries like what company you're working for what's your size and whether like why using Yeager what issues you have. There are some like response to that there will be like if we could do something like that for open tracing and maybe get a bit more data of who is actually using it and yeah. I think this this pivots into you know what I see is sort of the main thing we should be focusing on this fall which is there may be some core API changes that we need especially around making context propagation more useful and specifically thinking about metrics use case where people have a metric system that's separate from their tracing system and they're trying to dimensionalize their metrics in some way. So when you put a product ID into your tracer that you know propagates that product ID and then you can you know make metrics using it it's basically what we kind of do with baggage but baggage is just a it's like a very crude mechanism so I think we should have some focus API wise on that but really the rest of it is around just application developers part of why I think things are quiet is like the people who need tracing tend to be people working in bigger companies working on bigger systems but it's still like hard to adopt for just a variety of reasons better documentation announcing things and you know just being out there more to like help people office hours and stuff like that but then also things like sugary APIs we've discovered like simpler APIs agents doing more work on agents for languages that support them registry making it easier to find OT integrations just all the things that lower the barrier for application developers to kind of wrap their heads around this and get started I feel like we haven't put much effort into that and so that's the other reason why the project feels a little quiet and especially if we don't want to go ripping the API apart every three months like but still want to have like notable velocity for the project I think it all needs to be in that area of just building value so they're coming and using open tracing like regardless of what they think of the API they're showing up because you know the integrations are here for example that's the reason we've seen people for example bridge their homegrown tracing system to open tracing is because like they're sick of rewriting these instrumentation libraries and they just want to use ours so I think just focus on making that library ecosystem work really well just all the things that make this nice for application developers should be a focus I don't know do other people have thoughts on how important that is and also like what like brainstorming what we could do what what they think aptives need to make tracing easier I have lots of thoughts about this but I mean you already know them but yeah I think I'd like to see like an open source effort to automate the configuration of like modular pieces instrumentation so that you don't as a developer need to say oh well let me audit all of my dependencies in this particular service and figure out what instrumentation is out there and plug it in it's a little bit different than just a fully fledged agent like all our app dynamics or something it's more like a discovery process followed by some kind of automatic binding and I think like seeing that through to its natural conclusion you'd end up with a sort of meta API for discovery and configuration of tracing instrumentation or something like that but with some registry mechanisms and so on and so forth but I'd love to see something like that I think that the aptive experience of having to go and discover and and audit configure a bunch of tracing modules even if you don't have to write them is still kind of a hassle and it's very very much in line with the goals of the project to solve that problem yeah it's definitely a pain point for us and every other person who's ever done this ever I mean it's like a pain point I think so yeah totally it's super painful it's hard for me there's a lot of reasons that what I described is actually really difficult with various versions out there I mean it's not an easy problem but I mean one go on as you're saying that's like a layer we kind of try to stay away from because it's so widespread and awful so sorry yeah I know I mean I was gonna say like it's obviously hugely challenging because effectively writing a bunch of static analysis engines for every language that will be able to you know handle all these different formats of defining dependencies and yada yada yada I do think that having to register just a simple searchable registry is a good first step and that we can build on that to kind of make suggestions turn it into some sort of like hey this whole here's a tool that scans your you know scan your stuff and tells you what you can use yeah yeah there's a thing that we want to avoid people who make agents tend to put the whole kitchen sink in the agent and we could do a more modular approach where we have all this instrumentation it's all manual instrumentation it's all written by the community and it's all out there but can we then just write like standalone tools that do things like the dependency analysis and matching it up with like uh just understanding which plugins to install and then if you have another so that's just like a standalone discovery phase and then if you have a standalone phase that's just about automatically wiring things up for the people who need that phase uh then that's like a separate phase yeah and there's certainly I think there's like text acts and there's language languages and frameworks that work better for that part than others yeah I was chatting with someone about like oh it was you Joe about like dumb dumb node tricks and like agent-based node js instrumentation which could either be super hacky or like super not hacky depending on how you do it and depending on what native support there is yeah but then you're ultimately you know you're only hitting a certain amount of things and are you are you solving like the core problems it'd be really cool I also think to see just kind of better framework support for stuff like C sharp I know someone's been working on like net core um like net core two web framework instrumentation with ASP.net maybe it'd be worthwhile casting around and seeing if there's desire to have like net four five and above if if more C sharp people are looking to get into the project um or start using open tracing seriously yeah Ben is there a ticket for that because I'm kind of I want to learn more about it so hopefully there's some reading material about your thoughts on that oh gosh I don't know um but I can write something I think that it's been discussed in various places but I don't I'm pretty sure there's not a single place where it's discussed in any depth that's at least not a ticket but I don't I'd be happy to do that I actually think that'd be good yeah um we we do have someone right now working on java agent like stuff in java yeah right now just FYI uh so you'll see uh some uh call for a review of that project uh probably soon it has to get a little bit further uh using the existing java agent code base as a sort of example the basis to work off of but it's ended up I think taking a slightly different approach so just FYI something coming soon at least for java on this from yeah um so like yeah basically I think you know the apis are sort of like I mean we need to have the apis there needs to be an api somewhere but I really feel like the value of the project to me is the instrumentation like if we can just be the the best cleanest most normalized instrumentation out there and then also like the best educational resource giving lots of talks about tracing having office hours going to meetups good documentation just generally being an educational resource like those are the things I think that are really valuable that we can provide as a community when I think about the value open tracing the open tracing community can provide to the world it's it's focused more there if that makes sense and the the api is more just like table stakes like there has to be one with certain requirements and there wasn't one so we made one but it's less about that and more about like actually instrumenting all the software I would uh I may be biased but I would add like a third uh opportunity which is the post analysis uh phase that I was talking about those first two like the instrumentation especially um that's really focused on collecting information but then making it actionable uh is I think like a like a kind of a wealth of possibility yeah I mean more specialized analysis tools basically it's like if you've got enough of this system standardized you don't have to go out there and rewrite all of this stuff just to come up with a new way to analyze the data exactly like if we like if if Yeager query and and lights up and whatever whatever other systems would want to kind of align on this if we had endpoints to expose traces and a standardized structure then there could only you know you we would only need like one inch implementation of transforming it to like a service dependency graph or one instrumentation of or implementation of doing differentials etc yeah then it could just work across systems and actually could potentially consume across installations um I would love to talk to you more about this I can get you a feed with yeah there's been some of this work done through w3c I'll kind of point you at that maybe we can have a one-on-one wouldn't be great I'll uh I'll create a ticket for the high level kind of intention and then maybe we can connect awesome yeah um and I'll mention yeah it's not just the um open tracing there's other people who other groups who are not directly involved with open tracing for whatever reason but are also interested in this so but yeah we can we can see what the scope is awesome that's great uh yeah so open census haha this is up here on the list I think this is also something important the confusion between open tracing and open census has gotten to the point where it's nasty and has to be has to be dealt with I think for a variety of reasons um and uh just uh so people are aware there is uh a sort of summit kind of discussion meeting uh being planned uh I don't think like everyone in the universe can show up for this meeting but there is at least going to be some high level people from open census and the open tracing projects uh and the CNCF and various ecosystem players sitting down to figure out how these projects can collaborate better with each other um because we a lot of the individuals we all collaborate just in general on various tracing things and then there's like all this friction around these two projects uh so I'm curious to hear what people I guess I'm curious about two things one what do people find painful or confusing right now and two um what do what does the world people wish to see in relation to these two projects like if we could collaborate more like what do people have concrete visions of what they wish that looked like well given the audience of this call I think this is not a particularly fruitful discussion it's interesting to me at least going in there if I'm talking to people from census I want to understand what their goals are but it would be helpful to me to understand what what the goals are for people who are involved in the open tracing side I don't want to be like we passionately care about x and it turns out like no one cares about x it was just me so what I would like to see is if all the instrumentation that already exists for open tracing uh it could be used with sensors as a as a sort of the instrumentation of the tracer right and that requires effectively common api between the two projects even if that api like needs to change from what open tracing is today that change is probably not going to be super significant and the instrumentation themselves would be easy to adopt but if we if we could like yeah if the if open tracing would would become the api for that then it would be the outcome that I hope for so you're saying open sensors should adopt the open tracing api we can agree on a new api where basically it's changed but I think one of the things is like open sensors currently bundles api with instrumentation and we don't have a strict governance on how that api even evolves right like open tracing has way more governance and community review of the api than open sensors is because like so I think I'm sure these are things that are changing I know they seem to want more I mean I don't want to speak for the census people they can speak to themselves but they seem like they're sort of trending in the direction of trying to open the project up it's also kind of why this seems like a good opportunity talking to them so you're trying to bring Microsoft know the people in so it's not just an internal Google project so we started that document about open tracing goals to share ahead of that meeting we can share it here but I think we should I would prefer to just send it out to to that audience that is supposed to meet so if anyone else needs to review that please do that and let's send it yeah I would love to review that with you early next week Yuri before sending it out you have to go to the woods uh I mean I've looked at I I would I think it's very safe like there's nothing in that but I think it's very controversial I would rather just put it out there like I mean that must I mean you could take like literally 30 second look at it if you haven't looked at it but it's pretty okay I'll post it in the meeting notes yeah I mean I don't I don't want to like speak for the group but just just to make sure my my personal opinion that is known like I feel like very strongly that the two projects are damaging each other and the industry right now in their current state like that part I'm positive about and I think that the the greater good for sure and actually open tracing as well would be served by seeing like I'd like it to be like firmly documented that we actually would like there to be like one API that is pretty decoupled from the implementation of a tracer or a tracing system and that we're not like adamant about what that API is called as long as it's not just code-wise but from a kind of like product standpoint it's positioned as a separate entity I think that's really important it was that my only really significant technical consensus is that they haven't done that I think they really need to do that if they did I would be happy to see that no good tracing move closer or become the same thing that's the thing that I feel personally feel really strongly about and I can make lots of in the weeks technical arguments for why that's important going forward but but I I really would like to have as much of the goal stuff documented way ahead of the meeting which is why I want to send that out and I want them to feel some pressure to send stuff out too as the worst thing could happen is that we go to the meeting and spend like a lot of time discovering things that are static like goals for instance like that stuff should just be out in front way way in advance and hopefully we can get in there and already have like the main thing was you really need to debate outlined before the meeting and and I see like the next week as an opportunity to dig out those topics yeah yeah I I totally agree that we should have all our ducks in a row before going into that meeting and for what it's worth I think it's very helpful to focus on goals not implementation details because I think that is just we will just end up in the weeds and bike shedding I noticed we just because we're nerds it's like red meat if we start talking around API specifics or things like that it's just that'll just derail the conversation yeah so one one thing that might not be implementation specific but still go into almost contentious discussion is that even when we talk about goals I when I met with like Bogdan last time we had this sort of overall amicable discussion but it got heated at some point because for example one goal of open tracing was like we don't standardize in the data formats right and then they're saying okay well but if you don't then you effectively promoting vendor lock in because because we you don't provide any data format and so as soon as you pick a vendor in one service you have to pick the exact same vendor in the other service effectively right and so which is it's a it's a valid argument it's just not an argument against the goal of of having an API right but but that discussion will happen and so I wonder if we should maybe in addition to goals and non-goal sections have something like in that regard but it starts to get in into the like argumentative territory but I just don't want to spend another half hour like debating that point where it's like okay yes we pick different implementations of course you're gonna have a problem but it's not an API problem per se the API doesn't create that problem and I think that's I think you've hit the nail on the head is part of where a lot of this friction comes from is the you know many people feeling the elephant where all of these things holistically are important when you go to build a coherent tracing system that's going to have interoperability between multiple services potentially even multiple tracing systems yada yada you need you need all of these pieces right like you need a wire protocol you need some kind of data format you need analysis systems you need uh APIs uh like you need all of these pieces and because open tracing is tried to say like well that's like too much we're just going to focus on the API problem that leads these conversations into the ground where we say uh we're not focusing on that or it sounds like we're saying that's not important but to other people they think that is really important and so it sounds like we're disagreeing but usually what we're not saying is that we're disagreeing on the importance it's just that we chose that to be out of bounds for just the piece of the puzzle we felt like we wanted to work on like we just wanted to work on the instrumentation ecosystem in code basically and making sure software actually is producing this information and so the thing we cared about was the API and we were trying to keep that separate from these other problems it's not that these other problems aren't important though um and there is something to be said about having one organization that is like the kitchen sink maybe it's well factored but there's still at least like if we're talking about like I don't even know I don't want to call it open tracing or open this or something I just I want to make up a new term um but we're talking about just like distributed standard issue distributed tracing is like all of these pieces like so one bigger project that's well factored I think would be better than having a bunch of little projects each trying to focus on one part of it that confuses the hell out of users like they've got to come to open tracing the API and then go to Yeager to get the implementation and then like the W3C to get the wire protocol like it just and then the ITF for a data format like it just that's like really confusing to people I've noticed so I'm just gonna insert my slide from velocity uh it's partly your slide uh Ted but with some annotations uh I think this kind of clarifies I mean you know sounds like the decision to focus on client instrumentations but now it's been saying earlier it's you know stable mature state it could potentially grow beyond that yeah but we're discussing specifically like contention between open tracing and sensors right because like if if sensors continues to get in steam with a different API then everything that open tracing have done becomes useless right if like people instead start and then people now have to decide oh if I'm uh like instrumenting my stuff should there is open tracing open sensors yeah it's super hard decision like how do you have to bet on one technology to win effectively right so open sensors is like abc and open tracing is like a and so there's some contention about like open tracing not prioritizing or valuing b and c and I'm saying like if we want to align maybe we can sort of make a paradigm shift and make it known that you know b and c are important and imperative that we align on a b and c not just a yeah but I don't want to actually extend the scope of open tracing project because we will already have like a lot of work and not enough resources to even do like all the instrumentation and prs etc so ad and b and c is just like why if open sensors perfectly would drop there right yeah the way I've started it myself is sort of along these lines is that there's a set of problems that involve the instrumentation of software and it's it's a really significant chunky set of problems and then there's a separate set of problems that involve everything downstream from that and that actually can be subdivided into many different projects potentially and the 10th point is well taken that people just want to get shit done like the idea that they have to research first I'll have to understand and comprehend and then separately make decisions about each one of these slots is sort of overwhelming I think for people so I recognize that I think from a long-term maintenance standpoint making sure that regardless of marketing and how things are what things are named making sure that there is some clean separation between those two things is like it's vitally important that's the thing I'm trying to say more than anything and that's that actually goes beyond whether or not to merge projects or not it's it's more of like I mean if you don't merge the products and open tracing continues to exist then there's automatically some separation there but but I think from an engineering standpoint that separation is the thing that I'm really arguing for and I think the trouble is you have a project that does A, B and C it's much easier to kind of cheat about the boundary between A and B and poke holes in the API for the for convenience and and for like short-term trade-offs basically that assume certain details of B in the design of A and that actually I mean I generally think open census is pretty high quality software but they've done that in a number of places and I do think it limits the read the the reusability of APIs when you when you make those decisions so to me like the trouble with with expanding open tracing scope is that there'd be a similar temptation I to to make assumptions about oh well of course you're going to be using something like Yeager downstream when in fact there are many different ways you can use this instrumentation that there tends earlier points about content propagation have literally nothing to do with latency analysis and we need to make sure that those things aren't we don't box those out I think the industry will be better if you don't so that's been my argument for separating the thing all along it's just the the portability not just across analytical like latency and latency analysis system supportability across completely different applications of this instrumentation that's the thing that I'm really adamant about so you're saying stick to A B and C is a several problem there's also kind of a prime like there's all this stuff like the agents that they're talking about all this you know standardization of semantics all there's a lot of stuff like A itself we really just built the firmment of it we haven't done any of the higher level primitives that really make this like a nice UI for a programmer I don't mean a plain clip UI but like a programming UI or D what's the DX is that the new newfangled way developer experience of using apis is quite limited by your anyway I mean I think that's what we should be focusing on but it seems like it has to go yeah I think interesting getting kicked I don't know I think there's a lot of value in B and C and because the implementation is decoupled it kind of gives us like a that's like an advantage in terms of warding off the pitfalls you mentioned yeah sir A has a lot more depth and potential change but I don't know the I guess the downside is not putting any intention on B and C means we're also not influencing it yeah well we kind of are it's just I mean and that's that's what's weird about this is there's like discussion about how and we should wrap up in a minute just to be fair to everyone there is discussion about oh you know we don't we can't handle like the breadth of working on all of these things but the thing I noticed is it's like when I go to like the W3C distributed tracing meeting and we bike shed about what the wire protocol should be for hours it's like more or less the same people you know like so it's like even in terms of like the staffing of these efforts they're not like that different and you never know if like if we decide on the mission the people may come but I have to go uh pleasure chatting with you guys thanks yeah catch up early in the week yeah all righty I will end with a a final shill for the people who are left on the call and maybe you're not aware of the open trace sorry not open tracing but the observability practitioners summit uh this is a single day talk we're calling it ops for short uh that it'll be a single day talk single track conference uh in front of kubecon I can't announce the speakers yet because we're still finalizing that but I am very excited about all the possible speakers so it's going to be a pretty rad conference so if you can come to kubecon in December you should definitely consider coming a day early uh and going to this conference uh and the registration link is in the meeting notes for today so check that out let me totally crash and paste it into the uh chat window as well that's my my moment of shilling not too bad I have to go to thank you lovely love you seeing y'all hey y'all