 Okay, good already. Welcome everyone to New Year. I'll kick this OTSC call off. My name is Ted Young. I work at LightStep and spend a lot of time contributing to the Open Tracing project. Since there's no uniform list, I'm just going to call people's names out and go around and you can just say a little thing about who you are and a one sentence about your relationship to open tracing. That'd be great. So let's start with Manu. Hello everyone. I'm Manu, software engineer at Datadog. My relationship with Open Tracing is working for the Python implementation and helping contributing to that one. Great. Yuri? I work at Uber working on Open Tracing and Yeager, the tracing system at CNCF. Ben S. He just popped out. Warren? Maybe Warren can't hear us or can't unmute. Let's keep going. There's someone named Dan Kubrick. Yeah, hi. Dan Kubrick. I work with Chris Erwe at SolarWinds, so into distributed tracing and just kind of tuning in to stay abreast while he's out for a little bit. Great. You can see it. Dana Keynes, welcome. I work at Scow and we're primarily interested in Open Tracing with Python. Nice. E. Arnold? Hi, I'm Erica Arnold. I work at I'm an engineer at New Relic and also just tuning in to pay attention, watching Open Tracing, getting excited. Great. Nice to see you. Dave Anderson? Hello, Dave Anderson. I work with Derek at Scow and we're primarily interested in Python Open Tracing implementation right now. Great. We've got Jonathan Caldor on the call. Hi, I'm Jonathan from Facebook. I work on our distributed tracing platform called Canopy and I'm here mostly to hear about other people's experiences and talk about our experiences as well. We've got Gary Brown on the call. I work for Red Hat on first distributed tracing technologies. So Open Tracing, we contribute to Jager. Recently, I've been dabbling in Envoy and Istio. Nice. Can you see him? Pavel? Hi, I'm software engineer at Red Hat. I work on Jager and Open Tracing and I work with Gary Brown. Great. Then we've got Ben S. Looks like he's popped back on. Well, anyways, Ben Siegelman is from LightStep and he's also hanging around and works on the OTSC specification. So let's kick it off. We've got a couple brief updates about languages. The Java API is about to go out. So it's just been a little bit of time talking about that. It'd be great to get a Python update from Manu. And then after that, what I would like to do with this particular call, since there's nothing else on the agenda, is to just sort of have a roundtable discussion about a couple bigger issues around project structure with language maintainers and then sort of a dynamic linking issue that I can get into. So hopefully we can save at least 30 minutes for those two topics together. But to just start it off with the Java API, that's a pretty big deal. A lot of work went into that. And I believe we're now finally, finally, finally at the end. There's a single sort of patch issue that's outstanding right now, which I should link to. But that involves just having an old baseband as a sort of stub interface, which was solving some like dependency circles I think Yuri was having. So that's sort of the one final thing. Yuri, do you want to just say a moment about that? And then I think Pavel, you had some questions about that one. So it'd be great if we could hash that out and just get that thing in, because it's sort of the last. Yeah, the problem is that there is a lot of instrumentation which upgraded to 0.30 without actually changing any code because the remote user active span concept. However, once it was compiled against 0.30, there was like a reference to this baseband class recorded in the binaries. And now if you try to run the same binaries with 0.31, aside from that base class, basically everything would have worked because there's no API dependency still, but it still blows up because the class is not found in the new jar. And that manifests as a runtime dependency, unfortunately. So it's kind of nasty for people to discover that at runtime, even though let's say they upgrade like either Jager or OpenTracing API and then suddenly it blows up on some other library. So by simply introducing this interface as a dynamic class, that seems to solve that issue. Okay, sweet. And Pavel, I think you had some concerns. Have you had a chance to I didn't have a chance to play with it, but I think it looks a little bit weird. Yeah, I felt like I wasn't strong enough Java programmer to really grok how it works or what about the dependency thing, something to do with something to baseband, and then it just gets upgraded to the regular new span interface, which has all the methods it was looking for, and then it works. Is that kind of how the mechanism here? Yuri, did it fail actually on the loading, right? It fails on loading, yeah. But I think it can fail on the runtime because it's different class. It blows up on a runtime when loading the classes. Yes, it's certainly at the startup of the application, but still people wouldn't find it at compile time. So, I guess it seems like a small thing to give to give in just to have that class rather than basically cost paying again with the upgrade. Yeah, it's a pretty minor thing. It doesn't affect tracers or users to have this stub in there. I just I felt like I couldn't quite wrap my head around why it solved the problem. But if you feel confident that it does, I think we should. There's no great reason to not put it in personally. Yeah, okay. Well, let's move that discussion to the PR out of band from here, but let's try to get that in or otherwise resolved by end of day today. And I believe that that is the last issue that's been raised about the 31 API. Yuri, do you feel like you would be satisfied given the work you've been doing on Yager and porting it over to release on Monday if we put this in? Yeah, it would be good. Okay, great. Signing guys, finally made it. Okay. And then someone tacked on a single point here about update for contribute modules. Yeah, it was it was me. And mostly we started migrating all our Java tracer to the new API. But because in the Java agent, we are using open tracing content modules. I saw that some work was already scheduled and done to ship some release candidate. I just want to have an update about the remaining modules, the remaining content modules. Yeah, what is the plan now? In terms of time, of course, I think we need to to just and this sort of dovetails with the language maintainers discussion. But yeah, we need to just make sure every Java instrumentation we can find gets ported and possibly leaves a tag or branch for the version that works with the 30 API, I suppose, come up with like a tag OT V zero 30 or something like that. So people have a consistent way of being able to find that older one. That would be my suggestion for how we do it. But it'll mostly just be I think contacting the maintainers of each piece of instrumentation and asking them to do the work or doing the PRs themselves from some of the core groups that have Java staff who can push on that. But that's basically it. Okay, perfect. From that point of view, I think that we are using a couple of integrations that are not currently ported. So maybe we can contribute. We have to ask to Tyler that is helping us with the Java part. Okay, great. I'm going to hold him and okay. All right, I think that's it for Java. Let's move on to Python. Manu, do you have an update? Yeah, so currently, okay, because of Christmas and so on, we didn't spend a lot of time doing some updates. I just went to tell the action items for the next week. Because the first issue is that we didn't have a lot of reviews about the API specification and implementation. And just from, okay, of course, for me and from Carlos. What I'm going to do next week is addressing all the comments from Carlos that are pretty easy at the moment at this state. It would be great if someone, maybe people that are in this meeting, interesting about Open Tracing Python, if we can have some feedback about that. Because for us, actually, it's working. Internally, we are already using part of it for some services, not everywhere. And, but yeah, having more feedbacks would be great. Does the mock tracer support that the propagation? So for the Python implementation, no, I mean, I've just provided the basic tracer implementation. So the real tracer. I think, yes, it's missing. It's a missing part, but yeah. It will in any case. Okay, so for us, because most of Uber instrumentation is in the Open Tracing instrumentation library. So if that were, and I think that one has unit tests against basic tracer, I believe. Yeah, that works. Then that's probably a good validation. Okay. Okay. So that work is just basically in flight. And if people want to get involved, the Open Tracing Python channel on Gitter is a great place to ask questions or ask, hey, how can I help? And Manu here is currently driving that. Alrighty. Unless people have any other language API questions or, you know, sort of, nitty-gritty they want to discuss, I'd like to move on to a couple of looser roundtable discussions. So just one last call. Does anyone have any sort of day-to-day business they want to make sure they get done at this call today? Sounds like no, but we can tack something on to the end if you think of it. All right. So something I really would like to move and I would like to make a proposal soon is around some kind of new amount of structure to the Open Tracing project itself. I've been kind of calling this a language maintainer role, but to describe just sort of the issue that I see, I think there's just a sprawling amount of stuff involved in Open Tracing because it's just a huge project that spans all of the languages. And there's very few people who have broad expertise that is across all of these languages. So one way to divide up the work that makes a lot of sense is to sort of divide it up by language with the language API issues and PR requests around changes to that. And then sort of maintaining the ecosystem around the instrumentation for that language. Not necessarily doing all of the work, but at least sort of to kind of have a set of people who are kind of bottom-lining and making sure the work is getting done. And then there's a sort of set of maybe structural things around how our pull requests made templates for issues when someone makes an issue, have a template to start off with. If an issue or a pull request can't get resolved right away, what should we do with that? Should we create milestones? Should we close those issues, etc., etc.? There's just a certain amount of the process of moving through these issues and making these decisions that I think would be if there was a way for people to sign up to kind of bottom line those things or be officially part of it or a way to identify if you'd like to take on or take part in that work. I think that would help encourage people to kind of step up to the plate. And it would be great to start creating some working groups around those kinds of issues rather than just use this OTSC call to deal with some of those nitty gritty project details. So that's the kind of vague thing I feel like this project could use right now. And I'm curious, one, what other people think and two, have people had experience in the past with this kind of open source project structure and suggestions for other projects to look at, things they like, things they would hope we would avoid? So I'd just like to open the floor up to any comments on this subject. Okay, so one thing that I was thinking some days ago was mostly related to organizing each single project specification or the basic implementation so that issues are a good part of where we have discussions mostly. So we already talked about that, but I want to push that topic again because sometimes, okay, we have language maintainers, so let's say a set of people working around the Python API, let's say. The point is it will be great, of course, getting feedbacks from vendors, from whoever. I mean, and the point is sometimes we get discussions on Gitter and sometimes on the mailing list, it depends. My idea was what do you think if, okay, these language maintainers are also kind of moderators of GitHub issues where discussion, technical discussions happen instead of Gitter, let's say. So having the repository, the source of truth about the development of new features or feedbacks or stuff like that. Yeah, I certainly like the idea of making sure there's some official channel for where these discussions are happening or in general being able, for people in the community to be able to find the discussions. I think that's an issue. I don't know if GitHub issue, I mean, I think GitHub issues are a great, are a place to discuss it and a good place. They still, there's still this sort of, if you wander into an open source project that you're not already deeply involved with, I find the, if the projects of any meaningful size, the issues in PRs, because there's no way to sort of sort and order them manually, it's hard to tell what's big, what's small, what's moving fast, what's just sort of hanging around there. So I think one thing I would like to get a working group going on is like, how can we pave over those issues and kind of like the issues with the issues to sort of make sure when people land in open tracing, they can also find those discussions. So I guess it's, I would just add on to there. It's not just monitoring the discussions, but making sure people in the community are aware of them, having some channel for getting that information out there. I think we've discussed that before having an update section on the website or maybe just a regular blog post or something like that. Yeah, I agree. Cool. Ted, what kind of classes of issues do you see? Classes of issues. Because usually, like for example, on the specification, we have issues for quite a while having to do with let's say a new reference type or this debug ID. So yeah, they kind of linger and it's hard to see what we're going to do about them. But some other issues seem to be resolved fairly quickly. Yeah, I think some issues can get resolved quickly. Some other issues either something, they seem to come in a couple classes like this seems like a good idea, but we can't agree on a solution or maybe it's a good idea, but people aren't fully convinced yet. And in particular, there are I think some pull requests and issues where some people say, hey, this would be nice and other people say, yeah, I don't know about it. And then it just sort of trails off, right? And part of that is I feel like people don't know who has what authority or what the mechanism actually is to resolve debates like that or where they should go at that point. So just having at least, to my mind, some amount of responsibility placed on some people where you can at least say, hey, because this is Java, and let's say Carlos Alberto is a Java language maintainer, you can say like assign him to that issue and then it's his responsibility to keep poking the people to get them to agree or if they really can't agree to have some way of like then closing that issue or discussion down by saying like looks like we've gotten as far as we can get. This is like the final state where this landed. And now we're going to to put this in some place where people can find it if it comes up again. But we're not talking about it actively anymore. So let's get it out of the sort of open issue area. Okay, so a couple things I think we might want to introduce some taxonomy of labels in this case. I think we need to decide like if the issue reaches a point where no one is convinced or something or they're like, basically disagreement and we don't want to do we want to close it or do we want to just park it and label it sparked until further. So that's one thing. Another thing I think we should add co-owners files and then the third one is this is something we're doing a two bars part of our like my team OCRs. We have usually an OCR like respond to open source tickets within two days right like an SLA sort of we may want to make it last one week for for this one but at least there is some sort of like one of the owners should have a like the last response on the ticket or the ticket has to be reclassified into some other category. I think that sounds great. Yeah, you know from my perspective we're just kind of getting started with open tracing and there's really two things that I think cause a little bit of like I call it clean slate confusion. The first is sometimes it's difficult to tell where a specific language implementation is at compared to the spec and to do that right now I've had to kind of like dig through issues and make kind of assumptions that I'm not always sure are correct. And the second is I think sometimes you know for a vendor or for someone else if when they want to build on open tracing there could be some concern when you see a number of lingering issues because you know open tracing is evolving there's going to be bugs and language specific implementations but if it looks like there's like a critical one that would impact how our agent works which then bubbles down to our customers then there's some some concern on well how quickly our critical things address and I think is you guys are saying I think a lot of the ones that hang out there are things that aren't critical they're more like you know they're more broader discussions but it's not always obvious when you look at a list of issues that that's the case. Yeah for sure and I would also say Derek like I didn't put it on the list here but in general I would like us to form a working group or otherwise start to overhaul our documentation and website in general like I think we could just it's just time to to really improve that intro experience to people so they understand all this stuff it's all kind of there in the docs kind of but it's yeah I don't I think it could be just be better yeah and that's all normal normal things that at the stage yeah cool that's that's good and I think part part of that maybe is do you would you if there was a prominent location that explained like you know these are the maintainers for various languages or whatever or you know basically a directory of if you have questions here are the people who will respond to them I don't know what and maybe what how to contact them like what is the communication mechanism would that be a kind of thing you would use if you saw something like that would you do some reach out yeah yeah I mean I think in some cases like I'd probably see who just made the most commits to the repository maybe reach out but um yeah I think for just that clean slate like what's like the general state of this where is it at compared to the spec um yeah we'll we'll share as we get started because we're kind of yeah in that state we'll share some things that might help and contribute where we can awesome thanks for the feedback I kind of have some questions about uh how maybe the process of you know how how do you add people to these roles and then uh I guess potentially if you needed to how would you like remove someone from one of these roles uh personally I would like to see this kind of stuff be very open if someone just wants to raise their hand and say I'd like to help uh for them to be able to to to plug in and start start helping and so the maybe amount of responsibility any individual have could vary but the amount of total responsibility that the group per language has you know is is fixed and delineated um and so there's just some way to have a group of people per language who are maybe sorting sorting things out amongst themselves as to how it's working um but at the same time you can see it'd be easy for this to turn into a tragedy of the commons where uh you know there's a group of people and then no one picks up a thing or no one responds to a thing uh and I've wondered if people have had experience with with those issues on other projects and any advice on that front well typically you do need a list of maintain official maintainers right that's what co-owners kind of comes in place uh and uh they uh usually in all projects have the final say uh I mean so far we we kind of delegated this to the uh to the spec commission right uh but uh definitely um people on that list are not necessarily experts in all languages so uh I think co-owners are slightly different and that's the actual maintainers of the repo uh and like if someone wants to help there's like it's github no one's stopping you to actually jump in and um subscribe and this is basically all for opinions and then uh the uh the project governance should state how someone can become a a maintainer so we've recently added something like that to Yeager where you have a sort of like a voting process where uh maintainers can elect or delect other maintainers uh good idea and we may uh like rather than duplicating this in all the uh each each repos we probably should consolidate it into uh like the actual project governance consolidated in the specs repo uh just make sure that okay that it applies to individual languages differently uh in terms of like who are the people right because the the actual people will be in the co-owners in that repo yeah that makes sense um I almost wonder if we need a sort of uh so right now it I think if someone says hey I want to do this uh it would be great if we didn't need too much of a formal voting process someone who had sort of administrator privileges just can say yeah you know if you can convince anyone who has those privileges that you seem uh uh fine uh for doing that work then they could add you rather than having to wait for some you know OTSC meeting or something like that um but it makes me feel like right now we just have this sort of specification council and I wonder if we want to also maybe have just another administrator role which would basically be the same people uh I don't know maybe that's just getting pedantic um but as far as like who has the keys to all these various repositories and things if you need to like hey this person needs to get added as a you know they're a language maintainer so they need to get permissions to all these various repositories and stuff like that and can they become a language maintainer or not like maybe a list of like who are the people who have those permissions and can give them out can you hear me now yeah that's so frustrating this entire call I've been on my phone until now I heard everything I kept on talking like I called in three times that didn't work anyway I have a bunch of comments that from previous discussions I will keep myself or I'll tell you later etc whatever but in terms of this thing like I don't really want to create a bunch of structure that we don't need I think like the to me the most I mean I'm not I think we should do it like kind of on demand and it does seem to me like there's a problem for sure with certain languages where it's actually in the contrived area where I feel like it's most it's most prominent like there was a bunch of really good stuff that have been contributed to the Django and Flask stuff that is just like sitting there kind of dormant for months because I didn't notice it and you know oh well you know like that's a terrible situation like that that should never happen so to me it's like just having someone who's on the hook to just make sure that stuff like that doesn't happen it was pretty much as easy as like making one comment and then merging it wasn't a difficult PR and I just want to make sure that we have someone who's responsible for that for just keeping track of that and also someone who's as you someone said earlier like a decider for issues like we just I think there are a lot of situations they've been tracing where people involved in this project are usually pretty smart and reasonable which is great so it would be nice if someone just had decision-making authority even if the decision is to put something off but just like so it doesn't end up in the state that is in now where you have some issues that they're not going to be resolved we all know that but no one feels like they have the god-given authority to just say this is officially dormant or whatever whether that means we close it and put a little bit label on it so I'm more interested in just having an individual that has those sorts of those sorts of capabilities and I'm a little bit reticent to like create lots of structure around permissions models and stuff like that to me this isn't really a governance issue in terms of people being bad actors and keeping keeping bad actors under control it's more like an issue of making sure that we can make decisions quickly when any any one of us could make it if we were just granted the authority so that that's like my two cents and that's great I agree with a lot of that I can't remember who said it but I heard somewhere you should always aim for just a little bit less structure than you think you need and I think that's I would definitely want to avoid anything that involves voting or some like official process we should just think a little bit about like you know what are you going to put on the website around if you'd like to be take on this role you know how do you raise your hand and say I would like this and then who says okay fine you're on the list so just think a little bit about what that that process would look like like we're voting voting goddess in the United States like no one wants this yeah so so maybe it's just as simple as telling people to come into Gitter and say that you would like to do this and then we will just notice you and and and do something about that but or the OTSC can bottom line that for now and that's maybe some amount of tapping people on the shoulder and asking I think too it's not the kind of thing where I'd expect people to be volunteering even if they would like to do it but it's not like a huge responsibility in the terms of time commitment it just requires like someone to know that they're doing it yeah and for for the contrib repose I agree that's where probably the most like the biggest issue around like people making a poll request or an issue and then it like not getting responded to is probably much greater there than in the language repose just because of so many more contrib repose and there's this issue of well there's always going to be you know whoever is the sort of maintainer for that thing but you know people kind of come and go I think people might might love a big open source project that they're working on and like devote their life to it but if it's just some piece of instrumentation for some library that you're using at some point in your career you should kind of expect people who who write those things will then move on and the the maintainers of those instrumentations will have to like change over time and so that's sort of where I see things like you know the Django and Flask things for example where I think those were cases where the person who originally wrote that code had like moved on and wasn't necessarily paying attention to it anymore um so language maintainers being the people who kind of get those github alerts and are just paying attention to those things maybe it's just as simple as having an SLA for how long a PR an issue can sit anywhere before it gets assigned to someone for review that sounds like a good one I like Yuri's SLA idea a lot too it doesn't as long as it's written down anything would be good just it won't the SLA will not be six months you know like yeah for a person Django's SLA is basically if you are raising your hand to be added to code owners then you essentially taking on that SLA yeah and if those SLA's are easy for people to find when they make PRs or issues then that just lets them know when they have a question they can at least know like oh I was supposed to get someone assigned to this two days ago so I now feel empowered to like go like raise this to somebody and I can see on the list who I should ask like hey what's going on here I think that those are the kind of things that would make people who aren't part of the core community but are making a PR an issue just feel like they have a handle on what kind of next steps they're supposed to take sweet all right and I see it's now 9-11 and I feel like we've had a pretty good discussion on this there's also some dynamic linking I'd like to talk about does anyone else have anything they'd like to raise their discussion of language maintainer just one thing I actually be I'm not you know put you on the spot but it'd be great to hear from people specifically New Relic and and the the folks who have previously spent time at Avenida who are on this call and stuff to understand how just from organizational standpoint language maintenance word for the agents I don't think it's not the same thing when we're not writing agents here but it reminds me of that where you probably have some common data format and model and then you have teams doing this stuff I bet there's some similarities in terms of the way the stuff is structured and if there's some kind of like TLDR lessons learned like don't do it this way do it this way kind of thing about just how to structure the per language stuff in conjunction with some larger data model that that crosses between languages I would be curious to hear about it again that if you have if you feel like saying it now great if it's something that happens offline that's great too but I would love to hear the parallels between kind of APM agents per language and the situation we find ourselves in now the biggest one to me been with the especially with the contrib area I think that has just has a tendency to eventually kind of become a band and where because you solve a problem then you move on and then your organization probably stays on the same versions for a long time one thing that just might be really helpful for the contrib for example and the read me is like for python jango like which python version was this tested against which jango version was just tested against just because I think what can happen is someone might start trying to use those contrib modules and if they start having a pretty bad experience like oh it doesn't work for python three or oh shoot it doesn't work for this but it wasn't clear and then they had that same experience in other languages they can give open tracing kind of a bad name when it can just just be a clear like this is what it works against this is this is what we tested against yeah yeah um from New Relics perspective um this is very similar you're right there's a lot of parallels uh one of the things that I wrestle with um currently my job is to try to aim for consistency across all of our agents and one of the things that's the most helpful for me is just being able to track inconsistencies where they occur these legoges agents have been built over years there's lots of inconsistencies um but being able to explain them and just know the what happens under each situation and having that documented in a single place has been really helpful um to just start to reason about how to make them more consistent if that's the goal and I guess from the uh reporting from the formerly apnetic camp um so for us I think probably um you know consistency definitely a challenge and um I think one of the things that that I was really interested in when I first you know was kind of working on some of the open tracing docs is the semantic specification because when you start to get these I guess you know now they're in contrib but basically these kind of modules that instrument various items of functionality there's certain kind of guarantees that if you know that like a database instrumentation is always going to report these things or you know RPC HTTP you know client instrumentation is always going to report method and status like that um that really helps you kind of downstream so from a vendor perspective I think one of the things that helped us at least you know try to get uniformity is you basically have these internal spec documentations about like if you want something to show up as like an HTTP client then you'd better capture these things and then there's certain optional extensions of that but there's like a minimum bar uh so that's you know I think the kind of semantic specification might turn into a guideline for that in terms of organization we basically have like you know one direct code owner that is in charge of kind of the um internals implementation of it and so that's kind of vary from uh from runtime to runtime or language to language uh there might be other people who are contributing to that um and then there's there's you know it's all kind of obeying that broader uh kind of semantic spec and that's a little bit more like a almost like a wire protocol type view of things that open tracing doesn't really have the kind of luxury of but you know it's almost like if you just view it as the api is like the wire protocol but like one step higher then it's really really similar um and then I guess lastly to the version things that's definitely a huge problem um and so we have these kind of like matrix you know testing things where it's like run you know these versions in line with like these versions and basically like I guess with contrib style you'd almost want to pull those into your build and like run those on your matrix to figure out you know what's going to work on python 2 versus python 3 and so on but it's uh it is a challenge uh automation is probably probably the best way to keep track of it because stuff will go still fast I mean I guess there's there's other people working on this as well not just you know formerly apneta and uh and new relics so I'm always curious to hear how other people are organizing this too I think you've hit on all the major the major aspects of this though okay great that was I think a great uh set of um issues uh and targets for what this role should be that was raised I'll try to take this and synthesize it uh into a document and make a github issue out of it on the specification repo um in the next couple of days uh but yeah and if anyone else is interested in collaborating on this work of course see me the issue or you know contact me and get her all righty so we only have a little bit of time left about 10 minutes or so and I just wanted to first raise some awareness about another front opening up on the open for the open tracing project which is around dynamic linking uh so this is really kind of started with service meshes and proxies such as envoy uh in particular but also engine x um linker d and uh other pieces of software that essentially are what in the past I've called the network appliances these are pieces of software that generally you're not writing you're just deploying and you'd like to have the tracer that you use uh trace what's going on in these things because they're often in band uh with the requests going through your system uh kubernetes ingress another example of where this shows up and in all these cases you have a sort of way of getting that code distributed to you that is not that you are then mutating that code and compiling it yourself right you have a binary distribution a package distribution something of that nature and those projects don't want to bake in every single tracer ever written and be sort of in line to a sort of like a a downs they don't want to be part of however you're getting updates to that tracer and that other piece of technology uh if they bake all of that in uh to each project then they're one having to pick a winner and two they end up in their own version of like dependency help because whatever dependencies all these different tracer clients need they end up taking on all of those dependencies they may even be incompatible with each other and so what is looking more and more interesting on that front is some form of dynamic linking where uh perhaps there's some kind of uh set of some set of tracers that are baked into these things by default but if you roll envoy out for example you'd like to then uh dynamically link yager or light step or new relic or something like that into um into envoy at runtime and to have two distribution channels for how uh you're deploying these things and getting them onto your machine so that that's a thing that's that's coming up as we start to get more and more integrated with the uh sort of infrastructure layer that people are running uh running their programs on so my first question is does what I just said make sense to people uh or should I clarify it more and my second question is what are people's experiences with this kind of dynamic linking uh do people have suggestions uh for uh deployment models for this sort of thing in this new sort of docker image world that we live in is this a language specific problem problem though right so there's one part that's language specific which is the the three languages where we've kind of seen this come up so far are c plus plus go and java and they all have different kinds of runtime linking mechanisms so so that part is certainly language specific uh I don't think it will extend beyond those languages maybe rust but that's mostly what what people write infrastructure stuff in so it's going to be those three um the part that does seem a little cross language is there's some way there's some format for passing configuration to the tracer where you're going to write some kind of configuration thing uh in uh you know uh as part of your configuration for envoy or or link or d and then that has to get passed over to the tracer uh so that was one place where having some standardization that was maybe cross language would be nice as well yeah that's interesting to me I totally agree with that Ted about the um the static configuration of the tracers themselves is it's interesting because like in the specification we've struggled at times for instance with the idea of like it's I think most if not all like real tracer implementations have some tags that are passed to the tracer that then get applied to every span or something like that we don't have a place to like really reify that because the the way that tracer is configured isn't part of the specification it's just kind of a convention but we would have to I think well it would be beneficial if we could describe things like that in this um in the static configuration for tracers um I also think another use case that you didn't mention Ted but I think is sort of interesting there's a like I think there's a false dichotomy between open tracing approach and like a traditional agent that reaches in from the outside it is possible to write some kind of agent that um uses open tracing in fact I think there's some kind of fledgling project in the jvm world that does exactly that but configuring something like that would also be nice to um to be able to plug in an arbitrary tracer implementation dynamically and I can see I can see other applications of this for sure beyond just infrastructure software but they all need some way it would be great not to have different configuration for these tracers some language to language where I think I'd imagine that for almost everyone the the the parameter is for any given tracer it has more to do with where the data is being sent than what language is being written in something like that yeah yeah I agree so also the implementation is going to be specific to the to the like light steps tracer and jager will always have different parameters and that's not a bad thing like that's because they're different systems so you know that that's um it we won't be able to have a standard format so to speak no but it's more like you're gonna get a blob of json and then there's going to be like this key and then all of your tracer specific stuff will be off of that it seems like there's just a little bit of that um I linked by the way I linked to the uh the there's a pr for this in envoy that I linked to uh in the notes but I'll throw into uh the chat here just so people want to see a concrete example of what we're talking about here um that's that's one place where this is happening live which is also why I'd like us to address it as a group because it's kind of happening right now because people need it but it's happening sort of project by project so it's interesting I mean jager clients have always provided a sort of a configuration class which acts as a factory for the tracer we never recommend creating tracer manually and configuration typically either it can read environment variables or a config file um although like with the config file you run into like format issues um like most of our libraries sort of either punt on the format completely and let you do that or uh like in in in note it would expect like a dictionary or um javascript json file something like that so that that part of standardization is a bit difficult but that's like mostly limited to to the individual system and the the point about like having a sort of a common key and envoy configuration that seems like that's just an envoy concern like they can extract that key and pass what's under it to the tracer uh and but we would need uh I think at the open tracing level then to have this sort of the spi um interface saying that this is a general factory interface that can create a tracer yeah so so one example of where the standardization should be beyond just the project is so let's say you make a c++ tracer that could be dynamically linked and envoy offers dynamic linking to c++ tracers and then engine x open tracing is an open tracing module that also offers dynamic linking to c++ tracers and uh it would be great if you could just make one you know c++ tracer binary uh or tracer you know shared object file and it's going to work with engine x with envoy with anything else that's doing some form of open tracing you know uh dynamic linking so there's at least a bit you know that that's where I see the standardization you don't want people having to write a dynamically linked engine x plugin and a dynamically linked envoy plugin and so that's sort of what I would like to see uh get standardized and then the other question that's just sort of a bigger question is I'm not it's not totally clear to me what the best distribution mechanisms are for some of these things like if you want to add engine x ingress uh if you want to that has engine x open tracing baked into it now but it just has like zipkin on by default and if you wanted to add jager new relic to engine x ingress like what would be the actual mechanism for that that's more probably kubernetes to decide that but I'm wondering if people have any history with with how they they glue these things together at deployment time right now I'm just thinking packages and docker images I'm generally of the immutable camp they get in but anyways food for thought uh that's that's a sort of uh thing that's just starting to develop right now so again if that's the kind of thing you're interested in uh uh say hi and get her uh you can ping me uh ryan burn has been working on this stuff at the sort of c plus plus level and uh sir gay I believe was just starting to maybe look at uh a java version of this so we can make a pr against linker d uh so that linker d could support open tracing in java so there's some of this going on right now and I'll try yeah but but yeah for java we have a trace resolver I think so it's kind of solution for this yeah sweet yeah that's true but I also just wanted to make a quick plug before we finish the call that um I I'm going to try and like have a I'm not sure if it'll be this call or some other call but some scheduled thing in advance where we'll have people from various companies not vendors but companies using tracing doesn't have to be open tracing talking about their stuff um and just have that be kind of a rotating calendar like half an hour presentations it would be cool if we can get language maintainers then we can have language stuff in separate meetings when this meeting can be about that kind of high level stuff I think it'd be fun frankly and then it might also be useful um so I don't know that'll happen it might not be in the next call but certainly starting in March yes that would be nice um if the spec council could start talking about high level spec things again and not just the nitty gritty all right and I think that's the call uh uh for this week great to see you all and uh see you all on Gitter have a good have a good rest of your day next time happy new year right