 I'm Dan Dyla. I work for Dynatrace. I'm also a maintainer of the JavaScript SDK and a member of the governance committee. Hi, my name is Jack Berg. I'm employed by New Relic. I'm a maintainer on the Open Telemetry Java project and I focus a lot on metrics and logs, implementation stuff. Hi, I'm Aaron Clausen. I work for New Relic. I'm in the GOSIG. My brain just farted. No, I work for light stuff. That's why I wear their swag. So that's how this is going to go, huh? Yeah, GOSIG. And I've been working very hard recently on the metrics. So that's where my brain is right now, which is probably why I had such a flub already. That's okay. We would rather you focus on metrics. All right. So anyone here, if people are not ready to speak up or I get to ask some questions. So we will pick on you if you're not raising your hand. All right. I had most of our instrumentations contributed by outside people. That was actually how Amir got originally involved with the project was he was maintaining so many instrumentations that we were like, you should just be a maintainer. I'll go. One of the things that I saw with the double-edged sword of that community repo, the contrib repo is that it's also very difficult to maintain that, right? There are lots of contributors in the GOSIG that do contribute some really cool, interesting instrumentations or they want to. But we've had problems with them being maintained long term, right? Like making sure that security updates still get applied and that they are viable in the long term. So just being able to manage that has been both really awesome seeing the cool stuff that comes out of it, but also like everything that we accept creates more burden or more other toil. So that kind of leads into the other thing is just in general having the engineering hours across the board to maintain and to build on all of the cool projects that we're working with, right? So I'm very excited to see people here and see people using it and hopefully getting interest in contributing back to this project because there's a lot of really cool stuff that people are building and I'm surprised by it every day so. Yeah and one thing that the JavaScript SIG has done in the line of maintaining those the contrib instrumentations is sort of I guess what I would call distribution of ownership. So in the early days everything was just in the contrib repo it was on the maintainers and it was like five instrumentations, ten instrumentations and it was like okay we can maintain these and then it was 20 and then it was 30 and now I think it's like what 50 or something like that and it's like that's too much for four maintainers who are primarily focusing on the SDK anyways. So now what we have is individual contributors who own or maintain a single instrumentation in the contrib repo and the people that do those and are willing to say I'll maintain one, I'll maintain two, I'll maintain three like those people do not and if any of you are in the room thank you you do not know how much that helps me if you take even just one or two that's been huge. All right we got a question from the chat here for the panel how do you relate your work on a vendor neutral open source project to your employer or to the people that sign your paychecks. In other words if there's not someone above you that's already kind of enlightened to the value of contribution how have or would you advocate for making a certain percentage of your time be devoted to open telemetry. That's a tough one I think most of us here come from companies who are already enlightened I guess you would say to that but to those that aren't I would say the open source projects don't exist without contributors and everybody who works here I'm sure works at companies who depend on open source projects. So even if there isn't like direct obvious business value which for some there are but for some there may not be it's sort of the the rising tide raises all ships type of philosophy and I think that that can be really helpful. So I've worked at companies in the past that have you know that have had a lot of business value extracted from open source projects and they haven't really been maintainers on those and I've always thought that was kind of questionable so if your business has a strategic dependency on some sort of open source project I think you wouldn't feel comfortable if you had a proprietary software that you didn't have expertise in and have the ability to change in response to you know security vulnerabilities or other sorts of events so why why are companies more open to not having that understanding and the ability to change the open source projects that they have the same dependencies on so yeah I think it makes sense from a strategic standpoint to place engineers in a position where they they can they can make changes to those projects as needed. So I will echo what Dan said at the beginning is I'm currently working for a an outfit that really has embraced the open source but you don't always work for those companies and my one recommendation is start small it's a very big ask to say hey can I even get 10% of my time to dedicate towards helping and improving the open source projects but it's a lot easier to ask for hey I made a fit like this package had a problem that we we are experiencing I made a fix can I release that and go through the process of getting that and just training that muscle within the organizations so start as small as you can and work up if you can. I think like for me some of my time is like most of the time I'm working for a vendor for a specter but some of the time I just like change hat and then I just do the most professional work that I can regardless of where I work and I think like because I'm doing all these reviews and I'm so intimate with the code then when I do it I don't do it for the for my workplace but I just do the reviews and then I see how it pays back when I do my other work that's related to to my because because we use open telemetry in our distribution. I've got questions back here so hey y'all I'm Ariel from the Ruby SIG so just wondering from y'all what's your experience been working with trying to build instrumentations that first party support and libraries versus doing things like third party support and adding stuff like bytecode manipulation or in our case monkey patches really for a lot of these libraries and what's been the appetite of library maintainers for wanting to try to accept contributions and changes I think you were alluding to that a little bit just now. Also I'll kick it off so go is kind of in a weird position where it's very very difficult to add in support to something that doesn't isn't open to that right but there's also a wonderful ecosystem where people do think in layers right like in the net HTTP the built-in standard library it actually is built around that you're going to wrap this and and that's something that that happens quite frequently in that environment that being said there's a wide range of the support that we would have been able to engender some places getting first party support like at CD but then some just almost being almost to the point of like refusing to support kind of interoperability so I think one of the things you have to kind of gauge along with everything else is just how open your environment is to accepting those kind of kind of changes yeah I guess I work in JavaScript which is totally the opposite to go if somebody doesn't want their library instrumented then too bad we're going to do it anyway it's always easy to get it from the easy and air quotes you know relatively easy to get it from the outside but we've seen not too many that are willing to build in open telemetry directly exactly but they are sometimes willing to add hooks that we can then write a an outside instrumentation against that are stable and there's been some ongoing work in in the node ecosystem to improve that and make that sort of a standard thing which if you're using recent versions of node then then good for you unfortunately a lot of a lot of people run older versions where we can't always do that but the I think the the instability is probably the wrong word but the the the project has only recently gotten to a point where I think a lot of package maintainers feel like this is stable it's going to be around long term there are a lot of users and we're just reaching the point where I think we are going to have that critical mass of users where a library vendor looks at it and says wow a lot of my users do use open telemetry and maybe this would be really valuable for them so I think it's it's less about the technical aspects and more about convincing maintainers of of those other projects that the value is there and will continue to be there long term yeah in java we kind of we kind of run the gamut so there's the majority of our instrumentation is on the it's auto instrumentation so bytecode manipulation and as the javascript folks say we can just do it and so we don't we don't need extension points or anything built into the libraries we can modify the the code at runtime and you know a lesser extent is library instrumentation so libraries that provide specific extension points so that we wrap and then a very few number of libraries micro profile is one of them have you know embraced open telemetry and added direct instrumentation and I'm hopeful that as as the eight we have api stability around that the core signals that there's more of those that's that's kind of a future I look forward to in my head so we'll see where that goes though yeah and as a maintainer I think it'd be a lot easier if if somebody that maintains a library or a project is maintaining the instrumentation because they know it better than we do yeah I know this is Ted's baby so yeah yeah I just wanted to pretend to be a maintainer for a second and mentioned that we haven't really been pushing for library developers to add native instrumentation mainly because it hasn't felt like the project has been stable enough yet to do that not just the apis but also the semantic conventions I feel like we really want to get that stable before we start kind of like actively pushing that idea out there but it's really cool to see some people just dive on in and go for it yeah I mean I just tackle that a little bit we don't want to make a big push to say you should build this in and then you know move the cheese later and then when they say why did you do that you told me to do this and you said it would be fine and it's like well it was unstable yeah there yeah we want to only make promises that we can keep if we can handle that go find the envoy people and they have a lot of fun things to say about tracing open-source tracing we do have another question for the art did you want to go I just want to say yeah I haven't seen yet package that implements open telemetry directly I think there's going to be a lot of challenges around it because it will require people to upgrade the version of the package that they use to get telemetry or better telemetry like suddenly those two the features and the telemetry are coupled together which currently is not the case like they can just upgrade the open telemetry and get the latest instrumentation fixes without touching the version of the package the client that they use so I'm very curious to see how it will work and looking forward to it all right we have another question from the audience hi I'm too bad for Marvel technology we build at least my division we build data processing units and we're launching the open programmable infrastructure we the next foundation tomorrow and within that group we have been mostly official not announced yet adopting open telemetry for the chips so thank you I'm just a member but anyway the question is we're going to implement at least for us Marvel we're going to implement hotel collector agents on our chips probably dedicate an arm core to it this preliminary design mostly in my head right now my question is process pointers thoughts on how do we do that you know piggyback on the the answer just now about you know libraries and version packaging and all that among other things well one of the nice benefits of the collector is that it's kind of its own dedicated package one of the downsides is that it is something that is designed to run mainly on Linux it can run on other platforms so honestly my suggestion is just make sure that it has a processor Linux environment even if you're running a container a from scratch container that just has the binary it has the processor and the memory it's probably going to be a good experience but that also just means you also have to be able to integrate it with the rest of your platform there's a lot of unknowns there because I don't know how your platform is connected to everything else but you're probably like 90 percent there at least if you've got an arm and some memory an arm and a leg right well any metrics that we would recommend in extending so are you asking about like platform metrics or like so I know the collector has the contrib repo for anything particular for that I'm not I'm not a contrib maintainer so don't take my word as truth but that's probably where I would start but the collector is a process just like any other process running on a computer so define the metrics that tell you that this is good right like the system is running well and that that's going to be different for every platform and especially if you're going to be commingling those metrics with your platform metrics you have there's a journey there I can take you to a salesperson from probably a half dozen different companies to help you on that journey but it is a journey yeah it can be very individual one thing that I would say I've heard from Anthony Mirabella who is a collector maintainer and that really stuck with me is that he views the collector not as a like collector process but as a framework for building collectors so you have the the open telemetry collector repo which is all of the like infrastructure around the the collector itself and then the contrib repo which has all of the plugins and things like that and you can pick the individual plugins that make sense for you or contribute a custom one if none of the existing ones make sense for your environment and you can build a very targeted binary that can be much smaller where especially if you're code applying it at the platform level that is you know very important memory footprint CPU usage footprint things like that the more you can reduce the the ancillary stuff that you're not using out of the process the better as a side note to that if anyone here is in a regulated industry or has extremely stringent corporate security and yeah corporate security stuff we would love to talk to you about how to get the output of the collector builder a certified thing uh through supply and security stuff because I don't I'm editorializing sorry one challenge I've noticed in especially the enterprise is that they will prove a given specific version of a collector binary which is great but is completely identical antithetical to what you just said about the collector being a framework for building collectors so there's some work that needs to be done if anyone listening is interested in supply change security or any of those like Linux foundation projects then we should talk about how we can kind of get good binary or verified validated binaries out of that processor around that builder sorry yeah I'm sure it's not a challenge at a Linux foundation event to find people interested in supply chain security yeah I know I'm just going to go outside and I'm going to walk in the hall on screen supply chain security where's my supply chain security fanatics at all right I got another question from the chat what have been the biggest challenges with implementing the API and SDK across the supported languages in a standardized way how much does the SDK configuration feel native to your language oh man that's a big one um and obviously there have been a lot of challenges involved um the the biggest challenges that we have is making it feel native in any individual language because we could have you can go one of two ways you can say this is what the API looks like every language has to implement this exact uh you know method name with these arguments and all of that and it's the same in every language but then a ruby developer looks at it and says well this isn't how we typically name things it feels weird to me uh or a go developer looks at it and says this isn't how we normally name things it feels weird to me or how I normally structure my project or anything like that so uh if you read the specification which I'm I'm sure a lot of people here are familiar with the specification you'll notice it leaves a lot of leeway it defines the way that features should work and feel uh and the value that they should provide and how they should do that but it does also leave a lot of leeway for individual implementations uh to make it feel native for their users uh which I has been extremely helpful for me uh as you know a maintainer to say I'm looking for people who know javascript not necessarily people who are super intimately familiar with the way that other hotel SDKs or the specification work uh I will echo that um there is always going to be a compromise between being cross portable across like taking your knowledge from one language to another and the native way things would do uh python go any language has their own idiosyncrasies and and the go sig we do have compromises that have to be made to be able to meet what the api um documents ask of us um that being said it's it's a worthy tradeoff to have that knowledge also be portable so yeah for sure if you have uh an open telemetry a team in your organization using open telemetry and they're a python team and you're a javascript team and you're trying to build on their expertise the fact that the uh the apis are similar and feel similar means that that knowledge transfer within your organization to different teams can be much much easier although I would say that they don't feel entirely similar there's their ergonomics from language to language um yeah it's not entirely intuitive uh so I think of the specification is more of a a specification of functionality rather than like ergonomics and so you know you'll be able to find the same features in each each of the language but um yeah we we had some maintainers in java that you know just recently used to go SDK and api and they were kind of uh they were kind of surprised at how different it felt so um tradeoffs tradeoff all right got another question from the audience hello everyone uh my name is shubhanshu uh I'm part of adobe and I'm part of the observability team at adobe and we are trying to uh increase the adoption of open telemetry at adobe so we interact with a lot of engineering teams and so first of all I would like to thank the java community for building the auto instrumentation teams at adobe loves that that makes their tracing and instrumentation easy and on board to them uh so my question is for the other communities or other uh sigs in general like what are your thoughts on adding instrument auto instrumentation as part of the library and what were the key challenges you faced when you for specifically for java since that's ng and I know node gs is working on it uh on getting that rolled out well I'll just say before we get too far is that so we were supposed to have Trask here he couldn't make it and he's uh he's one of the maintainers for the open telemetry java instrumentation project which is where the auto instrumentation comes from and so uh you know I'm kind of on the other side of things in the in the java space I work a little bit with those folks but I work primarily on the the api and stk um so big shout out to the instrumentation folks on your behalf uh I know they faced a lot of challenges along the way to get the agent uh to where it is today and so yeah so for for the javascript sake I would say our primary focus has been on getting the uh you know the signals to a stable uh level within the stk uh because when they're not stable maintaining very advanced instrumentation can be difficult the api changes you then have to go change 50 60 packages which can be uh a challenge obviously we also have a lot of issues with different versions of nodes supporting different things and and maintaining backwards compatibility uh because the node uh the node community is not always on the latest versions I guess I would say um yeah I mean yeah it every every community has those problems um but a lot of our so we don't have the auto instrumentation the way that java does where you can just uh run a single command and it works but I know we actually have been uh working on that uh and I would say hopefully coming soon uh but until then if you can modify just the startup of your of your process you can actually get quite a bit um if you uh you know we have like uh if you're familiar with node there's like a dash r flag that's short for register which will load or short for require which will load a file before your application loads and it only takes like 10 lines in like a tracing j s file to do the setup work required uh in order to then when your application loads everything is automatically instrumented um so if you're interested in that you can come find me or come find us on the the slack and the the j s channel uh and we'd be very interested to hear about your experiences with that uh and I'm sure that many of the other language sigs would would have the same sort of sentiment I will say from a go perspective it's a very challenging problem because um just the way go is typically structured and the fact that we aren't allowed to patch out um out of library command so like I can't replace you know a database um the only thing I can do is do a do a wrap around that database so Joe has a go has a very particularly challenging environment for that kind of instrumentation um that being said I know there are some people that are just getting off the ground trying to work that in go um and it's it's monumental task for them um I also wonder if for go just the way we've built the the way the language is built if maybe just having easy ways to wrap um is enough because it's very common for go programs to start off with setting up a number of your dependencies uh right off the bat so uh the the the TLDR of that is um it's really hard and go uh it's not friendly for that so it might come hopefully I think auto instrumentation is uh super important like we're all very involved in open telemetry we know all the details and how to set up an SDK and configure it and register everything and how it all works but uh people that uh want to use open telemetry they usually it's just one task of other things they have to do they want to get started as quickly as possible as easily as possible they don't want to spend a few days learning all the details they just tell us give you something that uh is very easy preferably if I don't need to to change my code or create a PR or risk anything um it's a very good experience at least for the beginning so I think uh I really think we should try to to do it as best as we can very important I would also quickly mention I'm not super familiar with this so I won't speak about it for a long time but I know that the uh the operator project does do some like auto injection even in languages like JavaScript like where I said we don't necessarily have the no code modification setup but where the initial setup is easy to do if you're running in a kubernetes environment the the operator can do some of that injection for you and again I'm not super familiar with it but uh I know it's there it's node python and java right now um I would imagine dotnet will come once the dotnet agent gets more less alpha uh we have another question from the audience hello uh as far as infrastructure metrics are concerned um and the kind of recommended way to play within the infrastructure metrics ecosystem now that prometheus and the pole base system is kind of one the the dominant structure there what's the recommendation as far as um open telemetry metrics emission and collection from host space and points within the infrastructure is it recommended to always have the pole base method and then also how do metrics work work within the SDKs as far as that do we stick with prometheus and have that exposed there or um within open telemetry is it any different and how does open metrics fit into the metrics in open telemetry as well so uh just to heads up your story is going to be very different from everybody else's uh your infrastructure your application and everything are very personalized to your business model so this is not blanket recommendation for everybody everywhere um but my general thoughts are if you have something that's working keep using that unless it's not working right unless you have some extra feature that being said there is definitely an overlap between the push base metrics and the pole base metrics in what they are good at serving um and you can actually do both you're supposed to be able to do both with open telemetry um it's it can be difficult to set up but you can get it going at least in go it can be this it can start off difficult to set up we're still we're still working in that space but um uh the nice part one of the nice things that I think about the open telemetry platform in general is that collector is a tool that you can utilize to kind of aggregate both of those sets of data so if you already have an infrastructure that is Prometheus base or um open metric space which is sort of the evolution of Prometheus in a way the collector is there to be able to pull that as well as to receive data from your applications that you're doing in open telemetry metrics and that can all be aggregated and that's that's a wonderful tool um so if you're starting off fresh give open telemetry a try and see if it meets your needs if you already have something that's working give the collector a try and and use that as a tool to leverage what you already have so you don't have to rework it yeah I completely agree with that um I would uh I guess challenge the assertion that uh poll based metrics has won not because I think push base metrics won but because I don't think it's a question of of winning and losing they they they both make sense in different use cases and they can be used side by side uh one example I would give is a common deployment pattern we see is uh an application running on a host that has open telemetry built in that exports to a collector that's running on the same host that may be gathering host metrics and exporting those using you know whatever format otlp Prometheus anything like that and that collector may also be scraping Prometheus endpoints from different processes on the host which were already instrumented uh using Prometheus like he said if it's not broken there's no I'm not I'm not here to tell you that you should switch just because uh open telemetry is new and shiny like if you have something that's working then great but on the other hand if you have a problem or if you see a feature of open telemetry you're like wow I really wish that I could use that uh it's not like you have to switch all of your infrastructure it's it's it's I won't say easy to run side by side necessarily because like you said every environment's different but it can't be done and I would recommend giving that a shot yeah I just want to add that you they're reinforced that the collector is your friend here so um you know in kind of modern cloud environments you you have a variety of applications that emit data in a variety of protocols um the collector is is basically the solution to that it's it provides receivers that speak all the different protocols you you might be using and standardizes to an internal representation and provides tools to export to enrich the data and export it to wherever you want and so putting that in your environment really in a lot of ways allows you to regain control uh in in situations where um you know you you're using uh open source projects that are emitting in a variety of protocols and have limited configuration options around that so yeah and a lot of work has been done around and we're we're very proud of uh sort of the interoperability work that we've done uh with Prometheus uh I I think maybe I saw Richie Hartman here maybe I was hallucinating from the heat uh Richie's here he's been uh extremely helpful from that perspective as well from the Prometheus side so it's very much like a a community project uh we're gonna we have time for maybe two more questions maybe just this one um so we're gonna end at the beginning with a question from the chat I missed the intros and some of the start so ignore if this is duplicative it's actually not but I'd be curious as the panelists back stories around getting into open telemetry but not their full back stories so how did how did you all get into the wonderful world of open source observability yeah I I didn't plan to I started working at the Spectre and um we were originally we didn't use open telemetry and then one day we said okay open telemetry is here it's supposed to be great let's give it a shot and uh yeah they just rolled from there right it wasn't planned I just uh it just happened I think it wasn't planned it just happened is going to be a recurring theme here um I was uh you know I work at Dynatrace I was working uh on an internal team that was uh uh doing smoke and telemetry related work and as a part of that I was contributing to open telemetry and then I think uh as with a lot of open source communities you'll find that once you start contributing the the existing maintainer community is like yes more come in do more and it grows from there uh and you know you say yes to a few things and then suddenly you find yourself on the governance committee and you're like how did this happen uh so again I would say I I never really planned it it did just happen uh that said if uh you know if somebody said oh I have this this great opportunity over here and I would first question I would ask is can I continue my work on open telemetry because I probably wouldn't consider anything that would say no to that uh you know for me it's table stakes now for anything that I'm going to do in the future uh let's see uh so I worked in web services for a while um you know building out services in a uh service oriented architecture uh always kind of had a knack for observability and was interested in uh monitoring the systems that uh me and my team wrote and that brought me in to to New Relic uh so uh took a hobby and kind of went pro at it uh and then so New Relic I was I was kind of on the side always interested in in open source projects I tried to open source uh uh one of the main projects I worked on that was a proprietary project um my my company was having none of that uh so I was really excited at at New Relics interests in in open source and so they gave me an opportunity to go and spend a lot of time working on open telemetry which has been great so so mine's a little bit different um I found open telemetry because it solved about 90 percent of a problem that I had that was fairly unique um so that's where I started contributing uh but when my previous job was uh you know getting towards the end of a big project and I'm like I'm looking for something new I was I asked some of the maintainers at the time like hey is there any jobs that I could do open telemetry work full time and they're like hold on a second let me go talk to half a dozen people we'll get you something going really quick and uh that's where I found LightStep was able to uh interview there and and in working full time on a team dedicated towards the improvement of open telemetry so thank you um we didn't we didn't ask him to say that by the way that was all him that's that was everyone up here is hiring just assume that everybody is hiring I think yeah and if you're a hiring manager take note right people want to work in open source yes people love working at open source and open telemetry all right last question then again from the chat and this is a fun one to end on if you had all the contributor bandwidth in the world what would be the best use of that time right um how would that change maybe prioritization um I guess some of this is a scaling question right like if we suddenly like if everyone goes and tells ten of their friends and we get a thousand new contributors tomorrow like what is the best use of that time going to be where should people be focusing their efforts you know let's wrap it up with this one so down the line I'll start um reviews but like that's where we're lacking the most having good actual uh pr reviews both of documentation of what the code is doing um prs are great issues are great I love them they they save me time effort energy um but I know I spend most of my time doing reviews and if there was somebody who could if there was a team of people that could help speed that process along that could free myself up to to be doing more you know to be enabling more cool things going forward uh well a thousand new people is a lot I don't I don't know if I would know how to apply that many person hours um but so one of the things that I think about is or needs more attention is so I think the specification needs more opinions so um you know there's a lot of different parallel tracks going on of activity and um you know I think they move faster when there's people that are engaged and opinionated about uh the direction that things should go and so um you know logs and semantic conventions come to mind there so if you want to see those things stabilize engage with those those sigs bring an opinion to the table and I think they will accelerate as a result yeah I would I would say semantic convention stability is going to be uh huge for the project as a whole not just for maintainers if we wanted to see long-term adoption and use that's going to be something that absolutely needs to be taken care of and uh you know like you said a thousand people is a lot but with a thousand people you have uh you know call it a hundred or two hundred different areas of focus where some of those people may be experts in something and if you're an expert in uh you know databases then there's a semantic convention area for that if you're an expert in rpc there's a semantic convention area for that if you're an expert in uh host metrics there's a semantic convention area for that and all of these problems will need to be solved and if we're only working with uh the limited pool of people who is regular specification contributors uh they can only be deep experts in so many things so looking for outside contributors that bring expertise that we don't have into the project uh is huge I think there's a lot of creativity and good ideas out there and when someone is walking with something that he believes in and wants to work on it and wants to see that it's implemented at the best and like be something great then this is uh in my opinion this is like the best option so if a thousand people come so I just tell them pick the thing that you believe that you want to see that is implemented as best as possible and work on it and do what you do best all right and with that oh we have like 60 seconds why isn't open telemetry Ted you're banned Ted you can't come next year yeah I'm kicking what are you on kicking you off can we eluit is here right we we can vote you out of the gc all right so thank you that was all a joke for the virtual attendance we all have fun here all right so let's give it up for our maintainers this is a great panel thank you both people here and on the chat who ask questions now I would like to bring up Michael Haberman from Aspecto be talking about the unexplored world of open telemetry and service mesh