 Now we're recording Yeah Well, that's the first time we've messed that up. All right Moving on So I was curious Jonathan you mentioned unit tests and I was wondering if you've integrated Tracing into the development practice at all at Facebook because that I think people think about when they're they're writing their code And kind of integrating it into how they're testing and developing Or is it still just something that gets kind of layered in and it's thought more about as a production thing So Sometimes developers have to think about it. We end up having enough systems built on top of it that Effectively they get a lot of the benefits of tracing without realizing that tracing is happening and so What developers typically will see is like if they're developing some feature will see like some high level like Here's some aggregate breakdown of like your performance than where things are heading Without having to sort of think about sprinkling tracing in because we have enough of it at the base layers of the and or like, you know other profiling data pulled in like CPU sampling and Like on mobile apps, particularly like dynamic instrumentation of function methods. So It tends to be one of those things of most developers don't hit it Some developers when they're diagnosing will then have to go and say like the default instrumentation isn't giving me enough I need to go add in some custom instrumentation to figure it out But that and then at that point then they sort of are aware of the tracing libraries Regarding the unit tests so it I guess it We're all but so it's all built on top of the canopy paper and so can it be sort of have this notion of like a trace pattern and so effectively what we're trying to do is extend that out to say like measuring trace quality and measuring trace breakages by expressing patterns that say We expect an end to end trace to have these particular types these particular fields These are the fields we care about like if these go away the pattern fails and something has broken Which depends upon this? Ideally those patterns are then also used like directly for the metrics that you care about so you don't need to write them twice You can just simply say I want a pattern which is going to pull out this particular field from You know the end systems along the trace and then if that field isn't there that pattern will just fail and then we'll be like You know something has broken in this trace That does that's not integrated in the developer flow at all. So like we can still get into cases where like somebody commits a diff They break the traces that hits prod we realize that it's broken and then we're like, okay, let's go revert That's interesting. So it's almost a bit of formal verification almost though a very kind of simple version of that There's a seed stage company that I don't think is out of stealth or anything, but they're kind of doing this but with a Technology that can integrate and develop work flows so you can write assertions about about System properties that are only visible in the distributed trace and then they actually enforce those in band. So it's kind of like It's it's one of these things that Takes advantage of the context to do more than propagating trace and span IDs and they actually propagate a little bit of state all up if it's racing or something in order to To verify assertions down the staff so you can say for instance I'm writing a test up here and I want to verify the database layer that you know This field is equal to whatever and then they can assert that via this This like testing API and then and then run the test and like an integration environment before it gets committed It's pretty interesting. I think it if you have like really good hygiene around instrumentation It's a it's a my mind a pretty powerful thing to be able to do versus a normal integration test for everything Has to be programmed to emit things and you have to you know reach across a lot of boundaries We're just tests like rather simple and end behaviors So I like the idea that a lot although it does that leads pretty heavily on instrumentation in terms of the larger presentation time that you did which I thought was interesting the The couldn't go agent thing I don't like that works it means different things different people for sure But I have been thinking a lot about the fact that What you like Jonathan said that a lot of application developers aren't really aware of tracing It's certainly the same way at Google because it was built in a layer They don't know about if you know in some ways an agent is something that's there So you don't know about the tracing or something it doesn't necessarily mean that it's done through through interposition and black box In black box experimentation that can be done by instrumentation that's just linked and dynamically at you know on demand or just in time But I I would like to get to a place where We are for a common frameworks and common In you know per service application architecture you could get open tracing stood up without making any code changes I don't think that should be particularly difficult It doesn't necessarily require writing a bunch of crazy interposition code. It's the open tracing instrumentation already exists It's more a matter of discovering What's in the process knowing what's out there in the world and then doing the binding between those two things Dynamically, I don't know how the people have thought very hard about that But I think it would give some of the benefits of both worlds The explicit instrumentation has some real benefits to terms of maintainability. I think But it's a bummer to have to make code changes in order to get basic binding set up it in the in the main line Yeah, I'm curious on that front I see Pavel's on the call and I know Pavel you've done a lot of work on auto configuration for Java spring and I was wondering if you had any thoughts on on generalizing that auto configuration to just Open tracing Java in general and how feasible that might be just to avoid having to to Manually thread together all of the the packages Sorry, I wasn't listening last Yeah, I think it depends on the on the runtime You yeah, but the Java runtime in particular Well, yeah, but auto configuration is feature of the spring boot framework I'm not sure how something similar is feasible in plain Java No Because that seems like half of when people talk about agent it does seem like Ben said it's Part of it is just being able to take the existing open tracing Packages and get get them installed without having to write a lot of code It's not it's maybe a separate problem from then being able to dynamically insert call points or something like that You're not dynamically inserting the call points. You're simply Changing which packages are involved and It would be an interesting experiment to see what what would be feasible in Java on that front, but Perhaps another time. I think spring boot. Maybe I'm wrong, but I think spring boot of the configuration is just It uses bytecode manipulation behind the scenes Yeah, to exactly like load the classes and change the configuration So it's kind of similar to our open tracing Java agents So that's one approach on an unrelated note one higher level abstraction I think we could benefit from and maybe even consider baking into some of the API packages is we have these semantic descriptions of certain common patterns around RPC calls Database calls things of that nature, but when you want to go use them You have to sort of glue together a bunch of individual Tags or logs in order to to make that happen and one very simple straightforward form of sugar would be Something that allowed you to you know, that was a little bit more of a guide on what you had to put in there so rather than having to Refer to a spec sheet somewhere about What all the fields were you needed to add in order to record an HTTP call if there was just a function You could call that had all the fields required fields as a parameter there Just to make it clear and easier to people Rather than them having to do it themselves. So you would just say, you know Tag HTTP or something like that and then it would have five fields on it I've seen some people asking for something similar or more structured tags in general Not sure what people think about that Yeah, that's definitely something that we were doing We have a span API for go that does that I wrote that does that for Database queries and cash requests and external and incoming service calls and HTTP requests You know because we are looking for specific tags and it's really helpful to be able to formalize those for our users I think that is Is Something we're gonna end up continuing to provide in all the languages just and try to make it consistent Because right now we just have docs where we're like, here's all your tags. You better remember them So it's yeah, it's it's it's nice We have a we have a terrible API for custom SDK where you need to Propagate context between events in a span Which is a little harder than propagating spans and then and then you have to actually be very careful that you use the same tags in the start And end event for example in certain cases So we do try to simplify those we do duplicate certain Repeated values that have to show up in both events awesome Imagine a lot of that specific to your product, but is that that open if it's something you can take a look at Yeah, yeah, I mean there's basically an open tracing mapping from open tracings HTTP tags to our internal tags in my OT implementation for go for example And if there was something that came cooked up from the OT team where there's a several, you know standard Helper functions that people are using we would just map it to internal values that we index for ourselves Uh, yeah, it's just funny because every vendor is probably indexing each of the tags differently, right? And so like some people person's requirements are different like the number of required tags for a database query is probably vastly Different across the different vendors who would be doing this Yeah, this came up in the trace context working group Attempting to to find, you know What is the commonality if we're gonna have some common standard definition of an HTTP request and It does sound like a noble thing, but it was also instantly clear that everyone had kind of a Slightly different opinion about what was important and what was the correct way to slice it So it might be a little bit tricky to come up with a universal standard for that We are gonna take a shot at it All right. Well, it's it's nine ten So we should probably move on but I do want to use this opportunity to Promote trace identifiers. We're gonna make a PR hopefully next week to Java to add those So people who have a Java tracer keep that in mind that'll be a thing coming down the pipe and the hope is by Exposing getters for trace ID and span ID that'll make some of these middleware and other higher level abstractions Easier for people to write easier to integrate the tracing call sites into Secondary systems that they're already using so just heads up that that's coming up Okay, moving on to the next agenda item I want to make everyone aware that we're going to do a docket on Next so Wednesday after next June 13th What I mean by docket on is we we have some documentation on the website, but we feel that currently it's pretty insufficient It could be a lot more detailed in terms of describing all these concepts in general And then it would be great if we had on top of that sort of language specific documentation That described all of these contexts described all of these Scenarios that that you encounter in open tracing But in the context of the specific language because we think that would be really helpful for end users to have All of that detail at their fingertips when they came to the website at the same time We found the current Jekyll implementation of the website to be in Kind of difficult for people to use if you're starting from a clean Mac For example a MacBook it can take up to an hour and a half to install everything you need in order to make a change to the website so on that front Luke Perkins from CNCF has been kind enough to Create a Hugo version of the website That doesn't have that installation requirement because you can just get the Hugo binary and Start running with it and at the same time it works fairly similar to the current website where you're just writing markdown files so now that that's at the door we'd like to use it as an opportunity to Kind of overhaul the documentation on the website and because that's a little bit slow going We thought we would have a sort of hack day where we got together people who work on the various languages some copy editors and People who like writing documentation Hopefully some end users as well To sort of sit down as a group and just jam out as much of the docs as they can So we're doing some prep work there to get the new site set up with enough scaffolding And kind of bullet pointed versions of the different kinds of docs We think we need to see so people aren't operating from a blank page but what I would like to ask is if people are interested in this that They they sign up with sign up form So I'm gonna post that post that into the the chat right there and To find people that you work with in your company Who would be interested in? Contributing to the docs so that's people who either know how to write or people who know how open tracing works And if we can get a good group together, we're also going to try to get some documentation Experts from the floss community to kind of guide us on this and this is coming right up It'll be the week after next if we don't get it all done in the first session But we like this format. We may have a second one But put the word out, please So I'll be announcing this on Twitter and making a blog post and everything else, but I want to tell you all first and Though she's not on the call here I want to do a shout-out to Julie stickler from red hat who's been a total champ in getting this together Any questions on docket on before we move on to the final agenda item? Cool. All right So we've got Final agenda or there was a question. I think from Yuri about our playbook and SLA for Tickets and PRs. I agree. There's a lot of things. I've kind of gotten long in the tooth there But Yuri, do you want to do you want to talk about that? I'm not sure I want to talk about that. Just want to come up with some sort of a process for this Something I just am in the process of setting up for myself and could probably share Open tracing wide is Just some code to identify stale issues. I Decided stale issues for me mean anything that is is active that hasn't received a response from Someone directly associated with the project either someone from one of the language maintainer teams or another core member For seven days. So anything that's gone more than seven days without a response But isn't tagged in some way as being on ice Is stale and then I want to make sure that gets someone assigned to it in some attention So If I get something that automatically scrapes all of that I'll try to try to share it with you and others so that we can maybe track it a little bit better but Beyond that it would be great I feel like we have people who are interested in responding and managing PRs and issues But the process for for deciding who gets assigned to what I feel is still a little muddy Don't know if you have any thoughts on that Yuri and like how we could clean that at that part of the process up I Think in my view the problem is not like assigning someone but actually Come into consensus because there is certain like I think they do longest PRs. They're controversial and so we may We may need so like we used to do this on this call right to just go through some like Like important issues the VN opened So maybe we need some some time dedicated where there's like a group of people where we can just and at high speed discuss And make a decision on particular PRs and then just render it Yeah, sounds good. So we have All right, then you want to just gonna say yeah You That's probably better one one thing that we've done well that I've done other settings would be to have Some expectation around them. I don't know if you want to call an SLA or an expectation or whatever but there is an expectation that that The people who are responsible for certain repositories are supposed to respond to new comments on issues or PRs within a certain amount of time and then And then there could just be an understanding that if the discussion is getting kind of complicated that you're supposed to resolve that You know in a forum like this or something that happens more frequently than once a month or whatever It's I definitely think that get have this is so horrendous place to resolve like really Thorny issues that often take like 10 minutes in a meeting where you have all the participants and people can Can actually have a dialogue about it. So I like the idea of that but it seems to me that we We don't have And for certain repositories, we don't have that kind of There's no one who's responsible for just responding to new issues and PRs And it gets done sometimes but it's kind of that hawk like it almost seems like that would need to happen first And that you need to get to a sticking point before we would escalate or whatever you want to call it to Meeting like this. Does that make sense? Like I don't I don't know if other people feel that way but but some repositories I don't think are being watched that carefully necessarily That that's probably as much of an issue to me as as being able to discuss things in a forum like this Yeah, there is a part of that too I mean, I guess what I'm saying is we if we defer every discussion for Like an in-person meeting then it will become a scheduling problem and you know progress will get made But but if we don't use these sorts of meetings to discuss things that are difficult over GitHub Then they'll they'll get stalled because you just can't resolve certain issues very well asynchronously over get up so I Would just I think it would be great if there's a general practice within that core open tracing organization to Use GitHub to make sure that people are responsible for a particular language or a bus story Should respond within a certain amount of time to make sure things don't just kind of stall out By async channels, and then if things are getting difficult to resolve that that we quickly escalate to If it's just two people that are having no discussion They can have a meeting and then summarize and you know, and I'll be on get what the what the notes were but But it'd be great to avoid some of these threads that I've seen I'm sure I've also seen on GitHub open tracing or otherwise where you have some issue and there are hundreds of messages And then no one can ever come in and actually follow the discussion. It's it's not an effective experience Well, I have I have two concrete suggestions. I can make on that front one is We have a cross language working group meeting every week. It's usually at this time, but we can start varying the time up and That's mostly been focused on rolling out API changes that have already been approved and making sure they go in You know evenly across the different languages But because that kind of work is calming down with scopes getting completed and basically all the languages We could be using that time to Resolving some of these issues I think the only thing we want to make sure we do there is if we're going to discuss one of these issues on one of These calls that we promoted ahead of time So just to make sure that the people actually care about that Are actually going to show up to the call because it doesn't make a lot of sense to have a call and make a decision And then have someone who cares be like I object and they weren't there So as long as we promote it though, that is a weekly forum for having this kind of discussion The other thing I'll say when there's a log jam in the past, especially around API changes something That I found is it usually means the English language has ceased to be specific enough and a lot of these sort of long-tailed discussions are About issues that maybe they're important Maybe they aren't people are talking past each other and when we switched from English to Code and tests that's done wonders for kind of clearing up a lot of this We really saw this with scopes where it just seemed impossible to come to conclusions about what was A serious issue or whether this API was going to work in this scenario or that scenario and just switching to code examples and tests Really broke the log jam and all of that So that would be my other suggestion when things seem to be getting too long Just ask people to to actually start writing code to explain their point rather than continue to debate it in English alright, I think that covers that and we're out of agenda and We're almost done with the meeting Does anyone have any final questions or comments or things they like to discuss before we go away? Great. Well looking forward to seeing some of you at monotorama next week and for the rest of you Hopefully you show up for docket on I am gonna keep keep poking everyone on that front. So Give me a shout out if you're interested in organizing an in-person meetup We're gonna have one in San Francisco another one in Boston and another one in Munich So I'm in Portland wants to organize one or elsewhere. Let me know and Yeah, I'll see you all on the internet. Is there a chasing meetup at monotrama again this year You know I asked on I asked on slack if people wanted to do that and Haven't gotten much response. I'm happy to host it I think the way we'd probably want to do it this year is just go to a coffee shop or something nearby if people want Want to do it But if you're interested, yeah, please log into the monotrama slack and and make a post there I Feel like we sort of got away by just bringing a bunch of food in and taking over some space at the venue last year, but You know, I don't know if we'll get away with it I mean, I think that I Montrama people were fine with it But the venue and positive as they found out that we were doing that would have been really upset like I Learned afterwards that that was really bad And they just didn't notice so I feel a little bit like I don't know It doesn't seem super super duper moral or ethical or something to knowingly and last time it ignorantly did that But this time it'd be knowingly suggesting we do that. I actually can't be there for family reason, but Maybe Erica from the New Relic people if there's some space in Portland that's Like close to that venue that's available that could host it But I checked last year and like the public spaces are not that close or not that good for something So I think that's why I had to stick me a coffee shop might make more sense than anything else Sure, New Relic would be happy if you're interested again. We can explore it I didn't I'm not sure what the concerns were last year, but happy to talk about it more Oh Yeah I can tell that stuff mostly gets organized through slack. I'm on a trauma, but I'm totally happy to have one again It'd be lovely to see you all and talk shop on tracing. So even if just a couple of us. Yeah, let's grab breakfast on Let's let's just call it Tuesday Tuesday or Wednesday. Let's aim for Tuesday and See if that works Great Yeah