 Great. All right. Thanks for that great introduction So today I'm here to talk to you about distributed tracing and the struggle that some of you may be aware of and I guess all of you are going to experience at some point real quick Who remembers what they were doing in March of 2014? Most people don't okay one person. All right, doesn't matter I'm not gonna walk through the whole room, but in any case for me March of 2014 was when I started working on monitoring and observability I started working at a company that I'm sure a lot of you know that was working on commoditizing APM for the masses And since then I've had the fortune of working for a bunch of vendors that you love and some vendors you probably less love All focused on sort of that goal of providing a better developer experience particularly around incidents as well as monitoring and observability So, you know closing in on nine years later Here I am talking about distributed tracing which there is this great white hope of Cloud native observability in particular. So time for an awkward segue quite literally So for those of you who aren't aware The segue was revealed to the world around the time that a bunch of us were still worried about why 2k there's a lot of hype and fanfare and Really it was meant to change the world if you believed every bit of hype and media that you saw out there You know things like the way that we move the way that our Towns and cities were going to be built was all going to be affected by this brand new product and technology Lots of flashy features Steve Jobs himself even said that it was going to be just as important as the personal computer Which is a pretty high praise from from someone who had a pretty big impact on I'm sure all of us here But as you may have guessed from the tombstone the segue ended up Wrapping up this manufacturing in 2020 due to really a lack of adoption and a lack of impact particularly on all the things that was hyped up about and infamously The owner of segue unfortunately literally rode one of these off a cliff in 2010 to his unfortunate early demise So chances are if you're here, you've probably seen or heard some issues around the use of distributed tracing And much in sort of the same way that the segue is your was really definitely didn't live up to its hype There's definitely a potential for distributed tracing to not have the impact that we all hope that it's going to have particularly if we keep treating it the way that we have up until now as We've gone from monoliths and VMs to microservices and Containers our engineering teams have really struggled with the amount of data the complexity of the observability problems that they're dealing with and You know distributed tracing is sort of generally seen as this. Okay, cool for distributed environments We've got distributed tracing makes sense It's all in the name really just need to load in some some tracing instrumentation Load up a tracing back end and you know throw as many complex analytical features of the problem as possible But despite all of these huge investments both the open source community Bartek covered some of those earlier as well as on the vendor side distributed tracing really runs a risk of passing into a relevance and really Being used as a cautionary anecdote and some future talk by someone probably not me Much like I'm using the segue today in these particular slides So I want to talk a little bit about the kinds of problems that our engineering teams are facing In using distributed tracing, but importantly we really need to look at the organizational problems that we're seeing not just the technical problems Not just the hey, what can I build? You know a new crew engine? What can you build a new dashboarding widget to try and solve? Because that's really the the heart of what we need to be able to look at when we look at building and operating developer tools So The story that I've heard time and time again When talking to many organizations large and small about the use of distributed tracing Is that near any team who goes and implements this stuff really runs into a bunch of problems that really center around adoption? The adoption never really hits that critical mass and it starts to fall off rapidly after day one, right? It's the most valuable it ever gets the most Engaged it ever gets in day one It starts to fall off and a lot of it tends to focus on for key problems for key themes that I hear continually The tools are really only useful for power users You know the smartest the brightest the the most long-tenured engineers are the ones who can operate these tools They know the technology they know the data set they know the architecture and they can make use of it But oftentimes at the bottleneck for the rest of their organization And the part of the thing is that it's not easy to go into this oftentimes tracing tooling sort of slapped on on top Of everything else is happening. It's not purposeful in the way that we're using it I've heard continually that engineers really lack trust in what's going on with tracing Oftentimes because of really heavy sampling because of the cost because of the strain on the system because of storage and what and so on and ultimately because you're so Required to rely on what everyone else is doing with a tracing instrumentation There's no real collective or singular ownership around the instrumentation and data collection So you run into issues where you really need data from someone else and they haven't done their instrumentation and vice versa and this ends up with this sort of sorry state of affairs where Tracing for those who have invested heavily and have utilized it for a period of time really see it as a high-promise high effort But sort of low outcome low value Sort of story that they've they've spent this time and effort on so How do we move away from sort of that potential future of the lonely little segue in the desert lost And how do we think about really taking a step back and moving away from the sort of features and functions and all that sort of like Technical focus that we've really applied to distributed tracing up until now and really think about the outcomes and think about well How do we be purposeful about this? There's no real silver bullets. I'm not like up here to say hey You know pernisfree or anything else is about to announce some new magic feature There's really making sure that we start thinking about this properly particularly is a mass Number of organizations are going to start adopting distributed tracing So let's think about maybe the the number one thing that most people think about when they think about the use of tracing is sort of instance and responding to all that's so You know, there's gonna be lots of different variations of this diagram But in terms of what I hear continually from from organizations How are they using it today as I said a lot of the times distributed tracing sort of gets slapped on existing processes But particularly as you've moved into more complex architectures more microservices more containers more cloud native I've got all these alerts going off. Everyone has alerts going off But it sort of leads into this your big question puddle of like, okay, where do we look who's responsible? How many teams do we go get involved? Maybe you look at logging first. Maybe you look at your legacy APM first. Maybe you look at metrics dashboards first But almost no one looks at distributed tracing first No, I'm not necessarily saying that you should but oftentimes distributed tracing is used by that, you know One your principal engineer at the back of the room going wait. I have a theory. I bet I could get Yeager to work through this So and it leads to the outcomes you see at the bottom reduced velocity high frustration poor customer experiences All those kinds of bad things that ultimately make us feel bad and really just make us question. Why are we doing this? So if you take another step back and think less about the the instant workflow and more a little bit about well What should it actually do for us? I think about well, we had all of this sort of microservices complexity. We deliberately added this why we did this because engineers should be able to go and move And operate independently of each other sort of think about your individual team think about your your coordination Only one tier away from you But you've introduced all this additional complexity and now you really can't handle it so If we think about distributed tracing as the answer to like well, can we scale back that complexity? Can we really make it so that a human being can think about the where of what's going on and then start applying all the other tools that? We've used before it's not about distributed tracing replacing everything, but it's making The the haystack that we've got to dig into a little smaller Then we can have sort of a much more focused use of it and we can think about okay We can be purposeful as how we use it in those incidents and other use cases So if we think about a Slightly nicer diagram, you've got your alerts those alerts They can lead into dashboards let you know about things like your symptoms justify your investigation Your symptoms and your impact you can use distributed tracing to figure out where So you can go and narrow that scope the the size of that haystack You can better use your logging your APM your metrics whatever it is events exceptions all those wonderful things that we want to use But we also need to use and feed it back into things So for example everyone focuses on SLOs tracing is a great data set to be able to improve our SLOs It's a great data set to be able to help illustrate what symptoms we're having and the impact that we're having on our customers And so being able to sort of tie this together and make it a loop means that you don't have that Oh distributed tracing is only as valuable as it was on day one type situation So ultimately really quickly What we want to do is drive the adoption and realize the value of distributed tracing But to do so we need to be purposeful about what we're doing We can't just you know throw up a distributed tracing back end and have one engineer Who's really good at fiddling around with an advanced career language. We need to integrate in workflows We need to focus on showing where problems exist because that's Allowing us to negate the complexity we introduced purposefully into this situation and make sure it's useful for every engineer It was really quick But just wanted to highlight that chronosphere here is that booth g15 as well as out there in the corridor We've got one bunch of little enticements for you to come talk to us And obviously if you disagree with any of this feel free to come hit me up. Thanks You