 Hello, everyone. Welcome to Cloud Native Live, where we dive into the code behind Cloud Native. I am Annie Telesto, and I'm a CNCF ambassador, and I will be your host tonight. So, every week, we bring a new set of presenters to show you how to work with Cloud Native Technologies. They will build things, they will break things, and they will answer all of your questions. So, you can join us every Wednesday to watch live. So, this week, we have Jessica here with us to talk about open telemetry. And as always, this is an official live stream of the CNCF, and as such, it is subject to the CNCF Code of Conduct. So, please do not add anything to the chat or questions that would be in violation of that Code of Conduct. Basically, please be respectful of all of your fellow participants as well as presenters. So, with that done, I'll hand it over to Jessica to kick off today's presentation. Thanks. Yes. Hi, I'm Jessica Tron. I am currently in developer relations at Honeycomb, but I'm a software developer of 20 years, and development is what I care about. And today, today I have a program that I want to change. It's just this script that I wrote like a Christmas time last year for this cute little Christmas activity, and I want to make it more general. So now I want to kind of mature it into something more generally useful than just exactly one thing. And in order to do that, I need to like refactor it and make it more configurable, but it doesn't have any tests. I wrote it for fun. I didn't write tests. And so, it's really hard to refactor things without tests because you never know when you break something. And yeah, you could like make a small change and then run it again and see if the output's okay. But that's so tedious. Also, it's hard to write tests for because it's like a visual thing. It invokes randomness. It uses stuff relative to the current time. So, we'd have to like pull all those things out of the functions that are like reading the file system and reading the clock and calling ran. I'd have to put all that out to make it have deterministic tests at all. And that would be a big refactoring. But I can't refactor because I don't have tests. So, one way out of this loop that I think is going to be really useful is to use tracing instead of tests as the feedback loop of what is my refactoring doing. This is fun in a couple ways. One, it's going to be useful in production. Well, such as production is with a script that you run on your computer. But in other code, if you practice this, those traces will still be useful in production. They'll be useful every time I work on this program in the future. And the other reason is it actually makes something change with my refactoring. So, my goal today is to add some tracing and see something that I want to refactor and then refactor it and see the results in the traces. Even though it's not externally visible, the program is just still working, the traces are supposed to change as the internals of our program change, sometimes, as we choose. And so, that's going to be the feedback loop. Okay, Annie, can I share my screen? What do I need to do? Oh, there it is. Okay, okay, right, so here's my little main.ts. This program already has open telemetry configured for reasons that will become apparent, but it doesn't have tracing around its functionality yet. So, when in doubt, always do the thing and then see what it doesn't do and then make a change. So, I can run it in local mode where it's sending telemetry to a local collector and a Jaeger instance. And then, I can go over to Jaeger and what I'm looking for, Jaeger UI, find traces, is a span with a top-level main. It doesn't have a main, there's only this one trace. And if I click on this, you're going to see what this program actually does. So, what this program actually does is output a trace with pictures in it. Let me shrink this a little. And you can see, can you tell that that's supposed to be, it's actually supposed to be a Christmas ornament. And this is supposed to be a Christmas tree. And there's another ornament, another ornament, or whatever kind of ball-like thing you want that to be. Smaller tree, ornament, bigger, right. So, it draws pictures in the traces. It does some other stuff. And at the very end, hopefully I'll have time to pop over to Honeycomb and show you the entertaining things that this program does with open telemetry spans that aren't visible in Jaeger. But for now, it's enough to see in here that it's still working. It's still making trees and baubles inside the trace. See this little line that's a span event that's supposed to be a sparkle. Right. But we're not getting traces from the program describing its own execution. And that's what I want to start doing before I refactor. Yeah. And we're getting a few comments from the audience with the screens looking a bit blurred. Just to let you know, the audience is what we try to troubleshoot it and make it less blurred. We could not get it fully working as we would have liked to, but it is the best that we could do at the moment. So I hope it's working for everyone, but if there's something that you would like for us to zoom in, for example, more or something, let us know. But yeah, let's hope it works. We can also do this for you. Branch name, add tracing live. So if you want, let's see. If you go to, I'll just open it on here. If you want the code, it's available at honey. GitHub.com slash honey.com.io slash happy Olly days where Olly is 011Y. And it's the add tracing live branch. That's where I'm working from right now. If you want to go look at it yourself. And someday I'll have to figure out how to get restreamed to show my screen differently. But that could be, that could be ours. I did make the resolution a lot bigger or yeah. Yeah, I worked really well. The differences is better than the first time. Okay, cool. Right. So this one, this is a batch script. You run it from the command line, like clear it's dot slash run local. It's the wrapper and it prints some stuff and it makes a cow. Come here, cow. There's the cow. And then it makes that trace that we saw in Yeager with the pictures in it. That's its game. Because it's a script and not like a long running. It's a program like a web application. I have to explicitly start the SDK and wait for that to complete before I do any operation that could create a trace. Otherwise you get a trace ID full of zeros and you don't get a trace. And then after that, I have to explicitly call SDK shutdown before the program exits or it just won't send all the traces at the end. So that's something to know about tracing in a batch job or script. Something that's not a web app that runs forever. So now to add a trace at all, I'm going to wrap this main call in a trace and let's see, do I have a tracer yet? I don't have a tracer. So let's get one const tracer equals hotel dot trace dot get tracer. We'll name it main.js because that's the file we're in. Now I can say tracer dot start active span. You have to say start. I'm in JavaScript. I'm in Node. You have to say start active span, not just start span. That's important because if you want your child, your span that you create to ever have children, it has to become the active span. And the problem with that is that in order to have an active span, you have to put all the code, the unit of work that's going to be inside that span in a callback, bummer callbacks. And the callback receives the span as an argument. So we can call from there, we can get main. Main is an asynchronous function. It's returning a promise. So after main completes, then I can say s dot end. You have to end the span yourself. And today you're going to see the travails of adding tracing, parts of it are easy, parts of it are hard. Mostly if you know how it's working, then you're fine. That's kind of asking a lot too. Oh, and I'm going to be careful here. And since this is going to be the root span of my program, I want to be sure that it's transmitted, even if an exception happens inside the program. So here's my unfulfilled function. Receive nothing and the span. I'm also going to pass an unrejected function, which is going to get an error. And then I'm going to record the exception on the span and end it. Okay. So now instead of just calling main, I'm calling main inside an active span. And I end that span. It doesn't get sent until you call end. So it's really important to call end in order to get a span. All right. Let's run this. Unlocal. And this time I expect to go to Yeager and see a whole new trace. Oops. Okay. I passed the wrong thing to record exception. It's not telling you that in here, whatever. Okay. Record exception. Okay. That makes sense. Thank you, TypeScript. Okay. This time it ran the same. Go over to Yeager and see if we got a new trace. Find traces. All right. We have one with 552 spans. That's going to be like the one that it created with pictures in it, but we also have a trace with a single span and its name is main. And that's what I wanted. Okay. There it is. It's a main span. It's not very exciting. But it's a starting point. Okay. The next thing to do is what auto instrumentation can I add? Because this is no, there's got to be some automatic instrumentation that I can add to find out, kind of spy on what this program is doing. And that's going to tell me something about the program I'm modifying, as well as giving me a starting point for this bands that I can use to describe my refactoring. Okay. I need to know what libraries this project uses. I guess I could look in the package.json. I could also grep or import in source slash start up to yes. But I want to exclude local imports. Grab dash v. So I'm excluding quote dot. Okay. Clear. Do that again. No. Grab again. There we go. All right. At the top here, and it's not super clear. Can make it even bigger. Don't know if that helps. At the top here, it says import Axios from Axios. All right. So Axios is an HTTP client. I'm using that to call out with HTTP calls. That's something I'd want to track. Okay. PNGJS, that's obscure. That's not going to have auto instrumentation. And the file system, that looks useful. And then I have open telemetry. And then I have file system again. Open telemetry, open telemetry, open telemetry. And the word important. Okay. So I want auto instrumentation for Axios and file system. And when I want auto instrumentation, I always go to open telemetry.io slash registry. Registry. And then I scroll down until I find the search box. And then I type Axios. No matches found. Dun, dun, dun. What do we have for node? So if I look at JavaScript and type instrumentation. And I look at that. Okay. I can get the whole list of JavaScript implementation. And I did this yesterday. And I looked and looked for something that would be relevant because I'm like, well, if Axios is an instrument, so I could use a different library that is. And here I found HTTP instrumentation. And HTTP is like the built in library that does network calls. And Axios is going to use that under the hood. So this is actually going to give me what I want. So I click on that and I get these experimental packages. And it tells me this stuff. And I know what to do now. So I'm going to install that auto instrumentation. And then great. I'm going to add it to my tracing configuration. Here's my tracing configuration. I have set the logging to tell me if something's wrong. And I have the SDK set up with a batch processor where I've like upped the limits on everything because generally the libraries try not to interfere with the functioning of your program. But in this case, open telemetry is the functioning of this program. So I'm like, please, yes, use my memory. But then I don't have any instrumentations. So let's add those instrumentations, array, new HTTP. Did I import it yet? I did not import HTTP instrumentation from instrumentation HTTP. And now it will like me. And we will instantiate a new one. Those pass it into the SDK. And let's see if anything's different. All right. Run local again. Send to the local Yeager thinking about it. Let's take it a few seconds. You know what? I bet this tracing might even tell us how to make it faster. Yeager, hit back. Find traces again to get the most recent ones. Aha. This one, zoom in again. This one has four spans. So when I click on that, aha. So here's my main span. And here's an HTTP get and another one and another one. So I can see that we're making three network calls in this app. If I click into these and click into the tags and dig and dig and dig, I can find that the target here is the off end point at Honeycomb. So it's this program calls into Honeycomb to be like, hey, who is this person? Where am I going to put this data? And then the next one is going to a data sets end point for Happy Ollie Days. And the third one is going to, aha, off again. So the first one and third one here are duplicates. And I could definitely shave 270 milliseconds off the execution of this program by saving the output of the first get call here. So that's one thing to learn. That's cool. Let's try adding a file system. Instrumentation. So for the file system instrumentation, I go back to the registry and look for the file system. Ah, fs instrumentation fs is built in node library for file system. And this should tell me how to use it. Yes, it says require instrumentation fs and new fs instrumentation. So let's go through that game again. Open telemetry instrumentation fs. Installs that. And then we go to tracing.ts and this is something. This one gives me a lot of value for a little bit of typing. Instrumentation. And then we'll add it to the array here and we'll have two auto instrumentations. And that's going to do a lot. Two auto instrumentations. Run it again. And then we'll go look at Yeager and see what files this this is opening. Yeager back. Fine traces. And there's a big one and a small one. I want the small one. No, I don't want the small one. I want it to be bigger than that. One span, four spans. I don't have it yet. Maybe if I hit find traces again. Oh wait, here's one of 51 spans. Let's try that one. Oh, okay. So here's our main and look at all this at this real path sync, whatever that means stat sync, read file sync, blah, blah, blah, read file sync, read file sync, read file sync. Okay. So we're doing a whole lot of synchronous file reads. What file are we reading? I click on one of those spans and I click into the tags. It tells me nothing. It tells me nothing. This auto instrumentation is unsatisfactory. It doesn't tell me what file it's reading. It's got an internal span format, a library name, a library version, a span dot kind, and then all the process level stuff, which is the same for every span in this trace. So that's a bummer. If I do want to find out what files it's reading, and I also find it interesting how scattered the file reads are. For my purposes of making this configurable, I'm probably going to want to move those closer to the beginning of the operation of the program. Okay. So I can't really modify the auto instrumentation. That would be hard. But I can put spans around the read file syncs that I'm doing. I go over to VS code and I look for read file sync. I find, not marked down. That doesn't matter. Okay. I find image.ts and song.ts. Okay. And then we also have a question. Oh, yeah. So they ask, do you guys have any representatives for doing the same on our end? Yeah. If you want to look at the code for this example, it's at, I have that open over here. Nope. That's open telemetry. That's open telemetry. Honeycomb IO slash happy holidays is currently where this code is. In general, I'm using a zillion open telemetry projects. So this is all a bunch of open source stuff that I'm using. And if you want this particular example of using it, that's available on GitHub too. And Carlo, I'm sorry. I can't make restream work better for the screen clarity. Someday I'll try. But that's out of like 10 minutes. Right. I'm on a Mac pro. This is not a weird computer. Anyway. So there's two places that this read file sync occurs. Do you all want to do the easy one or the hard one? Do you want to like make progress quickly? Or do you want to see, um, see adding tracing the hard? I will take the first response. Let's see what you can say. My gut feeling would be the hard way. That's always more entertaining way. But let's see. Does anyone else agree with me or not? A little bit. Yeah. So thank you a lot for the links that you provided. So that's helping. Okay. Well, if nobody has an opinion, or maybe there's just a bit of delay, then I, oh, oh, okay. Rob G says the hard way. All right. The hard way it is. We'll start with the hard way. So here in song.ts, like maybe, maybe it's this one. We probably are going to need to instrument both of them. But okay. Here we go. We have our read file sync. So what I want to do is wrap the body of the function in a start active span. To do that, I need a tracer. I have a shortcut for that. Get tracer. So we will get tracer named song.ts because that's the file name. And that's my personal preference. I need to get this file name onto a spam. Okay. So normally what I do is I use my spam shortcut. And I'm like return tracer dot start active span. It's immediately angry because this is a constructor. So I can't return anything. Oh, okay. Well, that's fine. I'll just wrap. Oops. Name the span. Just a span song. I'll just wrap here. I need to put inside this function before the span dot end, the body of the constructor in this case. So I'll just move that up. Hit save for the formatting. And type script is angry. Property versus has no initializer and is definitely not assigned to the constructor. Well, we have the bane of every JavaScript program, which is the word this. And this is not in fact this once I put it inside this callback. So frustrating. I fought with this for a while earlier and gave up and did the easy one and it turned out to be useful. But I think in general what I need to do here is actually put this stuff in utility function. Function read. File. I'll name. And then we'll just paste this stuff in here, but instead of sticking in this, we'll return it. Okay. And then. If I call read file. Read file with the file name. Instead of. Instead of putting that file read directly in the constructor, I think that's going to work better. Because specifically once it's in a function, I'm no longer doing anything with this. Now I can do the standard. Method of wrapping the body of the function in start active span. Oops, I forgot to name it. Name it. Read file for song. All right. And then I'll scoot the end of the function down here. Oh, s dot n needs to be right before the return. Hmm. But I don't like that we're doing a bunch of this stuff after the span. So I'm going to stick it in our results is this and then end the span and then return the result that way. If this operation takes some time, it'll be incorporated in the span. Okay. And now the whole point of that, which was to add an attribute to the span, set attribute app dot file name, and we'll set that to the file name. Now I expect at least some of my read file syncs, not all of them to have another span on top of them. And that span should reveal to me the file name. That's happening. But that's being read. Okay. So run it again. Okay. I'm going to take a few seconds to make the cow. This is all startup. That cow was super early. Okay. It did some stuff. Back to Yeager. Back. Fine traces. See, I had 51 spans. Now I have 52 spans. That's a good sign. When I click into this trace, I'm going to scroll down and look for, what did I call that? Something about a song. There it is. It says read file for song. And it has a file name. Input slash song dot text. Okay. And I can tell that it's taking place. It's not quite the last file that's read, but it's close. There's one other one somewhere. That's kind of interesting. That surprises me. But that's interesting. Okay. So there's a read file for song, but it only showed up once. There's only one occurrence of song on the screen. So that was, that was the hard one, but it only got us one. So let's go do the other one. An image dot TS. There's another read file sync. This one conveniently is in a function, not a constructor. So this should be easier. Make a tracer image dot TS. This, the name of this tracer is going to show up. Let me show you where that shows up. Cause I find it interesting. It's here under tags for the span. We have hotel library name and that says song dot TS. So that's where the tracer name winds up. In this case, that tells me that the, because I named it this, that tells me which file the, the span was created in. Also, I can search for its name in the code. All right. We've got this tracer. It's here under tags for the span. We have hotel library name and that says song dot TS. So that's where the tracer name winds up. We've got this tracer. And now we're going to use it. To wrap the body of this function. In a tracer. So we'll call this one read PNG. Cause this is reading a PNG file. And then move this to the bottom. It's really important. One reason I put this in the alias is you have to return the output of start active span. Or else it gets munched and your function doesn't return anything. Probably just type script gets mad at you. If you're in JavaScript that can be really tricky. Okay. Oh, I don't think this either. There's too much happening right here after the span ends. So let's make that a result again. Okay. So now I'm doing the read file sync. Oh, right. The point was to set an attribute on the span, which is what's the, we call the app file name before. Let's use that again. I like to prefix. Pardon me. I'm going to sneeze again. Okay. In custom attributes, I like to prefix them with like app dots so that I can find them easily. It doesn't help as much in Yeager, but it does in Honeycomb. All right. So now we're adding the file here. Let's see which files we're reading from image.ts. Let's see how many spans we get. Thanks. Thanks, David. We're the bless you. All right. Yeager, what do you have? Fine traces. What's at the top? 86 spans. 86 spans. That says something. All right. So here we are at the top. We've still got a lot of these read file syncs at the beginning. Spoiler. Those actually wind up inside the first HTTP call. Or the first Axios call, I should say. But here, now we have read PNG. We have a bunch of read PNG spans. Read PNG. Read PNG. The browser tells me that I have 34 of those. 34 read PNG spans. Wow. Okay. Well, which files are we reading? App file name. Input slash don't peek. Okay. App file name. Input source. Input slash bigger tree dot PNG. Bigger tree again. Bigger tree again. Bigger tree again. Okay. It's, it's clearly. It's clearly being the same pile over and over. Yeah. There's a link. The question from Carlo. It's amazing. Thank you so much for great questions. Everyone. It looks like you're modifying the production code with what's the overhead you are adding? Have you measured that? How to evaluate the threshold where is worth as this traceability operations. They're imagining a fast and quick set of light operations. What would be the impact? Okay. So adding tracing to production code. There are people who measure the impact of this. It's very small. Tracing. Tracing production code is like a modern standard for production code because in this case it's just giving us a picture of what one program is doing. But in a distributed system, it gives you a picture of the overall system. Tracing. In production. When it's carefully curated like this is, you get like a strong picture of what's important in your app. An extreme level of tracing would be profiling. And profiling is it's known to add overhead in production. But these days we have continuous profiling where that's pretty low overhead too. Tracing is less overhead than profiling generally because it's not as high resolution. I'm only tracing the units of work that I said are important. A file system access. Let's see how long that takes actually. How does this file system access? This one is taking 273 microseconds. Wow. That's really small. Actually the whole read, well, okay. That's just the read file sync which opens the file. My span of read PNG, which is also gathering the data and doing a little processing on it. But it's also reading all the data. 1.29 milliseconds. So this file system access of 1.29 milliseconds. The overhead of tracing is going to be in the microseconds. It's negligible. So if you have super, super high performance code that loops around a lot, actually totally trace that because then you'll be able to optimize the code at Honeycomb. As part of our database, we've run lambdas like in production, if you make a query into Honeycomb that's like over more than a few hours of data, behind the scenes we're spinning up lambda functions to read the data from S3 for the database. And we're paying for every one of those. But we're paying for it because we want your query to come back in a second, maybe two seconds. So we need it to be incredibly fast. And we're paying for every CPU cycle for every millisecond. Those lambdas are running. So we optimize the hack out of those to keep this affordable. And we do that with tracing. So I would say if you care about performance, the overhead of tracing is more than made up for by the ability to see what's taking up time in your system. And where you're spending that. Okay, so that's my opinion on that. For this program, how long does this program take? It takes 1.8 seconds. Nobody cares. I mean, nobody cares about the overhead of tracing. Also, I don't really care that it takes a second. I think the node startup time is much longer. Great. And then there's a few other questions from. Okay. Yeah. So Rob asks, can you programmatic the decide if tracing should be activated during a request? For example, if you want to monitor a specific request, the critical given some runtime context. Okay. Yes. So that's fun. Tracing locally, like we're doing here, where we're gathering all the traces and using it for local debugging. You absolutely want every trace to be recorded in this local Yeager. As soon as I duck or compose it will be gone. But you want to keep every trace. In production. I'm looking at this and there's like 86 spans. If this were a batch job that I'm running in production. Okay. Actually, if it were a batch job, I'd probably want to keep everyone anyway. But if this were a web service request, or even a batch job that runs on a queue, and I'm doing a million of these in a minute, I absolutely don't want to keep 86 million spans. The generation of them is less of a problem than the storage. Just the storage gets expensive, especially if you want them to be searchable. And in that case, you do sampling. You can do sampling in multiple ways. You can decide at the beginning of a request. Okay. We're not going to keep this one. And then all of your span dot span creation and span dot set attributes. Those are all going to be no ops. If you decide at the beginning that we're not keeping it. And then there's, there's ways to decide at the end instead so that you can do things like keep all the errors. The trick with tracing is that you want to keep all of a trace or none of it. So you do want to be consistent. You don't want like randomly a couple of read files from this run and a couple of read files from that run. The point of a trace is that it tells the store a story and it's a coherent story. So in production, you would keep one out of 10, one out of a hundred. If it's a health check, one out of a thousand, whole traces. But one really nice thing about adding traces to understand the program locally is that it can still have that in production. And I get to decide by setting the sampling rate, how many stories I want from production, but I can always get to one. Okay. More questions. I think there was one more, I think, and then someone asked for the template demo. Say cold people again. But luckily Carlos sent it in. So I think that's their answer. So that's perfect. And then there was Ludovic with the question, what's your point of view on using agent or gateway to collect tracing on communities? Okay. I am going to go with. Ask me on Twitter or something because that's totally like out of scope for this particular topic. You can use an agent or a gateway on it. Well, one, I don't really care. I would probably pass that opinion to, or that question off to someone else. I don't have strong feelings about that one. And I want to focus here on like the local tracing and the, the how to add tracing to a note app in this case. Thank you for the question anyway. LinkedIn user. Okay. We saw that we can read files. We saw that we are reading the same file over and over. Bring that back. Okay. I need to expand this shrink this. And we're reading treat multiple times. So my next question that I have for my own app is. Okay. Why are we reading bigger tree? What's reading bigger tree? I can go with that in a couple ways of like what, here's my read PNG. What calls that? I can ask where to go. Okay. Read PNG is called I read image, but read images like just one. This looks like just a wrapper function. That's not interesting. So while in profiling where it does like every function call, it would create a level for this. I'm not going to, this isn't meaningful. Clearly at some point, I used it to print a bunch of debugging stuff, but I'm not now. So let's go outside of that. This is something I could do either outside or in or inside. Okay. It looks like read images called in four differences. So any of those can be interesting. And so I don't know like, where do I want to follow this tracing through? So one other thing that I can do. Oh, go ahead. There's another one again. Carlo. Thank you so much. Do you have any techniques or best practices to keep tracing code separated from the actual code implementation? In general, if you have auto instrumentation, you're not even going to see it. The calls to the file system just happen and just get it instrumented. That one's not actually called or it would show up. But when the tracing is about your code and you're making custom spans, don't separate it from your code. Now, I can say that, but I also will say do wrap it, create a little wrapper. Where did I put that? Here. Okay. So like here where we're doing read PNG. And I do this enough times that I have an alias for it. I say return start active span. And so that part, the name to type in each time and the name should be searchable in your code. You want to hard code that. But then I always have to remember to close the span before the return. So I've left out the part in these where, the part that I put in Maine, where if there's an exception, I want to catch it and put it on the span and end the span anyway. So I actually would recommend making yourself a little wrapper function that does what you want. And calling that instead of the open telemetry start active span. And that's, that's going to be, I don't know, it might be 10 lines. But that's, that's as far as I would go with obstruction. This part set attribute, put this in your code. If you don't have the S variable, you can also do this with hotel dot trace dot get active span. You can do that anywhere, even if you don't know where you are. For instance, I could do that in Maine. In a function. Maine doesn't know about the tracing around it. And this trace ID is actually really important. So when we, we put it to the console. So I would do dot hotel trace dot get active span, question mark in case it's null, set attributes app dot created trace ID and put this in here. Now, a lot of people worry about this is cluttering up my code. Well, one, so do log statements, but we log things because they're important when we log things. I would trace that I would add as add an attribute to a span instead, if you have tracing, we console dot log this because it's important. And adding a set attribute to your code says something about what's important in that function. And I find it actually express it. I mean, maybe you want a little wrapper function for this. Rapper functions are great because they let you like create little organizational standards, such as maybe instead of app, you add a different prefix to every, or just a consistent prefix to every field name. So do create wrapper functions, but don't try to like completely separate the tracing from your code. Make this searchable between the attribute on the span in your tracing system and the code itself, because that draws a connection. And this is how you get back and forth from your code to, okay, where's this in my traces? You can search for that field. And from the trace to, okay, I see that there was an error on this span. What code is that? Cool. Okay. Yeah. I think that's a good way to quote sample, give your Twitter handle. If someone wanted to ask a follow up questions or so, but you can also save it to the end. I'm Jessica Tron on Twitter, J-E-S-S-I-T-R-O-N. Okay. But for here, let's see how we got as far as, I don't know, I don't know how many of you have seen this. I don't know how many of you have seen this. But as far as finding that there are all these read PNGs of duplicate files, the next step that they take here is to find out what those are part of, either by going up or by going down. In this case, it turns out to be useful in a lot of cases to also, like as I'm moving up from the file, I also go down because I want to find like what's in action. What are all these file reads part of? And my main function, I have a couple of main steps, read image. Oh, well, we know what that's doing. We already have a record for that. Plan spans and send spans. And so if I go into like plan spans, which is a major step, it's doing a couple of things, but I could span, add a span for this, plan spans. This is a fairly easy one. Oops. Enter. Down, down, down, down, down, down. Don't forget to end the span before the return. And this will give us a span for just like one level down in main, and we can see how many of the file reads are inside of plan spans. And then we can kind of zero in on the areas that I want to refactor. And then I haven't done any. We don't have to get this done in an hour. Then I can try to shift the spans and the look of them as I do a refactor. Okay. Let's see if plan spans showed up. So this is, this is another thing that I do when I'm adding tracing is I just, every time I want to see a difference, I love to see a difference. Some people love seeing green tests. And that just like does it for them. And they use TDD all the time and TDD is excellent. It's a fantastic tool. It's a tool for thinking, a tool for design. And it's a, it's a great way to code. But sometimes you're presented with a program that doesn't have tests and you can't just do TDD on something like that. Tracing works for both. And I love seeing a different in my trace. I love my software telling me, Hey, check this out. I did something new. Oh, and spans right here. Okay. So if I like shrinks, no, no, no, no, no, over here, shrink. No, no, no. There. Okay. I can, here's our read PNG. That's at the beginning of main. And here is plans. So it's all of this incorporates all of those. All of those files. And I say, okay, well, it's somewhere in plan spans. And then I can dig into this. And I'm like, okay, there are three main components to plan spans. I'm going to cheat. I know which one it is. And, but I can see which of those are taking time. Which of those are reading files. Waterfall, build pictures in water. Oh, watch out. Oh, watch out. I think this is, this is the hard one. No, this is the hard one. And then I need to get this to the end, which is actually tricky. Where's the end of this function? 94. Um, And then some, um, person is asking also, is the tool you mentioned? TDD question work. Oh, okay. Oh, Okay. Okay. Great question. TDD is called test driven development. And there we don't make a change to the code until we've written a test. Um, you start with a failing test. Like if you want to add a feature, you start with the tests, the tests that the feature works and the feature doesn't work because you haven't added it yet. And then you add the feature and then the test is green. Um, and this is really satisfying. And then from, from green tests, when you have everything nicely unit tested, um, in automated tests that you can run, uh, with a push of a button or automatically on file save. Uh, then you can refactor, you can move your code around, uh, without having an effect on the outside and all your tests stay green. And that's really satisfying. In my case, I don't have tests to stay green. So I'm looking at the traces. Do they do what I expect? Um, and this, um, this is teaching me about how the program works. Um, and it's something that I can build in gradually. Um, which is harder to do with tests. And I don't have a tracer. So we need to do that. So as I, as I like, add, um, additional spans around things that are important. Um, I get a better idea of like the shape of the program and, um, get a better idea of like the shape of the program and not just the code, but the shape of the program execution. Like with these operations, I'd plan spans, which takes a long time, which is hard. Um, yeah, you should show me this now. Find traces. I want like one more span. Oh, 88 spans that look good. It's random. How many pictures it builds in, uh, to the trace. And here, uh, Oh, I think someone asked where they can learn about TDD. Oh, let me give you my favorite recommendation. Gpahill.org. Jeep. Um, Gpah is the best for teaching about. TDD. Um, Gpahill.org. Okay. My router is not making the connections. There it is. Uh, yeah. Highly recommend. Um, he's the great material for working in smaller increments and generally a great programmer. He's, he's funny and very real. Okay. Plans spans. We've, we've got a read. Ha, build pictures in waterfall. So build pictures in waterfall. Now most of the read files under it. So I know that it's part of build pictures in waterfall. That it's reading the same file over and over. Um, and at that point I can. And start. Moving those reads out. Um, which I'm not sure I have time for today because any, is it okay? What if I, if I show what, what these, what this trace looks like in honeycomb with its little. Yeah. Um, yeah, we have on the top of the hour. So nine minutes left. So, okay. Okay. Check this out. I'm also sending these to honeycomb. Um, with my local collector and. Yeah. Oh, it looks like I have two of them going on. So let me check my recent traces. Um, and let's get. Now you stop group by trace ID. Last 10 minutes. There we go. Okay. So here is my trace of everything. Um, where I can look at the file name. Here's a little easier to see. Oops, not servicing. Do it. Here you can see bigger tree, bigger tree, bigger tree, bigger tree. Okay. So that's the trace we're familiar with, but the interesting trace is this big one. And here is, yep, here it is. Here's the, the trees and the balls. And I need to zoom out to make it look better. There we go. Um, and the little song and some sort of colors here. That's okay. Uh, what I really wanted was this trace ID. Um, let me. Go to this board. So I get just one of them. Okay. And really what, so now I'm, I'm, I'm graffing the spins in this one trace. And here they are as a stacked graph. Um, and so it makes a tree in this case. And here, uh, in the heat map, uh, it makes a whole drawing. So this is, this is the game of, um, switch to absolute time for me. Um, of this, um, the program is that it draws a picture in a heat map. I feel like I should show you what a heat map usually looks like. Oh, let's see where's a random heat map. Um, let's see where's a random heat map. Um, let's see where's a random heat map. A demo can get a random heat map of latency. Um, front end. This is a heat map. It shows like higher is slower, darker is more. There's those requests. Wow. Some of them are really slow. Uh, but, um, the, the program that I'm working to make configurable, um, it's, it's reading PNGs and making a picture in, uh, which is some visualization art that I'm rather happy with. And then you can do things like say, Hey, what's different about these spans? And it turns out that what's different is their reindeer name. Those are dancer, um, pixels. And then you can group, uh, group by the reindeer name and, Oh, no, I need, I need to put it back in the time range. Um, custom 40 to 50. I, there they are. That's right in the middle. And the results. I'll see my group. Bye. And you can see that there's dancer and there's Rudolph and there's answer. I think this is really cute. And I want it to be configurable. Um, you can already draw your any PNG. Uh, but, but, uh, regardless when you click into a trace, um, you're always going to see the Christmas song and you're always going to see ornaments and trees. Um, and after I move ornaments and tree reading up to the beginning, then we'll read the files less, but also, uh, it'll, I'll be able to, um, accept a configuration file that says which drawing should be in here. These drawings are configured. Um, in here and no, no, here's the bigger tree. It looks like this, um, in a PNG. And this is the PNG that gets converted into, uh, Santa and his reindeer. So that's, uh, that's the fun. Yeah. Yeah. And then actually in here it's a little complicated, but the, the blue channel is like, how dark to be in the heat map and the red channel correspond to, uh, to the attributes on the span. Um, so that's how the, the PNG like becomes a picture. Um, yeah. Perfect. Such a great image. Um, came out of there. So lovely. Um, I have four minutes left. So if anyone has any questions, I highly recommend you to type them out now. So we have time to get to a few of them at least. Um, but yeah, anything else that you want to show. Oh, there's a question already. Uh, will it be possible for you to show the RG are in your code implementation? What's RGR? Um, maybe if the question asker would be. Um, a little bit more or what they actually want to see. Uh, anything else that you want to show us or, um, I, yeah. Yeah. I didn't quite get as far as like doing a refactor and seeing the trace change, but I did earlier today when I practiced this. Um, and it's really satisfying. And I think that this, Oh, red green refactor for TDD. Would it be possible for you to show read green for factor in your code implementation? Definitely not because I don't have any tests. Don't have a test framework set up. Did not write this for testability. I wrote this for hurry, hurry marketing has a thing going on and I wanted to be cute. And it has to be up before Christmas. So there was like this Christmas deadline. Um, in general, I think T, um, I think that the red green refactor and test driven development. As part of test driven development is a great, great way to code, especially when you, you intend the code to go into production, which this was never intended for, but often you have code that was not intended to go into production, but now they're like shipping. The customer service reps are like, Oh, we'd love to put this in our newsletter and send it to the customers. But I'm like, but it's not Christmasy. It's too Christmasy. It's too hard coded. Um, and so when you want to move a piece of code from, well, it works, but I can't easily change it because I don't know when I'll break it, right? It's too fragile because it doesn't have tests. I think traces are a great way to be like, okay, I can have some confidence that my refactoring is working and then I can move it towards something testable. For instance, if I get a full configuration file, the random seed and the time of day, um, that will deterministically produce a set of spans, um, for the output that, that draw the three pictures. Um, and that I can put a test around. And then I can put a test around the individual parts because all the file reads are happening at the beginning. Um, and all of the, the open telemetry send spans are happening at the end. Um, and that, uh, that will make each part in the middle testable. And so this is, this is moving the code to production ready, uh, testability wise at the same time, it's immediately making it more production ready by giving you visibility into what's happening at runtime, uh, because the same traces that are helping me while I'm coding will help in production at a smaller scale. We'll sample them then. Perfect. And then, uh, Daniel asks a compliment of the, this is interesting and then wanted to know where, where to learn more about observability and also ask, will a recording of this be available? So I can answer that it will be available. And it's going to be in the, um, club native meeting foundation YouTube. I think it just a matter of an hour or so. Um, so you can check out the recording there very soon. Um, but as the final question, I guess for today and after that we will wrap up, um, what do you think is the best ways for people to learn about these concepts? Um, uh, will I say that the CNCF has a couple other open telemetry specific meetups. There's open telemetry in practice. There's an open telemetry and user group, uh, and user discussion group, I think, um, and open telemetry is the modern standard for observability. So that's a great place, uh, to come and talk to people and learn more. Perfect. So that's where everyone will be headed after this. Always great to have the next resource ready. But thank you everyone for joining the latest episode of cloud native live. It was great to have a session about open telemetry today. And we also really loved the interaction as well as the question from the audience. Amazing questions, amazing reactions. Always very, lovely to hear that. And as always, we'll bring you the latest cloud native code every Wednesday. And in the coming weeks, we have more great sessions coming up. So tune in for those. Thank you for joining us today and see you all next week.