 Okay, switching to the second topic. So the second topic is on distributed go logging. And some of you have seen this before, it's been presented in a previous Pixie Knot meeting, but I'm gonna recap it for those who haven't seen it and for those who just need a little bit of a refresher. What dynamic, what distributed go logging is, is essentially a way to inject printf statements into your code while it's deployed, live. It's live, it's running and you kind of are inserting printf statements just on the fly without having to recompile, without having to redeploy, because we've all been there and applications running, something goes wrong. We wanna figure out what it is that's going wrong, but we look at it, we look through all our logs, we maybe have a hint, but we're like, oh darn, I just wish I had a little bit more visibility on this one function and I could see what was getting passed into it and what was being returned or something like that, right? And so we have with Pixie the ability to add these dynamic log points into your code, right? So here we're putting a printf statement and maybe we wanna see what A and B are in Foo. We can get A, B, we can get the return value, we can even get the latency of your function. How do you do it? So just one example, which is gonna be very relevant in a second, let's say we were trying to trace this function called sum. It's a go function. It takes two money values, left and right, L and R, and it returns the summation of these money values and it can potentially return an error as well. And if we wanted to trace this with Pixie, there's a few things you need to do. First, you just need to say what function you're trying to trace. Obviously you need to specify that. So we give the, for now we have to provide the full symbol name. We see it's this thing at the top here it says ends with money.sum. We say this is the symbol that we're trying to trace, that we're trying to attach a dynamic log to. And then we say in the body of this function here, we're just specifying everything that we wanna log. So we say, hey, let's log L, let's log R. Those are the two inputs. Let's also log the return values and let's name them L, R result, result and error. And that's it. That's all you really need to do. And then you deploy it and you should start being able to see what's happening live in your application. To actually deploy this, the last thing you actually do need to do is to specify the pod name that you want to attach this to. So the stuff at the bottom is just deploying this. And we say we wanna deploy this code to the online boutique checkout service, which gets, and then we upsert trace point which essentially deploys. It's really just a deploy function. It deploys our dynamic log. And then the data all gets collected the pixie way. We put it all in tables and then you have the power of the pixie platform to go query stuff, look at it, filter, do whatever you want. It's all kind of recorded and ready for you to look at. Three things I did wanna just quickly call up for the people who have seen this before. The three new features that we have, some things that were kind of requested was first, previously you had to trace only primitives. Now we can trace entire structs. It'll represent the entire struct as a JSON format. So you no longer need to do, you still can but you don't need to do like L dot foo to get to an enter, float or string. You can just say dump the whole struct for me. The second thing is we now trace interfaces. Pixie will automatically figure out dynamically what the type is and output the JSON format for that struct. And the third thing is we've made it easier to deploy by allowing you to just specify the pod name that you wanna attach to before you had to provide the U-PID which was an internal pixie construct that kind of designated the process. But now you can just use the pod name. If there's multiple containers or processes within that pod, then there's a way to specify which container and which process within the pod that you want to attach to. Best way to kind of learn this stuff again is with a demo. So we'll come back here and like this. Let me just set up a few things here. I'm gonna pull up this demo application which is called online boutique. You've probably seen it before, it's a retail website. And we went and added to this code. I made a small change to the code. I made a small change to the code and it caused the crash. I went into their code and I wanted to kind of play around with it. And I just added a special deal. I said there's a quote service that gives you the cost of how much it costs to ship something. And it says, how many items do you have in your cart? And then it'll give you a quote for how much it should cost to ship this. This is online boutique is a retail website. And I just went in and I said, I wanted to change it so that if you put more than 10 items in your cart, you get a special deal, you'll get a 99 cents shipping. Kind of get very discounted shipping. And I added this and I expected it to work, right? So I expect it to be no problems. But when I ran it, let's see what actually happens. If we run this, we come to their website, we come, you can add stuff to the cart, you can purchase it, you can purchase things. Actually, let's empty our cart here. And we add something to the cart and we place an order. It all works, right? And now let's go and add 10 things to our cart. And I'm gonna add 10 things to the cart. First thing to note is this worked. The shipping cost was updated to 99 cents. Awesome. The code change we made looks good. It's reflected in the UI. And as soon as I try to place the order, boom, it crashes. Why? It was completely baffling to me. Like why such a simple change should cause it to crash? The other confusing thing, it wasn't the shipping services itself that crashed. It wasn't the code that I changed crashed. It was a different service that crashed, which had me really baffled like what's going on here. So we can use dynamic logging to figure out what's going on in this. So let's pull it up. Let's pull up a script. It's similar to the one I just showed. And so let me bring this up here. And I'm gonna pull up the script here and we're gonna trace all the inputs and outputs to that money.sum. And we're gonna deploy this trace. And just to keep things fresh, I'm gonna rename the trace point that we're going to deploy. And so we're gonna hit run. And what it does is what it's doing under the hood now is it's deploying all the BPF probes, taking care of all of the distributed system parts of it for you, getting the BPF code automatically generated and deployed out so that we can insert this dynamic log point into the code so we can see what's happening. So now we're tracing the checkout, the sum function that we suspect it has a problem based on the error code that we saw. And so we're gonna come in here and we wanna reproduce the situation. So we're gonna add 10 things to the cart and we're going to place an order and it's gonna crash again. And I'm gonna rerun the script. So the first time I ran the script, it was empty because it hadn't traced anything yet. But I'm gonna rerun the script. And what we see is it actually traced that sum function that we were interested in. And also we see here that where the error was, it says one of the specified money values is invalid error code 13. We also see that we traced that. I'm gonna full screen this now. We traced that here. We see the error code 13. So we actually traced the case that was an error and we caught it in the act. We see what happened. And now we can kind of dig a little bit deeper and see, well, so what was it? What's going on with this request? Why is it special? Why is it creating an error? And so we pull it up here. We inspect a little bit of the contents and here we can see we said trace L and it's giving us the entire JSON representation of what was inside the struct. There's the currency code. There's essentially units, which is the number of dollars. It says it's $1. And then there's nanos which is kind of the number of cents in there. So it's saying it's $1 and negative one cents. So already kind of our spidey senses are going off. We're saying that looks really suspicious. That's probably the cause of why there's an error code. Just to kind of check it out, we see that the other value that it was trying to add was $67.99. That one looks good. That was the other input and the output is invalid because we got an error. Okay, so now we have a clue about what's going on. We got that observability. We were able to, without adding a printf statement or anything into our code, we got instant visibility into what's happening with this function, right? And so next we're going to say, well, so why did we get an input of $1 and negative one cents instead of $0.99 cents into our code? And we go look at the previous- Just to jump in a couple of minutes gives Zen three minutes before. Yeah, pretty much done. Yeah, one more minute will be done. So if we go look at the previous requests, we actually see that it was trying to do is it was trying to add $0.00 and 0 cents to the shipping cost of $0.99 cents. That makes sense. But the answer that it produced, so when it tried to add $0.00 and 0 cents to $0.99 cents, it produced $1 and negative one cents. And this was not our code. We did not change. This is actually a real bug in the, this is Google's demo application. That was only triggered once we hit values less than $1. I had no idea we were going to hit it. We just hit this for, you know, just by chance. But we were able to use Pixi to kind of figure out what's going on. And you can go back to the source code of the sum function. And what you'd eventually figure out, I'm just going to cut that to the chase, is they're missing a greater than, they had greater than it should be greater than or equal to to come into the correct clause here and produce a valid input. When you're trying to add zero to $0.99 cents. Again, I just want to stress that the use case was we, you know, I made a change to the repo. It caused a crash. It was like, it was frustrating. I didn't know why it wasn't even my own code base. But I was able to use Pixi to kind of just added a dynamic log point without having to redeploy, recompile or anything. And just look at what the input's that for. And just look at what the inputs that function were and it very quickly was, you know, able to kind of see, oh, there's some suspicious input here. And it really helped me figure out where it was. And that's kind of the vision that we have for this dynamic log point. So we're excited for other people to try it out and see and provide us feedback of how useful it is, how you can use it, use cases that other people find for it. So yeah, that's kind of it. Just to answer a few questions. So Bogomal has a question. Will the distributed logging that you demoed for Gobi available for other languages? Yeah, it's a great question. On the more immediate roadmap, we're going to be supporting C and C++ fairly shortly. And then we are looking at other languages as well. We want to keep expanding this out to other languages totally. Yeah, so I guess the other point I wanted to quickly add over there is C and C++ are the most immediate because we have a lot of share of functionality. And then there'll be features for Rust. So basically you're going to see support for compiled languages first and then dynamic languages later on because it's harder to do and there might be better ways to do it than using a VPF. Yeah. Cool. One last question. I don't know if there's any answer to that, but Rangaswamy had a question. DNS tracing and code tracing use the same code base? Yeah, I did answer that. I guess I can take that quickly. But basically everything in Pixie is utilizing the same infrastructure. And we basically use Pixel to control, which is the Pixie language, Python based language to control how code is instrumented and monitored. And there's underlying data collectors that go and store data in the tables and we can query it using Pixel. So everything is unified. Great, everything is unified. And yeah, behind the scenes we're using a VPF for both of them. Yeah, it's really powerful stuff.