 Hello, everyone, and welcome to Cloud Native Live, where we dive into the code behind Cloud Native. My name is Taylor Dolozal, and I'm head of ecosystem at the CNCF, where I get to work closely with teams as they navigate their Cloud Native journeys. Now, every week, we bring a new set of presenters to showcase how to work with Cloud Native technologies. They will build things. They will break things, and they will answer your questions. In today's session, we have Pierre Tessier from Honeycomb. And today, Pierre will be presenting the Open Telemetry Community demo. I'm really excited to see all of the things that come out of the Open Telemetry demo today myself personally. This is an official live stream of the CNCF, and as such, is subject to the CNCF Code of Conduct. Please don't add anything to the chat or questions that would be in violation of that Code of Conduct. Basically, please be respectful to all of your fellow participants and presenters. Be excellent to one another. With that, I'd love to hand it over to Pierre to kick off today's presentation. And with that, Pierre, please take it away. Awesome. Thank you very much, Taylor. My name is Pierre. Pierre Tessier at Honeycomb IO. I'm really excited to talk with this topic coming here today. It's something that's been consuming much of my free time. And just a lot of thoughts and process for the past several months. But it's coming to fruition really nicely. I'm going to talk about the state of today. So early October 2022, I'm going to talk about where we're going with this, and what's happening with the Open Telemetry demo itself. If you have not heard about it before, well, what is this thing? It's really a showcase of everything Open Telemetry has to offer for the community out there. How you use it, where to use it properly, different combinations of usage you can get out of it. What we did is we took a 12 microservice application written across 10 different languages. And we're going to show you how to deploy this thing, how it's instrumented. It's meant to be rendered and forked, if you will. There's a lot of vendors that are part of the Open Telemetry community. That's what helps keep it going. We have employers that are grateful to provide us with the way to contribute to the wider community of use. Well, these vendors also want to show off what they can do with Open Telemetry. And this demo is meant to be forked so they can do exactly that. We do have our own GitHub repo, of course we do. It's like all Open Telemetry 6. Go ahead and check it out. And we'll check on that a little bit more later. But I want to showcase what this thing really is. It looks like this. Like I said, it's got 12 different services in here, a couple of different storage locations as well. It uses Redis, which we don't count as one of the services, and also a Postgres DB, again, not counted. And I'm going to say again, actually, I mean, it's 13 services. I keep on saying 12 because I don't count the low generator as a service per se, but it is in this case here. So if you look at all the boxes, you'll count 13 up coming up here. Internet should not count either. Internet is loaded in generator if you want to get to year 12. And we've written across all these languages. The only ones that kind of duplicates front-end and payment on JavaScript, but it's fun. Front-end's actually written in TypeScript and payment is written in JavaScript. So although it's written in the same kind of output compiled, if you will, runtime and interpreted language, they do showcase a little bit of syntax differences between the two. Also I mentioned front-end also instruments the web browser. So if you came in here, it's a front-end developer like, man, I really wanted to insert the web browser. I got to cover. We're going to show how we do this later on today. And then we move on to the other languages across the whole gamut. Notably missing here is Swift. We do one day want to add a Swift client to this application to make it even better. And we're also going to be looking for a host to provide this application for people to use without having to install it as well. So that's coming soon. Plans for that to be determined. If you want to help out here, I'll get more info on how you can reach out to us and do exactly that. And with that, who am I really? I work for Honeycomb I.O. I run the solution architecture team over there. That's a team of people who are passionate about ensuring that other customers instrument their code, get up and running most of the open telemetry and get the observability they need for their applications. Honeycomb I.O., we are very much committed to supporting open telemetry. And we support all the various endpoints for receive traces, metrics and logs into our platform to provide what you need to do to debug applications should go sour in production or maybe just want to find performance improvements. So we help you do that. And also I'm a maintainer on the open telemetry SIG itself. I've been part of the SIG since it started and I'm really happy to be part of that due to the contributions that we do. And I'm diving some details on what the demo itself does. Some interesting stats as well to help you understand. First, a little bit of history. The SIG was formed April 26th this year. We had our first meeting on May 2nd. And I think Carter might be watching right now. Carter from Microsoft, he's been instrumental in really kind of kicking us off and getting it going and helping us move this project along and meeting our dates. And most importantly here, we're going to hit, I'm very confident, we're gonna release GA at the end of this month, the week of CloudNativeCon, CubeCon. So on that Monday, we're gonna have a GA release ready for everybody. It's gonna be less than SIG months since the SIGs inception. So I don't know if that's kind of a record or not out there, but we're really proud of the work we've done. We did have a massive initial contribution thanks to another one of our contributors, Juliano, who took the Google Cloud project. It was known as microservices demo. People knew it as hipster shop or online boutique. He took that and kind of ripped out all the old instrumentation, all the other stuff and got us a first cut of this running with open telemetry so we could use. But we took that, we refined it, we made it idiomatic and ultimately we're gonna be releasing 1.0 here shortly. The goals we wanted to do here was we really wanted to have a common demo that can be used by everyone in the open telemetry community. It wasn't just vendors, it was people and other developers that were spitting up these little mini hello world applications or maybe they were using some other applications or some other microservices things and it was getting kind of just fragmented out there and they were doing it a little bit differently. Open telemetry is a very evolving stack of technologies. It spans multiple languages and they're all kind of evolving at their own pace. Sometimes they change how you should do things and all these demos were not necessarily keeping up. So we wanted to create something that kept up with the latest and was common across the mall and we wanted to make sure it includes every single language that open telemetry has support for. We're one short right now, Swift, we will get there, but we do support the entire ecosystem. It's gotta be using the latest stuff, like I said earlier, it's really important we do that. And I think the biggest point here that everybody should appreciate, it needs to be well documented. People need to be able to just go to this thing, look at the code, look at the docs with that code and understand how to do it themselves. We need to enable the community with this demo and how you do that is with good, strong documentation. And that's really one of our big goals that we're gonna do when we release eventually. A couple of stats, and I really caveat this with as of this morning, the repo's got 150 stars, 95 forks and 34 awesome contributors. We just cut release 5-0 or 5.0-alpha yesterday. I'm not, no, it says alpha, I don't know. Maybe we're ready to drop that label and call it beta as we now have all services ready and kind of nearing future freeze here. You can use it, it works. I'm gonna show you the small little caveats of deploying this thing later on in the session. It works across a whole gamut. So definitely don't be scared of that alpha label right now, if you're interested and after you see all these are like, wow, I want to help. We do six, we meet every Monday at 8.15 a.m. Pacific time. Again, the repo link, we do have our own Slack channel, Hotel Community Demo is a Slack channel. In case you haven't figured it out yet, we're looking for Swift developers. We want to do this. We don't have anybody who will experience in this. And we're looking for perhaps a Swift developer who can help us do this. Bill does a small app. Let's get it in the app store and let's allow people to actually play with this and we'll have them point to a hosted version of the demo itself and it'll be great. People can actually use their phone and work with this demo and see the code, how it all works. So if you're interested in the rights with code, please reach out to us. We're happy to help and provide you whatever support you need to get going. Now with that, what did we do in the code? What did we actually instrument? So first off, when we did this instrumentation, we really wanted this to be in an idiomatic way for every single language and that was really important for us. It's just, we want to make sure that when people look at the code inside this demo, that's the right way to do it. If you were to go to open telemetry docs and you were to go to copy and paste code, it should drop right into what we have inside our demo. That's the idea. And as the SDKs evolve, as they perhaps add additional APIs and functionalities to allow you to tie in and make it easier to use, we're going to update the demo code to stay in conjunction with whatever is available in the rest of the open telemetry community. And this will be the proper way to instrument. And we're going to keep on doing that. So you could be rest assured that what you see in this demo will be, hopefully, will be how we want you to move forward with it and how to use open telemetry to the best advantage. I see here a link. Somebody's asking for that link in chat for the repo. I'm going to post it there in a couple of minutes. Just give me a couple of seconds and we're going to run through more stuff and I'll make sure I get just all the links when we get going for the repo itself. Sorry here, here we go. So what did we do to instrument? Well, for traces, all languages support trace. So they all have tracing support. We do auto-instrumentation where available. So Java, Python, node is kind of a weird thing. People call auto-instrumentation, people don't. I don't know where you want to sit on that world, but we did it, okay? It's pretty easy to do with node. And all languages also have manual spans. They will have all have manual span attributes. Almost all languages have span services, have span events. I believe all services do do span events somewhere there. Maybe Elixir might be the only one we don't do that for. I'll have to double check. And we also included span links. And we only did this for one spot, notably in the front end. But we wanted to introduce a concept and why you would need to use a span link to make sure people understood why we kind of did that. For metrics, we're gonna have Java go end.net with our GA release is the goal. We have a stretch goal to also include Python, though we're a little dependent on that SDK for us there. So like I said, we're keeping up with what the SDKs are doing for each language. And as they GA their support for metrics, we're gonna go ahead and add that in as well. For all of these metrics, we're gonna include the runtime metrics available to us. So like JVM, you get heat metrics, threads and stuff like that. Go, you get like go routines, I think. We're also gonna include on the HCP, GRPC endpoints. We'll include all the latency distributions for every one of them. Like kind of your typical things. And we're gonna get this dumped into a Prometheus, another CNCF technology. So it'll go into Prometheus and then we're gonna be able to provide you with a group on a dashboard to visualize it, all part of that same package. Traces are gonna end up going to a Jager. It gets deployed part of this demo as well. So you could use that. But like I said, because we made this very vendor fork friendly, it's easy for you to pipe your output to somewhere else. I'm at Honeycomb. I'm gonna show you how to do it for Honeycomb, but it's the same process. You wanna dump it into LightStep or if you wanna use Splunk's way of doing it or even data, or any of the other vendors, it's all gonna be very similar in how you get the data out to them from this demo. Now, one more thing we wanted to add, and this is a tricky one, it's context propagation. It's important, very important tracing to do this. So all the trace headers will propagate properly. Kind of already do it for you out of the box. Some languages like Go, you gotta explicitly configure it. But baggage. This is an often, we're not products often, but it is a discussed topic. Even when we were discussing it in our SIG meetings, it turns out that a lot of people perhaps don't use baggage the way it's supposed to be used. If you don't know what baggage is, baggage is a way for you to pass at a telemetry level, additional context, additional information about that request. Unfortunately, some people perhaps are using this to actually pass additional parameters in the request itself and use it downstream, and that's not the right use of baggage. You should be modifying your procedure calls instead. Passing that extra parameter properly when you're calling the various services. Don't attach it to your telemetry. What you attach in baggage should be context to affect your telemetry itself. So we use that and in baggage, all requests coming from the low-generate will be mark synthetic. And then we handle this later on in the front end to understand what to do with the trace. You actually break the trace up in the front end if it's a synthetic trace. We continue that baggage down to alert the rest of the downstream. But effectively what happens is the low-generated traders have their own trace and then we continue a brand new trace from the front end linking back to that low-generated trace. If you come in from the UI in the browser itself, that won't be an issue. You won't see that broken up stuff. You won't see the synthetic stuff at all and we maybe don't do the baggage stuff. So proper use of baggage kind of want to encourage people to don't use baggage to do things that you should be doing in your procedure call signatures instead. Use baggage to influence your telemetry. And that's what we want to do. Also all metrics will include trace exemplars. We're appropriate so you could use those in any platform that supports them. With that, we check real quick here for any more questions here in the chat. I see Taylor, thank you for dropping in the link to the repo in there. I see a question, which profile to use engineer DevOps? This is for all engineers, open telemetry. QA engineers, the operators, developers, platform engineers, SREs. We all benefit from additional telemetry in our production apps. And open telemetry is made to do that so that you could send it to whatever tooling you have to understand what's running in production. Doesn't even work. So one use case I like to use often with open telemetry when it comes to distributed tracing in particular when you add the right context is you can find those really gnarly bugs. I say this often, but things like maybe iOS version 14.1 using the French language pack located in Germany and that for whatever reason has a problem loading the resources because last week somebody made a change to the resource loader. Open telemetry provides you the context and if you have the right tooling in place you could find those kinds of really gnarly bugs in your applications. So moving on, let's get talking about deploying the actual open telemetry demo itself. So part of this, we wanted to make it really, really easy to deploy. It is a microservices application. So we got to think about I need to deploy a lot of things together. Naturally, we have two choices here when it comes to containerizing stuff. You have Docker and Kubernetes. We provide both avenues. There's a lot of services in here. So if you're just gonna spin this up in Docker you need to configure Docker to have at least eight gigabytes of memory. You could probably get away with six but I don't recommend it. Give it memory, okay? You're running 13 services. It's not a 13 plus Redis plus a Postgres. So really you're running 15 containers plus an open telemetry collector, 16 containers to make this thing work. Make sure you have the memory for it. It's simple, get clone, CD into the folder and then Docker compose up. If you do not pass in dash dash notion build this could take up to up and above 20 minutes built. If you're building it all from source it will take a while. It's working. It's just gonna turn away for a while. Definitely something you wanna run before you head out for lunch and come back. We provided a way where we give images, we publish images so you could run this without having to go build the images yourself, pass them, you know build flag and you'll get the latest images that we published to spin up. I wanna also really strongly note here this is Docker space compose not Docker dash compose. Docker made a change in how compose works and they're effectively deprecating Docker dash compose. Compose is now part of the Docker binary itself. Docker space compose. There are new features part of this that we take advantage of inside of this demo. So you have to use Docker space compose otherwise you may get in there. Docs matter, right? I said earlier, we're going to well document everything. There are documents about this as well within the actual repo itself under kind of docs folder. You'll see a deployment for Docker there. We're gonna do this in a little bit. I'm gonna show it off, how we're gonna do it. We're also gonna do this. This is really great. You need some 3.0. I don't know what version of Kubernetes you need. I'm sorry. I don't think we do anything special so you could probably go way back in Kubernetes days and it'll still work though. You should be running the supported version at this point anyhow. Add the open telemetry repo. Okay, I've not already done so. And then tell them install the actual demo itself and it's default settings will get it up and running for you. When the install is done, it'll give you a couple of commands you could use to do cube cuddle proxy to port forward the right port so you can actually access the demo. Another option is you can really deploy ingress routes if you want to so you could provide a way for you to access that demo through a web browser. The home chart does have several configuration options. There is also currently as of right now, a very tiny bug in the home chart in that the email service will not export. It spans by default. We're aware of it. We've just found it this morning. I'm confident team. We'll have a PR. We'll have that fixed shortly. I'm gonna show you how to get around it today but I'm gonna guess by the end of this week it'll be resolved because we are pretty fast moving on this whole entire world. So I've done a lot of talking, a lot of theory, I guess. I'm done with my slides. This is not for me to talk to you about slides. It's for me to show you how to do this stuff and how to run it in code and take a look at it. Before I do that, if there's any kind of wild questions or anything like that, please drop them in the chat. I'll keep an eye on chat here, moving forward. But let's go ahead and dive in to what this thing actually is. So first off, this is a repo for it right here. It's already been linked from a kind of structured perspective. Our Remy is really just that same service map you seen earlier with the legend and links to, again, documentation. We're really all about the docs here and all of these links are all contained within this docs folder up here. Inside of this docs folder, you got some screenshots, you got our deployment stuff. We're gonna have feature flags in this. You might have seen the feature flag service running. The idea here is we're gonna introduce some scenarios where they can break. Now by default, the feature flags are all disabled. You have to actually go into the feature flag UI and enable them yourself or perhaps modify a fork where you just enable it in your fork. So things that are coming out, you can understand them. Right now, we define two feature flags. Although just one of them actually does anything. We're working on it. We'll get more. We're gonna build up some more scenarios in here, some more breaking ways, if you will. If anybody's wondering, we're just the feature flag service is written by Tristan for those of you who know who he is in the community and Elixir, of course, because that's what he's wonderful at. It's a pretty simple Phoenix application and Elixir and it just serves up some flags and writes them through Ecto, I believe it's called, into a Postgres newbie. We're gonna actually look at this UI here a little bit. Other parts of this, of course, is the Docker deployment. I talk about the Kubernetes deployment. And I like to talk about these two tables right here. I'm gonna start off with the trace service features first off. I'm really proud of this table. We've actually accomplished all of our goals in terms of tracing coverage thus far because we don't see any more construction signs anywhere on this table. We can't do certain things like for C++ and Rust, there are no instrumentation libraries, so we don't do those. And we just didn't do manual spans in this Go service down here because we did them in this Go service up there. It's kind of all of those things are. And we'll talk about where the RPC context propagation doesn't exist for you. So you gotta mainly specify and go in C++ again. And again, I mentioned where we do the span links as well as baggage stuff. I'll show you how that all that works later on. For metrics, the coverage is much lower right now. We're working on it as we speak. There are PRs available to get stuff of this done. I am actually looking at this table and it may be not quite as up to date. So we'll make sure we clean this up here moving forward. But that is our goals to make all the construction signs go away, either replace them with completed or we're just not gonna do it for any particular reason. With that, I'm gonna drop into the last one here and we're gonna do this for any manual tags as well. Part of this is you're going to add attributes, manual attributes. The thing about distributed tracing is that you get good value with automated instrumentation libraries and automatic trace and instrumentation. But where you get a lot of value, where you go over that chasm and really benefit from it is when you all add context from your application, which only you know as the developer. I believe there is an OTEP right now that you should say, all of your own context should be prefixed with app.t. We did that here in our demo. So everyone of our attributes start with app.t. And although it may seem like it's app.t service name and an attribute name, that is not of it. It's app.context of what this thing is and where it comes from. And these are the ones that add service lab, these are the ones that cart service lab. So you see cart adds a full product, app.product.id. And that's kind of a schema that works well with many organizations. We would recommend you do the same thing for yours. And we just document what they all are. So when you're using this thing and when you're searching for specific attributes, remember where they came from. And you know why we set them in there and what kind of values they will contain. So that's a lot. I've said a lot. I'm gonna dive in a little bit of code right now and show how this thing works. So you can see how I stopped this just not too long ago. I'm actually run this in Docker compose. Oh, dash, dash, no build. And that's it, I'm just gonna run this. If you have not done this before, it'll go ahead and pull all those images down. So it'll take a few minutes for it to probably pull all your images, and then it's gonna start up. And you're gonna be off to the races and running. There are various UIs that are made available to you for you to go click on and go explore what is going on in here. And when you run Docker compose, we take care of exposing all those ports for you with the Docker compose command. So what are those URLs? If you will, I've got them all bookmarked underneath my thing right here. But you know what, we documented them for you as well. And they're all available right here. Oh, I'm sorry, I went to the wrong spot. I want to go to my Docker deployment and they're all gonna be documented right here for you. Web Store, Jagger, Prometheus front end, the Grafana front end, the feature flags UI, the low generator UI. Yes, we put a UI in the low generator because Locust is great like that for you. So I've got this running, right? Let's go look at what it is first off. So I'm gonna go to the demo app. I'm actually gonna create a new tab here. And we're just gonna go to the demo app. This is it right here. It's, if you've seen online boutique from Google Cloud Project, you might seem a little familiar. I want to say, this front end has been rewritten. I want to call out to Oscar Reyes who rewrote the entire front end and next JS, gave it a gorgeous, modern touch to it. It's now an SPA contributor now. He's part of our team. He's on all our SIG meetings, really phenomenal. The work that was done here for this. He's also done other stuff for us to do integration tests, but I'm particularly really proud of this. Gorgeous looking UI we've got here. You can see the OpenTel Venture inspired products if you will. I'm gonna go ahead and just click on one of these. I'm gonna add a chunk load of these on Canadian. So let's go ahead and find the Canadian currency code. They're not super cheap. Add them to cart, you can see them. I'll make a $2,400 purchase, no big deal. I'm sure there will be okay with that, but we're off. And I've generated data inside of my environment. Now, at the same time, we're also generating data through a load generator. So I didn't need to do all that stuff. I can go ahead and go find my traces that I wanted to. Remember I said we're running a load generator. So we could go look, see that load generator UI. I'm just gonna go right into Yeager right away and just show off. Hey, okay, I got 13 services going on here. Here are all the various services you wanna look at. I'm just gonna pull up front end and I'm gonna look for particularly the place order endpoint. Place order hits a lot of services. It's probably the one route that does the most things throughout the entire thing. So you go ahead and find traces on that. And you can see here, I've got 49 spans, 56 spans, depending on them. Some of them will place more products in there than others. And there you go, I've got my trace. It's all up there and running. And at the same time, I'm also setting the state of the honeycomb. Because I said, remember, this is meant to be a vendor. So I'm gonna show off what the vendor looks like to make this happen in the honeycomb. And you might've been looking right here and you might've seen it earlier. Now, inside of every repo, I encourage, if you're a vendor, fork this thing right now. Go and fork it right away. Just fork that repo. Go inside of source, hotel collector. There's a file here called hotel config extras. I'm gonna diff this file so you can see what I've done to it differently. And basically, it's the whole file. I've diffed it, okay? I've completely replaced the whole file. The idea here is this is any additional configuration you need to send us a honeycomb. Yes, that is my API key right there. That's okay. I'm gonna delete this API key when we're done. If you use it, it's gonna send data to like a demo team that we have in Honeycomb. It's fine. But this is your configuration. So if you're a vendor, add in whatever it is that you need to do to send your data into it. Or if you're just wanting to build some cool stuff and you want to use it in your own little thing, go ahead and configure this. Make sure you add your exporters into the pipelines. Right now, we only have a metrics and traces pipelines to find and you're done. And we'll merge that with the prior one. I do want to note that merging configs, maps, merge, arrays overwrite. So I have to re-specify the entire exporters, all the options and exporters here. I'm not re-specifying Prometheus. I could have, probably should have. But it's something to note there when you are doing these config merges is that the maps emerge, but the array elements will be overwritten and that's just the nature of merging the ML. And you also see several outputs here in my console. That's a low generator hitting this thing all the time. So, okay, I'll put this thing in Honeycomb. What does it look like in Honeycomb? Let me get you a screen in there. I'm gonna refresh. There it is, data's coming in. I can click on this. Now, I'm gonna just say, just give me the traces there. Trace.pandid does not exist. That's a fancy way in Honeycomb to say, give me all the roots. I'm gonna group this by root. And run that query. And we get them all right here. I probably only care about these checkout ones. So I'm just gonna filter on checkout right here. And I did that because these are usually the best looking traces that we get out of this whole platform. And you can see it right here. It's called all the different services and all the various stuff and how it works, right? We get span events on these things. This one here has got several span events on it, for example, so receive, get, quote, request, processing it. You can see the span events right there. Because this thing came from the front end, it's also got a span link to it. Span link is really just gonna give me back a low generator trace that started this off. So you kind of get a sense of what's going on and where. And yeah, we've got all the data flowing into Honeycomb. That's pretty cool. Really easy to get going with that. Now I'm gonna stop this Docker one. And we're gonna do the same thing again. It's the same environment, but in Kubernetes. Just give this a second. This is the CNCF livestream. We love coding. One beautiful thing about coding is sometimes we've got to wait for the things to do their things. And, you know, this is where I need my jeopardy music. I really do. I haven't seen any libraries adopt that MP3 while waiting, but, you know, fingers crossed that someone's gonna implement it soon. Right? Okay, that's done. What I wanna do, starting up that, I want to Docker, I wanna really kill this. Docker close down, just shut it down. Okay, it's gonna be moved. It's gone. EKS, or I have an EKS cluster running already. This is my helm configuration. Some of you who are familiar with helm, you can pass in value files when you install a component instead of passing in set and like a bunch of names. I'm gonna recommend you do the same thing for rendering of this. The helm chart today does not allow you to specify image names. That is on something we're gonna do. We're also gonna allow you to do resource limits so vendors can get really fancy with perhaps introducing scenarios where, you know, maybe you wanna add a new feature flag and you wanna have a pod crash. So on a memory leak, well, maybe you only get that pod 25 megabytes. So we're gonna allow you to do all these things. It will be ready for GA at the end of this month. Not quite there yet. Again, I gotta tell this thing, I open telemetry configuration. So this is effectively the same config you've seen before but it's under this open telemetry collector config block instead. And earlier, I mentioned there is a small bug. The email service is not exporting it spans. I get over that by passing in this little component to the helm chart. I'm confident by the end of the week you will not need to do this. I'm doing it myself because I need to get this out there. I want this to work. So first off, helm list, I should say nothing here. All right, so helm install open telemetry demo. This is from the hotel, no, I'm sorry, open dash telemetry and it's this dash demo dash dash values. I'm in the right folder. So I'm just gonna call that file already. And this is gonna go ahead and install it. I'm gonna bring this up a little bit because when it's done installing, it's gonna give you that cool little graphic. I don't know who wrote this graphic but I wanted to thumbs up. That is a really hot graphic, by the way. I'm all for ASCII art. Coming from my old days running a BBS. ASCII art was huge back then. So it's installed, it's up and running. I'm gonna flip over to this screen here. You can see here all the times here, the ages are very recent. Anybody's wondering, this is a tool called K9. It is a CLI based UI for Kubernetes. If you have not heard about it before, I endorse this tool. If not here to tell you to use it, use whatever you're most comfortable with. Absolutely, I love K9s. It's a really great tool. I'll go ahead and search for it. You'll find it's available out there. But it's a great way for you to get operating with Kubernetes. And why did I pull this up as well? Well, I gotta port forward some stuff. All this new is just deployed to Kubernetes, right? As you all know, Kubernetes network is gnarly. As in, nothing's exposed, unless you explicitly tell it to expose. We pop a bunch of services in here, but they're all cluster IP services. Nothing is done to be exposed externally. If you intend to expose anything externally, it's up to you to then write your own load balanced or load bound services or ingress routes to take advantage of the existing service or maybe even deploy additional services to take advantage of it. You would do this above and beyond the standard Helm chart. For vendors, what does this mean to you? Well, you can provide instructions to deploy these things on your own. Or you can sub chart the Helm chart and deploy them on your own. Up to you how you wanna do this and how you wanna communicate it to your users. But that's how you would effectively do it if you wanted to provide a way for them to get additional external access. This is a demo though. So I wanna be careful on how much we really plan to make this exposed. We do give you simple commands, which is really using the kubectl port forward command to get access to the various pieces of the demo UI should you want to do it on your own. And if you're deploying this to maybe a mini cube or you've got your own EKS cluster running or AKS or GKE or whatever you got. Maybe you're doing something on another platform. It's on you at this point. Mine's running in EKS. And they can absolutely run these commands but these are blocking commands and I've got K9s. So I'm just gonna come here and K9s and say, hey, front end, shift F. Yeah, I want port 8080. That's done. That's gonna get forwarded for me. I want the feature flag UI. We're gonna show up that UI and it's kind of here. So in particular, the feature flag UI is running on 8081. Kind of see it right there. So that's the one I want. Let's put that out there. I can expose Jager if I wanted to as well. Really all these other things if you wanna expose them it's all up to you at that point. But I got my front end and my feature flag servers exposed for me. So let's go ahead and check them out. I'm gonna first go look at the feature flag server. So I've got that bookmarked right here. It's local host port 8081 is what it's gonna be. Oh boy, that's pretty. Wouldn't be a demo if it would work. That's just how you know it's real. Oh my goodness. I'm just gonna check. Let's look at some logs on this. Feature flag server. I know why it's broken. And this is great. I'm so happy I hit that error. This happens. We're gonna fix it. We know about it. We gotta do something about it. The feature flag servers depends on Postgres. I did a clean deploy. The feature flag service started before Postgres started. And I'm gonna go in here and I'm gonna see a bunch of SQL errors because well, yes, Postgres just doesn't have the darn database. How do I fix this? I'm just gonna restart the pod. It's that simple. So I'm gonna delete it. Kubernetes does this thing. One of the best schedulers in the world. Resorts pod. This will work. I will need to remap my port four. This port four is gonna get terminated on me. So I'm just gonna re-specify it. And this is how you know it's a live demo. No. It took a second there for it to release a prior for it. So let's try this again. Two feature flags. Product catalog and shipping. This one here is implemented. This one here not yet. You put this on. Nothing's gonna happen. What happens with this one? Really, it's meant for a very specific product code. Some like O, L, J or O, J something to product code. If the product catalog service is trying to get that product code, get the details about it, it's gonna check the feature flag service first. And the feature flag, this feature flag comes back as enabled. The product catalog service is gonna spit back an error. Awesome. Love that. I'm not gonna show this off first. Well, actually let's flip this on. We'll go look at some traces here in a second. So it's on. Oh my goodness, I made errors happen. I'm gonna come to Honeycomb real quickly. First, we're gonna go back to this little world here. It's gonna run this query again for the last 10 minutes. You can see here, I stopped it in Kubernetes. I restarted it in Kubernetes right there. None of these here will show error equals true. At least not recently because error equals true is gonna be very recent and they're all right here. These are the ones that now I flipped on that product catalog flag. So when this is going out to go get this product to ID, which is randomly selected from a group of like 10 or something like that, the feature flag service will return true. We bubble up the errors all the way up the pipe. This is one of those demo scenarios. So I'm kind of targeting you observability vendors out there in the world. If you wanna play around with, what does an open telemetry error looks like? We did it for you. This is a great example of it right here of how you could understand what's happening. I believe we also spit out an event here as well to say, hey, we're failing on this because the feature flag was enabled. We do plan on introducing another feature flag, which is gonna actually create a latency, potentially even a memory leak. And the hope here is that we will allow vendors to make use of their tools to really discover why it happened. In that case there, it'll be a unique gnarly little situation where it's a specific user doing a specific thing and causing memory to leak higher. So some fun ways and things to think about. So again, feature flag service is meant to for you to enable and disable different kind of usage and demo scenarios. We sometimes call them hero scenarios within the actual platform itself. Now I'm gonna show off some code of how we made this all work, how we put this together. I'm not gonna show off code for every single service for every single thing. I kind of wanna focus more on the not so popular things. The things that I want people to really focus on when you're running this application, when you're using it yourself. So let me close a couple of these. I'm gonna actually start off, we're gonna go into the front end service. So the front end service has code to enable telemetry. Of course it should. It will actually has two tracers part of it. It's got a back end tracer, which is used by the front end's next node service part itself. And it's got a front end tracer used by the browser. I'm gonna show off the front end tracer first. See a question here about introducing fault injection. I'm not clear what you mean by that David. What we're trying to do is we're trying to make it with a feature flag. So if the feature flag returns the fault, then the service itself is just gonna explode. It's just gonna return an error. It's how we did it. So we're kind of injecting an error based on the feature flag's status. And we're doing that in code. So this here is instrumentation for the front end. It's a little wordier than node back end. Some of that has to do because we gotta do some cores URL stuff for the web auto instrumentation. We need to kind of wrap things up a little bit more manually. You don't have all the environment variables available to you in the front end. So you gotta specify a few more things. When you do this from a back end perspective, you don't need to specify much at all. In fact, this here is not actually, this is the instrumentation for the back end. And then we just wrapped it up with something called backend tracer to make our own APIs easier to use. But this is how you boots throughout the back end. That's it. There's not much to this. And we'll show you the payment service. Very similar to this here. How you do it is when you start up a service, you do a required module. So node is not an agent per se, but required module makes it like an agent. David, your question about chaos monkeys and the tools around that or chaos engineering, we thought about it. We did. It's kind of a big task, maybe too heavy handed for what we were trying to do. And we wanted to not have to introduce other dependencies into the stack if possible. So yes, it was thought about, but we were more focused on kind of try to keep a little bit one nimble than having to introduce additional dependencies. So it's kind of node.js. And then from there, what we did inside the middleware is where we did a lot of kind of magic, if you will. Now I want to discuss this, because this is where we look at baggage. So baggage is actually added at the low generator level. It'll add a piece of baggage. And all the low generator does, it has this baggage called synthetic request. They're gonna have a value of true. Okay. And here you have to use true string, not true boolean because baggage is passed in as an HTTP header. It's just a really long string. So everything here comes down back to a spring in all the properties, everything else. So we're gonna be looking for, we're gonna pull baggage out when the request comes in on everything within our API handlers. So this is effectively a next.js middleware. And we're gonna look to see if it's got that baggage. If it does, we are going to effectively create a brand new trace. We're gonna stop the existing, we're gonna create, we're just gonna create brand new span with no prior context. And we're gonna put this one into context. We're gonna make this one active. So it doesn't have a parent effectively. And we're gonna link it to the prior span. Note here, span links are really powerful to help you do these things. But you have to specify the span link at span creation time. This is part of the spec that's across all languages. I've heard people ask, I don't know the link until after my span is created. Yes, I get that. But you have to specify it at span creation time because it needs to be mutable. If you need to do bi-directional linking and then you would have to create an additional span on the kind of the one that you're being linked to and initially, and then add the details, the context from that one there. It's gonna be a little more manual code, can be done. I've seen customers do this. But this is basically what you wanna do. We're gonna add an attribute here at that standard request equals true because we're gonna check that attribute later on. And sorry, we're not gonna check this attribute. We're gonna add the baggage header. We're gonna push the baggage, we're gonna let the baggage continue because we're gonna check that baggage later on inside the payment service. So this is kind of how to use baggage, how to make it work. Why we did it in the payment service is, I will just show what we kind of did. So the payment service here and it's kind of charge. Right here, we check baggage again. And if it has a synthetic entry and it is equal to true, we just don't actually charge. We send attribute to charge or not charge. These are the kinds of things you should be using baggage for. Please, please, please do not use baggage to actually pass context to a downstream service. That's not what it's for. It's, or I should say application context or downstream service. You wanna pass telemetry context and keep it to that there. Things like synthetic request. So that was kind of getting up and running with Node.js, the front end. I wanted to really discuss that because the front end is unique in that it has two different tracers, one for the browser, one for the Node.js part. The payment service is a pure Node.js only service. And you can see here, this is kind of the idiomatic way of how you spin up and how you add instrumentation, open telemetry instrumentation to your Node service. You will just then do a dash dash require referencing tracing.js file when you start up your Node application and boom, they're off to the races. This will instrument most well-known frameworks, though the NPM world for open telemetry is growing rapidly. You could add additional instrumentations to here as well. You don't have to add a little bit of code to do that. A few other things I kind of really want to touch hard on here and really make sure everybody kind of got was I wanted to just cover a little bit of the kind of the known things about Ruby in particular. Ruby's different, Ruby's doesn't do GRPC, right? Ruby's different in that area there. So for Ruby in particular, we need to pass in, I'm sorry, I want to pull up the doctor's post for this instead, we need to pass in the HTTP port to export the traces. So if you're playing with this, you're using this and you're like, why did that happen? That's why Ruby just doesn't support GRPC for export of its open telemetry data and therefore we need to specify that way there. I don't have many other things that are really important. I want to make sure people capture with documented them all. I should perhaps show some of that documentation. I did not show our service specific documentation. I think it probably makes sense here as well. But under docs, we have services here. Every service will be documented. They're not all done yet, but we're getting there. For example, the checkout services, probably the most busiest of all services, we go through how to initialize the tracing provider, how to do it right. These are the idiomatic ways to do this and we do it for every language, for every service. And we show you how we do it. How do we wrap the GRPC calls with auto-insubmutation? How did we take care of the outbound calls? So these are the in-bounds and these are the out-bounds. How do you get a span? How do you add attributes? What about span events? All that is shown off here. They will all be documented and as you continue moving forward because part of the goals of this project is to be well documented. You will see us put them in here. And if we're missing something, please, please, please, create an issue. We're gonna get to it right away. I'm gonna end it there, talking quite a bit. If anybody wants to ask any further questions, I'm happy maybe to dive down and show you how we did that. Thank you so much, Pierre. If you do have questions, please feel free to throw them in the chat where you're at, whether it's YouTube, LinkedIn, or anywhere else that you're watching this. I'm sure that we have some way of getting your feedback. Please, please, please, feel free to ask those questions and we'll get those answered. I think one question that I have is what are some meaningful ways in which we can help kind of contribute back? I'm sure, obviously, being a demo, the biggest thing is going to be testing it and actually kind of poking at it and really taking a deeper look into some of these different aspects. But is there anything you think needs a little bit more testing or really that you're excited to see about this project really coming up? I would love for people to try this in more scenarios right straight up right now. I would love somebody to stand us up and host it. Let me know how the browser parts work because cores is an issue. It cores as a thing. I don't think we've well tested out the whole cores aspect. And I know once this thing goes on a web browser somewhere out now, the wild, wild web world, probably gonna break. So we'd love to have feedback on that one. I think we need a host. I'm sure others will go ahead and take care of that. But that's the first one. And another one, I made a call out, we're looking for sweat developers. If you can help contribute, I think we've decided not to move forward with Swift for our GA release, but it is something we're gonna tackle immediately post GA. I really like all of the languages that you were able to pick and source around this too, like seeing Rust, C++, Swift, all of those languages. I've done some work in Elixir in the past as well. So really fun to kind of see all those different contexts come together to give you that visibility that you just normally would not get on each of these, on their own with each of these languages. So major kudos on that front, that's awesome. Yeah, major kudos to all the contributors for that as well. We could not have done this without people in the community who knew those languages. We're familiar with how to instrument them. Just walked up and said, here you go. I rewrote this service for you in that language and they implemented the proto and they're off to the races. So it's pretty handy. All right, did we get any questions on the other various channels or are we? No, I think those are all the questions that we have today. So with that, I would like to wrap and give you a little bit of time back, but truly thank you so much, Pierre. Do you have any parting words of wisdom, affirmations or any fun code, snippets to share with us before you go? Because of this project, I learned as a Python agent. It's interesting when you run into this. I've been part of the open college community for a long time. I had no idea a Python had an agent. Although I think the agent was really new when it came out, but it's an auto instrument or thing and it's really handy to use. Please use it to write Python code. It's really, really easy to use and we encourage you to try it out. And we also use it for our load generator and everything else. So it's kind of one of those weird things that just happened to me because of this whole project. Interesting. I didn't know that either. I'd almost want to call it a secret agent, even though I know it doesn't work for any three letter acronyms. Python 007, maybe. Great, awesome. Well, awesome. Thank you so much to CNCF for hosting this live stream. Please join us at Slack. If you're going to be going through the wonderful conference in the month that we are going to have an open telemetry community day on the 24th, we're going to be really pumping it up. You'll hear stuff. I'm going to slip it out. Hopefully, Austin doesn't get mad at me, but it'll be called Hotel Unplugged. Really excited about this and I hope to meet you all there in person. It'll be great. It's going to be fantastic. I can't wait to see all in Detroit those view that will make it. And thank you all for joining us. So I put my background right there. That's where we're going to be at the end of the month. Yay. It's going to be so fun. Awesome. Thank you, everybody. I appreciate it very much. Awesome. Thank you, everyone, for joining the latest episode of Cloud Native Live. We enjoyed all of the interactions and questions that you had today. Thanks for joining and we can't wait to see you again soon. If not online, hopefully at one of the upcoming cube cons. Thank you very much, everybody. Have a good one.