 So, up next, by request, we have the amazing Adriana and the Fabulous Rees, give him a hand. And thank you so much for joining us here today. We are so happy to see such a wonderful crowd, and we are thrilled to be here talking to you about observing the pipelines. Now before we get started, let me tell you a little bit about us. My name is Adriana Villela, and I love solving difficult problems, whether it's as part of my day job, which is a senior developer advocate at ServiceNow Cloud Observability, formally LightStep, try to say that three times fast, or on a rock wall, which is where you'll find me blowing off steam when I'm not at work. I checked out a local gym today, super fun. Among other things, I am a CNCF ambassador and a second year HashiPorp ambassador. I work with Rees in the hotel and user working group. I'm a tech blogger, podcaster. I have been doing computer things for 30 plus years starting at age 10 when I learned basic thanks to my dad, so I guess that makes me pretty old. And oh, geez, I'm Rees. My name is Rees. I'm a senior developer relations engineer at New Relic. I'm involved in the hotel community as part of the end user working group, along with Adriana as well as some other wonderful folk. We work darkly with end users, mostly through monthly discussions, practical sessions, and interviews, and we're always interested in talking to end users about interesting use cases and problems that you faced using OpenTeleptery, so we would love to hear from you if you have anything to share. Also, I enjoy doing tech support things, so figuring out interesting problems and learning about interesting use cases, yeah, and we're excited to be here. All right, well, now that we've got the pleasantries out of the way, let's get to the meaty bits. So we all know that the software of today is orders of magnitude more complicated and then the software of 20 years plus. And of course that means that we have a new set of challenges that come with it. Now fortunately, because of the wonders and magic of observability, we are able to get better insights into how our applications are performing and how to solve those narrowly bugs that we get into prod, right, which is awesome. Now similarly, we also see that the way that we deliver software has evolved. And again, thanks to the magic of DevOps, we are able to embrace the practices of automation and codification of all the things so that we can ultimately shorten the amount of time it takes to get our code into production. So we've gone from quarterly, which I totally remember, starting my career 20 years ago with quarterly releases. We've gone from quarterly to monthly to every two weeks to weekly to daily and sometimes multiple times a day, which is amazing. But when it comes to the observability of our CI CD pipelines, it seems like we haven't made as much progress with that compared to with our application code, which is a problem because our CI CD pipelines are the way in which we deliver our code into production. So when our pipelines are working smoothly, awesome. We're happy, code's making it into prod. But when we encounter a problem, it means that all of a sudden the way in which we get our code into prod, there's a bottleneck. We can't deploy and things go kaka, and we don't want that. So as you may have guessed, today's topic is going to be on the observability of CI CD pipelines. More specifically, we're going to level set with some definitions to make sure that we're all on the same page. We are going to also talk about why being able to observe a pipeline makes a difference, how to make them observable, and finally, the challenges in creating observable pipelines. So we're going to start with some definitions. And there are three definitions that I'm going to cover today. So they're going to be observability, open telemetry, CI CD. I know many of you are familiar with these terms, have probably heard them many times throughout your careers, sick of them to death. But at the same time, I want to make sure that we're on a level playing field, speaking the same terminology, so we shall get started. So observability, or O11y, Olly for short, allows us to understand the system from the outside while allowing us to ask questions without knowing the inner workings of that system. And this is one of our favorite definitions of observability. Might not be yours, but this is the one that we're using for the purposes of this talk. Now, what does this mean? It means that observability allows our systems to emit enough information so that we can follow the breadcrumbs to answer the question, why is this happening? And in order for us to have that information that we can interpret to understand why is this happening, well, we need our good friend, open telemetry. Open telemetry, or O-Tel for short, is our vendor-neutral CNCF open-source framework that generates, ingests, transforms, exports, telemetry data. And the thing that I love about open telemetry is the fact that most of the observability vendors out there have fully embraced open telemetry as a standard. But most importantly, there are companies out there that are using open telemetry like eBay and Sky Scanner, and I think that proves to the actual success of open telemetry is having folks using it. Now, we can't talk about open telemetry without talking about its good friend, the O-Tel Collector, which is our vendor-neutral binary that is used to ingest, transform, and export data so that we can, it's basically a data pipeline of sorts. So it has many functions, but there are multiple, there are three main components of the O-Tel Collector that are important to this talk, which is first we have the receivers, which ingest data, then we have the processors, which are used to transform our data, so things like data obfuscation, add and remove attributes, filter data, sample data, batch data, et cetera. And then finally, you need to send them somewhere for analytics, and that is what our exporters do. So now we've understood open telemetry, observability, final definition, is CI-CD. Now, CI-CD is an automated approach to software delivery that is made up of two components. First, we have continuous integration, or CI, which is all about making sure that we are able to code, package, sorry, code build, test, and package our application code as soon as a code change is made. And then finally, CD, continuous delivery, means getting our code into production in a timely manner. And this is important to us because it allows us to get our features out into the world faster, our bug fixes, and other general updates. And it also provides us with a standardized approach for providing continuous feedback for our developers. So I'll hand this over to Rhys now to continue the conversation. You guys want to see that catwalk along? I'm kidding. So now they're all on the same page with those key concepts. Let's talk about why the observability of CI-CD pipelines matter. So when you have a healthy pipeline, your team is able to write, build, test, and deploy code as well as config changes into production on a continuous basis. You'll also be able to improve or achieve development agility, which means you can change your operations and minimize the amount of time it takes to figure out whether those changes had a positive or negative impact on the health of your application. On the other hand, when your pipeline is unhealthy, you may run into one or more of the following issues. So you may face slow deployments, bug fixes may not get out fast enough to curb user dissatisfaction, and small problems could snowball into critical issues. You may have issues with testing. So having to wait for a test to complete or not having enough time to test against different configuration variables could lead to delay deployments as well as challenges with ensuring that you have good enough application performance across your user base. And you might also have difficulty in determining underlying issues that may be causing technical debt. And pipelines, so while pipelines may not be a production environment that external users interact with, they mostly are a production environment that internal users, so SREs, software engineers, interact with. Having the ability to observe your pipelines means being able to prevent unnecessary long cycle times, which impacts the amount of time it takes to get a commit into production, reducing delay in pushing out new features and bug fixes, as well as reducing wait time for users. CRCD pipelines are also run by code that define how they work. So, and we all know that despite our best and most careful efforts, code can still fail. Making our applications observable helps us make sense of things when something goes wrong. So similarly, when we add observability into our pipelines, it helps us get a better understanding of what's going on when something fails. It also helps make troubleshooting easier by helping us answer questions such as, what failed? Why? Has it happened before? What has failed most frequently? What is the normal run time of our pipeline? Do we have any bottlenecks? And so where are they? And can we shorten the lead time for fixing pipeline issues? So in order to answer those questions, we're going to need to collect certain information about our pipelines. Some of that would be things like branch name, commutia, machine IP, run pipe, so things like scheduled or triggered by merge pull, Jesus. Failed step, step duration, build number. And now I'm going to hand it back to Adriana to talk about how we're going to add observability. Thank you. Awesome. So, you know, we have a pretty good understanding of the type of data that we need, why we need this data, right? How it provides us the observability of pipelines. But while we were doing research for this, we kept running into the same question. Okay, but how? Give me an example. I just want to forget an example. So let's go back to our original definition of observability, where our system emits enough information so we can follow the breadcrumbs to answer the question, why is this happening? And again, how do our systems emit the data through open telemetry? So we know that with our application code, we use open telemetry to enable observability. So it stands to reason that, hey, why don't we use open telemetry to enable the observability of our CI CD pipelines? Awesome. But again, but how? And so you might be wondering, okay, how is it that we enable these things? Oh, that's nice. Slicer a little bit out of order. Damn it. That's fine. So how is it that it is possible for us to enable the observability of our CI CD pipelines? There are a few options out there. So first off, we have SAS vendors that provide observability of CI CD pipeline functionality as part of their products. Some of these examples, some of these vendors include Datadog and Splunk. Now they're not necessarily using open telemetry to enable it, but it's worth noting that there are vendors that are out there doing that. And then you have vendors that have gone on and created some tooling too so that you can pop it into whatever your favorite CI CD tool is to enable the observability of pipeline. And so I want to call out Honeycomb Build Events, which does that, which then allows you to just pop that tool in and then you get some tracing information from your CI CD pipelines. We also have some SAS offerings. So Jenkins and Tecton have open telemetry capabilities. And then, well, you might be wondering, okay, what about GitHub actions? Well, there isn't a native GitHub action for observability of CI CD pipelines. However, there have been some lovely folks in the community who have gone ahead and created their own GitHub actions to enable the observability of GitHub actions pipelines. And then finally, our little spoiler bullet point there, there are also tools that are used within our pipelines that can provide us with the observability of, and further observability of what's going on in our CI CD pipelines. So you might be wondering, okay, what's an example of that? So we have tools like Maven. There's a Maven build hotel extension. There is also Ansible. There's an Ansible plugin. It's called the Hotel Callback. There, if you use PyTest for your Python, for testing your Python code, there is a library called PyTest Hotel, and then finally another one that's super cool, something called Hotel CLI, where if you do any bash scripting in your pipelines, you can add some open telemetry data to your pipelines through this lovely little tool. So that's awesome, but what does it all mean? What exactly is an observable CI CD pipeline look like? Because, so take this example, right? So suppose we have a Jenkins pipeline. We are building some Java code without observability of our CI CD pipeline. Maybe it's working. Maybe it's not. Maybe it looks like it's working, but we don't really know, right? So suppose we enabled some observability into this pipeline. So for example, while we're using Jenkins, so let's use this lovely Jenkins plugin, so that we can get some further insights into what's going on in our pipeline. Awesome. Then in our build step, we are using Maven. So do you know what? Let's use that Maven Hotel plugin, and maybe there's some additional scripting that's happening in bash. So we use that Hotel CLI to get further insights. Well, during the testing phase, there is this lovely, so if you're using Java, obviously you're most likely gonna use JUnit. So then there's this lovely little Maven plugin called the JUnit Jupyter plugin. It's an open source tool, it was developed by Dynatrace. So that's a nice little option to get further insights into your pipeline. When you are packaging up your code and use Artifactory, well, Artifactory generates logs, and there is this lovely little receiver in the Hotel Collector called the Fowlogs receiver, which allows you to tail your logs from the collector so we can gain some further insights when you're building your packages. And then finally, when you're deploying, suppose you are using Ansible to orchestrate your deployments, you can use this Ansible plugin so that you have further insights into your deployment steps, and if you happen to be doing some additional bash scripting as part of your deployment, you can again turn to the Hotel CLI. And then of course, all of this goes into a collector, which ingests the data in the native OTLP format, where it uses the Fowlog receiver, and then there's also this Git receiver as well, which if you're using any of the main Git providers, you can get some further insights into what's going on in your Git provider tools, so that now you have this bigger picture of what's going on with your CI CD pipelines, and of course, all this information should go somewhere, so we wanna send it off to an observability backend. That's awesome. What does this look like in real life? We have shamelessly taken this lovely screenshot from the Hotel Collector contrib repo, which shows what your traces might look like if you were using a Jenkins pipeline alongside, and then also using the Maven Hotel build plugin, so that gives you an idea of what the data looks like. And that was specifically in Yeager, which is both of you recognized anyways. So that is all well and good, but you may have noticed something that we did when we were putting this together, which is that while we agree that adding observability to our CI CD pipelines is important and OpenTelemetry provides several extensions, the space as a whole isn't super well-developed, and also what if you're not using Python or Java? So it's complicated. While we recognize the importance of implementing observability into your CI CD pipelines, there's still a lot of work to be done around standardization. I think there was an OTEP OpenTelemetry enhanced proposal open earlier this year around semantic conventions for CI CD pipelines. It's still open, obviously. I'm not quite sure. There was a comment on it we just saw from last week, so it sounds like things might be picking back up, but it was kind of quiet for a while. So OpenTelemetry isn't built into most of our CI CD tooling, but there are some movements, and they're slow going, but there's some initiatives that are open, and there are also homegrown options that Adriana touched on earlier, and if you're interested in contributing more to OpenTelemetry and this is something of interest, please definitely go see the OpenTelemetry folk at the hotel booth in the Pavilion. I think we have a link to the, yes. Yeah, we do. To there, to the map. So what do we learn today? LV pipeline means we're able to deploy code and config changes into production on a continuous basis, whereas an unhealthy pipeline could lead to slow deployments, testing issues, as well as technical debt. I forgot the last one. And adding observability to our pipelines can help us resolve some of these issues. We can use OpenTelemetry to add observability today, but options are limited at this time. However, things appear to be moving in their direction, and again, if you are interested in contributing, definitely please come to the OpenTelemetry booth and stay tuned. To wrap up, we would like to thank Mike Kitten, Taco for being such an amazing model. Hopefully you all enjoyed watching. And we have a few things. Oh, these are some handy resources that we referenced for our talk. And I believe the slide deck will be available online, but I'll leave it up since for some pictures. Okay, hold on one more second. And while you're here. Oh, stop. I don't look for my slides. While you're here, check out the hotel booth in the Project Pavilion, sorry, I can't read. I can't read sideways, can't read my slides. Also check out the ServiceNow booth and learn about ServiceNow Cloud Observability. We are in booth N26. And the hotel booth. In the Project Pavilion. The Project Pavilion, yes, okay. And yes, there's also a party tonight. New Relic is hosting with Honeycomb and Observer IQ, so please come by. The event brought my seat, it's sold out, but it's not, just come. Bring your badge. If you're coming, yay, if you were planning on coming, it's sold out. It says it's sold out, but it's not, please. Tell them they're with you. Tell them they're with you. Oh yeah, just say you're with us. But I think you have to bring your badge, so make sure you have your badge. And I think, oh, yes, and then your podcast. Oh yeah, I have a podcast called Geeking Out. I produce it with my 15 year old daughter. She does all the video editing and she designed the logo. And Reese was on one of the very first episodes, so extra reason to check it out. Yes. And finally, check us out on our socials. We would love to continue the conversation with you and to connect with y'all. Thank you so much. We really appreciate it. Thank you.