 Yeah, gonna dive into this pretty quickly and try and do, I think I was given about an hour and a half. I think is what we're set for time, so we'll try to get through here pretty quickly. And there we go, all right. So like software delivery itself has never been like more critical to success of an organization in a business and pretty much every industry. You know, it's what really kinda sets, you know, how well you can perform in that, really sets you apart from the other organizations and other companies that are out there. But those expectations for like delivering something that's high quality have only gotten higher. It used to be that there'd be the smaller groups of people that would have a really high performing team who got the Netflixes and such. And now that's become the norm. And so those expectations have made things a whole lot more difficult, I think, for a lot of engineering teams. But also at the same time, like the landscape of those tools has gotten bigger and bigger. Like this is the CNCF kind of cloud puzzle. There's probably one way to look at it. And in fact, somebody should make this into a puzzle. It's like more and more complex. And this keeps changing, more and more things keep getting added to this. So how are developer teams supposed to kind of figure out what is really that necessity? What do they really need to succeed and beat out the competition? So hi, ladies and gentlemen, my name's Jeremy. I am a Devereux and Community Professional. The reason it says independent is because I am looking for a new job. So if you wanna talk about that and what that means, we can chat later. Nice little plug and you can find me on Twitter. All right, so in this period of immense change that we have and the global economic uncertainties that we've all kind of felt over, especially over the last couple years has presented a lot of impacts for companies of all sizes, whether it's startups all the way up to enterprises even. But in this environment that we have, the stability and reliability itself has become increasingly important to businesses. The ability to provide that reliable, stable platform to customers has become that critical kind of value metric for how engineering teams get measured and it's only gonna continue to increase as we continue to see things change over the coming months and years. So one of the core tenants of DevOps and a solid pipeline is really this idea of continuous delivery. It's why we're here. It's a CD foundation. If you didn't know, CD stands for Continuous Delivery Foundation. That's why we're here. This is just a very core principle about even what DevOps is. And so a couple of weeks ago, I put out this question of what does continuous delivery mean? How somebody might wanna define that. Anyone wanna take a quick stab? Everybody's still asleep? Good, all right. Well, I got one response. Tracy Miranda, fantastic person. She provided just a really good definition around it's the software development practice in which teams release software that changes software changes to users safely, quickly, and sustainably. It's a really great definition. And it goes very well with what Jezz Humble had to say a few years ago at DevOps Day Seattle, where he talked about like it's that the ability to get changes of all types, including new features, new bugs, or new bug fixes, getting all those things, those experiments into production are into the hands of users safely and as quickly as possible in a sustainable way. In a recent talk given at Fintech, I think it was Fintech DevCon, Charity Majors, who's a CEO at Honeycomb, talked, or CTO at Honeycomb, talked about like a list of these modern development practices. And with the continuous deployment, kind of continuous delivery point, she kind of focused heavily on the importance of deploying as fast as possible and as automated as possible. And so over the last few years, we have seen a jump in companies that are starting to understand that importance that Charity spoke about, but too often we see that's been done with kind of this grow at all cost mentality. And when that gets in place, that's kind of starts introducing more and more challenges and we've seen the cognitive load necessary to keep those systems going and in place has gotten more and more overwhelming for DevOps, Dev and Ops teams in order to put that together. And it's led to a lot of cut corners. I think we've all seen, at least it seems like that we've had more and more outages across a lot of the core systems that we use in the industry over the last few years. And I think it's a direct result of this kind of grow at all cost mentality. So some of you probably familiar of the different kind of industry reports that are out there. There's a state of DevOps reports, state of cloud native, CDF, has their state of continuous delivery report. And they're all kind of great for understanding kind of the industry as a whole, but they also are dealing more on survey data. And so what I'm gonna talk about is based off of a report that was done, it's the state of software delivery report that was done by a company called CircleCI that kind of dealt with actual raw data and kind of able to take that along with what some of these other reports go on survey data and understand actually what's happening with actual verifiable anonymized data. So before we kind of even get dive into some of these metrics, I wanna stress like there is no one size fits all for metrics because everything depends, which is a common DevOps answer, it depends, depends on what you as a company are doing. What are you building? Who are your customers? Is it more internals, more external? Like there's a lot of things that depend on that. And so there really is not a one size fits all. So kind of the four that we're gonna deal with are duration, mean time to resolve, success rate, and then throughput. Those are the four that have been identified as kind of CICD benchmarks for what a high performing team would look like. So and then I'm also gonna talk about the optimization steps, some of those that would be necessary in order to take that strategy and input a lot of these things in place in order to get to that kind of the holy grail kind of way. And hopefully we'll do it without getting concussions as well. All right, so duration as first one here, it really is that foundation of what like a software engineering velocity would look like. But this unit of work that's kind of spoken about in the definition doesn't always mean that you're deploying to production. That could be something as simple as running a few unit tests to kind of get you things on a development branch that you can then start to make some other changes. So it's not just a deploy to production. And so it's kind of probably the best way of viewing this is as kind of this proxy for how efficiently your pipelines are delivering the feedback on the health and quality of your code that's so essential to your development teams to be able to make the changes necessary. And the core kind of promise of most software delivery paradigms, everything from agile, DevOps, it's all speed. And so that ability to take the information you get from your customers or stakeholders and respond quickly and effectively is at a core part of the duration. And this rapid feedback and delivery cycle itself is also just, it doesn't just benefit your organizations and users, it's also gonna add benefit to keeping your developers happy, engaged in kind of this uninterrupted state of flow. But when we exclusively kind of focus on speed, often comes at kind of the expenses of stability. So a pipeline that gets kind of optimized to deliver fast, unverified changes is really nothing more than a highly efficient way of delivering bugs to your system. So it's important to kind of make sure your organization is not exposed to a lot of that risk and your customers don't feel that, is you've gotta be able to move quickly with confidence. So having your pipeline guarded against any potential points of failure, it's important to have a robust test suite. So we'll kind of talk a little bit about that as well. All right, so the duration of benchmark, good rule of thumb that Paul Duvall and his work in 2007 kind of set the stage is keeping your builds no more than 10 minutes. Many developers who use CI follow that practice. And so it kind of keeps you uninterrupted. Anything longer than that can kind of interrupt the flow. That's probably about enough time that once you hit push and things start building, you go grab a cup of coffee, come back to see what the results are, as opposed to maybe 30 minutes or longer and now you're losing kind of that attention span of your developer teams. So among the workflows that were observed in the data set from this report, 50% of them completed in 3.3 minutes or less, 25% were completed in under a minute and 75% under nine. But then the 95th percentile, which isn't reflected here, but it was right around 11 minutes, which did reflect a lot more of the 95th percentile. A lot of those workflows are taking 27 minutes or more to complete. So many teams, when we think about ways to improve this, many teams are still kind of continuing to bias towards speed rather than robust testing. So the number one important kind of thing you need to be looking at when you want to improve this performance across all four of these metrics is enhancing your test suites with more robust test coverage. So that's, you know, there's a number of things here. I'd say one of the biggest things on this is like code coverage and the pipelines identifying any of those inadequate testing spots that you can help alleviate some of the potential challenges from pushing fixes that weren't adequately tested. So, you know, balancing workflow and, you know, test coverage that prioritization is gonna have to take center stage. So, you know, that means asking, you know, which of these features are really, you know, part of that critical path that we need to make sure that we're putting in? Where can you afford more, you know, experimentation, maybe on different pipelines? So all these techniques kind of here on optimizing, they're really kind of helping getting better buy-in from, you know, a lot of your important stakeholders on that right balance between, you know, test coverage and workflow speed. It's okay to have a longer workflow than 10 minutes if you're getting good quality testing feedback. It's also important to know that like when you think about like duration for builds, it's gonna depend on what you're building. It might take longer if you're having to compile something, it's a big application, you're having to compile it as opposed to you're just kind of pushing some website changes. So understanding kind of what your builds are and setting your metric accordingly. So yeah, so teams that are focusing only on speed, they're not only gonna spend more time kind of rolling back all of those broken updates and debugging in production, but they're also gonna face kind of this greater risk to their org's reputation and stability. So optimizing those workflow durations means you're really combining all of these good test practices with efficient workflow and you all get to dance together. All right, so meantime to resolve. This is a metric, it's indicative of your team's resilience and your ability to respond quickly, effectively to feedback from your CI system. So it means it really is that best indicator of how mature your org's DevOps kind of adoption is. It's a really quick way to see that. You know, the way that users in most organizations kind of see this is nothing's more important than your team's ability to recover from a broken build. So customers are definitely more likely to, they're definitely more unlikely to notice any of those kind of incremental small changes that steady stream that kind of happens maybe throughout the week, but they're definitely gonna notice when they can't access the website or access that application. So that keeping the meantime to resolve, keeping those errors from the system is any recovering from broken builds is extremely important. Martin Fowler kind of talked about this, the quickest way is just to get back to the last known green build to start with and then you can adjust from there. And the whole point of working with continuous integration and ultimately that helps deliver that continuous deployment continuous delivery mindset is you're always developing on the lat a known stable base. So getting back to that green build is extremely important there. It is recommended aiming to fix broken builds on any branch in under 60 minutes. It's kind of what the data shows is what that high performing teams look like. So depending on the scale of like that user base that you have and how critical that application is, your target may be significantly lower or higher, but this is kind of that sweet spot of what it looks like for a high performing team to be in that spot. It's gonna allow your organization also to avoid like the worst outcomes of prolonged failures. So when looking at the data from the report, it showed that 50% recovery was kind of happening in 64 minutes or less, which was an improvement from previous reports, which was about 10 minutes longer. The top 25% was in less than 15 minutes and 50% of them, or I guess the top five, fastest five was in less than five minutes. So that's typically you're seeing that with smaller, obviously smaller builds, but probably less feature driven applications. So the first kind of step to lowering your recovery time is to treat like the default branch as the lifeblood of your project. It has that kind of main chalice of what everything should be. So your organization is gonna make sure that, while red builds are inevitable, getting your code back to green is always gonna be immediately important and it should be everyone in the organization's top priority. A vigilant culture that you kind of have in place where everybody is poised to leverage, getting back to that green build is gonna provide your best chance of hitting those metrics and reducing the impact on your customers. Recovery speed itself is completely bound with pipeline duration. So the shortest possible time to recovery is going to be the length of your next pipeline run. So if your builds, your duration and you've identified that your acceptable duration maybe isn't six, maybe isn't 10 minutes, maybe it's 15, well your mean time to resolve can't be any less than 15 minutes because the build has to run again so you're never gonna hit that metric. So it's an understanding how that fits together. So along with mean time to resolve success rate is that other kind of indicator of stability within your application development processes. But the impact of your success rate on your customers and developers is really kind of very according to a couple of different factors. One of those is like was the failure on a default or dev branch? Did the workflow involve a deployment which there could be an error in the deployment not really the build? Or how important is that app or service? So that's gonna influence how important in what your success rate when you set your metric is gonna look like. The failed signal isn't necessarily indicative of something that's gone wrong as well or a problem that has to be addressed right now. Far more important though is like your team's ability to take that signal in quickly which comes from that duration metric and then remediate that error effectively which is that mean time to resolve metric. Failed workflows that are delivering fast valuable signals are gonna be far more important than having a passing workflow in the high success rates that have little to no actual information, actionable data for development teams. Maintaining the success rate benchmark of 90% or more on your default branches is a reasonable kind of target that was seen through the report and in previous reports of mainline code of kind of a mission critical application where your changes should be merged only after passing a series of well-defined tests. Keep mentioning tests. It's really important for software development. Topic branch failures are generally less disruptive than those that occur on a default branch so if your development teams are doing a lot of innovative work and they're doing it on dev and kind of topic branches which is where that should be happening your success rate there is gonna matter less because that's where you're testing things or you're trying things out. So having a metric on that you have to hit a certain success rate on your dev branches is not as important as actually having it on your production kind of mainline branch. So looking at the data, the success rate on the default branch was 77% on average with 67% on the non-default branches. The trends over the report data kind of showed that success rates on those defaults have pretty much held steady and so a good standard is that that 90% we're still kind of coming in underneath that but we have seen like the pattern of the non-default branches, the success rate there starting to get higher and so we're starting to see that teams are starting to do a lot more test testing on default branches and doing a lot more development there and less on the production branches which is what you want to see. So throughput is kind of that last metric and traditionally that kind of reflects the number of changes that your developers are making in a 24 hour period. It's useful as a measurement of team flow as it really is tracking how many kind of units of work are moving through your pipelines but when you're performing at or around kind of recommended levels throughput really kind of puts that continuous in what continuous delivery looks like because the higher your throughput the more frequently you're performing work on your application and getting it out ready to be deployed. Of course it also doesn't tell you pretty much anything about the quality of your work that you're performing because it really is important to kind of consider how rich again your test data is and as well as how your performance on those other metrics like success rates and duration to really get that complete picture of how well your teams are performing because as with duration a high throughput score means little if you're frequently deploying broken code. So an ideal kind of throughput benchmark really is it depends. Typical DevOps answer I know but it really is going to be subjective to your organization goals. If you're a large cloud native organization that is deploying a really critical product line you're going to want a far higher levels of throughput than a small team that's managing kind of legacy software or a new startup that's working on a new thing. You're probably not gonna need to be deploying having a throughput of hundreds a day. You're not the Amazons that, what is it? I think thousands, tens of thousands of times they're deploying a day like that's just you're not gonna need that. So it really is going to depend on your organization and how you set those type of work, the resources, all of that. So in the data the kind of the median workflow ran about 1.54 times a day but the top 5% was running seven times per day. So when we look at some of the previous reports and I'll have it aggregated data to kind of show that here in a second but there wasn't much change from the previous reports there. Overall the average project had 2.93 reports that were run, I'm sorry pipeline runs in 2023 which was a little bit more than 2022 but while it wasn't much when you look at the sharp decline in meantime to resolve it is reflecting teams are committing to smaller more frequent changes to limit a lot of the impact that is had to the organization and to the customer as well. All right and throughput is really like it is the most dependent on the other metrics from any of the others that we kind of talked about. How long your workflows take to complete. If you're doing something that's gonna take you, it maybe takes an hour to compile your build which if that's the case then that's your metric. You can't do 30 deployments in a day, just statistically not possible. So understanding what that benchmark is for your org is gonna be extremely important when you look at all these metrics first. So improving that teams throughput, you really have to first address all of those other underlying things before you can get to how successful you're getting on your deployments. It is also a trailing indicator of a lot of the other changes you have within your processes and environment. So rather than setting just some arbitrary, pick a number out of a hat, you really gotta have something that reflects your business goals, your needs, capture that baseline across all of these metrics to understand how you're currently performing and then make the necessary adjustments. Achieving that right level of throughput really means staying ahead of a lot of the customer needs as well. So really understanding your customer and have a good feel for your product. All right, so when we look at all the data, you can kind of see these metrics over the last few years, see where some of the change is and when you measure and then optimize these metrics, your teams and your company are gonna gain kind of a really tremendous advantage over organizations that don't which can really set you apart and have that high performing engineering team that you really, everybody really wants to have right now. So you can get the report. This is a link you can go out, either QR code it or type it in there at the bottom and you can kind of get the report. You don't have to sign up for anything, just hit the download button that CircleCI has out there and definitely recommend. I'm gonna post these slides as well so that you'll be able to come back and reference them. Okay, so thank you, appreciate it. And I'm gonna be around the real rest of the day and the next few days, so if you wanna talk, talk about what you're all doing or have some questions around some of these metrics, would be happy to chat with you. So, thank you.