 I'd like to thank everyone joining us. Welcome to today's CNCF webinar, DevOps from a Different Data Set, what 11 million workflows reveal about high performing teams. I'm Liddy Schultz. I'll be moderating today's webinar. We'd like to welcome our presenter today, Michael Stonkey, VP of Platform at CircleCI, and Ron Powell, Technical Content Marketing Manager at CircleCI. Couple housekeeping items before we get started. During the webinar, you're not able to talk as an attendee, but there is a Q&A box at the bottom of your screen. Separate from the chat box, there's a Q&A box. That's where you want to go. Please feel free to drop your questions in there, and we'll get to as many as we can at the end. This is an official webinar of the CNCF, and as such, is subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that would be in violation of that Code of Conduct. Basically, be respectful of all your fellow participants and presenters. Please also note that reporting and slides will be posted later today to the CNCF webinar page at cncf.io slash webinars. With that, I will hand it over to Michael and Ron to kick off today's presentation. All right, thank you so much. Well, let's get started, I guess. That's probably the best thing to jump into. So I want to talk about kind of a giant dataset for how we analyze performance and what we see in terms of teams and the industry as a whole. And so this is going to cover basically three different sections where we have kind of the setup a little bit about the data and how we gathered that data. And then we have the data itself, which is kind of the readout of here's what we found. And then we have insights based on what you found in the data. What can you infer from that? What can you correlate with it? What can you kind of extrapolate from the data that we see? So from a setup perspective, what are we talking about and how does this work? Well, there's a lot of data out here on DevOps and team performance and things like that. And a lot of this started from Puppet's work starting in 2011, coming all the way up through 2020, I believe the 2020 report actually came out today. So, you know, and I've been a part of this report for a number of years and I've always been interested to hear how team performance works and to get those responses back. This is done through surveys. But one of the things that was really interesting to me when I came to CircleCI was now I'm not just talking about surveys, now I'm talking about actual datasets that are here that I can look at to see how teams are working and what they're really doing beyond just answering some questions. Not that questions are bad and they can tell you a whole lot about how teams are working. But this is a different angle to kind of look at the same set of problems and the same area and see, okay, what are teams doing? Well, how do they behave? What are the, do they actually push code as often as they claim they do? Do they, you know, what does their CI workflows kind of look like in terms of duration or how often they're running them? So I like to think of this as performance derived versus performance described. So in a survey, we're asking participants to describe their performance. They tell us how they feel about it, how it's working, you know, what they do. And there can always be recency bias there. There can be, well, I'm excluding things that I don't like and so I'm only going to write about the things that we're really good at. Whereas when it's derived, what we see is the set of all things going on the platform which does skew numbers in different ways because a lot of those questions are phrased and for the primary application or service you work on, tell us about, you know, X, Y and Z. And so for us, we wanted to derive that and which means that we look at, you know, if you're an organization, we might look at everything going on in your organization and it turns out if you have one of those, you know, repositories that only gets run in CI once a month and is usually sitting there in a red state, we catch that. That may not be something that enters your mind when you're filling out a survey question. So this data set, it was 44,000 organizations and 160,000 projects. And so this makes it about 1,000 times larger than all the state of DevOps surveys that have been done. And again, this is just a massive amount of data and that's what makes it interesting. And so we started doing this last year where, you know, when I came over to Circle CI, this was one of the initial questions that I had was, hey, can we slice and dice this data set? And Ron and I took it apart and kind of looked at it and we had some help from the data team and, you know, it was a lot of fun. And then this year it was kind of, well, what's changed? You know, I mean, 2020 has been a year like no other. So we imagine there's some inputs that, you know, would definitely change things over time and we'll definitely see some of that in the data. But I don't think, you know, so kind of in year two, you can look at what's changed in the baseline, I guess, from year one. So this is our second year of analysis. Basically the data set's a little bit larger in a 30 day set than it was a year ago. And that's mostly from growth of Circle CI on just our customers and usage there. And so in a lot of cases, you know, there were four classic metrics that were in these DevOps surveys and DevOps reports. And that was basically your deployment frequency, recovery from failure time, your change failure rate and your lead time. And for us, we took these numbers and kind of thought, well, how do these map into a world where software developers are using a CI platform versus, you know, just kind of the overall software delivery as a whole? Because they're not, those are not one-to-one analogous in all cases. In a lot of cases, they're quite close. And in other cases, you may have to do a little bit of, you know, mapping to kind of, to kind of get where those metrics come from. And so we kind of took the state of DevOps report metrics that were the four on the far left of your screen. And then we map them to the metrics that we use for Circle CI, which is on the right of your screen. And in the middle, you can see, you know, kind of a little bit more of a description here where deployment frequency kind of changes to how often you initiate your pipelines, lead time to change, turned into pipeline duration, change failure rate was pipeline failure rate and MTTR was time for red degree, so recovery time. And so we refer to these in pretty simple terms as throughput, duration, success rate, and recovery time. So now let's get into the data. Here's what this data tells us on throughput. So this is how often do you push code that triggers CI? Originally this was kind of how often do you commit, but then we realized that commits can sometimes be grouped and pushed into CI once and not always be running all at the same time. So most projects are run to, you know, run to basically run their CI per push to the Git server. Some are doing it on a schedule, but most often it's usually per push up to the Git server. And so what we see is, you know, at the 50th percentile, you're basically getting a 0.7 times a day, people are running, which is not a lot. So a lot of things are not happening as often as you might have imagined, but then you can see that there's, you know, at the 95th percentile, somebody is running CI on a project 32 times a day and, you know, it's above that even higher. So, but the mean is really sitting around eight and that's because of those 90 plus percentiles. There's a lot of projects that are sitting there, you know, at the higher ends too. So, but most projects are not really deploying dozens of times per day. They're not even validating their software dozens of times a day. And so we can derive that, you know, a lot of the descriptions of high performing teams, it doesn't mean those aren't true. It just means that it may be one project that's really moving that fast, but a whole bunch of other projects that they have really aren't moving that fast. And so you kind of have to look at the total sample of everything going on within an engineering department versus maybe just their single focus or their primary thing. And so that's kind of why this gets different is because you have these questions phrased in, what's the primary application or service you work on as opposed to us looking at, you know, say every repository in your GitHub organization, which again, you're going to have different characteristics when you look at it from that sample. So how does this compare to 2019? Well, you can see that, you know, things are a little bit, are a little bit, I guess I would say a little bit, they're kicking off a little more often in 2020 versus 2019. Sorry, I was at the point seven versus point eight was throwing me off because that's the one that went the other direction. But yeah, so people are really leaning into CI in 2020 and that's not surprising to see a little bit more focus on automation, a little more focus on standardized delivery throughout the software lifecycle. And so those leveraging CI are doing well and some of them are doing even more with CI than they already were. And so while 2019 CI was very popular, I think the investment in automation has continued through 2020 and we'll definitely see that as we move into the insights portion toward the latter third of the presentation. There are also fewer developers worldwide pushing code and that's one of the things we're seeing as well, which again, if you factor in 2020 overall in the economic climate, you can imagine that there are probably fewer software developers pushing code, but that doesn't actually necessarily equate to less code volume, it just equates to fewer contributors on the, I guess, in the commit logs. So duration, how long does it take to get results? This is something that we take near and dear to our hearts because we want you to get feedback as fast as you can from when you're running your validation suite or whatever you're doing with your CI system. And what we see is 5% of builds finished in less than 12 seconds, which you think, well, 5% that doesn't sound like very much. It's about 500,000 builds in this sample. So it's not a small number, even when it says something like 5%. Then you ask, well, what can you get done in 12 seconds? And the answer is probably not a lot, but some people can do quite a bit. If you have many unit tests that can be parallelized, maybe you can run a whole unit test suite. If you're uploading an artifact into an S3 bucket to get it ready for deployment, you can probably do that in less than 12 seconds, assuming the artifact isn't too large. And so sometimes these are steps in a larger set of things that work all together. And sometimes it's, all we do is run our tests really quickly and that's really cool. So in 2020, you can see that around the 50th percentile, you're right around that four minute mark. And that feels pretty good. I mean, that's some good engineering going on if you're getting meaningful signal in four minutes. I know as a developer, I would be often thrilled if I could get my first set of feedback in four minutes of whether or not something I did was working or not. And then you can see that the 95th percentile is sitting here at 34 minutes. There are a lot of companies that would be very, very pleased with even a 34 minute return for signal, depending on what you're building and what your test scenarios are and the complexity of those and the thoroughness of them, you can be all over the place. One of the places I used to work, I was looking at seven or eight hours to get this kind of feedback. If I could get that in 34 minutes, it would have made every developer we have very, very happy. But like I said, half of all bills are finishing in under four minutes, which I think there's a lot of people really engineering to try and get their test suite durations lower. And that's really important. I definitely have seen teams that have kind of SLOs around their test time. And if it gets out of bounds, it might be four minutes, it might be eight minutes, it might be 10 minutes. But if they get past the amount of time it takes to get them the feedback they want, they kind of stop engineering and just start re-engineering the test suite to make it go faster or remove flaky tests or fix flaky tests or invest in more parallelization or in better setup handling, things like that. I just can't put enough emphasis on how much fast feedback will increase your overall throughput for your development team. So any year, there's not really a big significant change here. You can see that generally the durations have gone up just a tiny bit. It could be that people have added more tests and so they're doing a little richer validation than what they were doing in the past. The one thing that is interesting to point out though is that the mean is actually a little bit shorter. And so what you can see is, while at many percentiles it's running longer, somewhere in there those longest running ones are maybe not as long as they used to be because some people have done some engineering or there's just a little more weight at the bottom end of the scale because people have invested in it. So again, not a significant difference from 2019 to 2020 but pretty cool overall to look at things, adding a little bit of time, but in the mean still going down. From a success rate standpoint, we're looking at how often does your pipeline complete with the green status? A pipeline can compete with either red, green or error and error just means incomplete. You couldn't do something and that can happen in any CI system. It could be that an external resource was not reachable. It could be that you were putting in an impossible scenario, a test divided by zero and blew up the whole thing, things like that. But for us for success rates, at the 50th percentile, we're seeing 61% of all pipelines are passing in green. And you can see that even at the lower and higher end these numbers become less meaningful in that there's a group of things that never ever fail and there's a group of things that always fail. And so for those, once you throw those out, you can see that kind of in the middle, 54% of builds are passing so that roughly one out of every two builds is passing. And so basically what this also tells us is that sometimes people are just dabbling with CI, like maybe they're setting up a project, it fails once or twice and that's the end of them using it for a while for that project. Maybe they don't get back to it in that this month of sample. It could have been, one of those 20% time projects and somebody just didn't have the time to get back to it until the next month or a few months later or they started a different repository where they solved the problem in a whole new way and this one just sits there as registering as a failed pipeline for quite a long time. And yeah, some of our samples saw no failures within a month as we covered. So the delta between 2019 and 2020, very little difference, which I wouldn't expect there to actually be a lot of difference here. We got a 1% improvement year over year in terms of the success right at the 50th percentile and the rest of it's pretty much identical. If you dig into it a little bit more, like if you slice and dice the percentiles a little bit, you do see that toward the higher end things are improving where you're getting 89% versus 86 or 100 at the 85th percentile. But again, this is, while interesting, I don't think it's that significant overall year over year to say that behavior has changed in a massive way from 2019 to 2020. Recovery time is the time a pipeline sits in a failure state. And so in the fifth percentile, you'll see that it's at two minutes that it'll sit in a failure state before it's recovered. A lot of times in those scenarios that could be from multiple contributions running in parallel where developer one pushes a bad commit, developer two pushes a good commit, the bad commit fails really quickly and then the good commit succeeds right afterwards. It could be the same developer. So at the 50th percentile you're staying about an hour of time until it's recovered properly. And then after that, you start to see things that are probably more attributed to, you push some code and then you go home and come back the next day and check on it or you come back over the weekend and check on it even. And that's where that 14.85 hours comes in as kind of the, well, I pushed some code at the end of the day, I'll look at it tomorrow or I'll deal with it tomorrow. And 14 hours is about the time of the end of somebody's workday until the next start in a weekday scenario. Yeah, and so this is kind of calling out that giant gap between the 50th and 75th percentile is that it looks like people just kind of go home. And so that's interesting because I think if you look at classic continuous delivery philosophy, you would say, well, you can't go home if the build's red. But I think there's also kind of some pragmatism about that where when you say the build, is it the most important build or is it a scratch build or is it a test build or is it a feature branch build or whatever? And so I don't think I'm here to judge somebody who decides to go home and then come back the next day to work on their build if that's acceptable for their business climate. So, and this has changed overall in 2020. Recovery time has definitely decreased and so it's moving the right direction for 2020. And again, I could hypothesize on this a little bit and say maybe it's because more people are home and more attentive to what's going on with the code and there's maybe less. Oh, I was going to look at that but I ended up getting a coffee with somebody for an hour and a half kind of thing at the office. So we have seen a fastest recovery time improving year over year. And then the last thing we want to get to is kind of the insights. So, we talked a little bit about how we gathered the data. We've talked a little bit about the data and what it told us itself but really this is the more interesting part of the discussion is what can you derive from this data or what kind of things jump out as. And in this part, I'll invite Ron to come in and have more conversation with me because I'm tired of talking to you all just straight up. I don't need to be a broadcast and we'll get some of what's interesting about this data set I think is really what we'll start to find out. So, 2020 has been a year like no other like we've said and I think that we just can't we can't look at this data and not think about well, what's the impact of the global pandemic? What does that have on performance? So, Ron, I don't know if you want me to walk through slides you just want to kind of answer some questions really quickly but what have you seen for the pandemic? I mean, I think going through some slides or just answer your questions be just fine. The pandemic, I mean, like, you know what I think with this slide here representing throughput on the left we have our data from 2019. We recorded a month of data for the month of March, April and May. So, in the middle of this plot are points for March, April and May and then on the right is our August of 2020 data. And so, I mean, many of the percentiles we see that the maximum throughput was reached in April or it would have been in May. And so, I mean, we think that this absolutely makes sense in face of global economic uncertainty. I think that it is very wise for us to take our engineering teams and shore up business interests. To me, this looks a lot like, you know automating as much as I possibly can my workforce is about to go home. And so, I need this to be done as quickly as I can that application as quickly as possible. It also looks to me like resolving bug tickets resolving issues like making sure that I can really lean on the stability of my platform. And so, and then we also need to get this done right away, right? So, one thing that we also notice through this period is that throughput increases, you know on Saturday and Sunday and well outside of the normal hours of operation, which is also expected. This was not something that any of us were relatively prepared for. And so, to see so many teams jump on this opportunity to really try to build their engineering organization their use their engineering organizations to shore up their business makes complete sense. Let me look at the next one. Do you think that on the weekends maybe some of that was even just people kind of toying and being more hobbyists because they couldn't go out and travel or do, you know, go out to the bar or whatever else they could have done? You know, I like this as an idea because I also think, you know what the economic uncertainty and the pandemic caused was a lot of folks to be at home. And so the folks at home, I mean the techno files that sort of group is going to lean into the internet lean into the projects that they work on. I know that I went out in times of anxiety I lean into the things that I do really well it is, you know, developing applications it is resolving bugs it is those types of things. And so, you know, some of this could be, you know just wanting to like create some stability and some familiarity in your life while the world is rapidly changing. And so that would show up as well as evening hour operations it would also show up as things over the weekend. And so the, you know all of that together I think combines to seeing, you know the throughput increase significantly for those times. And then, you know while the year over year is higher it's not as high as the month of May. The month of May was when we saw our peaks. Yeah, yeah through April and May we definitely saw our peaks. You know, I think it was late April, early May was when kind of the peaks happened. And, you know, that was primary lockdown for at least most of North America most of Europe at that time. And so we just saw just a whole lot more code going on the system. And, you know, as we entered into that that time of uncertainty, you know we didn't really know what was gonna happen on CircleCI the platform either. And so we were watching and learning from it just as much as everybody else. I think one of the other interesting things that I was able to notice as, you know I run the SRD area of Circle is we could see that the amount of traffic the scheduling like the patterns on the traffic throughout the week had changed. They used to be like kind of peak windows where they were, you know maybe 12 hours throughout the day and they ended up being more like 14 by the end of that where you know you were getting an extra hour or two of peak traffic time every day on the platform which was again, maybe it's because people aren't commuting now and so they're using that time to work or, you know whatever it is but. So after, yeah after April or early May depending on how you're looking at the graph throughput falls just a little bit. And, you know, that could be people being like wow, I'm burning myself out or I'm working too much or I need to do other things or, you know, mental health problems or mental health issues, anxiety can, you know, raise pretty high when you're sitting at home for multiple months in a row and I think people need to understand that and take a break and give themselves the space they need and it looks like that happened and we definitely saw, you know even in June the traffic definitely was lower and people were starting to take days off even if they couldn't, you know go travel and take vacation it was just wow, I need to be away from the computer for a little while which is all great. And so what can you tell me about duration here Ron? You know, I think that the duration you know, the mean looks like it falls I mean it falls a little bit but, you know, for 50th percentile and above we see duration increasing, you know, year over year and then having the longest runs around the time of April and May when we saw the peak throughput and so, you know, this makes sense to me if bit shoring up your business is the main goal in this type of environment then I would imagine that looks to me like adding more tests to my CI suite. Now fine tuning my test might be the second thing that I do and so seeing a drop right after that might be representative of, you know first I write all of my tests then I increase parallelization or I start running tests on larger machine sizes or I make those types of optimization changes but right away I need to make sure that like the application that I'm creating is as stable as it possibly can and stability to me looks like running tests and so it makes sense to me that I'm seeing this increase during this time in which the throughput was also higher and then it also makes sense to me that it would fall slightly after that and then that we talked a little bit about duration increasing year over year but not to an enormously large amount this is in seconds on the Y axis there so the 2,000, 2,500 seconds it's not incredibly large but like that we were still able to pull out signal from the months that we thought were of interest here and that that signal says that things were running slightly longer like that does look to us like that this supports the shoring of those business interests and it supports that through this is likely due to adding more tests. Yeah I think another hypothesis that we could have on this is a lot of organizations have a lot of side projects I would say where they're doing something and maybe those are running a little less often they probably don't have as detailed a test coverage as the main important business features and so those are probably the things that are really dominating the lion's share of the CI for each of those organizations and so that you can see that raising the duration a little bit too and of course they're investing in that and adding more tests and all that because a lot of companies went back to basically what's my core principle what's my core business objective during this time of economic uncertainty. So a lot of this says basically what Ron just said. So yeah we see a concentrated effort starting in April an optimization where after the duration goes up you start to see people saying wow I don't like my duration creeping up I'm gonna start working on making these tests run a little bit faster or removing the tests or giving me signal that is unimportant or whatever. What's different about success rate through the pandemic Ron? Sure success rates I mean when you know we saw that the really the interesting places in success rate are the mean in the 50th percentile there are some things that are always green there are some things that are always red and so what I see is like to get an uptick in success rate in 50% success rate 60% is about what we measured year over year the that success rate says that people are innovating you innovate like innovation is going to look the signal that that provides is a failed build at least on occasion like innovation should look like red to green, red to green, red to green that type of cycle but no one expects every commit to be perfect every single time. And so a nice healthy innovation environment for your engineering team does look like somewhat high degree of failure but when we see things like a peak when we see things like a signal that where the success rate is increasing on teams the 50th percentile of the mean we expect that to be your average engineering team. To see that increases to me that innovation is likely stalling maybe it's not stalling but maybe it's taking a backseat to what we described earlier shoring a business needs and resolving issues, resolving bugs resolving issues, resolving bugs often does it is, I mean, you're going from red to green you're trying to get more stability in your system and so we would expect that that behavior would also see a peak in innovation or not in innovation but in success rate. We see that fall down a little bit afterwards and the expectation is that once you get your business stable again now we're back to feature branch development now we're back to doing these innovative things now we're back to testing optimization strategies right Stonk you mentioned that that was also likely what occurred after we write all of these tests then we go in and we optimize those things that would also be available to engineering teams if innovation was something that was something that was not necessarily the highest priority as teams were moving towards making sure that stability of their product was the was of top of mind. What do you make of that the red bar on the bottom just kind of trailing off after April? You know, having really low success rates and then having, you know, have seen that decline there I think that there's, you know all of the lower side of this I think is people testing continuous integration tools and so I'm only going to see a handful of commits they're likely not going to be very successful those first five commits and then I see no more action on that as folks are evaluating whether or not that CircleCI or and I would imagine that there were probably all automation tooling across the engineering space probably saw a lot of people go where is my fit, where can I get this I need this for my team and the, you know, decline there that type of behavior and it is oftentimes difficult to describe the behavior that's in the 5th percentile or the 95th percentile but I think that this is probably a clear sign of folks recognizing the opportunity for automation as they send all of their employees home and trying to, you know push commits into as with onto as many tools as possible to see what they meet their business needs. Yeah, I think the other thing that fascinates me about success rate is that, you know success is the state you want to reach but there are many ways to failure there's only only one way to succeed and so if you have something that's failing you could push another commit it could fail again because maybe your fix wasn't good enough or it was incomplete and you could push another commit that could happen again you could do that a few more times and so maybe you've pushed five commits that were all failures but then the sixth one was the success well, now you still have only a one in six success rate and so sometimes that can trail off on the bottom which generally would also like on that 25th percentile line there the reason that I think about that is it could be that you're actually doing high levels of innovation and you're actually seeing failures a lot more or it could be that you're, you know brand new to this whole space and you're trying to figure out is my syntax right he's just working business testing what I want it to do. There's a lot of reasons I guess for failure rates to go up but that's not what we're seeing which is interesting so, you know other than on this, you know the bottom line there success rates were basically the highest on record throughout April and May again, it seems like when developers kind of stopped going to conferences for a while and they stopped traveling and they started working on all their software and core business and showing it up and it looks like, you know the results were seeing in spades on the platform. Yeah, so our hypothesis was that people were working hard on core business stability throughout all of this. I guess Ron, what can you tell me about recovery time? Yeah, recovery times I think is interesting, you know because much of the narration that we've been able to apply here like each one of these metrics like exemplifies a different kind of aspect of the narrative that we've been able to come up with. Recovery times decline. I mean like for, you know as you see in the 95th percentile we're getting year over year drop but that drop, you know that comes in May or in April and May those I think are, you know is reflective of if stability of your business is the ultimate goal for the commits that you're pushing at those times then I would expect all hands on deck to be able to resolve issues that much more rapidly. Now those aren't often the times that are the paradigms that engineers work in. Recovery time is one thing that like most teams optimize on as much as possible, right? Getting your signal back as quickly as possible and then mitigating issues as quickly as possible. Like that is the feedback cycle that is at the heart of continuous integration and rapid application development. And so everyone wants to see a decrease in recovery time but you see signals like what we have here in this plot that look like this those types of drops I think come from the ability to bring all of your folks back on, right? There's this isn't the DevRel team that's out giving talks at conferences. This is all of the engineers that would be attending that conference as well. So like none of that travel is happening. All of those folks are back on and everyone is trying to get to a stability state that is acceptable as they try to then, you know make the changes and incorporate the new world that we all exist in. And so seeing recovery times drop around that time makes sense to me because I think that those are the opportunities where we're trying to solve the problems that are most core to our business. And we have the most of, you know the largest amount of our engineering effort going towards that. And I think that the signal clearly shows that. Okay, great. Yes, you can see that kind of since the pandemic has started we've been seeing recovery time improving. And again, people being more attentive I think is kind of just a quick summary of that. Yeah, I would say that we kind of have a hypothesis as fewer distractions working at home. And I do put for some values of distraction because I do not want to diminish any one situation working at home. There are children at home that are usually in schools there are relatives to care for there are other healthcare things going on that are very difficult to work with. And so some people are able to do way more work because they throw themselves at work while they're at home. Other people are able to do way less. And neither of those is right or wrong. I think it's just a difference that we see throughout the industry. And, you know, especially for all those parents who have had their kids home since March, I'm here for you. I see you it's hard. Some branch information stuff that we came up with. There's a couple of things around branches that we found interesting. And there were a few hypotheses that we wanted to test as we sliced and dice the data. And the first one was kind of, you know throughout June and July of this year there was a lot of social unrest and a lot of Black Lives Matter, you know protests throughout the country. And some of the branch information that kind of came up this was an older discussion that I think really got resurfaced during June and July of this year which was, you know this term master throughout computer science curriculum and throughout tooling is it really something that we have to be using? Or, you know, can't we find better language? That's more inclusive in less, you know comes with less baggage for all sorts of different people. And, you know, we wanted to look into this and this was one of the questions that, you know, as the data scientists were slicing and dicing this, I said hey, let's see if the use of a master branch has decreased year over year because of this. And, you know and we had definitely seen some press information about it. You know, we saw Git changing that you can change even what the default branch is by default in the upstream Git. GitHub had of course published some things about how they were gonna change the use of master. And I can tell you that realistically we have seen almost no real decline in the use of master branch. Parts of this kind of, I don't say make me a little sad maybe is the right way to say it but I think also not all the tooling supports it yet. And so as the tooling, you know gets there by supporting it or defaults to better naming and better answers I think we'll see more of that throughout, you know the future. But I also think organizational inertia and tooling inertia is very difficult to overcome. And so if you have a bunch of automation written around, you know, checking for tags in a certain branch and then it does things, you know when other software assembly processes and things like that that's not free to change. And that's actually quite expensive. And I know that at CircleCI we had gone through that and, you know, we basically had that had to go through an architectural review and get a decision record written and get, you know changes put in and we felt like it was worth it. And you know, in the end it didn't turn out to be weeks and weeks of effort or anything like that but I think there are definitely software shops where that could be monumental amounts of effort in terms of, you know, getting something done. And I think in a lot of other cases it's just, it's a very simple change. And I think, you know we'll hopefully see more of it in the future. Ron, did you want to talk a little bit about kind of innovation in future branches? Well, you know, one thing that we talked about earlier was the success rate being about or not the success rate. Yeah, success rate being about 60% and you know, there's I hope that there are some folks in this group that are like, I could never have my master branch or my main branch sitting 60% success. And so like that and that's clear. And so we do have a little bit of ability to look in and see, you know, what is the default branch and then what is non-default branch. And so like, you know we have looked in the past to try and see if we could get things like, you know a branch name deploy and maybe that was doing deploys but because we don't have a lot of visibility down into the configuration level of those types of branches for those pipelines it was more than just speculation. And so, but just being able to split success rate by the default branch and the non-default branch showed us that the default branches in general are significantly more stable that you're not seeing developers pushing, you know innovating on the main branches. And you see a lot of that innovation happening on the future branches. So success rate 80% on default at the 50th percentile. And then it's higher for, or it's lower than that for the non-default branches. And that 58% being around 60 is what I would expect to see for teams that really are trying to, you know run their CI on all commits and that their commits are trying to build new services, new features like that's going to be a success rate that is going to be reasonable but no one wants to see their main service in the red. And so clearly when we saw that we split those apart seeing that the success rate is much higher for the default branch makes sense to us. And this is, you know there are many different types of Git flows that organizations are using now. Trunk based deployment though is not something that is even the ways that trunk based was originally kind of ideated a lot of the teams that are doing a version of trunk based development, you know still kind of feature branch for, you know for a number of commits before they commit to the main line like all of those things I think are going to like increase as we see more types of Git flows for different organizational needs. There I mean as our tooling becomes more sophisticated as our VCS becomes more sophisticated we do get these opportunities to have a more mature Git flow paradigm that we operate in and we will start seeing more differentiation from these different types of branches and ideally, you know success rate of your main line branch 80% might be too low but that might be a bar that your company is unwilling to see get that low. Yeah, I think in a lot of companies what we'll see is, you know main line is the thing that everybody's reason everybody relies upon. And so that being red means that every other developer is possibly blocked or unable to do their work. And so, you know mainline you want to keep it the, you know 90 plus percentile if you can do it. And then you'll see these other branches the reason you have all these other branches is because no one wants to be the person who broke the mainline build. And so they have these other branches where they're testing that and validating okay my change looks good before I put it into mainline because the last thing I want to do is have, you know every other engineer standing around saying hey can you fix this so I can do work? And so that this supports that, you know we do see, you know feature branch development and you know a lot of people say well feature development is in or feature branch development is an anti pattern and I guess I would say yes sometimes but it also depends on the length and the duration of that feature branch. You know, if it lasts a couple of hours long enough to do a pull request and do some validation and get it merged I don't think that's really anti to any of the trunk based development you know, kind of philosophy if it lasts for years because you have a feature branch that's you know never integrated you have your own set of problems that you will find out about eventually. They're coming. Yeah, yeah and so you can see that the duration on default branches is faster than every other than non-default branches at every percentile. So basically the mainline branch people work on test engineering they work on making it fast they work on making the feedback good when it's non-default branches they're slower and that could be because you're adding new tests or learning how to write new tests or you're putting in sleep statements and print statements to figure out you know what values are where and when but that's just part of I guess developing on multiple branches. And we also see recovery time lower on the default branch in every percentile and this again makes sense if you think about it when there's other people that rely on mainline they're watching recovery time they're watching mainline a whole lot more to make sure that fewer people are impacted by a red build and so we're gonna make that red build the shortest duration we can and so you know recovery time increases and that's just again this is kind of just classic continuous delivery philosophy and it does all those things I guess are shown in our data sets which is cool. And then what you know we were able to ask a little bit about some development practices and see what works and one of the things that we were able to see is you know when you look at organizations and how many contributors are in the organization success rate really doesn't correlate with company size at all. If you have 800 if you have 800 contributors and you're good have order you have four we can't really tell that just by looking at success rate. I don't know if that's interesting to everybody I found it interesting. Duration is longest for teams of one which I think makes sense because if you're a team of one first off you might not be a team. So, but secondly you might not care if it takes you a long time to get feedback because there's no one else waiting on you and I think there's a social construct there where when you know you're in the way of somebody else you will do extra work to make sure that it's you know faster better understood, taken care of in a better way and I think that that's kind of the things that we miss when you're one. Recovery time decreases with the increased size of your team which I found which was up to about 200 and the reason I wrote that is after that the data gets a little fuzzy and the data sets small enough that you can't really draw a lot from it. We don't have a ton of teams that are over 200 engineers that use us all at once on the same repositories. So, you know there's many companies that have more than 200 engineers but they're usually not all working on the exact same repository. So, but recovery time decreases. So basically the more people you're impacting with a red build probably the faster you are to recover it but I think is really what that's telling us. Mike and you think that that's more of the social construct than hands on deck or just a combination of those two? I think it's, I mean I think it's a social construct but I mean I know the last engineering team I ran when certain things were read there were literally 70 engineers that basically couldn't merge. And if they can't merge, they're sitting there blocking or queuing change or their context switching into something else and now whatever work they had committed for that day is at jeopardy. And I think that I think it's the social construct more than the you need 200 people watching the build. I think it's the person that puts the commit in there is like really watching their build to make sure it's okay because they know there's 70 or 100 or 200 people that are gonna be relying on that build for the rest of the day or overnight or whatever. Longest recovery times are from teams of one which again kind of supports our, you don't have a social construct here. I know like for my little chat bot that I play with if the build turns red I might not get back to that for three weeks because literally no one cares except for me. And so basically in summary like performance is better with more than one contributor like from every outcome that we can measure you're better with you and you have more people to work with. And so the takeaway here was that software is collaborative which I don't think is a new thing to think about but I don't think we talk about it a lot when we talk about data and metrics. And this is a thing that we were able to prove out in just looking at the outcomes it's the success rates of team sizes and number of contributors that you have you can see that putting more people working on a project is going to give you better outcomes and whether that's because it's purely social construct or whether it's because you get new minds and new approaches I don't think it's one or the other I think it's an and and there could be a slider there depending on how much of value you're getting out of either of those and there's probably other indicators as well. One of the other questions that we went through with the data with was how does this is don't deploy on Friday? Is that a really a real thing anymore? When I hear like we don't deploy on Friday is I definitely think more about kind of IT from a while ago in general but that doesn't necessarily mean that we've moved past that it could be that my bias has changed because I work in a technology company versus an old manufacturing IT background and things like that. But to look at this we saw there was 70% less throughput on weekends just in general. And so when you think about a seven day schedule Monday through Friday there's a whole lot more traffic than on Saturday and Sunday not surprising there's a lot fewer people working my AWS bill goes down I'm happier. But we see 11% less throughput on Friday and we do all this in UTC measurement because it's the only same way to even talk about dates and times on this planet. And so we see 11% less throughput on Friday's UTC but we see 9% less on Mondays. And so the reason that we kind of do this Monday, Friday comparison is Tuesday, Wednesday, Thursday are the only days of the week where the 24 hour period is a business day for everywhere in the world. Because Monday you know you're actually it's Sunday in North America for portions of it and Friday you're the weekend in Europe before you're the weekend in North America. And so but when you look at this and you kind of normalize it out if you see well you're 9% lower on Monday and you're 11% lower on Friday we can pretty much say that that's not a significant change between Mondays and Fridays more people take off on Fridays too just generally looking at the way people do holiday scheduling. So we don't really see that there's a significant drop off on Friday in terms of workflows. I think that kind of tells us that this we're not seeing a whole lot of people holding back from pushing code on Fridays which makes me happy. I guess you should be able to push code hopefully when you need to and you can reduce your risk by not doing something or by working on other ways to reduce your risk. And hopefully when people are using CI and automation they found other ways to mitigate that risk in some way. Mike you showed that there was a 70% drop. What are the workflows that are being run on Saturday and Sunday? A lot of times it's scheduled jobs. And so whether it's we run a job that submits a report to a manager on a Monday morning so it's in their inbox when they start or it could be we have a certain set of tests that maybe take a very long time to run because they're very intricate and very complex. And so we only run those on weekends because we don't need to run them on every commit. It could just be that certain repositories just run every hour no matter what throughout the weekend on the weekends. And so they're still just being counted on the weekends. It could be the hobbyists playing on the weekends. It could be it's crunch time in Cyberpunk 2077 is supposed to come out and the developers just keep having to work because they keep moving the date back. So it could be any of those things. Excellent. And the last thing that we have kind of on here is just language trends. And I love throwing up language trends mostly to start debate among other people because you can always pick your favorite language and fight for or against it. And in this sample size, which was about a month long this was the percentage of jobs we saw anything below 0.083% just doesn't register which is interesting because languages that are quite popular even still like closure which is what a lot of circles see eyes in for what's worth. It doesn't even show up in terms of registering here and you'll also see that a lot of I guess there's a lot of markup and markdown and then kind of I would say simpler languages which maybe is not the right way to describe that but things that necessarily don't have a full test suite all the time like HTML. There's a lot of people writing tests to validate HTML they're usually validating something that generates HTML more so. So this is the languages in our sample in terms of throughput this is how often things are going on to the platform. Ruby is one of our most popular, you know people that using it on circles the eye and that's been that way for forever and I think that, you know that's not a big surprise for us. Type script and go have both jumped up year over year in terms of where they are in the rankings here. So definitely those are growing pretty quickly Docker file sneaking its way in at number 20 a lot of people building Docker artifacts and so you start to see Docker files is just one of the main things that jobs are circling around. Language success rates. I find this one interesting because again, I don't think a lot of people are running tests using CSS. I think they're doing simple things with CSS maybe assembling it, maybe running less or doing some of the other kind of tooling to build it out. Shell kind of the same way I don't think a lot of people are running very advanced test suites in Shell. I mean you can use bats or you can use other tooling to do that but I would bet in most cases it's a shell job that's doing something else with maybe another artifact maybe putting it in S3 or getting something from an external resource and prepping it for the next step of a pipeline and a lot of times Shell's just exit zero no matter what happens unless you've explicitly written it not to. Docker file is kind of the same way and so even when we get down to a Hashicorp configuration language in number seven so there's a lot of Terraform scripts that go through CI and things like that but Go is kind of the first language that I would say that usually comes with a healthy test suite with a lot of applications and I just find that interesting that goes above TypeScript and JavaScript and then you get back more into the kind of dynamic languages and you get to the static languages more toward the bottom and then at recovery times Go being number one was that was surprising to me just that apparently the Go developers are just on it they wanna make sure that their pipelines are green and maybe they have because they have lots of contributors and they wanna make sure that everybody else can be active and successful. JavaScript ping right there is not surprising to me and then toward the other end recovery time being longer for C++ or Swift that doesn't surprise me either both because sometimes those languages are quite complex to work on solutions but also because you have to build an artifact before you can validate it and so in some cases you have to wait through that build time which just means your duration is longer which duration isn't input into recovery time because your recovery time has to at least go through one duration before it can be recovered. And then duration, you'll see things at the top are shell and hashy-corp configuration language whereas at the end of bottom end you have a lot of the a lot of the statically typed languages and then Ruby doesn't show up super well this time but that's okay. Last year it was really weird because PHP showed up in a really good light in almost all of these measurements which I thought was a weird anomaly but it did let me tell everybody that they should be using PHP for everything which I enjoyed because I didn't mean it but so this year I think PHP groups much better where I expected to group closely with Python and Ruby in most cases on most of these charts and it does. So for a couple of final thoughts when you kind of map what the outcomes are from our CI data set and you put them back kind of overlaying some of the academic research that's come out of the state of DevOps reports usually map just kind of if you're at the 50th percentile in our data set you're showing up between medium and high performing in a lot of cases just if that's if you're at our 50th percentile so that's just if you're average at using CI more or less with circle CI you're gonna be between a medium and high performing team once you start to make a little more investment in it you can shift up the high performance very quickly and very fairly with it's a fairly straightforward path I won't say it's easy because writing tests and making sure that they have good coverage and things like that is not easy but it's a well-known path so if you're average using CI you'll be on the right path is basically where we're at yeah our most frequent CI users have more outcomes on all four metrics and so like the more you're leaning into this the better you're gonna do in terms of output of both your CI measurements but also just your traditional DevOps outcomes the way they're measured and configured and more collaborators means better outcomes and I think that was one of my favorite things just because I love putting a human element on software and so if you have more people to work with and you can do a little more whether it's collaboration or socializing or whiteboarding or whatever to get your code through it leads to better outcomes and we can show that with the data that's kind of the end of the presentation just reminder circle CI is hiring because we are hiring like mad whether if you're interested in this data set or you're interested in engineering how we get this data set or the platform that circle CI runs on feel free to apply I just wanna say thanks Mike and that was Ron and so I think we can switch over to Q and A All right, we have about two minutes left so if anyone has something pop it in the Q and A chat and thanks for a great discussion y'all let's monitor that see what we've got and we'll go from there must have been a great detailed presentation no questions left we do have two minutes if anybody wants to ask anything though Michael and Ron, thanks so much for a great presentation that's all we have time for today thanks for joining us everyone and the webinar recording and slides will be up online on the website later today and we look forward to seeing all of you at another CNCF webinar soon All right, thank you so much Have a great day, thanks so much Bye Bye