 So, welcome everybody. I am Dr. Donna Foster. I will be your lecturer today and here in the lecture hall. And we will be talking about measuring the health of your CNC project and going beyond Stars and Parks. And I will be getting tested. Also, the slides are posted on schedule already, so the links and stuff is already there. Before we dive into the topic, I will tell you just briefly, a little bit about me. I have been in the technology industry for over 20 years, working on open source projects at companies like Puppet, Intel, and now at the end, we are responsible for our open source community strategy within the author's program office. I'm also on the steering committee for the Linux Foundations 2D Group, where companies collaborate on ways to run successful open source program auditions. I'm a board member of the UK, which is focused on invalidating and sustaining the leadership in open technologies. I'm a governing board member and maintainer for the Linux Foundations Chaos Project, which is focused on using metrics to evaluate the health of the resource projects. And I also have a PhD from the University of Greenwich in London, where I spent a few years researching how people collaborate within open source program. Now, long before starting my PhD in getting involved in the Chaos Project, I've had a passion for data and more specifically for measuring the health of open source projects. As part of my work within the CNCF contributor strategy technical advisory group, we wanted to help CNCF projects get better at measuring the health of their projects and making improvements based on metrics, which is what led me to this topic. One of the how-to docs that I've written along with the rest of the contributor strategy tag is the Project Health Measurement Guide, which you can see on this slide. And the presentation will dive into each section of that document to talk more about measuring the health of your CNCF project. I'll start by talking about some general approaches to project health before diving into several key measurements like responsiveness, the contributor's velocity, releases and inclusivity. I'll wrap it all up with a few final things to think about, links to some resources and some final thoughts on the topic. Now, before we dive into the metrics, I wanted to talk about a few approaches to project health that are important to think about before you decide how to measure the health of your project. I see a lot of people getting hung up on what I would call a vanity metrics or popularity metrics. I know it can be super exciting to hit big numbers for your project, hundreds, thousands, whatever, stars or quarks. While popularity measurements like stars and quarks are interesting, they can be exciting. They don't actually provide any real insight into project health or indicate areas for improvement within your project. So, they'll be to celebrate those milestones that might be fun, but don't fall into the trap of thinking that they actually indicate anything meaningful about your project. For CNCF projects or really any open-source project, we encourage you to focus on project health instead of popularity. Project health metrics help us understand which specific areas of a project are healthy so that we can make improvements and then measure the impact of those changes to see whether they made any difference or whether they made the difference we were expecting. And if not, we can address something else. Equally important is that project health metrics also help indicate where you don't need to make changes. If some area of your project is always healthy and doing really well, you're better off focusing in other areas where you do have things to improve on. What makes all of this so challenging is that every open-source project is a little different. And the interpretation of the metrics will vary quite a lot depending on the project. For example, what's healthy for a very large project like Cooper-Neggies won't be the same as what you'd use for a smaller project. The measurements that I'll talk about in this presentation and the guide will be to the agenda page, arm dot, one size for all measurements, but instead are designed to help you think about how to apply some of these metrics as a way to improve the health of your project. Also keep in mind that data and metrics dashboards, including DevStats, which is what we have for all the CNCF projects, they aren't going to be perfect. And you'll find maybe some inconsistencies or inaccuracies. So you'll always want to apply a bit of call and sense and a bit of a reality filter based on what you know about your project. One of the hardest things about measuring project health and metrics in general is that you can look at so many different metrics that people just tend to get overwhelmed. CNCF uses DevStats, which we have for all CNCF projects, and it has approximately a bajillion different metrics and different ways of looking at the data. DevStats dashboard can be really overwhelming, especially for people who aren't familiar with that. So we recommend picking a small number of metrics as your focus and providing those in an easy to use way that any member of your project can understand. One of our focuses for writing that project health measurement guide is to help people narrow it down and start with just a few metrics that are likely to be important for those projects. Now these don't include everything that you might want to measure or every option for each of these metrics. So think about this really as just a starting point. And again, there is no one's eyes that's all, but you think that these are probably a reasonable place for most CNCF projects. One key indicator of project health is responsiveness to community contributions. Projects that are responsive to use PRs and other contributions are more likely to retain both new and existing contributors. People will find bugs, right, or want to contribute fixes back upstream. So as a project, you'll want to make sure that maintainers and other contributors typically respond in a timely fashion. This shows people that you appreciate their contributions. And when people aren't having to spend all of their time following up and chasing things down and trying to get a response, they'll hopefully have more free time to make additional contributions. Now, most projects that we're done have struggled with keeping up at one time or another. And this can be difficult, but it's also really important. When response times go up, meaning it takes a lot of work to respond, which usually is a sign that you don't have enough people in certain roles in your project or possibly not enough contributors over time. And there's going to be a good opportunity to have your maintainers or other users take a hard look at some of these up-and-coming contributors and recruit them or enter them into the mission leadership. Since a lot of us work at companies, in my office, we seem to bring other employees into the project to help out with the work plan. And I have a couple of examples about how you can measure responsiveness using DevStats on this couple of slides. One way to look at responses is to look at the first human response to a particular contribution. This is an indication of how long someone has to wait before getting some sort of feedback. Now, this example is for the onboard project. And it's looking at PR time to engagement dashboard from DevStats. Now, most people get a response in less than a day. So these numbers look pretty reasonable to me for a media-assessed project like on point. And they've been pretty steady over time, which is also what I like to see. Now, if these lines start moving up with increasing time to first response as the months go on, then that would be an indication that you probably don't have enough people working on the project or not in the right areas of the project. Now, a very small project with only a few people working on it, and maybe a lot of those people live in similar time zones, those will have slightly longer times to first response. And a big global project like Kubernetes would typically have shorter times to first response. Now, you see these red lines that are the vertical lines. And those show releases. And if you hover over them, you can see which release it is, which is nice to have because sometimes release timing is something you really need to take into account when looking at responsiveness. In this case, there isn't a big change in responsiveness around the release. But if you see a big increase in time to respond, that could indicate that almost everyone's working on the release, and not enough people are spending time keeping the rest of the project running. Other things you can look at would be important conferences when key leaders or maintainers are on vacation, holidays, other things that might impact responsiveness. And this is where interpretation comes in because things might happen in your project that have an impact on responsiveness. By interpreting the results in light of what you know is happening in and around your project, that could have an impact on responsiveness. And, yeah, so interpreting the results is really important based on these other things that are happening around your project. And you can focus on improving responsiveness if and when there's a real problem. But if you think something else might be impacting responsiveness like conferences or holidays or something, it might make sense to wait and see if it goes back to normal after some big event or holiday season. Another way to look at responsiveness is to look at how long it takes to close issues in PRs. This gives an indication of how long it takes for someone to get a resolution to their issue of PR. So as an example, this is the issues aged by Repository Group dashboard in DevStats for all of the Envoy repos. And the yellow blocks at the bottom show median issue close time. And the green line running across is the average number of issues opened. And in this case, issues are typically closed in a few days, which is pretty good. And I like the way DevStats has them both on the same graph because the more issues people are opening, the harder it is to keep up with them. So it really helps to look at both of them at the same time. And you can see that in November and December, there was a big spike in the number of issues opened, but time to resolution stayed low, which is good. And as you can see that in the spring, the number of issues opened spiked in March, followed by an increase in median time to close, which isn't unusual after a spike like that. But it might be an indicator that there aren't enough people working on issues to keep up with the demand. In this case, it looks to me like it was probably a temporary issue since the median time to close went down after this and seems to be sort of flattening out. But it would definitely be something to track over time and keep an eye on. And again, like with the previous slide, you'll want to interpret this data in light of all of the other things that might be happening within your project. It's important to look at whether people are sticking around in your project. Stable or increasing numbers of contributors indicates that people want to continue participating in your project. And it's a good indicator of project health. A decline in contributors can sometimes indicate that there are deeper issues that should be investigated within your project. Maybe it isn't as inclusive as it could be, or maybe you have some toxic contributors that are driving people away from your project. Again, you'll want to think about how you interpret these numbers. For example, most projects see a decrease in December and November if you have a lot of U.S. contributors because people get busy with other things over the holiday season. You might also see a temporary increase in contributors shortly after a big conference or other contributor event when people are really excited about the project. That's when existing contributors maybe get re-energized or new people who learned about the community are starting to contribute. And again, you should interpret the metrics in light of what's happening in the project. This is the DevStats contributions chart with the blue bars indicating number of contributions and the red line is the number of contributors. Now, this chart is mostly useful to get a really quick snapshot into overall contributor trends. But if you see a big decline, you'll want to look deeper to understand what's happening. Now, this is the chart for Prometheus and it looks pretty good. It's mostly stable except for the holiday dip in December. Likewise, the decline in contributors in July and August is pretty common since a lot of people are taking these contributions and I know that Prometheus has quite a few European contributors where this July, August dip is also even more pronounced. And I'd still keep an eye on it though to make sure that it bounces back after the summer holidays but I wouldn't pray too much about that. Now, this chart is way busier than what I like and I'll describe it because I know you can't read the text on it. But the information that it shows is really important and I would encourage you to look at it for your projects. New contributors are people who are submitting their very first PR. And casual contributors, also called episodic contributors are defined as someone who hasn't submitted a PR in at least three months and hasn't had more than 12 PRs overall in the project. And this helps you see whether you're regularly getting new contributors into your project and whether they're sticking around. So this chart has number of PRs on one axis and PR creators on the other which is a pretty useful way of looking at this data. Now the yellow line which is mostly the bottom one has new contributors and the green line which is kind of just right above it has the number of PRs from new contributors. The orange line is which is right above the green line. The orange line is episodic or casual contributors and the blue line which is mostly the top one is the number of PRs from those episodic or casual contributors. And again this is an example from Prometheus where you see the same seasonal dips but one of the things to look for on this chart and why I think it's really important is that the line with the number of PRs from either of these groups hopefully is greater than the number of contributors. And this indicates that at least some of these people these new and episodic contributors are actually sticking around to make more than just that first PR. And you can increase the chances of getting new contributors by having excellent onboarding documentation or training to help new contributors get started quickly and easily. You should also do everything you can to make them feel welcome and included which can make a big difference in your ability to attract and retain developers and contributors. Also having easy and clear contribution processes makes it possible for these casual contributors to kind of swoop in fix a single bug and this lets end users participate in your project. And it's an eventual source, eventual source of more regular contributors over time. Your project will be a lot more appealing to potential contributors and users if you have contributions from a variety of different people and organizations. For your project are there enough contributors that if one of them won the lottery and retired on a beach tomorrow could the project continue with minimal disruptions? What if this person works on a part of the project that is really complex and requires a very specific skill set? Would the health of your project suffer if certain people left the project? You should also look at organizational diversity as part of project health. If all or most of the contributions are from the employees of a single company you aren't getting the innovation and diversity of ideas that you would get if you had people from more companies participating. If your project is dominated by contributions from a single organization it might help to understand whether there are barriers that you're putting up to contribution like maybe planning and road mapping being conducted in private or governance with no clear path for others to move into leadership roles or new contributors just not feeling welcomed or included. And again this requires a fair bit of interpretation to understand potential issues and make changes that improve the health of your project. The DevStats companies table can shed some light on organizational diversity. This is a really good example from Prometheus where you can see that no single company dominates the project which is what we really like to see for CNCF projects especially the graduated ones for sandbox projects. We often see a single organization making the vast majority of contributions which is to be honest just to be expected in a lot of cases for less mature projects but as that project moves up into incubating and then graduating, graduated we expect to see contributors from a wider variety of organizations. There are a lot of other options to look at contributor data and organizational data for your project in DevStats especially if you want to dig in and really understand something. But this should be enough to get you started. Active projects will have a steady or increasing flow of contributions and this is probably what most CNCF projects should be looking for. On the other hand a decrease in project velocity could indicate that a project is mature and maybe requires a little less work or it could indicate that a project is no longer interesting to anyone and might eventually need to be sunsetted or archived because no one's using it. Decreasing velocity can also indicate potential project health issues if you're getting fewer contributions because something's happening within your project that's driving people away. This could happen again if you have toxic community members or if people don't feel welcome and safe when participating in your community. The contributions chart that I mentioned earlier is one good way to look at project velocity. Another interesting way to look at project velocity is using the CNCF projects health table and I'm just showing info for commits here but the stable is huge. It has data about PRs, issues and a bunch of other types of contributions. One of the things that I like about this table is it just has little arrows to show whether things are trending up or down across a wide variety of metrics and the arrows make it easy to see the trends at a glance but it also because all of the CNCF projects are on this one table, it lets you compare across other CNCF projects which can be helpful when deciding whether the trend you're seeing is seasonal. In this case, most CNCF projects have metrics that were trending in the downward direction in the three months when I ran this in early September because lots of people were out on summer holidays and again, like with everything else I've talked about so far, these metrics need to be interpreted based on the other things that are going on within your project. Regular releases provide peace of mind for users that they'll be able to get new features, bug fixes and security updates in a timely manner. A regular release cadence also helps your contributors know more about when they can expect their contributions to be included in a release. From a project health standpoint, I look at whether an open source project makes regular releases and quickly patches security vulnerabilities. Projects that take a proactive approach to addressing security issues and releasing bug fixes are more likely to be secure and more likely to be healthy. I don't have any new charts to look at with this one since we've already seen that the release info is included in those red vertical lines on a lot of the other charts that I showed you earlier. Diversity, equity and inclusion is something else that I look at in my project health. This is a bit harder to measure and it usually involves mostly a manual assessment, so you won't find this in DevStats. We have to find some metrics within the chaos projects, diversity and inclusion working group. I've included the link to the metrics on the bottom of the slide. And you can look at those for ideas about measuring various aspects of diversity and inclusion and a lot of cases using surveys and other methods. And you can look at code of conduct which applies to all behavior across the entire project, not just at events. And you want to make sure that people feel safe when reporting code of conduct violations and that you address those violations in a timely and satisfactory manner. You should also make sure that you address issues with difficult or toxic community members quickly to minimize the damage to your project. And the number of contributors, especially new contributors, can indicate that you might have difficult or toxic contributors as we talked about earlier in the contributor section. Does the project have diverse leadership from the very top? Are there mentoring programs and other ways for diverse folks to move into leadership positions? Projects that make a concerted effort to bring in new people from a variety of backgrounds and have programs in place to help them grow and increase innovation and just have a healthier community. And again, this isn't tracked in dev stats, but it's worth the effort to track it manually and to help understand any lack of diversity, especially within your leadership ranks. There are some other things that you should think about when measuring project health and some options in dev stats that can be tweaked to help you understand overall trends and then dig into potential issues as you look for them. Assessing project health is one of those things that can easily be forgotten or neglected. People just don't remember to do it if you don't have a regular cadence for looking at and reporting on your project health metrics. If your project has regularly scheduled and frequent releases, running metrics for the period of each release is a great option. For projects with more sporadic or infrequent releases, it might be if you have specific issues or concerns around project health. You might even want to look at them every month. When looking at any metrics, you should pick a window of time to use for your measurements. I like to start with charts that cover a year's worth of data to get a feel for what's happening overall in the project and then dig down into more granular time windows to get more details. And you probably noticed that in some of the earlier examples, I tended to use a year's worth of data. And if you can change that, there's a dropdown in the type right-hand corner of most of the dashboards. And there are loads of ways to set the dates using absolute dates, relative time ranges, depending on how you want to look at the data. One of the other important things to know about DevStats is that you can set the period of time to use for the graphs. Things like issues and PRs, it usually defaults to a moving average over seven days for a lot of the charts that I looked at earlier, which can look really busy and make it really difficult to spot trends. So I like to start with a month, which is what I showed you earlier, which is not very granular. But it's easier to get a quick look at the big picture without getting distracted by the noise. But if I see something I want to better understand like maybe a big spike, I might dig into the week or days to see when it happened. And then probably look in more detail at the repo and other things happening at the same time to get a better understanding of what might have caused it. And once you understand what happened, you can decide if there's something you need to do differently in the future to address some project health issue, or maybe it's just a one-off situation, kind of an anomaly that you don't need to address. You should also think about the scope for looking at project health. In some cases, especially for larger projects, you might want to look at some of these metrics by repository group or subproject to better understand whether the project is healthy across all of the areas. If some areas of your project are experiencing issues, those might be masked by other areas of your project doing really, really well. So looking at smaller areas within the project can really help you identify specific places that need improvement. It can also help to drill down into different repository groups if you're trying to better understand the potential project health issue. Before I wrap up this talk, let me leave with a few resources that you might find useful. The CNCF contributor strategy tag has a governance working group and a contributor growth working group, which provide guidance and templates for contributor experience, sustainability, governance, openness to help people develop strategies for maintaining healthy projects. The to-do group has a bunch of guides that have great info about all aspects of creating and managing healthy open source projects. The chaos metrics community has defined a whole bunch of metrics and they have other tools that might be useful for understanding how to improve the health of your project. These are all great starting places for understanding best practices for open source projects to help you evaluate project health. Interpretation has been a common thread throughout this entire presentation. Every project is a little different. So the way these metrics should be interpreted will vary based on the by project, based on the size of the project, maturity, loads of other factors. It's also worth mentioning that no metrics are perfect. I mentioned this earlier, but if something looks out of place, it's worth investigating and verifying the data. The needs of open source projects also change over time. So each project will need to determine what metrics work best for them and how they will interpret the results as they continue on their journey to understand project health. With that, thank you for coming to my talk.