 We'll be discussing building a community metric strategy. Thank you for coming on. Thank you, everybody. Can you hear me in the back? OK, awesome. So you can find a copy of this presentation. It's already uploaded. So you can find it on FastWonderBlog slash speaking, or it's also uploaded on the FOSTA site. So it's all there, so you have it already. Now, as you'll notice throughout this presentation, community metric strategies are indeed based mostly on cats with measuring tubes. So that's the primary thing you can take away from this presentation. I have actually spent more than 20 years working in the technology industry. And for most of that time, I have indeed been focused on open source. And metrics was always a big part of my work with an open source projects. As director of community at Puppet, we used our metrics dashboards to provide updates and recognize community members. Before Puppet, I was community lead for Intel's open source technology center, where I did similar things with Mego and Tizen metrics. I just finished a PhD. And as part of that work, I did a whole bunch of analysis for the Linux kernel, mostly looking at mailing lists, but also at source code data and a few other things. I'm on the governing board and Emma maintainer for the Chaos project, which Brian mentioned. And I will talk about more later in the slides. And it's a Linux foundation project, open source project based on community health metrics for open source projects. I joined Pivotal back in October, so relatively recently to focus on open source strategy. And right now I'm mostly working on our open source strategy for contributing to Kubernetes. So of course I've been digging into some Kubernetes metrics. Since we're talking about community metrics strategies, I am gonna start with how I think about community. Since this drives how I think about metrics. And I tend to define community a bit more broadly than how I see some other people define it. So we'll start with that. When I talk about people, I am indeed really including all of the people who work on the project. So I'm talking about core product developers, release managers, quality assurance, localization and other people who contribute to the product itself. Application developers who are maybe using the API or other features to write applications on top of your product through applications, modules, extensions, things that build on top of your project. Users, people who actually run your software and with any luck maybe provide some feedback. Vendors, so the companies who are creating their products based on your project and hopefully giving some engineers back to the project as a part of that work. And then other contributors, people who are working on promotion, organizing meetups, doing moderation, documentation and all of the other stuff that happens around open source projects. Some of these people contribute as part of their day job on behalf of an employer and others contribute on the free time. But they're all really important parts of your community. Now before we dive into the strategy, let's talk about some of the ways that you can use metrics. So metrics allow you to measure progress to see if you're achieving your goals. So using metrics to measure whether you are successfully achieving your goals is a big focus for the rest of this presentation. So I'll talk more about that a bit later. But as you're defining your goals, it can also help to use any existing metrics that are already available and just sort of lying around to learn more about your community and explore in more detail how people are currently participating. And this can be a really valuable input into your goals and strategies. Another use for metrics is to look at your community over time, to spot trends, gauge interest. You may notice that people are showing a lot of interest in something you didn't really expect. And knowing this might help you encourage further contributions in that area. On the other hand, people might be showing less interest than you had hoped in another area. And this gives you an opportunity to dig a little bit deeper and talk to people to understand maybe why they aren't as interested in something that you thought was really, really valuable or really cool. However, one of my favorite things about metrics is that it tells me more about who is contributing to the project. The people will make the community. And for me, the people are the most important element of the community. So my metrics do tend to focus more on the people than anything else. And metrics not only tell you who contributes to your project, but they also tell you where they're contributing within the project. And this allows you to learn more about your key contributors. And I like to use this also as an opportunity to recognize them. It's nice to recognize that person who maybe answers so many questions on the mailing list, or maybe you contributed something awesome that a bunch of other people enjoy using. Now, when I say start with the goals, what I don't mean is to start with your goals for the community. I actually think you need to take a few steps back and start at the very top. So what's important for your organization or for the project as a whole? Now, this is typically been a company in my case, except for lots of big companies that are doing open source, but it could be an organization like the Linux Foundation. It could also just be a project like the Kubernetes project instead of an organization. And you need to start by looking at what the organization or the project hopes to accomplish and what its goals and strategies are and its objectives. And then you can take this down a level to start to figure out what your goals are as a community. And as I mentioned earlier, looking at some existing metrics can also help you explore what people are doing now as input into defining your metrics or defining your goals for the community. What I think the most important part of putting together strategies and plans for your open source contributions is really aligning them with these overall goals for your project or organization. If your goals for the community don't support the overall goals for your project, then you aren't likely to be successful because the community is just doing random things that nobody cares about. So it's worth the time to figure out what you actually want to do and how it supports the rest of the project or your organization. And once you figure out what you want to do as a community and can tie it back to the bigger organization or the project as a whole, then you can start looking at using metrics to measure your progress. Now some of you might be thinking that starting with the goals really isn't that important. So I'm gonna spend a couple of minutes talking about what can go wrong and trying to convince you why the goals are important. But I think it falls into a couple of different areas. So first, you just won't know if you're successful if you aren't measuring the right things. And if you aren't measuring the things that are important for your project or organization, you won't know if you're making progress in the areas that you actually care the most about. And second, measurement actually impacts behavior. And this is something that is really important to think about. People will do different things depending on what you measure. For example, if you publish metrics that focus on the number of mailing list posts and that's the focus for your metric, you're likely to start getting more mailing list posts because we're vain and we wanna move up in the rankings. And if what you're trying to do is maybe get more people to contribute code to your project or get more people to do something specific, then additional mailing list posts probably aren't going to help you all that much. Now, I love data. So measuring open source project success is kind of a hobby that I'm pretty passionate about. I'm not sure what that says about me if this is my hobby, but such as it is. But as a result, people tend to ask me a lot of questions about metrics. Usually the questions are focused on how to decide what to measure. And people often ask for examples of projects with the best metrics. But I really don't think that's a good approach. What you measure depends on your goals and what you're trying to achieve, which may be completely different than other projects. So I really prefer to encourage people to start by defining their goals. And ultimately you need to look at your strategies and plans to come up with criteria that will help you measure whether or not you're successful. For example, if you want to improve the performance of a particular piece of open source software, you'll want to have success criteria and measurement based on specific types of performance. If you want to gain influence within an open source project, maybe you need to measure increases and contributions or the number of employees moving into positions of leadership or positions of influence. And as with any good strategies and plans, the outcome and the results should be measurable so that you can actually tell whether or not your efforts were successful. And this is where your metrics come in handy. Once you decide on your success criteria, you need to make sure that you can actually get the data that's required to measure these things and start measuring it now to get a really good baseline from what you have. And there are loads of tools available to measure this stuff. So there's lots of tools available to gather contribution data about open source projects and some of the commonly used tools can be found in the Linux Foundation's chaos project and which I'll talk about later. And many big projects already have dashboards using these tools and other ones to display contribution data for the project. Now that I've talked a lot about focusing on the right metrics for your project, I'm also going to recommend measuring other things that just aren't even important to you now and going a bit overboard on your metrics. Because I love data, so I just want more data. But you may not spend much time looking at this data now, but it might help you later because there are a few things that can happen. So the obvious is that your project goals might change and something else might matter in two years that doesn't really matter to you now. And if you've already been capturing a bunch of data, you can probably track progress towards this new goal with some existing historical data. And the other piece, and I think this is probably more likely and maybe a bit more important, is when something unexpected starts to happen. So this is actually one of my favorite things about working in open source projects is you just never know the direction that people will take with the stuff you're working on and how people will use that software. So with a really robust set of metrics, then you can dig in and learn more about something that you didn't expect to see. And you might need to adjust your goals based on some unexpected things that are happening in the project. So now that you have a whole bunch of awesome metrics, you actually need to do something with them. So historically, most of the places I worked, I've reported metrics monthly. If you work in an organization that doesn't really get community or get open source, monthly is probably a pretty good cadence. When I worked at Puppet, they were a bit more savvy and so I think I reported metrics quarterly up to the exact team. But being able to show frequent progress, and again, the stuff that you're showing people is, are the metrics that actually demonstrate whether or not you're being successful. So being able to show frequent progress to help justify what you're doing and show that you are indeed making progress towards accomplishing your goals. As I mentioned when I was at Puppet, I did a short report. I think it was about quarterly for the exact team that was focused on actual participation in the community. I don't usually bother tracking things like people who join just to lurk. I only try to count people who've actually contributed something, whether it's code or docs or answering questions. Any of those sorts of things. We also did a blog post on the Puppet Labs blog once a month that focused less on the numbers and more on recognizing contributors. And for this report, there was a lot of human filtering that happened. We sometimes highlighted someone who made some important but small contribution that just wouldn't have shown up in the top participants list. And we also tended to filter out people who had behavior that we did not want to encourage. People who were using up lots of people's time and not a lot of actual output. And instead, we put a heavier focus on the people who were helpful and answered lots of questions from other users. Now, I also wanted to provide some example metrics that show how metrics fit into your strategic efforts. So at Pivotal, as I mentioned earlier, my current focus is on putting together a strategy for our open source Kubernetes contributions. So I'll focus my example metrics on Pivotal and Kubernetes. Lucky for me, the CNC app has metrics for all of their projects, including Kubernetes. So this was really a great starting point for me to better understand the existing Kubernetes community. They use dev stats to build their metrics dashboards, which you can find at a link at the bottom of the slide. But first, let's start with a quick discussion about vanity metrics. So when I talk about vanity metrics, I'm referring to those metrics that make you look good, but that don't actually tell you much about whether or not you're really successfully achieving your goals. So I'll admit it, like anyone else, I look at these metrics and it feels pretty great when you or your company are moving up in the ranking. But I try really, really hard to keep these vanity metrics out of my strategies and plans since they don't really tell you much about whether or not you're being successful. For example, let's say that your organization is moving up in this ranking. It looks great. But what if you're moving up because you have a rogue employee or possibly group of employees who are gaming the metrics by submitting a whole bunch of trivial one-character changes, each as a separate commit with a separate pull request. So this is gonna drive the commit or pull request numbers up for your organization, but likely at the expense of your reputation as you annoy all of the other contributors. Now I know none of you would do this. None of your colleagues would do this. But this is actually a real example of something that's happened more than once within the Kubernetes project. And we have this discussion in the contributor experience, say, on a fairly regular basis because companies, people at employees at companies do do this. And I know this is kind of an extreme example, but there are plenty of other ways that people can participate that would drive numbers up without actually helping your organization achieve its goals. So I just encourage you to be a little, to be aware of the vanity metrics. Now in Kubernetes, most of the work is done within various special interest groups or SIGs. So the Kubernetes contribution strategy that I'm working on at Pivotal is focused on coming up with specific strategies for the SIGs that are important to us so that the various teams across Pivotal can understand where their contributions to Kubernetes can have the biggest benefit. And be aligned with the SIGs that are going to be important for not just current work at Pivotal, but also future work at Pivotal. And because Kubernetes is such a massive project with daily changes, there is no possible way that one person can possibly understand every single detail of every single thing that's happening in the 30 plus SIGs in the dozen or so working groups. So to be able to build a strategy for our work within Kubernetes, I need to be able to find out who I can talk to to better understand what we've done so far in addition to what we might wanna do in the future. And also, Pivotal's not, we're not huge, but we're not a small company. So finding these people can be a real challenge. And one of the things I did in my first couple of weeks at Pivotal was to start to dig into the existing metrics that the CNCF already has to understand our current progress and who's already been contributing within, and who's been contributing to which SIGs within the Kubernetes project. And this helped me find some people to reach out to and gave me a good baseline for where we were already active. Now here are also a few snippets from a Grimoire Labs dashboard that came from Biturgia with a few more details showing Pivotal's contributions to Kubernetes over just the past three months. So before looking at the Kubernetes data, I had never really used DevStats, so it probably has some features that I don't know about. But I found it a little bit limiting compared to the dashboards using Grimoire Labs as far as really digging in and figuring out what individual people were doing. What I do love about these Grimoire Labs dashboards is that they're built on top of Kibana, which means that I can filter my view of the data on any field available in the database, which is pretty cool. In this case, I've just filtered it on organization name to get the GitHub commits from Pivotal. But I can also filter on a single person or a repository, time zone, a bunch of other stuff that's in the database. And I find these really useful for better understanding how individuals are participating within a community. For example, if I look at contributions from Honus, who's up at the top there, from our London office, you can see that he's removed a whole bunch of lines of code, which indicates he's been contributing to existing files, right? So he's been removing things, adding things, and contributing to existing stuff. And he's done this across a number of different repositories. Ben and Leah, on the other hand, who work together on Kubernetes for Windows in our New York office, have only added lines during the time period that I used, mainly because I could get this as a good example. I shortened the time period drastically. But it corresponds to the Kubernetes enhancement proposal that they just submitted for Windows node support. And you can also clearly see in the top, you can see the time zone split, right? Between our Bay Area office, our New York office, and our London office, which is where most of the Kubernetes work happens, which is kind of cool to look at. But again, this really helps me understand who's contributed to Kubernetes from within Pivotal and which areas we've contributed to, but with a few more details than what I could manage to extract out of the DevStats dashboards. Now, when getting started with metrics, there are a couple of different approaches that you can take. Before you start building anything, it's a really good idea to look around and see what kind of existing dashboards already exist that you can use. Many projects already have these dashboards, like the CNCF dashboards for Kubernetes that I used as a starting point. And you can also take the DIY approach. So I've also done a fair bit of this and built my own dashboards, gathered my own metrics. Requires a bit more technical know-how, but it's a good approach for people who have the time and resources to build and maintain your own metrics. If you have really simple needs, you might even be able to just easily hack something together with the GitHub APIs. Those are pretty easy to use. For most people, the tools within the Chaos project, like Grimor Labs tools, for example, Augur is another one, are a pretty good starting point. DevStats is also an open source project itself under the CNCF, so you can always use that. There are loads of other tools. It's a tool called Measure that Jeremy Garcia and Datadog has been involved in and presenting about, but loads of different tools you can use. By far the easiest choice is to pay someone, so Biturgia does this. Bold disclaimer, I'm on their advisory board, but you can get companies like that to build and maintain the dashboards for you, which requires way less time. And Biturgia, for example, uses all open source tools, so you're also supporting their contributions back to open source projects, which is pretty cool. But in general, I recommend kind of doing a little bit of all three, if you possibly can, and doing them in this order, because you can use existing dashboards, see how much of what you need already exists, requires a lot less time. Then if you have the time and skills required, hack something together using some existing tools, and which, if nothing else, is a really great learning experience to help you figure out exactly what you need. And then if you want someone else to maintain your dashboard, you can pay someone to do that. It's also really important to think about other metrics and measurements that aren't necessarily in a traditional community metrics dashboard. There are loads of things that I've used over the years to measure whether or not we're successful that you might not find on a traditional dashboard. For example, when I was at Puppet, one of the things I tracked was whether we were increasing the diversity of speakers at our annual conference, which was one of just several measures that we used to determine success towards becoming more inclusive and more diverse within our community. And loads of these diversity and inclusion metrics aren't often found in traditional dashboards, but they're a really important part of measuring whether you're successfully achieving diversity and inclusion goals for your project. The Chaos Project, again, also has an entire working group devoted to diversity and inclusion metrics. For projects that are really important to your organization, you might have goals around having specific people move into leadership positions, or especially in key areas within a project. And the leadership positions available, so they vary widely depending on the project, but they could include becoming board members, maintainers, committers, reviewers. But it's important to measure this separately because these positions rarely, if ever, appear on the community metrics dashboards. And many people who move into these leadership positions spend quite a bit of time reviewing contributions, which means that they often actually have fewer commits themselves. And this means that the dashboards may actually show them as having decreased contributions when they're actually doing more within the project and having increased influence. So it's important to look at that a bit separately. In some cases, you might have specific features that you want to contribute or specific issues that need to be fixed. In these cases, you might wanna measure a specific item like a specific pull request merge, a specific bug fix, or some other contribution that you need outside of the metrics dashboard. So in other words, metrics dashboards are great. I love them, I love data. But you'll likely need to find other ways to measure other things outside of your dashboard to build a complete set of metrics that help you determine whether or not you're actually achieving your goals. Now, I mentioned the Chaos Project before, and if you're interested in working on metrics, I would encourage you to join and participate in chaos. So it's an open source project. Woo, several people in here are in the Chaos Project, unsurprisingly, since this is a metric stock. But it's an open source project, it's under the Linux Foundation, and we welcome anyone regardless of your skills. So while we certainly have people writing code to build metrics tools, we also have a lot of people coming up with metrics and writing descriptions for them. So we also love people who can help us promote chaos, present it conferences, and all sorts of other tasks. So this is something that, personally, I'm just incredibly passionate about. There's the link to get you started. We have three working groups. So there's probably something of interest to you if you're interested in metrics. We have one focused on growth, maturity, and decline of open source projects. We have a diversity and inclusion, and then we have one focused on metrics that don't fit anywhere else. And we have mailing lists. We have a general weekly call. Each of the working groups has various calls that you can participate in. And as I mentioned before, anyone with any variety of skills can participate in a chaos project. So that's my shameless plug for an open source project that I am particularly passionate about. And with that, there are three minutes for questions. And if the next speaker wants to come up and get set up, that would be cool too. So thank you. So what questions do you have for me today? Hi. Hi. Sorry. How do you begin to measure, say, the diversity of a particular project? On any axis of diversity? So the question was, how do you measure diversity within an open source project? I wish I had this awesome magic bullet for that because the reality is it is super hard to measure diversity and inclusion within a project. So within the diversity and inclusion working group, we actually have seven different focus areas that we're looking at. So we're looking at things like, are the events for your project, do you have attendees that are diverse in various ways? And we have a whole bunch of dimensions of diversity that we've defined. So it's incredibly complex. A lot of times it comes down to, one of the ways you can measure it is by actually surveying people and asking them how they feel within the project. Do they feel included? Do they feel like it's a diverse project? You can measure attendee demographics for events and participation. Yeah, it's a, because it's such a wide topic, there is no single answer to that. I'm gonna encourage you to go to the diversity and inclusion repository for chaos and have a look. Yeah, I just, I was kind of interested in that because surveying people from an underrepresented group, you're liable to get the answer. You know, I prefer not to say I'd have thought in quite a lot of cases. So yeah, it does seem to be hard. Yeah, it's really hard. There was a talk this morning in the Python Dev Room that tried to answer the previous question. So I suggest that you check it out later on because there's a video of it and you can watch the slides. So that was a recommendation for a talk in the, Yeah, it was a recommendation. It's already done, it was this morning. Okay, cool. In the Python Dev Room. Luckily Fosdum records everything because they're awesome. All right, microphone in transition, we have another question. One more, and I'll give you the microphone. Oh, you're hitting my backie-packie-packie-packie-packie-packie. I get that on your way. One more question and then I'll give it up. Yeah, so one small question. You have to talk a little bit.