 Welcome everyone to our buff. I'm super excited that we all found our way here. It's quite unusual format. So let's see, let's make the best use of this as we are all figuring out how our tools work and how video conferencing work. I'm going to share my screen because to set the stage and start the conversation for our birds of feather, I prepared a few slides. What I want to talk about is the challenges around open source metrics and to recap what those challenges are and these are only samples. I'm sure you have many more and I would love to hear your own challenges afterwards. Then I want to talk about how together we can be better stronger faster in having open source metrics specifically around the chaos project. And finally, I want to have that open conversation with you all. A little bit about myself. I recently last year earned PhD and my focus was in open source corporate engagement in open source through this very qualitative work interviewing participating in open source. We heard that especially companies want to understand open source communities through metrics and we had a session at the open source leadership summit three years ago in Lake Tahoe. And from that session, there was so much interest in open source metrics that we started the chaos project. So I'm one of the co funders of the chaos project. And what that is, I will cover later. After I graduated, I joined beturgya as the director of sales. And what what really defines me is my mission to help open source communities companies foundations become professional in their use of open source metrics. And I do this through the work in the chaos project and through my work in beturgya. So that is a little bit about me. When we look at metrics, the definition of a metric is a method of measuring something or the insights obtained from this measuring. And so the way I think of metrics is we have our open source communities we have the project data. We have methods for collecting the data for analyzing it and then we have the insights that come from that. Now, this is an ideal situation and there are several challenges to making this happen. Challenges I group in legal and ethical challenges, organizational challenges and technological challenges. The legal and ethical challenges that I hear time and time again are around laws like GDPR the European general data protection regulation that empowers the data subjects or in our case there would be our community members to have ownership of the data and have a say in how that data about them is being used or not used and even deleted. So that's the challenge if you want to have metrics about what is happening in our open source communities with our members. Data privacy, especially if we go into diversity and inclusion, there might be a chance that we out someone. So those are concerns and then informed consent. This is something that especially is embedded in GDPR when someone gives us permission to analyze their data. Usually there has to be an opt in, but there are a few cases where opt out is possible so rather than our community members saying yes, it's okay. There can be a gray area in open source where we say our community members have contributed to an open source community knowing that all of this is public. And so because we're analyzing this to help the community, we have legitimate interest and we can opt out or we give our community members an opt out process where they can say, oh no, please do not analyze my data as part of your analysis. So those are some of the legal and ethical challenges that I keep hearing about going on to organizational challenges. Why do we measure open source communities? Why do we look at the data? That is something that should be answered by our metric strategy. We should know why we are starting to embark on a metric journey. And knowing what it is that we're looking for and why we're looking at it, can we actually get actionable insights because our communities produce a lot of trace data. This is just data that is created as our members interact with each other. And so it's starting to analyze that is like opening a fire hose of data. An organizational challenge is buy-in from the community. I have seen communities that embark on a metric journey themselves, so obviously there is buy-in. In other cases, the metrics might be generated by a foundation or a company where it's a little bit more negotiation necessary with the community to make sure everyone knows what is happening on board with them. Now, moving on to technical challenges. Obviously, we need to get quality data and getting that data into a format that we can analyze and making sure that there are no inconsistencies in the data we collect, making sure that we can access the APIs of the tools that our open source communities use. All of those are the technical challenges around getting data. Managing identities is a very interesting challenge because our community members use different logins, different usernames in different collaboration platforms. On emails, in Slack, on IRC, in GitHub, it's even on Twitter. And if we assume that we want to understand our entire community, we need to somehow consolidate all of these different identities or profiles so that we can say this is actually the same person, which then gives us insights to the contributor journey, their activity throughout the different parts of the project, and so on. And finally, we need to be able to do reporting visualization. That is how we make the data that we collect insightful and actionable, and how we do that is a challenge. Now, after looking through all of these challenges, I want to recognize that there are some super experts who can manage all of this. But those super experts are rare. Being able to look at an open source community and say, hey, this is healthy community because of these aspects. There are some people who can do this. But as we heard in the keynote this morning, there are a lot of people coming into open source or tech in general who don't have that background, who don't have that knowledge. And so let's build together a tool set to go to be better stronger and faster around open source metrics. And so that is what the chaos project is about. It's to create analytics and metrics to help define community health. And to do that together, we started this project three years ago at the Linux Foundation with academics to really provide the rigor with tool makers like the Georgia to have software that provides open source metrics with the practitioners from various companies and foundations who actually use metrics in their day job. So that is the background of chaos and how we got started moving on to better developing practices together helps us analyze communities at scale through metrics and figuring out how to do that. By having shared practices or shared language around our metrics gives us the ability to compare across communities to transfer our analysis and analytics skills. Having open source metrics can reduce our bias in decision making. We have to rely on the grapevine to know what's going on in our communities. We can actually use data to manage our communities and some communities have grown so large that there's no way a single person can know what's going on everywhere. Through the right metrics, we can actually get those insights we need and even share decision making because we have an objective foundation to work from and so if we have a governance model in our open source communities where we have shared decision making the metrics can help inform those decisions. So that is better moving on to stronger chaos metrics we define chaos metrics in the chaos community. The work is organized in working groups and there are five of them. First inclusion is a work group that looks at how do we understand how welcoming our communities are and how well do we include people in our processes. Risk is looking at business risk, at license risk, at the risk that communities go away because no one wants to be stuck with a project that they then have to maintain. And so understanding that risk is important. Evolution is a working group that looks at the growth, maturity and decline of communities and defining metrics around especially activity, what's happening on issue trackers and commits and different channels and how's that evolving over time to understand that. We have value metrics, value metrics, what's the value of an open source project in on society as a whole to other open source communities to the individual contributors that are in the project or the value to businesses that come to rely on the open source software in their innovation streams. So what are the metrics we can understand there. And the last working group common is one we started because there are metrics that multiple of these working groups are relying on, like information about contributors or what is a commit or what are forks how do we measure that, which has multiple implications for other working groups we combine those in a common working group to define that. So better stronger faster, you're moving on to faster that chaos project is producing software to collect data from communities present the metrics analyze it. The grimoire lab project is used by companies like Red Hat and Hashi Corp and Ericsson and others. And auger is a software we have for piloting new metrics and trying out ideas. Craig it is another tool we have for looking at the commit history. Moving on to one last slide on lessons learned. So having had these conversations for three years now about community metrics open source metrics. I hear lots that we should listen, we should collect everything when we start our metric journey, because we don't know yet what metrics might be interesting in the future and having that data already allows us to start our metrics strategy later and we already have that foundation, especially when we start to implement changes in our communities to then be able to say this was the baseline from before and here are the changes in our metrics afterwards. Also ask questions of the data to achieve your goals. We use the GQM or the goal question metric approach and chaos to say you have to start with your strategy you have to understand why you're starting to look at metrics. And to choose the metrics so that they support your understanding of whether you reach the goals you have for your open source community. Just looking at the easy to get metrics doesn't actually help you make any improvements. Also tell a story with the data and just saying our commit count went up or down is not interesting enough or not insightful enough you have to actually know the context of the community. Understand why the change is happening and if you don't know why then start to dig a little deeper. But let's say the number of new issues or issue comments is going up. Could be that there was a release that is super buggy and we have more book reports could be that there are more users suddenly who are asking questions or it could just be that Google some of code is around the corner and students are asking questions because they're interested in participating. You know the context tell the story avoid gaming of metrics. If we use metrics to incentivize our community members, then there is a chance that they will focus their attention more on improving their metrics than actually advancing the community. And that can actually be detrimental to the community as a whole and value all contributions. We should not be limited to get up issues and commits just because those are the easy metrics to get. There are several resources I can recommend. We started a podcast where we like to share stories and experience around open source metrics community health. And if you have an interesting story, please reach out. I'm happy to get you on the podcast to tell your story. Look at the metrics. It's all on our website. Look at the software. And then if you also want to be participating in the chaos community amongst peers, then you're welcome to join us there. Join the main list and our regular calls. They're open to everyone. So together we can build together we can be better stronger faster about open source metrics. And I would like to have a conversation with you also. If you could ask your questions in the question section, right in the chat or even raise your hand. You should be on this platform so that we can actually bring you on stage so you can ask questions or just make a comment about your own experience. This is a birds of feather so I'm not the smartest person in the room we all have our own experiences to share. And I encourage you to take that opportunity. I'm going to stop sharing my screen. Okay, we have question here. How do you think the metrics discovery and storytelling process changes between metrics for the company's purposes, such as use in marketing performance, etc, and metrics to tell a story about the community's health itself. Again, I don't have to be the one answering this if anyone has thoughts on this question. Please raise your hand. I'm happy to have you come on stage and share your own experience from my own perspective. Let's talk about open source metrics in an organization. There are business goals, objective strategies that need to be met. And so the metrics should support whether the company is achieving those goals. The company has an open source programs office that open source programs office has its own goals and reasons for existing and being able to show yes, we are accomplishing why this hospital was created can secure funding. Make sure that the program is not being cut in the future and even get more funding for an expanded operation. It's it's important to have that alignment with the goals. Community health on is something that is interesting, maybe in that context as well. Because that might be something that introduces risk if it's missing, or that the company just cares about as a good will kind of thing. But I think the key focus in organization is to really line the metrics with the business objectives. And if you're doing a report for an open source community, then we are much more concerned about the longevity of the community and telling the story from a very different angle. On the chaos podcast on chaos cast, they have an episode coming up with Ruth cheesley, who is the community manager of the multi community. And she had to do internal reporting, but then use those same metrics analytics to craft them into a blog post and share them with the open source community. And the nice thing about that is it creates a dialogue with the community members on where the community is going what the project is lacking or being good at and it can actually drive some changes. So, let's see. Here's another question. What are user metric proxies when your project has no server? And I assume this question is how do we know how many users does our project actually have because an open source as we know is we don't like telemetry that much or having a call home from the software. Because that could be considered spying on our users and might be unethical. So I see. When you raise their hand. Do you want to jump in here and talk about this. Hello. Am I here? I'm curable. Great. I like this question. Thank you for asking this question. I think it's really important because we're entering an age now with security becoming one of the most major factors of what metrics you choose or don't choose to view in your customer value journey. And coming from an LGBTQ standpoint in my past I spent a lot of time talking about like how do you count people who are closeted or aren't available in your community. And in a lot of ways it's very, very difficult to tag a number to someone who explicitly doesn't want to be seen. And it's kind of similar if you don't have a server or you don't have the technical capacities to find that number. Maybe there are ways qualitatively to get that number and it's OK if it's not intensely accurate because accuracy doesn't matter so much as usefulness. So if you can find a truth that's great. Get good then get better. If you can find a more useful truth switch over to it. And so some some ways of getting some insights to a number of users. These are not exact sciences. But I've seen people use number of people asking questions on Stack Overflow or issues and so on as indicators of more of your users user activity. In the case of LibreOffice I just had a conversation with them. They're looking at estimating how many computers there are and what percentage of those computers they have based on surveys and experience. So those are those are some ways that user counts can be estimated. If it's software like OpenSSL and we know it's in almost every server then we can count how many servers there are or there are at least estimates out there to use. So those are some thoughts on getting a sense for how many users there are. Creating a user community is a valuable thing to do. I'm having the engagement of those users and that can give you an idea but I also realized that not everyone is actually willing to join a user community. Many people are just happy using an open source software and never declaring that they use it. Let's see here's another question. Once we are in the beginning, what kind of metrics can we get to know our users profile? We used to have a registration but developers were not registering and not trying out our technology. So I have to make a few assumptions here. It sounds like a registration for developers. So this open source tool is for developers who are the users. So they're not the developers of the actual tool but the developers using that open source software. To get insights on who those users are, unfortunately there's no good way to do it besides asking them to self-identify. At least I have not seen anything in practice. If anyone here on this buff has any ideas, please jump in. Share your insights. Here's another question. Are there any plans to move to cloud in a sense of GitHub application or software as a service? Talking about Grimoire Labs. The answer is yes. There is a cloud version. It's called Codron. Codron.io is an early alpha version of Grimoire Lab where you can just plug in my answer here. I'll put in the chat. So if you go to Codron.io, it's a neat platform. You can just enter your GitHub organization or user. And it starts collecting data about that repositories inside that organization. And you can add multiple organizations or whatever project repositories you want to analyze. Codron also supports GitLab, any generic Git repository that is publicly available, and Meetup data. And in the future, Codron will support more data sources and be providing other kinds of analytics. But it's built on the same Grimoire Lab technology. And so I understand that I didn't talk about Grimoire Lab during the slides. So I'll just give a little bit background. Grimoire Lab is a tool set developed by Biturgya based on 15 plus years experience in doing research into open source communities and projects. And then for eight years ago, Biturgya was started from this open source research team to actually provide metrics and insights to companies because they were asking for it. They were willing to pay for it. And about four years ago, the company completely rewrote the metrics platform. And this is what we now call the Grimoire Lab software. And it's a collection of different tools to ingest data, clean the data, have processing on it, do the identity matching, and then have their consistent reporting and dashboarding. And so Codron.io uses the same technology under the hood, but scales differently in the cloud and makes the management of the platform much easier. So thank you for that question. Again, this is birds of a feather. So I don't have to be the one talking here. I'm happy to let any of you come on stage and share your experience and your stories and ask the other attendees questions. So here's another very interesting question. How can I know if my numbers are good or not? Where can I find bench marking? For example, we have 100 people in our telegram community, but only 10% are engaged in answering questions. Is that good? So the question for that. I see Vinnia raise their hand again. I'm happy to have you come on stage if you want to take this question. I see myself here as a facilitator. I'm happy to talk, but if anyone else wants to talk, I'm happy to give space. I like this whole stage thing, but I kind of feel like it's adding a lot to like power distance or whatever. So it's like literally stepping up to a mic in a real place. There's just a certain level of anxiety to it. It's kind of fun, but I don't know. Well, thank you Vinnia for stepping up to the mic. Yeah, I hope my mic is actually good. So kind of regarding this, I feel like it's an important question to ask. And I kind of think that we should spend some time getting multiple answers for this. Because ultimately determining the activity level of a individual community depends on where you come from. I'm a marketer slash brand community manager. So I spend a lot of time looking at power users of which only about 2% will be there. But on the flip side, I also spend a lot of time running love our lurkers campaigns in order to get that 40% of people who are just reading to actually raise their hands and say I'm here. So it largely depends, I think, on the value that you're providing essentially an invisible audience. How do you determine what's actually useful and what's not useful? Because you can't force people to make comments. I think it's an interesting question. And benchmarking is going to have to be based on week over week activity in your own group. Not some statement about your gourmet or anyone else in this group saying, no, yeah, 10% seems fine or 40% is a little bit high. National benchmarking seems important, but you have to ask yourself, is it a vanity metric in relation to the community that you're running? Absolutely. Benchmarking against your own history is one way to know whether your current 10% is good and improvement, or if there was a loss in activity, which could be a warning sign. Yeah, especially as your community grows, that number will change. Smaller communities tend to have larger participation per user base, whereas as you get larger and larger, your lurkers start to outpace it. So you'll see that percentage go down week over week. Yeah, and then if you actually want to benchmark something that during my PhD, we have tried. I wasn't the one actually doing the work. We had someone on our research team who was pursuing this idea that we can compare different projects and we can just choose one that is similar to our own. And to say we take a benchmark to a similar, so if we were to take countries, there's no point in comparing the GDP of a really wealthy OTC country with country that does not have that infrastructure and the history of having free markets and everything. They're just very different. And the same is true in open source communities. We have very different governance structures. We have very different collaboration patterns. Even projects that use GitHub have different ways of leveraging GitHub to get the work in the communities done. I mean, in extreme case, some use the pull request process for getting contributions or even as a maintainer doing contributions while others just push straight to master. And measuring that activity level means something different in the communities. So as we do benchmarking comparison, we have to make sure that we compare apples to apples. Another related comment on this is quality models. Since the beginning of open source community measurement and open source metrics, there have been attempts to say a healthy community has these characteristics and we can build a quality model that takes the metrics in defined thresholds, like 10% of active participation is good. So it gets a green light, whereas if it's less, it gets an orange or red light. These quality models can be very powerful for managing a program of open source communities for keeping tabs on many different projects and to say, oh, there's something going on here. I should look into this. The work of defining quality models, however, and there are several in the academic literature, the problem is that everyone has slightly different criteria. And so there is no unified open source quality model that has been transcending or found useful by someone else than the person or the group that created it. When we started the chaos project, actually, we had a tool contributed to the project that did exactly that, that took the metrics in and then had the stop light, red, green, yellow for the project. And you could parameterize it was really awesome. But the way the tool worked didn't make sense to others. And so right now it's in the attic. If anyone wants to continue that work, please let us know. We are happy to take the software back out of the attic and have a community around it. So I see there are some chat messages. And again, if anyone wants to come on stage and ask a question, I believe we still have 10 minutes left. So plenty of time. I'm just reading the chat right now. Okay, so it looks like in the chat, we are, Vynia and Carol and Ilvanini are still talking about the, how do we benchmark? So that's, that's awesome. Let's see, we talked about several things. One thing that I find really interesting is around gaming of metrics. I'll just keep rambling on until someone interjects, raise a hand, ask a question or something. Gaming metrics is something that from the very beginning of the chaos project has been discussed. And the idea is that you get what you measure. Or if you measure, you have to be careful what you measure because you might just get that. So you get the sentiment and it shapes behavior. Once we start measuring behavior, people will respond to that and adjust their behavior. The, there are several things we have discussed to mitigate that. One is to not provide metrics at the individual contributor level. Because if it's just a group effort, then it's not as strong an individual incentive to shape the behavior to improve the metrics. Then also be very mindful about what changes you want to make in the community and what metrics can indicate those changes are successful. And then try to guide the participation in that way. But then also reevaluate after a while because the goals may have changed. And the behavior that we were trying to incite at first might no longer be the behavior we need in that moment. Or we have reached our goal and it's time to focus on something else. So we have to reevaluate our metrics strategy and change the metrics that we highlight and focus on. Okay, here's another question. I think it's an awesome topic. What plans or tactics do you all recommend once it comes to brass tax to ensure that the gamification metric doesn't infringe on natural organic activity in the group. People usually worry about gamification backfiring. Now I'm not familiar with the term brass tax. So I'm sorry that might just be that I'm not a native English speaker. So if you could elaborate what brass tax is, but going on into into verification if any of you here on this birds of feather call session have thoughts on this please write in the chat or raise your hand and take the microphone. So gamification what I talked about so far was don't provide individual metrics, because then people don't have an incentive to game their own metrics. Be very mindful about what metrics to highlight and focus on, and especially what incentives are tied to which metrics. And then reevaluate regularly whether those are still the goals of the community whether you've reached the goals or should focus on something else brass tax when the rubber meets the road. Okay, so that's on gaming. I'm also asked often about the ethical and legal challenges and I, I can reiterate and expand on what I was sharing earlier. The best strategies to be as transparent and open with your community as you can. Because the, if you look at the legal side of things and again, you should probably ask the lawyer for your specific context and goals. But the legal context seems to be that unless someone complains, we're good. And by being open transparent and providing ways for people to, or for our community members to opt out and have a say in the community metric strategy. We can mitigate that upfront. So I've been some messages in the chat. Yes. And the patches metric where people then create smaller, almost reveal patches just to update their up their metrics. Good, good example. Thanks for sharing that. I know another community was having plus one plus two upvotes and the game turned into waiting for someone experienced to make a plus two upvote and then doing your own plus one right after that. So you're reviewing code or I don't remember the exact context where this happened, but it turned into a not so useful exercise. Because you were earning status by doing these reviews and providing plus one plus two votes. I also like vinnia vinnia is also active in the open source in the chaos project and vinnia if you're okay with coming on stage again you have a really nice metaphor about dashboards and reports. And before I say your metaphor that I learned from you, I want to give you the chance of maybe sharing that yourself. Yeah, can you hear me. Yes. Right. Yeah. So this isn't actually my metaphor I want to be very clear this actually came from Chris Mercer he goes by Mercer at measurement marketing.io. He's a measurement marketer. And I have found that this is one of the most important ways for communities to look at how you're building out stuff. And I think of a dashboard I think of a dashboard on a car right you're going 75 miles down the highway. You want to know how fast you're going, whether you should slow down speed up stop or if your engines on fire. Those, that's for maybe five metrics. So what are those four or five metrics for you. Then, once you discover that your engine is smoking, something happened, you pull over to the side of the road and you ask yourself, Okay, what happened. What's the point where you open up the hood and you have a whole bunch of extra metrics. Was it oil was it gas was it fluids. Did something happen to my carburetor or was it my belt and understanding how that engine runs understanding how your communities engine runs gives you an understanding of what you should check first what you should check last, and what to do about it. There should be a chief few metrics that tell you whether you're doing well, keep going, hit the brakes, slow down, and your reports, your detailed inclinations and everything like that, they should be out of your site, while you're driving. When something does happen, that's when you open up the hood and you take a look at your report metrics. Thank you. I, even though the metaphor might not come from you, it will always be attributed to you in my mind because I learned from you. Yeah, and I probably shouldn't say this here necessarily, but I am working on getting Mercer onto a chaos cast so we can hear it straight from the horse's mouth, so to speak. And we'll kind of discuss that a little bit more on chaos cast as we move forward into the future, I guess. I'm still looking forward to that. Yeah. So I think we have about a minute left. If you want to stay on. If you have any thoughts. You can do that. If anyone else has any ideas, any questions. Last minute thoughts, please use the chat or the questions feature here. In any case, I thank you for joining us for this birds of a feather in this very new format. I appreciate all of your questions and engagement participation. We can head over to the slack channel with pound sign to track community leadership Ospo to do and continue the conversation there. If, and I'll be there for the next two days just hanging out. So, pain me there. You can also start an individual conversation. Yeah. I just saw a clapping hands rise up. How do you do that. Oh, that is so much fun. Oh, there's a new, you know, I was trying to figure out how to do emoticons on individual chats. It looks like they just do the Facebook live thing going on here. I have a question. I hope you don't mind but just something super simple. What is everyone's vote on whether the last icon is prayer hands, or two people high fiving. I kind of want to know where people wait that I honestly don't know. Yeah. I think it was prayer hands for the longest time and I thought it was two people high fiving, but it makes sense because the sleeves are both the same color. Anyway, when I, when I look at the emoticons here on my Mac I think it does say their prayer hands as the official name for them. Yeah. I never thought of them as high fiving. Yeah, a lot of people just started using them as high fiving in a few of the other online conferences I've had since COVID. It's interesting. Yeah. Well, I also think of them as thank you, or something like that. I guess the Mac says hands pressed together. Let's close the session. We're out of time. Thank you so much. Also, great. Thank you to Amy. Amy has been the host in the background, helping me with this session. So thank you very much. Have a good rest of your conference.