 So welcome to this Birds of a Feather session, focused on people who have what I call specialized questions that you wanna ask about community health and sustainability at scale. And so I'd like to start with just a brief discussion of what kinds of things have occurred at scale and what kinds of projects or groups of repositories that we have looked at at scale. So first of all, what do I mean by at scale? I'm interested in, so when I say at scale, we're usually, I'm in the thousands of repositories, usually 2000 or more, has many as 25,000 repositories that we're trying to create signals for and understand at any one time. And usually these are open source program offices inside of organizations that have these kinds of questions and this diversity of projects. And also companies that are part of the open source community who manage thousands of projects at a time. And I'd like to open a chat discussion since that's how you're able to participate. And just ask, when we say at scale and when we talk, when you decided to participate in this Birds of a Feather session on health and sustainability at scale, what does at scale mean to you? And I believe you can use your chat to get to sort of typing in what at scale means to you. Okay, attendees, apparently you need to raise your hands first in order to be called on. So for those of you who just are joining us, this is a Birds of a Feather session on health and sustainability at scale. I introduced the topic as large scale organizations that either use a number of open source projects in the thousands and wanna understand ongoing health and sustainability of those dependencies at scale or open source software companies that look at tens of thousands of projects that they have an interest in and wanna understand at scale. And I'm throwing out a question, thinking of a Birds of a Feather session being a little bit more conversational. My understanding is you have to raise your hand to ask a question in the chat, but I'm interested in what you think health and sustainability at scale is, what is it to you? Looking to the participants to ask a question, sort of state to you, what is it? What does that scale mean to you? So I don't know what it means that no one's raising their hand in a Birds of a Feather session. Let me try to think about, can, is everybody able, can somebody raise their hand just to make sure that that part of it is working? And so I've got a couple of people who've raised their hand, I don't seem to have the power to do anything with the hand raise. It seems that the raise hand function requires camera and mic not text. Okay, so you can ask, you can answer my question via text without raising your hand, but you can ask, you can speak I think if. Okay, and Dylan, I am not hearing you. That would probably be because I was not talking. Oh, okay, great, thank you for helping to sort out this process. Well, I guess since I'm on the spot as a co-founder of a small startup at scale to us means something with a business substantially more established and with access to much larger resources than what my co-founder, Vinny, and I currently have. It's a little bit, I guess ephemeral, a concept for someone that's still kind of as small and growing as we are, and I guess something that we're still interested in learning more about hence why we were on the call. Yeah, yeah, so that's a, so there are others who are in similar positions that have a sort of interest in understanding what it looks like to be at scale. And let's see, and then I have another, there's another person who accepted. So, all right, so Shriram, I think, are you the other person having a hand raised? I think, okay. Yeah, can you hear me? I can hear you now. I think what's happening is the person who's the host is the person who can give you the control, not me. So Megan's helping us out, so sorry. All right, no problem, I'm glad we can test it. So do you have a, I mean, is your interest in, or is your interest for definition of what at scale means different from what Dylan said? Sorry, I missed part of that, so I didn't get to hear everything Dylan said. I apologize, but I am interested in that kind of scalability for metrics. I am involved in the chaos project. We do an apps ecosystem. It's a work we're changing the name, but it's very much linked to scale the onboarding and various other projects we're doing, not at IT Renew, but at the Canom project. So, yeah, this is something interesting to me. Yeah, so that gets to sort of the question of what chaos is. And Sharon, you're clearly familiar with it. We are a project that provides both definitions of health and sustainability metrics as well as tools that let communities look at the health and sustainability by gathering and analyzing a bunch of data. So the main area where the metrics are developed, I've just put it in chat, and our general community site is just without the slash metrics trailer. And so chaos does these two things. One is sort of try to help define, provide standard definitions or reach consensus in a large group of industry participants about what does it mean? How do we count commits? How do we count newcomers? What are the things that are parameters? Newcomers is a really good example of, and newcomer, what we call it, what I call stickiness. Okay, you get so many newcomers a month. How many of them make a second contribution or a fifth contribution on a periodic basis in that those kinds of metrics can help people who are running large scale communities or startup open source communities determine if the new project that they have is scaling well and also share practices that they've had for scaling the projects as well. What we do with chaos and the tools that we have on chaos, and the tool I work most closely with is called Augur, is we gather, in the case of Augur, up to, I guess, 25,000 is the most we've gathered in a single database of repositories with lots of different information where we can look at relative newcomer participation, newcomer stickiness. We can also look at some of the metrics that we've defined around elephant factor. We have a license scanner that lets you know what licensing risk is, sometimes at scale, understanding licensing is really, really important. And so when you have a large scale enterprise, for example, if you're a startup firm and you have a large number of open source projects and you wanna understand the relative health, one way to do it is to just sort of get a survey of the landscape on some of the basic metrics and statistics, and another way to approach it is to take this giant collection of data that Augur lets you create in a database, and actually, I'm just typing the link to Augur in there, and actually starting to use some of the Jupyter notebooks that we're distributing to analyze things like pull request stickiness. You can also do competitive intelligence. So if you have a bunch of projects you're competing against, that is another way that people define scale. In the case of Shoram, who, am I saying your name correctly, Shoram? Yeah, sorry, the last bit there got crumbled. What was the last thing in the case? Am I saying your name correctly? Can you hear me already? All right, all right. Yeah, yeah, it was just that little bit. You were perfectly clear up until then. So yes, my name is, yeah, it's Shoram, for now it's Shoram, or Shree. I'm mostly known as Shree in the open source world, so. Yeah, okay. Oh yes, yes, Celine pointed out that I mistyped my, I actually mistyped the chaos group in our, on the internet there. So thank you very much, Celine. So the group that Shoram is working with is looking at kind of a, I would call it a orthogonal view of metrics definitions, and it really is kind of organically an at scale problem because in the case of the GNOME community, you have, I don't know, how many different applications for that desktop? We have a large number, and there's part, I would say like, I'm gonna put out a guess here, but there's nine that are part of the release of GNOME, but hundreds of others right there. Right. Yeah, and I would imagine that in the case, so in the case of this sort of orthogonal, specialized desktop app system, it's really hard to keep track of even hundreds of different repositories that are material to the utility or usefulness of that ecosystem, right? Right, right. I will also wanna add that it's not just GNOME, but essentially organizations, yeah, it's basically organizations that are public facing projects, like desktop projects, that are very complex because we're just basically platforms of various types. And so this effort is sort of because it's different from corporate-sponsored projects, but we have similar problems, but we have this additional thing where we deal with the public a lot more closely. And the consumer is using the tools. Right, exactly, exactly. And so how we interact with that public can be somewhat complex because it's not a narrow, I wanna say a cross-section, right? It's basically the Linux enthusiast, which poses an incredible amount of challenges for us at times. Yeah, yeah, yeah. So it's a public-facing environment of highly technical people who are not necessarily contributors. And so part of the responsiveness needs that specialized part of the CAS community has is to understand that there's some alignment between the public satisfaction or dissatisfaction and the developer bandwidth or commitment or ongoing contributions to a particular part of that ecosystem. Yes, exactly, exactly. And it's not just that, but quantifying those interactions through GitLab specifically, since that's been what our communities have been using. But we do a lot of quantitative things, even parts of abusive, like the abusive uses of emojis, right? Or some of the things that we can actually quantify. So we can quantify both negative interactions and positive interactions through emojis or some of these other things. And this is for us ways that these are things that are different that I feel that we have in other more controlled environments is that we don't have control over the people because a lot of them are mainly anonymous, right? So which gives them some freedom to do some free speech. I'll put that gentlemen. Yeah, yeah. Oh, that's, so that's a very different, you know, and I was listening to Dylan earlier. His interest, they're the interest that his, the firm that they're at has is in understanding, you know, what does scale look like and how would they help people understand its health and sustainability at scale? In your case, I think what I hear is that there's this public pressure piece of it. So scale to you means how much sort of help desk type of input can you manage? And then what are some metrics that you can put around that? Are you, is this your group? Is your subgroup looking at sort of those kinds of metrics? Or what are the kinds of metrics that are really critical for something where there's a lot of public input, which is really distinct from the kinds of things that most open source projects are attending to like contributors and commits and things like that. Right, I, well, at the moment, we're still putting together these metrics. Still in the very beginning stages, but we're essentially breaking down the problem into what are the information we're looking for based on roles and various other things. So if you give me a moment, sometime here I can try to put something together based on the work we've done, but I can't, I can't, I can't, I can't immediately. Oh, oh, oh. Oh, oh. First time, first time. I didn't prep myself for that. Yeah, no, I didn't know what I expected you to. So that makes, that makes perfect sense. Are there other folks who have different interests in what scalability is? I see some folks from Ubuntu here. I see a few other academics like myself in the room. Are there other kinds of sort of dimensions of scalability that folks may want to raise their hand and share with us as a, just examples of what brings your interest here. Again, my idea being my birds of a feather, the more we share about our particular challenges that we're facing, I think the more interesting the discussion might be. Okay, hearing none, I do have a chat question. So Selene is really interested if you look to your chat and I'm just sort of providing the sort of sports commentator where you look because it's not always clear in the system. So interested in the SCMS Dylan's working on, but by us so being able to measure community sentiment probably will be huge when it's working. SCMS, I don't know what SCMS is actually an acronym for. When we have Dylan raise his, Dylan, why don't you raise your hand again and see if Megan can give you a little bit more of the floor. And then that way we can, first of all, I don't know what SCMS is. Right, I lost track of that acronym, which is. So yeah, Venya Logan and I are in the process of introducing a new metric system in with the most recent chaos community metrics launch here. And what we went ahead and designed is the social currency metric system that is essentially a way to quantify qualitative data. That's essentially what that is. We are in the process of working with one of the, these students going through the Google summer coding program here that chaos sponsored and in the process of really, I guess, spinning things up and our big question is we have this implementation that is nearing what's essentially a minimum viable product. How do we take that MVP and continue to spin up the number of clients essentially that we can onboard, that we can serve and that we can support as that comes out of production and everything. And the way that we've kind of, the conclusion that we've come to is a sort of passive training program where essentially we do rounds of videos with that we would provide for free. Our big thing with the SCMS is basically being a free supported platform for anyone that's interested in the concept with more intensive lessons kind of locked behind that, that paywall, that being less of training, more of like a consultation sort of thing with more in-depth trainings available to those that choose to go that route. And so that's kind of how part of how we're trying to solve this question of scalability is allowing those that kind of want to DIY this whole thing to purchase basically lessons that will teach us how to do them if they're not getting what they want from the free ones and just trying to be as transparent in how we actually implement this process as possible. So that's kind of what the social currency metric system is and how we're kind of envisioning scalability at this point in time. Yeah, so for those of you not familiar with KS or for whom KS is a new thing, there's really two dimensions of scale that are being talked about. Dylan is talking about the social currency metric. And when social currency metric systems actually a very different way of thinking about scale than what some of the other folks like Shuram was talking about where he has hundreds of projects that he needs to kind of keep track of because there's high demand. That's one sort of category of scale. The social currency metric system is a way of, it's a very different than really every other chaos metric. And so far as most of the chaos metrics work to define a discrete kind of thing like issue comments and provide parameters that can be used to interpret those issue comments. So that any tool that implements the chaos metrics will give you consistent answers and parameters. And it becomes kind of tool agnostic in a very open source way. The social currency metric system in contrast is a very substantial kind of a metric that takes into account. I would say, Dylan correct me if you think this is too elaborate, but I would say probably constitutes 20, the weighting and aggregation of 20 to 25 distinct discrete metrics into a score of sorts to help understand the level of influence that different or the level of social currency that individuals in a particular ecosystem might have. Sorry, I started talking and forgot that I had muted myself. No, I'm sorry. Yeah, no, that's, we do have these, these five main metrics that we look at that we kind of have a 201 implementation that we were in the process of drawing up that breaks these down a little bit more. But some background on the SCMS is it's a result of I am by trade an anthropologist, venue by trade is a communication specialist and an anthropology student as well there. And so we were kind of coming at this from a rather different perspective both in terms of scalability and in terms of what the actual purpose of these metrics are than a lot of chaoses, the other metrics that they support and that they've helped to foster. We look at our five metrics first offer transparency, utility, consistency, merit and trust. And these seem like sentiments that would be more or less impossible to analyze. And they're definitely fairly difficult to do so, but by taking a look at qualitative sources from channels such as Facebook, GitHub comments, support emails and tickets and everything like that, you can get a bit better an idea of how people think, feel and are speaking about the organization that you're attached to. And that's, it definitely is a bit more upfront a little, yeah, I guess diagnostic if you will about the people that are working with the organizations that you're attached to. So is it trying to, so what's the ultimate goal, would you say of that perspective on scalability? So the ultimate goal is our goal at the start of this is to make the internet a better place, which is fairly ambitious, but it takes someone with just crazy enough to think that they can potentially change things around to make things to try and hit that ambition. That's kind of why we're all sitting here today with the various foundations that have coalesced into this very comprehensive organization and subsidiaries. But what our eventual end goal is, is to give businesses and organizations ways to quantify how they can be better serving the organizations surrounding their brand. And that's kind of the end goal here and what scalability for us looks like is essentially being able to take on more organizations. And while ideally we would like to give everyone one-on-one attention and all of that, and that might be kind of a scalability in terms of improving from just Venian myself, that's kind of still part of what we're trying to ask ourselves is what does this, is this a scaling of our company or our technology both? We're still, I wish I had a clearer answer than a shred of the shoulders for you, but we're still at the point where we've been very pedal to the metal with getting everything set up that we don't have as good of an answer for you there as well. So I'm just gonna share with people. So scalability from the perspective of this social currency metric system is really, I think kind of a horizontal scalability problem where there's a lot of different things. And I'm just sharing the proposed metric here so people get an idea and I'll also share it in the chat window. Oops, hang on. Go back here, go back here to the chat window. And that's the actual currently proposed social currency system. And you can see that it's just, there's a lot of different data gathered here. It's a fairly complex concept. Would you agree, Dylan? It's definitely very different. I would, it's something that's seen academic use for, especially in the field that Vinny and I met in anthropology for 40, 50 years, but it's definitely a new form of data analysis for kind of more privatized organizations, which thankfully means that it's very different. That doesn't necessarily mean it's unapproachable for those that don't have a background. Right, right. And so, and I think that it's very well articulated here. I think when it comes to implementation, it's more complex, for example, than when you think about scalability, one of the first metrics, the activity metrics that we've all looked at, or many people have looked at historically, might be something like code changes, which is a very fundamental metric that we did a lot of work at the very beginning to define with some clarity in the evolution working group. And it's a very straightforward definition and lasts about a page and really is just about this fundamental idea of volume of coding activity. And I think when I contrast it to what I would call a metric that's trying to scale across a number of things and aggregate and weight and measure them collectively as this concept of social currency. Do you agree with that, Dylan? That that's what it's trying to do? I had a little bit of a disruption in connection there, part of the way through it, but it sounded, from everything that I caught, it sounded pretty accurate. I'm sorry. So yeah, it's a no problem. So it's a multi-dimensional perspective as opposed to a discrete metric. It's talking about standardizing, collecting, analyzing communication traces, which broadly construed could be just about every other metric that is a communication trace. So commit may not be considered a communication trace, for example, but an issue or a comment or a comment on a pull request or a comment on an issue are all types of communication traces. And measuring all of these things could in and of themselves be a metric. So each of these could be in the historical work of chaos over the last three plus years. Each of those individual things that are part of workgroup, the social currency metric system is like a discrete metric. I suppose I would say, and I'll stop sharing my screen for a minute. I've shared the links if you wanna browse around. With that in mind then, and when I think about scalability from the, so there's really two perspectives on scalability that we've heard about so far. One is a complex metric, like social currency. The objective of social currency again is to what level is it trying to understand like where does it help understand, where does it help explain or define the health and sustainability of an online community? It's clearly where I'd call a scaled metric. It's extremely, got a lot of stuff in it. But what does it do to just tell us something about health and sustainability? That's not, I hate to put you on the spot. Just trying to get a little bit of discussion going around that. Not a problem. It's one of those things where the end goal, what it's supposed to do here is it's supposed to be included in like marketing conversations and such or conversations about interesting trends that are being seen in the qualitative data surrounding your company. Essentially say every Friday or bi-weekly, whatever your organization has a meeting where they talk about interesting trends that have been seen in someone's on interesting Facebook thread or a question that really sparked some good conversation and everything. Our goal is to capture the more meaningful conversations that people are having and to narrow in on the useful we call it sentiment, but the useful trends within those conversations in order to pull out what exactly you're doing well as an organization, what you as an organization could be doing better generally what people think and feel about what you do. Which is again, a pretty ambitious goal and something that's going to be fairly difficult to implement at first if you've never done something something like this, it's going to be difficult to kind of wrap your head around and everything, but that's at the end of the day, what it's supposed to do is give you an idea about the sentiment of those that are participating in these branded communities. Shreya and Celine asked a few really interesting questions that folks can see and I'm wondering if either of them want to elaborate or if we should provide an opportunity. Do we have metrics for inclusive language watching for master blacklist, et cetera? And this is a question by Tim at the Octo project at the Intel. One of the things that the chaos project is just doing is a badging project, it's a badging project for all of, I'm just going to side, we'll come back to the other thing, but this is an important question. Diversity and inclusion and using inclusive language like for example GitHub just announced they're going to get rid of the word master in the names of repositories. And so diversity and inclusion badging is a project that's part of chaos supported in part by the Sloan Foundation in Mozilla where the diversity and inclusion working group has defined a set of characteristics that I can't remember. The exact repo for that. But they've defined a set of event-based and participation-based kinds of metrics and the diversity and inclusion metrics are really distinct from the metrics that exist in other metric working groups because they're difficult to measure from electronic trace data. And the badging program is intended to follow the model of the open source, the Journal of Open Source software where there is peer review on every single awarding of a badge. So in some cases, in a lot of cases you can get a badge put onto your project simply by asking, filling out a questionnaire or making sure that certain files exist. In the case of the badging program we're putting together for diversity and inclusion those questionnaires, but there'll also be a peer review of the application for the badge by other members of the community. And the badge will have sort of a lifetime that I don't think has been fully determined yet somewhere between one and three years so that the use of inclusive language, I mean, use of inclusive language watching for words like master and blacklist are certainly part of the discussion in the diversity and inclusion working group. And I encourage you to look into that working group and ask about the badging program there. Now jumping back to some of the concerns. So communities suffer from tribalism at least out in the Linux enthusiast space without completely understanding everything I've read. How do I deal with weaponized campaigns against a project so the choice they want is more trusted? And then also wondering about tribalism and measuring that. Are there tools for visualizing networks within a community? Clashes can be easy to see in some visualizations. So I'd like to, a lot of, so from a birds of a feather perspective I would invite Shiram or Selene to talk a little bit more about those questions. I am able to talk about them for a while, if you like, but I think we have about 10 minutes left in this session. So I'd like to have either Shiram or Selene have an opportunity to perhaps elaborate just a little further on what you mean. And if you don't want to, it's okay. I'm just gonna give you a second to raise your hand and get called on. Okay, so hand raise. Okay, Shiram. Okay, so the desktop space is not exactly logical in terms of how they support. Support is like one would prefer one or the other, but there are people out there who are looking for weakness and will exploit anything to show that, yeah, you got the look of surprise on your face. I live in a world of good intention. So a little bit more about that. So when I do engagement, I'm usually in forums or like Reddit or like on our Linux or if I'm on or we're moderating r slash r cano, since I represent cano, I don't have a KDE representative here, but they will probably about to say that's similar, similar experience. Or in Twitter, there are forces always looking to somehow, I try to get a good word for this, but they're looking to knock us off, right? Because we're generally, at least for cano, we're generally the most installed, the default choice. And so there are a lot of tribal, the sort of tribalism comes in is when there's something wrong, right? They will augment that wrongness to be able to say, this is not the right project. They're trying to influence what should be the default because the project is changing things, right? Changing culture or things like that, whether we put, say something about Black Lives Matter, that was four days of dealing with incredible amounts of negativity in the sense of it. So the sense of the project, I mean, I just want to understand that the project itself is in those cases where the project takes a sort of a view toward diversity and inclusion. Exactly. Are these are the moments where people seek to weaponize that perspective and actually reduce the, try to subvert that project or try to replace that project with a competing one, but it's a, you view that as an opportunistic kind of attack? It is an opportunistic kind of attack. And it's not related to just, our stance on diversity. It could be as simple as a perception and that money was squandered, like a foundation money was financial thing. It doesn't matter if something can be used, it can be politicized. So there's a lot of politicization that happens out in the space because- What is the forum where this politicization takes place and Micah's echoing your sentiments about toxicity? And I think we could all have a beer together about that, but where do you see this actually? Where does it happen? Where does it happen? It happens usually on social media of some sort. I see it a lot on Reddit and there's no rhyme or reason because I mean, yeah, it's Reddit is a very toxic place, but it's also a place that we've actually, we looked at our numbers and we've been increasing, like at least in our own forums who triple more than tripled our subscribers. So, but that and Twitter, I can't make a post by anything related to diversity without having to deal with multiple days of issues with comments and toxicity. And just to bring it back, when your metrics is about how much our audience trusts us, then as your byline and socially constructed online, I start thinking, oh no, because I don't know, because there's so many loud voices, they're bombing up their minorities, but they know how to, there's an opportunity to manipulate and if there's an opportunity, they will. So I'm just gonna, we have, if I understand correctly, three minutes left. I would like anyone that has a specific link that they're willing to share where toxicity takes place in their ecosystem or an ecosystem they're aware of that you don't need to have to call your own out. I think the caste community, we have not discussed this issue so substantively as an obstacle to participation. So this kind of tribal aggression behavior isn't something that frankly, we've looked at it all. Earlier in my own career looking at open source, I did a lot of analysis of networks to under, network analysis and waiting of networks based on communicate both work and talk. And in that work, certainly we could identify the core periphery called extra periphery, contributory. And we can also do certain things like, maybe you should turn off your, yeah, there you go. So there's a lot of network analytic techniques that we can use that I've used in the past to identify types of leadership, types of participation based on the controlled world of how people do things in open source. I'm really interested in, like I heard Reddit, I would love to know about, and I heard Twitter, I'd love to know about subreddits or Twitters or Twitter hashtags or Twitter accounts where this kind of toxic tribalism takes place because I think that combined with analysis of the networks, which we have one Google summer code student focused on this summer could be really helpful for scaling questions. It sounds like scaling is, in this case, once you've scaled, then, and you have a voice and credibility, people have a desire, you become a point of attention. And when you take a stand on something with a social concern, that can cause you problems. It looks like there is a keep the conversation going. Please visit the number two track community leadership OSPO to do on our Slack workspace after the session ends. So this session is recorded. I think the Slack suggestion is a great one with 12 official minutes left, as I understand it, and Megan correct me if I'm wrong, but I believe that we are supposed to end at 510 on this schedule. It would be great if people would be willing to share with us. I will provide my own email address if you don't wanna share something, oops, and I can't even, apparently I'm not able to type my own information or the information of my project. So join that Slack channel, and I think the app ecosystem is interesting. I think some of these issues that we've identified are interesting, and I appreciate this birds with feather session went a completely different direction than I expected. I'm glad that it did. Thank you all so much for participating. And I look forward to connecting with you each in the future. Megan, am I right that this is the, I haven't done it, 510? I'm just trying not to, I don't wanna, the invite that I got was, it was originally scheduled for a bit longer, but I think we ended up at a little, oh yes, a little, correct, but you can go over a little, oh, I can't go over a little. So with communities, only a few hundred people, badly behaving people can be dealt with individually. Thousands more though, it's very difficult. And I think, you know, there's, I don't wanna call out any communities, but certainly, I guess there, I think most of the open source world is aware of some of the communication challenges that occurred in the Node community. Number of years ago, certainly some of the kernel community has had toxicity called out, but it sounds like from what some of you are saying that there are communities that have, that kind of get attacked in a very tribal way. And this is a phenomenon I'm just very interested in knowing more about. And is that Slack channel the best place for the chaos community to kind of look for? People to talk to about that particular part of open source. Does anyone know? You can type it in chat. Okay, and Sherom says, one thing about currency normalizing is about what this kind of issue is about. So a little bit more into that social currency metric myself, but thank you all. I think there have been some really good suggestions in chat and I'm very interested in what Megan shared for the Slack community. I'll just repeat that right here at the bottom. And invite you to participate in the chaos mailing list at chaos.community.slash-participate. I'm curious if I spelled that right. Yeah, I did. So we have a number of regular meetings, mailing lists. Encourage you all to participate and thank you for participating in this session. Again, if you wanna chat outside the session and get started with chaos and you don't wanna do it publicly, there's my info and I appreciate all of your time. It was great to have a discussion on what I expected and talk with you all soon.