 Well, as you can see, my slides are currently not presenting. So I guess I'm going to start my presentation because we only have 30 minutes, and then hopefully by the time we get to the metrics part, it's going to magically appear. I don't like the odds of that, but I think I'd rather have a productive conversation than nothing at all and just staring at a blank screen. So thank you for joining me today, which might be a very different presentation that I was planning on the benefits and pitfalls of open source software metrics with a particular focus on measuring health and risk. For those of you I haven't met, my name is Sophia Vargas. I am a research analyst and program manager within Google's open source programs office, where I support a number of open source measurement related tasks from looking at overall engagement across Alphabet to understanding what's happening in our projects and our communities. I'm also an active member with the chaos community where we design metrics, publish metrics and work with external projects on measuring health and sustainability. Still no slides. And so I'm here to talk about project health, which you might have seen as a topic that's been coming up a lot in various events and such. I'm most certainly been part of that, but I've seen many others give talks on this. And I've been really excited to see the adoption of using metrics to make more data driven decisions versus just gut decisions in a lot of these spaces. And so we've seen project adopt them to track participation, retention, productivity, and particularly companies using metrics to better assess what they should be using, how they should be using it, what they should be contributing to, investing in. And for many different stakeholders, metrics can be really valuable tools to understand what's going on in a project, when again your gut or you're not there in person to better understand and assess the scenario. There are many aspects of a project and project development life cycle that we could look into. For the perspective of this conversation, we're going to be looking at kind of a range between understanding health versus risk. So if a project is at risk from people or declining participation or maybe as a company using a project could increase your risk portfolio, this is really a gamble here. So the first thing I like to talk about is all the caveats health is not well defined or is risk. This is because it can mean very different things to very different people, whether or not using it a personal or a professional context. And I guarantee your company is probably going to be a lot more conservative than your personal, I'm growing in my AV support, we're scaling here, or the use case in which you apply this project, it could be part of a critical service, it could be part of a back office utility, that if it shuts off, maybe it doesn't provide any disruption to your business. I'm just going to keep going, okay, I'm not here. And whether or not you're looking at the perspective of the community or the asset the community is producing, you could be going for a stable code base to a diverse and active contributing population. I do want to have another big caveat for this conversation. Here I have a big slide if you imagine it that says risk is greater than security. There are a lot of security talks here today, I counted at least five and they've been really popular across a lot of open source events lately. This is not a security talk. So if that's disappointing to you, I'm sorry, I mean I'm disappointed for other reasons right now. Mostly I want to talk about the other elements of risk. There are other ways that you can introduce risk into your project, into your community, into your company by using open source or participating in open source software and other types of asset-based communities. If you are interested in security, I highly recommend looking at the open source security foundation, open SSF. They're building a number of great tools, frameworks, best practices, metrics like popular scorecard on how to assess security and security posture within an open source project and package ecosystem. Nope, still no slides. Well, I'm going to keep going. I mean it's like it's not great, but this is a hard problem. It's a hard problem for many reasons right now. Say something fails is not predictable. The data itself is messy, which is kind of a problem if you're building a metrics program where in theory the metrics are giving you feedback signals, but if you can't trust the data that are behind the feedback signals, then can you contrast anything at all? And then when you're building any kind of risk model, everything is subjective to your own case, to your own context and all the elements that we discussed before. So what is wrong with the data? Can we just assume that the data is real, accurate, actually showing things that we believe in? I've been an analyst for a long time in my career and in the last couple of years has been an introduction to working with data around open source communities and projects and version and control systems. And I learned about a whole new slew of data problems that I was not aware of, say inconsistencies between naming and schema coming out of APIs from the same platform or say how you design your pipeline to pull from these platforms if it's too large or how you've designed it is too large. You could be subject to rate limiting, which means that your data set is not comprehensive and is actively missing things. When you count things matters, when you're looking at version and control systems, you're looking at a current state, not a historical state. So if you're looking at trends over time, which are very popular for my role, that we are actively pulling things in time so we can see how things progress because if you pull it again five years later, any prominent forks that have emerged into the main branch are now the history of that project, now what was there five years ago. And I also looking at metrics around open source communities were often focused on people and people based activities. OK, wow, OK, now I'm going to start presenting again. Yeah, sorry, not you miss my cat slides. This is my little chaos agent Moby. So he's a great descriptor for a lot of my slides. So yeah, so the last started here. Thank you so much, AB team. OK, so the last one I wanted to make here on data problems was how you're counting people and I personally have multiple accounts on GitLab and GitHub. Why? Because of my personal professional context. I don't want some things going to my personal email or to my work email and vice versa and also there's a lot of automation, which is great and can be really powerful tool to scale processes and functions, but also does that really attribute to time based time on hands on the keyboard time, depending on what you're trying to measure? If you think if you're wondering how big of a problem this is, this is one of my favorite things to quote here. But the growth in bots on GitHub has been incredible over the last couple of years. So thinking about any kind of metric that we might talk about in the next two slides, say it's last time to update or last activity. If it's dependent bot, which is one of the most active bots, does that really count as a repository being active? Probably not, but that can really muck up your metrics. One of the other favorite ones is Kubernetes has an auto close policy on things that have been open too long in their pull requests, which means that if you're tracking mean time to close and a bot is closing things, does that metric actually mean anything? So be aware of how metrics or automation and scripts are being used in your community as well as a general best practices actively remove them because they could really muck up depending on what you're measuring. So let's come back to risk. How are we looking at defining risks in your own context? Again, non-security related risk. The question here, what could go wrong if this isn't your project so you don't have complete control over the tech, the process or the people? So I'm going to start rattling off some metrics now. One of the first and foremost is licensing. Some would say that open source is defined by the license that governs it. There are many different kinds of licenses that are created for specific intentions around use, consumption and contribution. So the first question for anyone who's about to use or release a project is, is this license compatible with your engagement model, with your intended usage model? If you're looking at it from a more of a portfolio and a consumption perspective, are all packages are using licensed? One of my favorite things to do is to troll through someone's GitHub and look at all the repositories. Are the licenses attached to all files? Sometimes they're aren't. So knowing that can potentially introduce risk if you're using any code base in your company that does not have a license to attach to it, should be a red flag or any legal team. How many distinct licenses that can create more complexity? And if you're wondering, OK, how do I even know what licenses I should be considering? The open source initiative has a set of approved licenses. They review new licenses all the time to help assess their relationship and applicability to the open source ecosystem, as well as helping you avoid things that are not well-defined because if they're not well-defined, you don't know how they're going to be defined and applied in a court of law. Another element of technology risk, you'll see a lot of suggested metrics around activity and reactivity as indicators of whether or not a project is being maintained. Last activity is a popular one. I have mixed feelings about this just because you can have a very functional project that has a really stable code base that hasn't changed in a long time and is still providing a valuable service. So maybe in that case, a better sign is reactivity versus constant activity in that if there is a new issue, a new bug that's been filed, is anybody looking at it responding to it or addressing it? Also, maybe if you start using a project and you submit a pull request because you want to actively change something, is anybody looking at it? If nobody's maintaining it, these are indicators that there might be lacking of that. Within your own context, I'm going to clearly I have multiple ways of looking at this. There are many kinds of things you can measure. So the goal of this exercise is to help you pick and choose the things that are most applicable for your own context. Say in the case of activity and reactivity, maybe you really only care about a subversion. You're only using version two point something, or maybe using it in a specific language or operating system. Quality metrics mixed bag as well. So helping to narrow down what could imply quality, even necessarily how you might measure quality, might be something like the number of bugs that pop up after a release. Another fun fact, typically the latest release of a project might not be the most stable. This is probably because you think of all the flood of bugs that happen once a release goes in. So maybe you want to wait a little bit before you consume the next release. But also, does this project have a history of preserving an integration and compatibility release over release? Or are you potentially introducing more breaking changes into your software? And then also just in terms of your own internal metrics, maybe how much time are you spending on fixes and patches as a result of taking advantage of this net new software. For additional context here, if you're curious of the kinds of bugs and issues that are coming up, you can look at where they're reported, what tags are associated with them. Fun fact here, I will have a little bit of security pointers. A lot of projects don't actually disclose security in the forms of bugs and issues because they're purposely trying to obfuscate their vulnerability, not saying, hey, we've got a target open on our backs. But they'll try to do this in a bit of an opaque methodology, either in closed doors. So ideally, this metric would be how quickly are you responding to security issues and vulnerability issues. But often that information is not available. Dependencies, clearly a popular topic in the broader slew of software supply chain. Being able to identify all of your direct and indirect dependencies, there was a talk by Sonotype, Brian, Brian from Sonotype about an hour ago, who mentioned that I think six out of seven vulnerabilities were coming from transitive dependencies, not direct dependencies. So this is a great way to keep your bookings up direct, sorry, keep your record up to date, maybe looking at S-bombs and you get the number of upstream dependencies. There are tools now like Devs.dev, which was released by the open source insights team at Google last year. So basically you plug in a package, see all the dependencies that are currently known, and they've actually been importing some of the CVE and vulnerability history there as well to help you understand what you potentially are introducing into your system through this dependency. Also ways to minimize this risk are how many versions do you have active in your code base, understanding when the dependency itself is relevant to you, is it during the test phase, build phase, or run time. And many other layers of dependencies, it's not just code, it also could be a service dependency and infrastructure dependency tooling. We could keep going here and you can kind of get the picture. You can't talk about risk without talking about cost. In my former life, I used to build risk models on infrastructure sourcing methods and often it was about, okay, cost as much as you do this, it costs as much as you do this, and what are the potential things that could go wrong that would increase cost. Typically risk can always be associated with the cost from the perspective of a company. So in this case, I'm focusing a little bit more on understanding what your costs are in a baseline and then the risk being how much variation is happening on top of that. So understanding what it costs for you to support this project, either as the presenter or the user from the tools, infrastructure, events, people, labor, training services, again, depending on your context, you choose the cost that makes more sense for you. Clearly complexity here can really change this equation and that's a very difficult thing to measure. I loved if you were watching the Angel Diaz interview earlier, the mention of forking, I was thinking about that, this slide as he mentioned that, in terms of how you're using a package in your system is really gonna change this equation from, again, is it part of a critical system or not? If you fork the code, then it makes it harder for you to take advantage of upstream versions that might be maintained, that might be being patched. And this isn't a bad thing. Forking and customization is one of the great appeals of using open source software. You consume it and make it work for you. But the more you fork it, the harder it is for you to stay up to date with the current version. So that essentially increases your maintenance costs to maintain it, which is exactly what Angel mentioned a couple, like 30 minutes ago, an hour ago. I'm losing track of time. And then again, from a technology perspective, what would it cost to replace this thing, to rip it out, to build it yourself to if your cost portfolio keeps going up, then maybe you do pay for a service to maintain it depending on what you want to keep investing in. Open source software is dependent on people. And these people could be coming from a variety of backgrounds. A lot of them could be and probably are volunteers with varying incentives. So this means that you need to interact with care and consideration to the community's needs and engagement preferences. So there are suggestions around tracking and understanding the role of the community, the health of the community, because without them the project would not exist or not continue to be maintained. Clearly, you need a sense of scale for the number of contributors. Here I've put a little bit of an asterisk because when we're typically measuring contributors, we're typically measuring what we can measure, which is data coming out of versioning control systems, which typically is code related activity, whether or not it's a pull request to review or documentation updates. But there's a lot of work that happens outside of that that isn't necessarily easily traceable in these systems. So I'm putting that as a caveat because the number of contributors is really more than the code base, more than the engineers, everyone that it takes to run this project. But for the sake of execution against these metrics, you typically end up focusing in on the things you can measure. I like to look at distribution of work. So I recently did a project where I was looking at some of our projects within Google and I found that the top 10 contributors by pull requests were doing anywhere between 30 to 70% of the work with hundreds in the long tail submitting one to do pull request year over year. So losing a handful of contributors, it really depends on what bucket you're pulling from. And so understanding the work done by what percentage of people can give you a better sense of the what people like to call the bus factor or I like to call the circus factor because it's slightly less violent, but how many people would have to leave the project during the circus and never come back for the project to not be able to be perpetuated. So this is an idea to kind of track how many people are really part of that core group and looking at the retention and attrition over time, giving you a better sense of is this project able to unwrap new contributors? Is it able to replace maintainers? Allow maintainers to take breaks and step away because we don't want them to burn out. We wanna give them an off-run plan as much as we need to unwrap new people. And then in terms of civility and community, does the project have a code of conduct? I also considered adding, does this project have a code of conduct enforcement committee? But that's a little bit harder to come by depending again on other aspects that we want to go into this talk, but whether or not a project has one can be an indicator of whether or not this is important to the community because you cannot predict how a toxic personality is going to impact the health and productivity of the community. We know that it's a risk. We don't really know how we can measure that coming into it, but whether or not a community is proactively setting grounds for acceptable behavior can be an indication of this. Clearly looking at, we talked about most of this context already, but diversity within company, background, location, I'm thinking about this again from the perspective of risk. So if say a project is primarily contributed to by a single company, that could be both a positive and a negative risk indicator depending on how you're looking at it. Negative from the perspective of okay, is this project gonna be too aligned to the goals of that individual company or maybe one of the company decides to pull back and then there's no contingency plan for how to perpetuate it. So typically you wanna just look for more diversity representation for both signals that the code base could be surviving beyond the whims of any one individual organization. Within a company, I'm always thinking about it, within your own context, how many people are you requiring to support this thing as a service or in house? There are a lot of metrics out there. Some are better than others clearly, but I think it's all really depends on what you need and what your own context is. This is a plug for the chaos community where we define metrics and methodologies as well as have software tooling to express these things. This is a great way to just kind of get a sense of what's out there and available. It could be overwhelming, but if you have a general sense of what you're looking to do, this could just give you more inspiration of how these things can be defined within your own context. So I wanna take a little bit of a left turn. Just wanna check my time because we had a bit of a strange start. When you start to collect metrics, you have to collect data. And if you're collecting data from systems around open source projects, you're probably collecting a lot of PII, personal information, emails, names, locations, companies, and other information around the history of that person's engagement with a platform. This information is not particularly controlled or regulated. If you're within a project, there might be project-specific policies, but if you're now a company aggregating a lot of information on behalf of your organization, now as a company, you are subject to regulatory and compliance and the policies of your own company. So the more that you amass, and depending on where you sit, you might end up being accountable for how this data is processed or reviewed or audited. So basically a general check that the data itself could have licensing policies and practices around it. It's generally good practice to review any of the privacy and policies attached to the platforms that you're aggregating from, say GitHub's privacy policy or your own company's privacy policy or a project's privacy policy. An example here is in the chaos community, we actually did develop our own data policy as a reaction to our DEI badging program that we've been running with events where event participants and organizers had to submit a lot of pretty detailed, sensitive information about the demographic profiles of their participants and speakers. And we realized with the collection of this information, it would be behoove us to create a standard policy of how we're gonna treat it, how we're gonna protect it, how we're gonna preserve transparency and trust within those that we're collecting data from. We did not have to do this, we felt that it was generally good practice to be open and transparent about our own data practices. So basically what I'm implying here is that there is an inherent risk with collecting data that has a sensitive nature when we don't exactly know how these things are going to be interpreted in a legal scenario. My general suggestion is that we're probably gonna expect more regulation. So if you know how to comply with standards currently, that's probably a good thing. So I wanna end with a couple of recommendations. I threw a lot of metrics at you. Clearly you could have an entire practice dedicated to collecting metrics alone. And this was just a subset focused on risk. Again, there's the whole security component that we just left out. So there's a lot more things that you could collect. So I think depending on the scale and practice that you're pursuing, it's always good to try to focus your initiatives and have goals or else you might just be left in a C and without enough people to interpret what you're looking at. Ways to narrow that down are determining scopes that are more relevant for your context. Instead of looking at a specific project, maybe you're looking at the specific version of that project or specific components of the version of that project. And then again, because we're thinking about systems based on people that don't necessarily have explicit expectations associated with them as the maintainer and you as the consumer. So understanding who's responsible for what isn't always necessarily a clean arrangement, especially when every project has their own practices, processes, governance, model, engagement style. Something that I've been encouraging of with maintainers that I have regular conversations with is wouldn't it be great if you just put it out there how you want people to engage with you? You probably don't want a random business trying to send you an email when something breaks in their context. But if this does happen, what's the best way and most constructive way for them to share this feedback with you? Is it using a mailing list? Is it using Stack Overflow? Is it creating an issue in their GitHub repository? And so if project maintainers can be more open and transparent about how they want to receive feedback or maybe they don't. Again, there was a wonderful talk in the keynote today from Sarah around how maintainers can feel about their own projects. And there are some cases where they don't want to do this anymore. And so maybe they need to be very open and transparent about I'm not maintaining this if someone is interested in maintaining it. Here's the steps you should follow. So in general, we need a little bit better communication on both sides as well as for net new consumers of open source to recognize that these are non-standard or contractual based systems. You're interacting with people typically in their personal time who have their own preferences for how they want to engage with you. And if you're looking at a bunch of metrics as someone who's personally been in this scenario, I often just have more questions which I think is actually a good thing. If you look at your metrics and you don't really learn anything then maybe you need to go out and measure more or just assume that you're only in steady state which may or may not be true. So often when you see a signal you have to dig further in and really understand what's happening. So never just trust them at face value is sort of my general encouragement here is to look at more things, ask questions, dig into it, see what's happening and if you can, go talk to someone who's on the ground living and breathing this experience. As a consumer, it's important that you invest in what's important to you. Again, I think that was mentioned again in the keynotes this morning but if this is important to your product or to your company, consider getting involved. There are many ways to be involved from financial contribution to developer contribution to event sponsorship, to mentorship, to many other kinds of programs and engagement style. So see what can be again most appropriate for that community and find ways to get involved that this is something that you want to preserve. And at the heart of it, people are your best resource. Do not let them burn out. We are looking into more methods to proactively detect signals of burnout but it's still difficult to do especially as people work in these spaces in both personal and professional time. That's some of the trouble that I have looking at the data across Alphabet is we can't easily distinguish these spaces. We see activity on GitHub but depending on the time of day we can't necessarily assume whether or not this is a side project or part of your core work. So asking people to do more or less in these different spaces, it's difficult to read those signals. So by proactively ensuring that there is a steady stream of people coming into this community that are supporting it and allowing people to step away when they need to to ensure that they stay engaged as long as they want to and have the ability to hand the reins over. And anyone who shares data, you always have the possibility to influence what you're measuring. I've seen this happen with leaderboards when someone will put up, hey, we've got the top 10 contributors by pull requests. And then suddenly they find that people are breaking up their larger pull requests into smaller components so that they can get higher stats. Is that what you wanted to do? I don't know, probably not. But in some cases you are trying to incentivize a change of behavior or growth and awareness. So just mostly acknowledging the fact that especially when you share data about human behavioral based systems that you potentially influence what you're measuring. So I kind of ran through that quickly because I had to say that was a little bit awkward on time. I think I have two minutes. Is that true? So I think I can take questions because no one is kicking me out yet. I'll answer that for two contexts. Within Google we have many. It depends on the project and what they're interested in tracking some of both roles, some we have supported as Ospo. But in theory there are almost too many things you can track. So we try to do it program specific in terms of say like the security team is actively looking at security metrics across the projects and repositories that are in our own organizations. But other projects have more robust engagement goals or say increasing participation in certain areas and in case they'll build their own dashboards then to sort of track those things or track against their own goals. We had one team that was looking at trying to ensure a similar contributor experience from those inside and outside of Google. So in order to do that we were looking at pull across response rates from those in known entities versus unknown people in the community to see is their turnaround time their response time similar to our own? Does that give us an indication of feedback that we need to update our documentation? So it can be very specific to a project as well as we have some larger scale programs that are looking more aggregately across the company. Within Chaos we've seen all of the above as well. We have a couple of software tools for more labs and auger that we're working on to try to create more tooling available. But you can just pull data directly from the get of API and the rest API is pretty easy to work with. Yeah, so I've seen most people build their own dashboards. So I've probably seen too many dashboards I think is the answer to that question. Anyone else? Yes? Yes, I am a symptom of the size of my company that we have someone who's just looking at analytics and measurement. I think right now I've seen better tooling happening externally in terms of public data sets. So we published a GitHub archive which is then used in tools like DevStats and OSS Insights that are basically just populating a comprehensive GitHub event stream so that you can sort of generate some of these metrics on your own without setting up pipelines. I will say that the data is getting worse. It is not scaling with GitHub and so when it was created in 2011 it was doing a pretty good job of collecting everything and now in 2022, we're down on the 80% range. So it's a pretty good estimate if you have no support at all to build anything. OSS Insights is a tool and if you are using a project in the CNCF they have their own dashboarding tool running on Grafata called DevStats. So I would start there as a place because if not then you are investing in time to build a dashboard. I mean this is, it can be a whole team of people which clearly you may or may not have support resources too. Other companies I've seen have leveraged more of their other business analysts functions. So if you can make the case internally about the criticality of open source as part of your infrastructure then ideally you can start to lever some of these other support services because you probably do have other business analysts which is kind of my role essentially. It's just I'm hyper focused on open source. So ideally you could get some other support because I would say as a team of one this would be hard to do on top of all your other work. So then being very focused and trying to leverage things you already have. We try to do as much automated as possible. A lot of the manual collection tends to be on some of those other styles of contribution. Say participation that's not collected in automated like system based. So like say board members or event sponsorship on participation that sort of one off other styles of contribution that aren't necessarily recorded on GitHub. Within the chaos community a lot of our assets are published metrics not software. So in that case we have explored using just markdown files to literally just write up what people's roles were. Just as a way to just provide more transparency of who worked on this and also to provide credit around people that spent time on it. But that tends to leave a lot of gaps still. So we focus a lot of our automated data collection on these larger systems that are giving us trace and activity data. But the softer ones are usually other ways that people are engaging in projects that we wanna give credit for and we wanna count. But we currently have no system to do that. Sorry, repeat that question? Yes, in some contexts. So it's a most certainly insecurity investment but it's more around teams that are already working in these communities are pretty well versed in it. And so we've used it to indicate where things are changing where we might not necessarily have full visibility. Say in very large teams where we are contributing across a large number of repositories we might not necessarily know how many people are engaged and what aspect at any given time given that there's a lot of personal engagement as well. So we can look at sort of our own footprint as a risk profile of just whether or not we still have the number of people we think engaging in these spaces because they're doing so again in personal and non and professional context. So we look at it mostly from a population perspective. There are other elements that we're looking at from a security perspective. Cost, not as much, mostly because we're very much invested already. Isn't a question of like what we should use but mostly how we minimize what we're already using. Controlling versioning that's something that we do as well. That's more based on policy versus risk analysis and that we have a policy that restricts using a larger number of versions on things. So we try to do things more proactively in policy than retroactively in risk assessment but that informs what we put in our policy as a way to reduce our risk. Any other questions? I think we might be out of time and getting if you're okay with it, I'll take one more. No one's kicking me out. Yes. This is Moby and this is Newton. Moby's named after Augustus Mobius, also a mathematician. He made fun shapes in Euclidean space is how we named him. He's not very smart though, which is why he's our chaos agent. So I usually end up using him in most of my slides. If you caught the risk one, that laptop was later damaged. He removed the B key. Just found it later under the carpet. So thankfully he did not disrupt my partner's work in the moment. It was more after the fact when he was missing part of his keyboard. Thank you so much for your time. I'll be hanging out around.