 Welcome, everyone, to our panel, what our OSPOs have learned about measuring project health. I'm really excited to be able to moderate this wonderful group here. So I just thought we would start with introductions if you could tell us a little bit about, like, where you are, where you come from, how you got here. And we've all met in the context of the to-do working group. And we often use an icebreaker in our intros. And so the icebreaker that we'd like to start with for today is what kind of music do you listen to when you work? So the intro and the icebreaker at the same time? Yes. Yes, okay. So I'm Dawn Foster. I'm director of open source community strategy at VMware. I'm involved in a number of different initiatives. So I'm on the governing board of the chaos project, which is, you know, focused on metrics, which is kind of why I'm here. I work in the OSPO at VMware. I'm co-chair of the CNCF contributor strategy technical advisory group. I'm on the board of Open UK. And I've been doing this open source thing for 20 plus years. So it's been a great career. And so I kind of love it. And the icebreaker question, what kind of music do I listen to when I work? I listen to a lot of instrumental metal. So with bucket head in heavy rotation, if anybody is familiar with bucket head. I also listen to a lot of German dance metal because I don't speak any German. So I find the lyrics not distracting because I don't know what they're saying. So that's what I listen to when I work. That's a good pro tip. Hi, my name is Emma. I work on Microsoft's open source programs office. I'm a principal technical program manager, which is kind of a mouthful. I've been at Microsoft for about three years previous to that. I was at Mozilla for seven. And I actually came to Mozilla as a contributor. I was a software engineer for a good decade. But I came to Mozilla under their project of the time called WebMaker, which was all about teaching people about technology. And I really wanted to teach people about open source. And so that kind of began the transition in my career from engineer to working in Ospo years later and kind of trying to help other people. I also, I'm on the to-do group. If you're not part of that already, encourage you to join. Meet people and learn how others are involved in Ospo's. I also am on the government of Canada's open stakeholder forum. So I help advise the government on meeting their commitments to open government, which I find very satisfying is trying to make a difference in the world. And for music, I kind of listen to everything, not everything. But it depends on the day and my mood. But when I really want to get stuff done, I listen to cello. I'll look into Spotify for like deep cello. And that's when I'm really, my kids literally know, don't go in the room, mom's playing cello. She's trying to get things done. So that's a good question. Oh, I got one. All right. Hello, everyone. My name is Sophia Vargas. I'm a program manager in Google's open source programs office. And my focus is on research and analytics. I support open source projects, programs, and questions across Google and our internal and national stakeholders. I guess I've been in that role now for three years. Before that I was in a product support role. And before that I worked at Forrester Research as a market research analyst. And I feel like starting in research, I kind of ended up going toward open source because I just have more questions. It was just an interesting space for me. So starting in market research, progression of the data center industry and the technology vendors and open source just kept coming up more and more in our own understanding of the landscape and the progression of the landscape. And so I kind of don't say to follow my nose, but I follow my questions or I just have more questions. And that's where I found things that were most interesting. So by 2020, I ended up on the open source team where I got to pursue questions around understanding open source, its role in our company, how we interact with it, and the progression of the role of open source across the industry. So that's kind of how I ended up there, I guess. In terms of other open-sourced things, I also work with the chaos community. I started doing that in 2020 similarly when I joined Ospo at Google. Because one of the things where I was really interested in metrics because that was part of my role and it was fun to find community practitioners. So I found a lot of support in that group. And also thought it was really important to start to participate in open source as a researcher of open source. I think it was kind of an awesome thing to go from being someone who covered data centers and I was never going to be a data center operator as a data center analyst at Forester. But at Google, I had the opportunity to join a community and participate in an external community. And I thought that was really one interesting and two critical part of better understanding it and my own research. In terms of music, I kind of have a pretty big split where if I'm doing anything writing related, it cannot have lyrics. So I generally go like full classical, big fan of like Chopin or Rachmaninoff. So like creamy, sad, depressing music. But if I'm doing any sort of like query or logic-based work, I kind of go for dance music, kind of something with a hard beat. Like there's this one project where I only listen to girl talk for about a month before I finished it. So I feel like it really depends on the kind of work. And that's the music I listen to. I love this question. I'm just going to answer that question. So I listen to white noise pretty obsessively. I'm currently listening to a rainfall. Highly recommended, but keeps me focused. So this is about project health and metrics. So I thought we'd start with the question for you and what you care about. What do you think health is? What are some of the themes? I guess for health, I like to sort of ask the like, is it behaving like we thought it would question? Because you might release open source or contribute to open source projects for a variety of reasons. And those reasons can be anything from, I just thought this was cool to, I want to start a movement or create a new language. And those things have very different goals and expectations. So the first being, is the thing behaving as intended or people behaving around it as you would have liked. And then once you know what that is, then you have something to measure against it. That was a great answer. I was thinking about that. Kind of building on that a little bit. When maintainers of open source projects at Microsoft ask, or ask this question, which happens all the time, is my project healthy? I ask similar questions or I'm starting to ask similar questions. It's like, what is your contribution focus? Who are you hoping to engage around this project? Is it users? Are you trying to have user adoption? Are you looking for contribution? Is this like to market and increase brand value? So I think first understanding that and then how it aligns to people's business or product strategy. And then basically are you accomplishing those things? Begins to answer the question, but then the second part would be what is the experience of those people? Do they feel successful? Do they feel safe when they're collaborating on your project? And then that's just a cycle. Who's your collaborator? What's the business proposition? Are people happy and feel safe? Is something to continually revisit? I think about the recycle symbol. That being for open source. Yeah, and just everything they said. So to build on that. When I think about health, I always like to think about trends. So is the thing that you're looking at getting better along however you want to define better? And so I try to look at health as far as I'm concerned. There's no such thing as a project that is healthy across the board. There are parts of the project that are probably more healthy than others. And so the idea being that if you look across a bunch of things that you care about and look at the trends across those things, how are the trends going? Are things improving or are things declining? And the metrics help tell you what to do about that. And so why should we collect or care about project health? Yeah, so I think it's super important to care about project health because obviously if we're working on things, we want them to be improving. We want them to be getting better. We want to be able to measure whether or not we're achieving what we set out to achieve with a particular project. To add on that, I think as someone who's been working with various projects on this, I like your comment about it being a cycle. I don't think this work is ever done and I think working with projects and understanding their health forces the project leaders to remember what they want to accomplish because we might have started with a particular goal but that goal can change over time. And so by creating or revisiting a metrics program or how we're defining and measuring health, it's also revisiting what we're trying to achieve in this community with this project or what we're investing in because those goals change over time. And so I like that this exercise forces you to think about that again and acknowledge how if you're actually still on the same path or do we need to revisit what you're looking at as health and how we're thinking about it. I would just add I was just trying to think about the milestones on a cycle. One of the things that we try to emphasize also is that communication upwards within your organization that describes that health and the value that it's creating and then outwards towards your community letting them know where your time is and the things you care about so that they don't necessarily waste time on things that might not be accepted as a pull request. Just keeping that communication open is really important in this part of health. I think it's important to have healthy projects kind of for obvious reasons but also I feel like it's a flag or it's a good prompt for when to not end an open source project but to recognize when your priorities have changed, your resources have changed and to make decisions at that point in time about what happens next and that's not always about continuing to invest in health it might be transitioning to a community member it might be archiving it and making like encouraging people to fork or whatever it is so it's important but it's also important to recognize when it's not a priority anymore. So for ASPOS and artists perhaps starting in this idea of collecting a project metric health how would you what are some guidelines you might have us think about or what to look for when thinking about metrics for projects that we are publishing to the open source. From my perspective I tend to try to take a more strategic approach so I try to look at not just the ASPOS but what is the whole organization trying to achieve and then how do the efforts with our open source projects fit into that because if you can talk to your executives about the things that they care about the things they are being measured on the strategies and objectives that they have and talk to them about how your open source efforts fit into that and support the work that the rest of the organization is doing then it feels like you are in a much better position as an ASPOS. So you would recommend starting there in terms of I try to start a big picture and then work my way through it but making sure that the work that we do as an ASPOS supports whatever the goals of the overall organization are because fundamentally if you don't do that then I have seen lots of ASPOS do this and describe the work that they are doing in a way that sounds an awful lot like charity to executives but if you do that as soon as there is a downturn in the economy you are something they can cut because it is not essential work it is not important to the business so if you can describe the work that you do in a way that shows it is important to the business then you are more likely to be able to continue to do it. That is all true I would just add on to say how I have approached this specifically in our ASPOS to align the types of metrics that we are encouraging people to use with the business strategy but also for me what that is meant is taking one or two two of the things that we have taken recently are the openness of scorecard value so security obviously matters getting maintainers and engineers to understand where they fit in this standard is a great tool for also speaking up to decision makers and showing the impact of that work some others have been things like security response time completing a code of conduct course because the safety of employees and their ability to respond situations is very important and that is something you can speak to employment legal type folks and there is a number from there but I am saying experiment is really important to pick one or two and experiment there to put up a big dashboard and there might be some hidden gold in there but it is better to focus on a couple that has been my experience I agree with both of those comments and perspectives I think what has been helpful for some of our teams is focusing on specific goals and whether or not it is something like improving responsibility to external contributors or looking at how we are evaluating processes versus looking at our internal metrics of productivity against our own engineering practices and tools we also I like metrics a lot is a way to evaluate the success of programs and initiatives say you want to try to grow your community now we explicitly contract that over time and so if you have something set up in place then that is a great way to see are we actually making progress against those goals are we actually impacting anything or making a change in our community so having the alignment with the business and having those goals explicitly allows you to look for those things but within that we like to I like to kind of encourage a specific focus just because it can be pretty broad sometimes and then sort of as a second layer to that what we have attempted to do is because there is quite a large portfolio of projects and teams at Google is to try to connect our internal community a bit more we have had some projects that are approaching similar initiatives on say improving our like issue triage rate in the community recognizing that we have I don't know maybe multiple issues systems internally externally and so it is a little bit messy sometimes so how are the ways that we are measuring improvement against that to ensure that external members are achieving the same kind of experience and responsiveness but then being able to say oh wait another team is doing that and being able to connect them and they can learn from each other and learn what has worked well between them so I think as ASPO we can't always do all the things but we can serve as a connector within the organization because we individually talk to all these project teams and so it has been really helpful to be able to say oh you should do a talk to this team they are doing something really similar they can tell you what has worked well for them and what hasn't worked well and how they have had to adjust their program because that is going to be a lot more real versus me kind of giving them that story but I also like I didn't do all the things so I think it is better have them talk to each other you just reminded me of something I think a role of ASPO at least one of the roles that I find myself taking is also teaching how to use metrics for their project house there is a lot of things that I look at from the strategic point of the company but something I might hear a lot from the maintainers I am doing this on the side of my desk there is all these PRs being opened how to use a metric like new contributors to identify folks that might be really interested in stepping into a role like triage github triage role so helping them connect with one or two metrics that help the immediate pain points that they are having I think is also an opportunity that also has to do that teaching and get each one teach one something like that and so then how would you give advice for people who are looking to get metrics around the health of the projects that they are bringing into the company open source projects that they are ingesting so that was like publishing open source what does ingestion look like I can start I am thinking there is a lot of things that we look at when we are looking at especially if we are looking at incorporating a particular open source project into a piece of our own technology into a product that we might ship there are a lot of things that we look at emma mentioned the OSSF scorecard that is something that we tend to run on those projects and we look at whether are they keeping up with security vulnerabilities they have branch protection like there is a whole series of things that we look at as part of the scorecard I look at who controls the project is it owned by a company are they a competitor of ours are they the only other user of this piece of software or is it under a neutral foundation where there are loads of people participating there is lots of companies who are the people contributing to it does the community look healthy there are a lot of things that I try to look at across some of those projects especially if it is something that is relatively important that we are looking at incorporating you haven't mentioned it yet but licensing well in terms of like one of the most fundamental things we have to consider just whether or not what we want to use is going to be able to use properly internally in a hospital we are kind of on the hook to ensure that we are using all licenses properly so to kind of limit that we usually have a set guideline of things you can and can't use and if you want to use things that are not on that list then you have to go through our team and a full consultation so they understand the ramifications of what that means for them and how to continually apply with licenses that require more things than just usage plus all the things that Don said so we actually made quite a bit of tooling to help us flag software we are using that they have any vulnerabilities or risk associated with them so a lot of that is in our tooling including licensing and that will send flags so we have a business review program so if there is a concern raised through the tooling and I can't remember the name we have released this specific library which I will try and remember in post later that you can use to do the same review process that we flagged and then we will bring in legal to review that and we have been historically looking for business reviewers like within the organizations to help review that but we are moving even away from that we don't have a use and not use list Microsoft has come a long way they used to be afraid of open source and it is really less so so it is more about those automation triggers and then putting a person on it when we need it no specific metrics would you say that the health of the metrics that you are using to evaluate health of projects that you are publishing into the open source are different than the metrics that you are using for the ingestion of open source like where is the overlap is that for all of us everyone I would say they are different I think that definitely things that are when we are releasing things out into the community there is just so many more considerations that you have to have in terms of our own perpetual involvement in the project and what we want to see happening and the choices that we make in order to do that and recognize that it doesn't end after you release it and understanding what that ongoing commitment might look like and trajectory versus I think understanding our other policies internally restrict what we can apply on things that we bring into the company and so that list is a bit shorter just practical again based on license based on security and risk in that sense but then also say limiting number of versions that come in so that we know exactly what we have at any point in time to kind of reduce the risk by again policy versus trying to go back and check everything and so because of that there is a little bit less criteria but more harder to get around criteria on the ingestion piece than when we release something maybe we will have a broader perspective Yeah I would just say same on release and we are especially especially mindful in asking people to have goals for their project again collaboration goals who they want to collaborate with is kind of a repetitive thing but it's a good check and balance to give people those questions because they might not have thought of that or some people might think of open source as a way to get so that I can do something with this project that I don't have time for people will help me in getting people to have a set of questions that they ask themselves and goals is a really important part of our release process as well so I will just kind of echo that Yeah we spend a lot of time talking to internal teams who want to release something into open source to make sure that they are really doing it for the right reasons because you do have a lot of people with sort of naive impressions of how open source works so they think they are going to do this for marketing and they don't think about the fact that this is a long term commitment that they are making on behalf of VMware and so we really try to make sure that we are doing these things for the right reasons and that the people are going to be around to maintain them and that they actually want them to continue over a long period of time Last question and then I got to open it up to the audience What is the one thing you want to highlight about metrics for the crew here and Emma you should go first because you haven't gone first yet Well yeah I was just thinking about how to frame it so metrics are a great tool but you are still whether you are talking to decision makers you are working with maintainers this is a human problem in that in that asking someone to do something for the sake of something you know have collaboration rules for your project because I am an expert and I am telling you you should have that or telling a decision maker that it is very important to contribute to the upstream those are difficult conversations and I think what I have learned is and I did a talk I call them curiosity hooks I try and use data and metrics to get people to ask more questions that will influence their decision making and their kind of alignment with our work so trying to give an example that contributions to the upstream are important decision maker might already believe that but showing them the data for example this team contributed x number upstream contributions which impacted 300 internal projects that is data that we are able to pull out recently it is not about convincing but it gets them asking more questions and better questions and making your conversation richer with more potential so trying to get people curious about your work instead of trying to approach people as not understanding as often they do or they have some insights into it giving them the curiosity through metrics so here is a metric for this and here is how it turns up with our business more often than not you will get more questions than asking people or trying to pitch something so I think that is the one thing I have learned and I am continuing to explore so that is just an early kind of a ha moment for me and I am continuing to explore that continue to be curious I think for me the most important part of thinking about metrics is being able to tie them back to strategy what are you trying to achieve and how do you pick a few metrics that show whether or not you are successful so by picking a few metrics that people care about whether or not you are helping the organization achieve their goals I think can be a really good place to start I think both of those are excellent recommendations and I feel like I would love to have someone like you Emma I feel like I am often in the case where I am trying to support too many teams at once and so being able to create curated examples of how hooks to get people interested I am often not in that role I am often in the step 4 where I am trying to make sure that the project leaders have that information to bring to their higher ups which generally puts me in a position to have more generalized information that I can share with more teams but that is also sometimes problematic because how many failed dashboards have you seen I have personally seen many oh wait there is a dashboard for that all the data hooks are now dead because nobody looked at it and it is this constant cycle where we try to use dashboards as a way to create generalized information for a team to come in and use it to create these stories and often they just aren't serving enough people with the right things and so I feel like something that I have been attempting to do now in this middle place versus being that person is trying to find those people within a team, trying to find who is going to make the best use of this and then I will support them on what they need because I know they know how to curate this in a way that helps them inform their programs and strategy and how to communicate that to their managers so I am in more of that encouragement role where I wish I had time to do that but I don't because I am supporting too many projects so I am sort of now in the place of how do I find the right person and then I will design to their needs and generalize I run the risk that it serves nobody's needs so how do I kind of meet them in the middle so I feel like finding your design point is really helpful for any of these sorts of exercises and I always feel like anytime you start publishing data around a project I am always curious about what are we inadvertently encouraging what kind of behavior are we incentivizing by showing a ranked list of contributors inside a community are we encouraging behavior that we want or are we encouraging things that we don't want and I think being cognizant of anytime you do share information or share anything that is sort of real time like that of what are the broader implications of publishing this information and ensuring that you are encouraging the things that you want because it is an incredible tool to encourage and incentivize behavior but could also easily incentivize things that you didn't plan for Very good point Great, thank you I thought we'd open it up to the audience Are there any questions from the audience? I have a quick question I'm pretty loud Sorry, we have a live stream Do we have another mic for questions? Okay, come up We can repeat it I'll hand it to her If you ask a question you've got to come up, take a microphone So I have a really quick question I really enjoyed your panel on metrics and what things that we are measuring impact and all of those particular different sorts of spaces and caveats Now in the current climate that we are living in I'll just call it a transitory job of functions that are happening What sort of things should we display, share, emphasize to help our case to keep our jobs Right, that's kind of what I was thinking of I hope that can I think it's a question on everyone's minds So I guess I'm talking first accidentally I don't have the answer I think would be the first one My hunch is a lot of Dawn especially emphasizing strategic alignment I think has never been more important and for me that's security especially however we can show that impact strategy but also security and be able to speak to the top line goals organization I think is probably most important I'm also stubborn in some things like for me safety and open source and psychological, physical that's really important and I'll never I won't sway from that I think in that way that doesn't have to put you at risk though I think you just need to work at finding your allies within the organization and that that work aligns with their strategic goals so that you move from being I mean our OSPA works across the company but it's really important to have those organizational partners that will speak to your work for you and collaborate with you on whatever those solutions may be so those are the two things I feel like I'm just going to like Don has been saying but I feel like the ideal would be if you start this work with business strategy and alignment in mind then your case is already made I think there's definitely been scenarios where I've been brought in to help justify work where if they had only done this exercise before they started it then we would just be proving that they're making their own case versus how do I justify business value for something that I never actually developed a strategy around to begin with and that's where things start to get a little bit sticky because we have to go in and figure out what that is and that may or may not be direct to the company so then we have to figure out well what story do we tell are you working on patches upstream that directly impact product features and functionality and conformance or are we generally trying to ensure the sustainability of an ecosystem working generally on Python even though we don't use that particular package or so it's kind of trying to find the connection back to why it's relevant for either your company or your customers if something you stop supporting who's it gonna break and that's maybe the like the inverse case is always the most awkward thing to talk about but I always like to look at well like what happens if we stop if you stop doing this work tomorrow who's gonna be most impacted by it is there can we point to potential loss and revenue potential loss and functionality that would be the result of stepping back and stepping away from this particular community function or role and if you don't have a strong case to make then the question is is this work actually aligned with the business and maybe it is something that should be removed which is always uncomfortable with something you've been doing for a while but ideally you can translate to another role in the company that could still serve that function and just would be more aligned and I think because we're all a bit strapped with all kinds of cash that I think we are gonna see a narrowing and a focus of prioritization but I think the thing that I've really been pushing on is again we talked about this a lot yesterday but like we can't be too narrowly focused on what we think we use and the recognizing that all these pieces are interconnected and so even if the relationship is indirect pulling away or pulling out of say like community building functions in a project is only gonna detract from the health of that community and so even if you're not thinking about the only thing about the technical piece that you use you have to kind of if your managers don't understand that you have to educate them on the value of the health of the entire project and ecosystem and people that are all part of that and I think sometimes it does require some storytelling which is I think where OSPA hopefully comes in handy because we can help other projects build that story, build that case and help them articulate that value if it's there because it might not always be Yeah and I think just to build on that I think that you really do need to be laser focused on business value when you talk about the work that you're doing to especially to senior executives now that doesn't mean that you don't do some of those chopwood carry water kind of tasks within the community you do you continue to do those but those are the things you do while you're waiting for a PR to merge while you're waiting for something else to happen that's not how you describe it to the executives and so I would say be really really careful about how you describe the work that your open source program office does when you're talking to the people that get to decide whether or not you continue to do the work. Other questions? Come on up. Well I think you guys I want to know some specific example of whether retired or failed metrics on an either upcoming or scaling process of an open source project in particular and why you retired the more you realize it was a failed metrics. Failed metrics. Failed metrics. Failed metrics. I feel like you talked about this a lot in your chaos talk yesterday. You see what I mean? Well there are no approaches. That's true. Well it's more that like and Don knows the story because I told it a few times but when you don't have full awareness of project processes and tooling sometimes you think that something's a good metric you say look at response times and close rates a lot or really popular ones or time based metrics and then if you apply the amount of pull requests or issues that are closed in a project to the Kubernetes project where CI robot closes everything suddenly you realize your numbers don't make any sense and so recognizing that if you don't understand the sort of tooling and process of a project some metrics are just not going to work in those cases because they don't make sense because there's active automation that's going to really muck up your numbers. In other cases it was I was looking at a lot of activity based metrics in terms of how many events were people creating and trying to try to gauge it in terms of amount work like work done in the community or productivity in the community and just kind of get a general volume of work and year over year I was looking at it there was a couple of people that were just having outrageous numbers of push events and so I just because I have their username reached out and found out that all of that was automation so again robots really messing up some of the numbers that I thought were quality measures of amount of time spent on open source none of them actually were these weren't even labeled under automated accounts I was removing bots I was looking at people based accounts and they were still scripts running and duplicating and causing a whole lot of noise and so I stopped actually using almost all of my activity counts and I used them to think about like kinds of work or diversity in work or engagement level because they don't want to catch things like watch events so I still consider event type but I'm now mostly looking at how much a person is spending in that so like how many days that person active or how many days in a month they active or how many months in a year are they active versus looking at their activity counts in any of those given moments because I can't actually tell whether or not the activity was hands on keyboard or something they wrote six months ago yeah I have an example from I knew a bunch of people that worked on an open source team at a very large company and their head office liked to measure numbers of commits so the developers were measured on number of commits to open source projects and their boss would get a call if their numbers of commits dropped or changed in some way so this is a terrible metric but what it encouraged them to do was just everything's a commit I've changed one line that's a commit I've changed one of the things that's a commit so it just encouraged all this really bad behavior among their engineers by measuring their performance on number of commits which is a ridiculous measure so similar story for lines of code we don't use that one no lines of code that's not the measure some lines of code are good yeah I mean the the only thing that I would say is assumption that people understand the value of the metrics you're proposing and the value will bring them out of the gate in the talk that I did I just have brought metrics to people to suggest them but that's not again it's like bringing them something to be curious about oh I saw your repository had the next number of whatever your metric is is a much better more successful way of having a conversation the bot is a great one I was really glad to learn about that I can't think of specific for me it's always approaches different approaches work different ways but I think the chaos project is a great one to visit if you're looking to learn about what metrics people are trying failing improving on I think that's one of the great things it's a really great community I'm just checking in I feel like I want to amend my lines of code one because it wasn't necessarily we discontinued it it's more sometimes you think you propose a metric for a thing but then it turns out it's a better measure of something else so it's also not necessarily requiring a metric but repurposing it so I ended up like looking at lines added lines removed as the amount of turn in a project because that can kind of just tell you how much is the code base changing over time and that is actually something interesting to measure to know whether or not this is stabilizing there's a lot of active change happening which maybe from thinking about project life cycle that's something interesting to pay attention to at any given time so we didn't like it as a productivity measure because clearly we all know the case where removing lines can sometimes meet a more efficient efficient progress versus something that had a lot of extraneous extras things in it but instead repurposing it for something that was more more valuable to the metric or more aligned to what that metric could tell us I love that because in accessibility there's a curbside effect where with a curb dip in the sidewalk was created for wheelchairs to end up working for strollers and buggies and that kind of thing I like that that is true as part of metrics is really about learning so learning a word for something else makes so much sense Any other questions? Yeah When is it? It's 6.50? I see it was 3.40 but anyways We're having too much fun Okay last question I'm more like an open source maintainer not I don't work for I didn't even know about OSP I've posed before today but as an open source maintainer I guess I'm curious like what not obvious advice would you give to someone like myself who doesn't necessarily think about the role of your jobs in your organizations but it's actually obviously very important that you guys exist to be able to make sure that my projects can be used in those organizations so obviously there's metrics maybe the not obvious thing is just here's the list of metrics but I'm just curious what your advice would be Is that as a maintainer what you'd want what an organization that you are working for as a maintainer would want or as a maintainer like an open source project that other companies use Let's say I'm an independent I'm going to think on that as usual I think it depends right so I think I think metrics can be useful as a way to to understand what's working well in your project and what's not and how that might be perceived by companies so for example you know if you look at the ratio of closed pull requests for example do you have like is there a big pile of open pull requests that nobody's dealt with or a big pile of issues that nobody's dealt with that's not going to look good for a company when they come and look at it do you have a whole bunch of security vulnerabilities that were never patched is another thing that companies are going to look at so that's another another thing that you can measure and then there are some things that may work against you that you can't control so are you a sole maintainer of a project and you're the only one working on it that's a big risk to a company like me to adopt it so yeah there's some metrics you can look at and I think there's also some maybe less tangible things that you can think about too this isn't really a metrics one but something that I look for if I'm involved is sort of has the project or say the person who's maintaining the project have they published their explicit goals of what they're trying to achieve in the project is this something that they want adoption they want contribution or maybe they don't like I think some projects are just out there to be experimentations or demos they won't they won't accept patches and they won't and it's sort of understanding what you want from us too in terms of like what are you trying to achieve with it as well as expectations of how best to engage with you as the maintainer I think that's something that I feel like I felt fairly acutely not not involved in this but seeing how some companies have approached maintainers with the sort of expectation that they're there to respond to their needs and these pieces bug or this issue of this patch not recognizing that that person is there in a volunteer capacity or in a hobby role and part of that is okay maybe those companies have known better but on the other hand I've been encouraging maintainers to be more open about how they would like to engage or not engage with their user populations how do you want to know about bugs or issues you want it on your project you want it on stock overflow where like where do you go to get that and where's the where's the best way that we can respect your expectations and your processes in the project and how to engage with you in a way that's where needs and your expectations versus just assuming you're going to be there on github responding to my bugs or issues so that's something that I feel like I would I would love to see us get to that point just because I feel like there is a lot of misaligned expectations and open source between both creators and consumers and so the more that we can be just upfront and transparent about how how we want things to happen and how we want to engage then hopefully we can incur better practices across company and user relationships. One quick echo on that is the experience that I had actually first of all communication putting in the read me all those things that you just heard we had one team that was trying to contribute to an open source project they really wanted to do everything right and the maintainer wasn't responding and they were asking if they could fork it because we try and avoid hard hostile forks that kind of thing and when I went into research also I can't say enough about qualitative research we're talking a lot about quantitative but reaching out and it's just a student that it you know was on their side project and they couldn't really respond it wasn't that they didn't want to and they were really surprised when they looked you know so there's just a lot of but you know maybe he could have put I'm a student I will check on this once a month and then you know then this team is not worried about forking because they're trying to meet a goal like it was very stressful and the team was trying to do everything they could but so plus one on the communication but that's a real story of it being confusing well thank you thank you everyone and if you want to join the conversation we have a regular working group where we have these types of conversations and otherwise the github link is on the slide please join us