 I feel like I have to whisper now. So essentially, this is about community health and vulnerability mitigation, and one of the things that has become really sort of flavor forward in open source in the last year, year and a half, are the dependencies that are related to projects. And the problem or the challenge for open sources that, first of all, identifying what all the dependencies in any given project are, and then deciding how far down the dependency chain. We do have folks on the back end, actually. We do have folks on the back end now, yeah. Can you turn down the volume a little bit? Can you turn down the volume a little bit? I think I'm almost causing feedback for myself. OK, that's probably better. So effectively, anytime you're importing something or have a dependency, often there are then transitive dependencies and tertiary dependencies and circular dependencies that sometimes exist. And so managing this is important because suffer vulnerabilities are injected in each of these things that I depend on and maintaining an awareness of where I am possibly going to have a dependency and where an actually explicit vulnerability is important. And so just knowing your vulnerability is often very difficult because these dependencies are not easily surfaced or visible. Does everybody agree with that? So kind of the birds with feather is looking at some brainstorm about where or how we can start to address this challenge in a way that isn't utterly overwhelming. I know, for example, on this effort project and in some of the embedded systems, software builds a material right into the way that the code is compiled. So you have a very complete inventory of everything, both at development time and at runtime. But most open source is not held to that level of safety critical, safety criticality. And so we have to go back and try to discover what our dependencies are. The things that we've started to use with chaos are a metric called upstream dependencies, which are essentially, these are the dependencies that your project has on other projects. We also use a metric called Libya, which is how old is the, I may be running air traffic control for the airport right now, I'm not sure. So Libya is, you know, so if a library was released yesterday and the version that you have was released two weeks ago, then your version would have a 13-day Libya calculation. So how old is the version that you're using of a library compared with the most recently released version? And so that is a useful, heuristic measure for, you know, do I have a lot of out-of-date dependencies? The more out-of-date dependencies you have, more likely you are to have a vulnerability that's in your code somewhere. And obviously there's a vulnerabilities list, but a lot of projects aren't paying close attention to that. And then there's also the OSSF scorecard, OSSF, I always feel like I say too many Ss or not enough Ss. The scorecard, which we've implemented, and we use that to get a sense of how rigorous the different components of our project are and the different components of the projects that our project depends on are. That's also a useful measure to try to get our heads around this dependency problem. But the core challenge kind of is from this sort of set of complex interdependencies that exist. And so I'm just curious how do you think about these problems or how do you think about addressing these problems on the projects that you participate in? So I'm looking at some DevOps metrics. DevOps? Yes. Okay. And maybe there are some of the capabilities that include dependencies that can help to measure. So in DevOps metrics, you try to measure the performance of the software delivery and then you measure in terms of the technical capabilities. And some of the technical capabilities include testing the dependencies. So maybe we could look at the daughter research program that has a lot of the technical capabilities of DevOps from Google. Okay. And then they have some metrics for dependencies. There is not only measuring the dependencies itself, like the amount of dependencies. It's like how do you deal with the dependencies? When do you test? What do you do with the test? What triggers the test of dependencies? So maybe those kind of dealing with the dependencies metrics can help you to analyze the risk of having the dependencies. Because the dependencies will exist. So maybe only measuring the amount of dependencies is not going to bring you good understanding of the risks. Maybe measuring how do you deal with the dependencies can add you more value to compose an understanding of the vulnerabilities. From a DevOps perspective, I'm not a DevOps person. It's okay. Forget the DevOps things. It's like I just wanted to bring where did it come from. But it's like the idea of understanding how the team manage the dependencies and not only measuring the dependencies itself, the number of dependencies. So when you say the manage them, one of the things that's come up in the discussions we've had on the CAS project are open source program offices. For example, they don't want to be overly restrictive of what dependencies a development team can incorporate. But they also want to try to build an awareness that each time that you include a new dependency, you're creating something that has to be checked and managed. And some effort ideally would be made to know that that dependency, for example, is a going concern. So it's in this line yet exactly this. And instead of only measuring the amount of dependence, you measure the practices that the team do with the dependence. When do they test? How do they do that? What triggers the test? What do they do after the test, the dependency? Yeah. When you say test, do you mean just an ordinary software test and that it passes with a dependency in it and a new version? Or go ahead. I have an idea related to what Bianca was saying. So things like the dependabot and other kind of spots. Can you measure your dependencies and if they have a dependabot kind of thing, how quickly they close those pull requests and things like that. So you can actually measure how actively they're maintaining their own dependencies, which are those tertiary, right? Yeah. Yeah, that's a good idea. So look at the dependabot pull requests. I know on Augur, I get dependabot alerts all the time. And off, I would say maybe half of the time, give or take 10%, there's a fix that won't break anything else. But the other side of that coin, half give or take 10%, is there's actually a substantial amount of work because the dependency change introduces breaking changes. That's especially problematic in different ecosystems. I mean, I'm sure you've all seen this. In NPM, breaking changes seem like almost automatic with any dependabot update. Python environments, less so. So it seems very ecosystem specific. I like that idea. Of course, the project has dependabot turned on, but is it still optional? I know I signed up for it voluntarily. But I know also sometimes GitHub starts to include things. OK. All right, so it sounds like maybe they're starting to turn it on automatically. OK. Hello. Oh, Steven Jacob from Rochester Institute of Technology. How much? All right, OK. Good to see it. Steven, this is Kelly Blanco from Auckland University, New Zealand. Bianca and Marcos from Northern Arizona. So it's a very academic crowd in the room. No one's threatening the show. There's a mic no one's threatening the show. Oh, OK. So dependabots, things like that, that's a good idea. Lib year metric that you have. Yeah. You could use that in like a hierarchy as well, right? Like what are the Lib year of your dependencies? So maybe some metric. Yeah. For each of your dependencies on how updated their dependencies are. Yeah, and we have like, we have pretty good data on that. Like we can, when we run a project, we calculate the Lib years for every import. So it's like, if I run it every week, I've got like Lib year data. You can see trends and things like that. But when it comes to fixing these things, I still feel like most open source projects that are not large scale CNCF type things struggle to have the resources to actually resolve all of these dependency aging problems in a timely manner. What about this is a question like, have you combined any of these types of dependency metrics with looking at your community metrics that you have, like the community health of your dependencies? Does that matter? It's not necessarily from a vulnerability standpoint, but more. What is the relationship between community health factors like pull request acceptance rates and issue response times and the extent to which the dependencies on a project are aging continuously? Right? Yeah. Yeah. I haven't done that, but that's a really good idea. Like I have all the data. Yeah. That's a good idea. That would be a really interesting correlation to do on a large scale. It's a good suggestion. In the community architecture assignment, we have our kids do? I haven't, but I wasn't accepted to RIT. No. I just struck me that I should probably send you that to show them what we have them look at. Do you describe it a little bit for the people? So I teach online. I forgot the repeal. There's a microphone by you, I think. Yes, I see that. I probably wouldn't have hit you if I remembered there were people online. So we have this, I've been teaching open source contribution at RIT for about 14 years, and then we opened an Ospo two years ago. And so we have students run large scale community architecture assignments. So I'm just going to try to find it as I sit here with my phone. But it has them look at not only like these days, a bitergia scan, but we have them go through maybe 10 or 15 pieces and we have them do blog posts on what they found and we have them do that three times. Since working in public came out, we have each group do one on a stadium and one on a federation and one on a, not the toy, but what's the third or fourth one she does? Stadium, toy. What is it? Do you know working in public at all? Oh yeah, and not egg balls? Yeah, she has these four categories. So we have them do that. I'm just going to sit down because it feels really weird warding over you on my feet. It's probably going to take me a minute to find this thing because I don't have a laptop and because my brain is essentially dead. But essentially you take the four categories that are not egg balls. We have them run through 15 different categories of stuff and we have them do one each on a, so they see the difference on that and that's their training to then go do their own when they're trying to figure out what projects to contribute to by the end of the semester that will have run through it with a group three times so they can say, oh that sounded like a great project but now that I look at it, it's a train wreck so maybe I don't want to try to contribute to that. Oh that sounds like a pretty interesting way to have students start to explore open source. Yeah it's because we're not, my students aren't exclusively CS or SE students right there, they're all across the board so the traditional, if you will, teaching open source community way of getting students lost, productively lost in a large code base doesn't work if some of your students aren't SE or CS students. So in general, those classes and the summer of code that Said started that other people are going to start to adopt, they pick one partner project and they will hop onto stuff and do team assaults of various upgrades or improvements or fixing dependencies or whatever. My students learn in a group how to figure out what would be a good group to work with and then they go and do it on their own. So the idea of looking at community health to see how that corresponds with dependency maintenance. Or even just is this a community I want to work with. I am a newbie student or a semi-experienced student I heard about this FOSS project while it sounds like something I'd like to contribute to but I'm going to be spending weeks beating my head against the wall or is it a place full of trolls or does this look like a community that's going to be responsive to my assistance. So that's kind of like the thought process behind that. The Callaway coefficient of fail is part of there. Do you remember that? I don't. Oh God, Tom Callaway's spot. Did this huge... Tom Callaway, he was at Red Hat, right? He's now at AWS, but he was at Red Hat for a very long time. I met him at Red Hat, but I didn't know the Callaway, what is it called? The Callaway coefficient of fail. Okay. This is a real thing. This is a blog post he put together that was kind of a tongue-in-cheek how you know you're doing open source right or wrong. And it is old. So that like one of the pieces of it, in Windows you get X points of fail right away. So we don't use that quite as one of the things we do anymore, but... Well, I mean, yeah, certainly certain languages are not very friendly on Windows. All right, here we go. Comm March public assignment. I finally found it. It's opening up. So we run Qualdron. They calculate Callaway's coefficient of fail. They look at the project's IRC channel. They look at the hazard. They look at the source code repository. They look at the mail list archives. They look at the docs. They look at what are their other comms channels that the project uses. And they look at the project website and our blog. And then they try to answer a whole list of questions that we have. History, purpose, and goals of the project. What was the initial commit? The latest commit? Is it a benevolent dictator for life or another structure? What's the structure? And obviously now that we use these egg ball categories, some of that's already pre-answered. What's the turnover on the core team? Right? Is it the same 20%? Are they staying the same over time? You know, who's got where to the unique knowledge, highest amounts of unique knowledge line of project? Who has commit access? You know, how many total who approves patches? How many people who's responsible for quality control and bug repair front and back end developers? Are they the same? Are they different? I'll read through the Callaway coefficient. Does Callaway's coefficient have anything about dependencies? It does. I think so, yeah. What are the major bugs or problems that have risen during development? How has the project participation trending? Would the project survive? The BDFL was hit by a bus yesterday or now. Would the project survive? The core team are the most active 20% left project. Do they have official onboarding in place? Do they have documentation available? How good is it? These do sound a lot like the things we measure in KS. I guess it's all from Callaway. No, not all of that is from Callaway. The Callaway thing was kind of satirical, but to give you the Callaway coefficient of fail, if it will load... Of course, it doesn't load. It's something you can Google. That's what we have them do. We do cauldron because it's easier for them. They just put an URL. Boom. Because we're not looking for them to develop. Modern things over time. We're looking for them to do a quick scan to see whether this makes sense or not. Okay. So kind of what I'm... The things I've heard are looking at depend about sort of as a level of responsiveness to the dependencies that change. There's a system called CVEs, which most of you are probably aware of. And so some of these metrics are intended to sort of not rely on the CVE process, but simply to help a project understand where they're at and by maintaining sort of up-to-dateness, long-standing issues that appear later, things like Herk Lead or the log4j issue that occurred. If you were maintaining a certain degree of currency, you wouldn't have been susceptible to those specific issues. And there's a sort of a hypothesis that's untested that if you do maintain a certain degree of currency with your dependencies, that you're less likely to be tripped up by some long-standing vulnerability that only is identified years after it's already been around. But of course, if you can be aware of CVEs, if you have a full software bill of materials and you know what all of your dependencies are, then any time a CVE is issued, you should be able to query that list of dependencies and know if your project is susceptible, which is obviously dependable to some of that for you. It's kind of to some degree what dependable does. Dependable doesn't catch stuff for every language, though. How a coefficient of fail original blog post is now in your email box. Oh, thank you. I'm feeling like a broad blog post, like the Goggins proxy of fail subsuming Callaway might be in order. Anybody else have any thoughts on dependencies? Anything else you want to bring to the table? Is there anyone, any questions online or is anybody still online? Okay. Yeah. Well, I think if there's no further discussion, I will just say thank you for coming and you all come back now. Thank you.