 Hello, Open Source Summit Europe, and welcome to our discussion today of safety, critical systems, and licensing, standard measures, and implementations from the CHAOS project, a Linux Foundation project. My name is Sean Goggins, and I will be sharing the work of the Risk Working Group from the past year. A good way to start thinking about CHAOS is to look at all of the different metrics that we have. And I'd like to thank our Asia-Pacific Group for putting this together. You can see that we have five major areas where metrics are being developed, common, which are metrics shared across all of the areas. Evolution, which is commonly known as how your code changes. Value, diversity and inclusion, and risk. Today, we will be focusing on risk. The CHAOS Risk Working Group today, we're going to discuss in four parts, and each of these parts carries with it a dimension of safety, security, and sustainability. In the four areas are metrics, ecosystems, the ELISA project in CHAOS, and TAG classification. First of all, when it comes to risk metrics, you have four major categories, code quality, business risk, licensing, and security. Code quality, really boils down to in the risk area, dependencies. And there are four categories of dependencies that our working group has identified and started to build metrics around. The first are runtime dependencies, the second are development dependencies, people, and the fragility of the overall system of dependencies. At runtime, we have a hardware ecosystem, so for example, in the case of AutoGrade Linux, there's a piece of hardware that we rely on, and there's also a software ecosystem. And the runtime risks for dependencies involve both dynamically and statically linked pieces of software. When it comes to development, this spans beyond safety critical systems and includes development. So there are compilation dependencies, you require libraries that are built in and they have dependencies, and so we have a network of navigable dependencies very far down the tree. You also have distributed systems that rely on restful APIs being intact. Those are two different kinds of software development dependencies. People are the presence or absence of people. A project can be a person, in which case one person in Nebraska is responsible for maintaining a dependency that you have, and that's important to be aware of. Finally, in the area of dependencies, there's this question of fragility, and fragility spans the other three areas. First of all, do you have access to the dependency and some control over? In other words, does it take upstream and downstream contributions? Are you aware of or knowledgeable about the issues and errors in that dependency system and have the ability to update it? And finally, what awareness do you have of the coverage of testing in each of the dependencies that your software has? So these are areas where the risk working group on chaos is developing metrics. Business risk is a separate area, and we started to examine risks associated with the pandemic. There is a noticeable decline in open source software contributions associated with the global pandemic that has me here instead of with you in Europe. There are also business risks around the number and diversity of contributors, and there are two factors that relate to how diverse that community is. One is the elephant factor. So how many different organizations are contributing to a project? And the other is a bus factor. How many individuals? And in general, if we have more than 50% of the code contributed by a single organization or a single individual, those are considered business risks for the ongoing sustainability of the software. Licensing is a separate area and we'll get into that, but we need to know how well covered a piece of software is for licenses. Are the licenses declared? Are the licenses OSI approved? So there's a growing phenomena of licenses that are not open source in spirit, but open source in naming and branding, but not compliant with an OSI standard that is actually how we determine whether or not software is open source. And finally, does the licensing associated with the project that you're using have some consistency with your business model? Depending on whether or not you're focused on services or software products, different types of licenses obviously serve those specific needs in different ways. Finally, security is an entirely novel part of the risk space. We focus from that perspective on the CII best practices, also open SSF, security threats dashboard and security metrics. And there are some links here in the slides that I'll share where you can learn more about those factors. Really the big thing we want to talk about are dependencies. It's a grand challenge and there's two types. There are safety critical, safety centered dependencies and sustainability centered dependencies. And when it comes to safety critical systems, we're really focused on runtime, hardware ecosystems and software ecosystems. And for sustainability, which spans most open source software, we're concerned about development time dependencies, things that are compiled into the system, distributed dependencies, those APIs I mentioned, and then how much knowledge different people have that control the authority, fragility and the ability to update. It's been mentioned also in our working group that containers are a dumpster fire. I did not say that and I will not tell you who did. But obviously, as the growth of containers and the distribution of software using containers emerges and becomes more prominent, these containers are dependencies that some of our systems have within them and it's much more difficult. We're not very mature in the way that we evaluate the sustainability, safety criticalness or sort of stability of container distributed software if we're going to incorporate it into a production ecosystem. So those are things to keep in mind. At the ecosystem level of open source software, we're really concerned about loosely categorizing five different types of open source dependencies in the ecosystems. There are frameworks, languages, libraries, databases and web and application servers. So we try to put different categories of software into these groups, different pieces of software into these categories. And with those categories, we have different criteria that emerges metrics in the chaos project for assessing the characteristics of languages, frameworks, libraries, databases, web and application servers. And these categories enable us to not just collapse all dependencies into one type because they are different. We can make a better sense of the space if we put them in the categories. We still need to learn how to rank within each category. And when we're measuring within a category, sometimes we're looking at things like CPU load, direct and indirect dependencies. And we're working on developing a matrix for weighting each of these considerations when it comes to understanding ecosystem dependencies in open source software. Alissa is a project for understanding what the bugs are, what their stories are, how they became important. And it's really focused on safety critical systems. And there's a couple of really great links in here. We've got some fuzzing and static analysis that's being used. StressNG is a good tool. And a lot of this comes out of the security ecosystem. So there's a lot of work happening in the LLVM sea laying area. And we also want to do summarization of tests as they're run. So whether or not the bugs are known or not known, evidence around commits. So we mine mailing lists. We look at bug fixes per hour on a stable version. Commits per hour on the upstream. And how long it takes to fix a bug. The longer it takes to fix a bug is some indication of whether or not the community that we're working with or that we depend on is attentive to defects that emerge in the software. And again, these are all metrics that are developing in the chaos risk working group. Finally, one of the things that we're doing in risk is classifying the tags that are placed onto bugs or issues and onto pull requests into process checks, solicitations. So a process check is I'm reporting on a requirement that's not met or not in the workflow. Solicitation is looking for some action to be taken. Classifications where we prioritize the bug in some way. Affirming is signaling that the task is done and can proceed to the next actions in some workflow. And negation is a claim perhaps that this isn't something that we need to address. So some labels that we've identified that are extremely common for process checking, affirmation, classification include the ones that are listed here. And what we're doing is actually mining 13,000 open source repositories in the CNCF and classifying the tags using these categories so that we can come up with a more general scheme in the chaos risk working group that we can share with the larger open source community. So what tools are available is a question that you might have. One of the tools that chaos supports is Auger. And Auger has a risk dashboard that allows you to take a look at some of the business risk, licensing risk, and testing risk areas associated with a particular package. So if we're looking for business sustainability, forks are a signal. The number of committers every week is a signal and you can see these particular projects or this project has some healthy information. Also the CII best practices badge is something that we look for and we can look at the total level of status. So in this case, the project is 36% complete. We provide the CII best practiced ID for the project and then a link to the badging website. OSI approved licenses. Again, this is important if from a business perspective you want to make sure that you're actually using a valid open source license. In this case, the project for the licenses that are declared is 100% OSI compliant. Here we have the number of forks listed specifically at a point in time and also the license coverage. So a best practice is to have a license declared on each file and signed off for each file in a project. One of the model projects in this area is the Zephyr project, which is 100% of its files with license declarations. This particular project is less mature. It has a 6% license coverage declaration. So not many of the files, only 6% have a license declared at the file level, but this is something that over time can be corrected. And you're also interested in knowing, I suppose, and I would be, what are the licenses that are declared at the file level across this project? And you can see that there is a very diverse number of license declarations here in this project. And depending on your business purposes, these licenses may or may not be consistent with what you're trying to accomplish. Auger also gives you some information about how you're drawing new contributors. So one measure of risk when you're considering the business sustainability of a project is, how many new contributors do you have in a month? And here, looking at the Zephyr project, you can see there's a fairly consistently high number of new contributors. And we break it down by the type of new contribution made by a first time contributor. And pull requests and commits carry a large volume. You also can see that there are comments that we count on both issues and pull requests and also the closing of issues and pull requests. How a project does with its pull requests is also important. So how responsive are we when a pull request is open? And how many of the pull requests are ultimately merged and not merged? If we look at the 20% of the slowest to close pull requests in the Zephyr project, we can see the mean days to first response in 2017 was 10 and a half days. Four pull requests that were ultimately merged and 4.2 days for pull requests that were ultimately merged and 10.9 not merged. So you can see that the not merged pull requests tend to stay open longer. Perhaps there's a little bit of a social dimension involved there. And as we go through time, we can see that the time to first response becomes faster and faster and the time to close remains very consistent for pull requests that are ultimately merged. And we also look at the mean days to the last response in a pull request. And so these are the ones that took the longest on the Zephyr project. Down at the bottom here, we see all pull requests closed on the Zephyr project. And we can see a high number of them that are merged and not merged. And the mean days to close in the case of merged pull requests is pretty short. Like less than two weeks across all years. And the time to first response is very very quick. It's less than a day in 2018, only a day on pull requests that are ultimately merged in 2019. And so these kinds of metrics are the sorts of things that you can get out of the tools that the chaos project provides related to the risk ecosystem. If you'd like to get more information about chaos, you can look up the risk working at github.com slash chaos slash wg-risk or the auger tool at github.com chaos auger. And you can also look up our whole community on chaos.community. We have a number of weekly calls and bi-weekly calls for the working group and the entire community. And just a few credits here. All licenses reserved and all that good stuff. I'll be happy to be online for further questions and anything else that you'd like to know about risk in the chaos project. Thank you for sharing your time and we'll talk with you soon.