 Hi there, my name is David A. Wheeler and I'm here to talk about the Core Infrastructure Initiative or CII Best Practices Badge one year later. First some background. It is absolutely not the case that all open source software is insecure or that all open source software is secure. Just like any other kinds of software, some open source software is relatively secure and some frankly is not. That said, the heart bleed vulnerability in open SSL that was demonstrated in 2014 showed that at least some widely used open source software doesn't follow commonly accepted good practices for software development and really needs investment for its security. As a result, the Linux Foundation created the Core Infrastructure Initiative or the CII in 2014. If you go to its website, the CII will explain that its purpose is to fund and support critical elements of the global information infrastructure. But what exactly is critical? Frankly, that depends on who you are and where you are. If software is used, then it is critical to somebody and so while the CII does absolutely focus on some specific, especially critical elements, it is also transitioning to doing more than just point fixes, it is also working on some holistic solutions to improve open source security much more generally. Which brings us to the CII Best Practices Badge. The Best Practices Badge is one of those more holistic approaches to improve open source software security. The basic idea is fairly straightforward. It is fairly uncontroversial that open source software tends to be more secure if it follows good security practices. But how can we encourage those good practices and in fact how can anyone know that these good practices are being followed? Indeed, what are those good practices? So the Badging Project approached this in a fairly straightforward way. We identified a set of best practices for open source software projects and it is important to note that we are talking here about best practices for open source software projects. This is the production side, we are not talking about the consumption side, they are actually different projects and initiatives for that, but we are talking about projects that develop and produce open source software and of course those best practices are based on a wide variety of existing materials, existing practices. We also created a web application so that open source software projects can self-certified against those criteria. If an open source software project meets the criteria it gets a badge and the great advantage of self-certification approaches is that these scale. There aren't one or five or 20 open source software projects, there are literally millions so we need a process that can scale to millions. The process of trying to get a badge is no cost and it is designed to be independent of size and products and services and programming languages. Now self-certification as an approach has its pros and cons, but we believe we have mitigated some of the cons of self-certification, for example we use automation. We don't just depend on the answers of humans, we actually go out and analyze the projects to look for whether or not certain claims are true. We require that the answers be publicly displayed and that allows criticism and the Linux Foundation both spot checks and, if necessary, for false claims can actually override the answers. This is what the homepage of the badging application currently looks like, so if nothing else after you listen to this talk please go if you are involved in an open source project, go get a badge, go to bestpractices.coreinfrastructure.org, push on that big green get your badge now and start the process of getting a badge. Currently there is one level of the badging criteria we call that level passing and the idea of passing is to capture what well run projects typically already do and it's important to note that this is not some set of criteria that says something about projects should do x but no project actually does this, not at all. We are only interested in criteria that are already demonstrated widely among real projects. These are something that real projects do and that they have a benefit. We examined many projects, many data sources and ended up with 66 criteria and then grouped them into six groups which are listed as below and were from basics all the way through to analysis. And once people answer those criteria we need to figure out some kind of scoring system. So after various debates here is the scoring system that we use. Most of the criteria are must or must nots and they must be met, in some cases they may be not applicable that's fine but if they apply they must be met. A few are shoulds, they must be either met or they can be unmet but that has to be justified. And finally several are suggested and those simply require, we simply require that projects consider them. It's fine to not meet those but nevertheless we're at this point appealing to psychology. People don't like admitting they don't do something if it's fairly obvious they should be doing them. And the suggested criteria are ones where there may be a number of reasons why they're not being used now but this at least nudges projects in that direction. In some cases we require a URL in the justification to point to additional evidence. General availability of the badging application was announced in May 2016 and at that point there were a number of early badge holders because when we were developing the criteria in the site we interacted with a number of projects because we wanted to make sure that these were practical criteria that they made sense and you see listed here a number of the early badge holders. Perhaps important to note is one of the holders is in fact the badging application itself. The web application is an open source offer project and yes it does receive its own badge. We think that would be hypocritical to claim that certain criteria are a good idea if we couldn't achieve them ourselves. Node.js, Linux kernel, curl, GitLab, OpenSSL and Zephyr all achieved a badge. It's interesting to note that while the current version of OpenSSL project does have a badge, one of the OpenSSL team members went back and tried to determine how well they were doing against those criteria before Hartbley and while it turns out to be difficult to work out the history and what was true several years back, the best estimates are that in fact the OpenSSL missed a third of all those criteria which would have been had this project existed at the time pretty good indicator that something is seriously wrong with the project of the time. That's a lot of criteria to miss and as I said I'm grateful and it is a pleasure to report that in fact they meet these criteria now. The good news is that CII badges are getting adopted. If you look at the graph of all projects from May of 2016, we now have a growth and now have over 500 projects that are pursuing a badge. Now of course some projects are just starting some have provided more data but even when you split it out by various levels of completion you can see that it's been growing over time. And there's also a lot of daily activity which again indicates that there's quite a bit of adoption of the CII Badging Project. Here are some additional baggage holders that I didn't list earlier and I think it's important to note that there's a wide variety of different kinds of projects which is a great thing. You've got things like Apache Libcloud, OpenStack, clearly Cloudspace, you've got CommonMark which is implementation markdown, software that's typically considered for security like a new PG, you've got NTPsec, LibreOffice and Blender which are application level applications. All the way down to Libset Comp which is very much a utility. As of 2017, 028 there were 60 badges and more since. But it's not just getting badges, the process of trying to acquire a badge has convinced projects to make changes to improve their results which is fantastic and really it's better for everybody. Here's a couple examples of changes that have happened because of the CII Badge process. For example, OWASP ZAP is a really widely used web application scanner and the PM lead said, hey the process of trying to get a badge really helped us improve the ZAP quality. It helped us focus on areas that needed most improvement. For example, they had serious weaknesses in terms of automated testing and in the process of getting a badge they needed to essentially really implement properly automated testing and that of course is going to have a lot of dividends all the way down the line. CommonMark changed to support TLS on their website, published the process for reporting vulnerabilities. The lead for JSON for modern C++ noted that hey I really appreciate that there is some formalized QA which even hobby projects can follow and they made some specific changes in order to get a badge. But of course all of this raises a question, what are the biggest challenges today for getting a badge? The good news is that now that we have started to collect a lot of data we can begin to answer that question. That's what I'm going to be talking about for the next couple of minutes. So all we did is we looked at all the projects which were at 90% or more but not passing which is 100%. That turns out to be a set of 52 projects and then we looked at all the criteria which were musts who absolutely had to do them and were either unmet or question march which simply means unknown. They haven't answered yet. A lot of people don't want to answer until they've met the criteria. And once we sorted them we found that there were 10 top challenges. So here's our top 10 challenges. Number one is tests are added and all the way down to number 10 which is documentation interface. And if you look at this list a little more carefully you'll see that in fact you can group these into six basic categories. These categories involve testing, vulnerability reporting, analysis, HTTPS, knowing secure development and documentation. If you change this from 90% or better to 75% or better the top 10 list has a slightly different order. This set is essentially the same. There's a little tweak one moves down one moves up a little bit but this is essentially the same list. So while I did cut this analysis off at 90% it's not really all that dependent on that particular value. And so with that I think we can pursue in looking at these six areas. So first let's talk about testing. The number one issue, the number one criterion that was missed was the requirement that the project must have evidence such that tests are being added in the most recent major changes to the project. And number four the project must have a general policy that has new functionalities added tests should be added to an automated test suite. Fundamentally automated testing is important. Of course that's important for quality but it also supports rapid change. If you know you've got a good test suite you can easily add a new capability and be confident that your testing will quickly detect if you've messed up something else. So it's important for security for a variety of reasons. One of which is that it supports updating dependencies when a vulnerability is found. In general software depends on other software and if one of those other components has a vulnerability found you want to be able to quickly update and quickly release your system. But you can't do that if you don't know if that update is going to cause your system problems. Now it's interesting to note that we made a conscious choice in the badging application. We don't have a coverage level that requires a certain level of automated testing. Instead we chose to focus on simply do you have automated testing and are you continuously working to get it better. If you have it and you are continuously working to get it better even if it's imperfect now you're in the right direction. But if you don't even have automated testing or you're not adding tests as you add major new functionality then clearly there's a problem. And these are some of the areas that a number of open source software projects are falling down on. Next up vulnerability reporting. One is the project must publish the process for reporting vulnerabilities. It's the second most common problem. And if private vulnerability reports are supported the project must include how to send the information in a way that's kept private. Basically tell people how to report. What's interesting is that these are in principle very very easy to do. You can add one sentence to your read me on GitHub or whatever and simply explain here's how to report vulnerabilities and just tell them. If you don't tell security researchers how to report vulnerabilities though you have an unfortunate problem where the project has to figure out and do a quiet dance of how to even accept a report and to process it before the report can be even submitted. And that's the wrong time to have that discussion. That kind of decision should be made ahead of time. Of course if by asking projects to tell everybody else how to report the vulnerabilities that means the projects have to decide how to accept vulnerability reports. And that's a good thing. Because then once the projects decide they can actually just go do what they've decided when the time comes. Third most common problem is that the project sites must support HTTPS using TLS. And there's still a number of folks using only HTTP for important things like downloading software. That's not a good thing. And even in our D we even provide on our site some hints for ways that you can possibly do this. You can get free certificates from Let's Encrypt. GitHub, GitLab, SourceForge, many of the hosting sites already provide mechanisms to help you do that. And if you're using GitHub pages we even allow a particular approach which has its pros and cons but at least gets you beyond the anybody in a coffee shop can intercept and cause trouble. So and we've also been quietly talking to various hosting systems to encourage them to support HTTPS. Because once as hosting systems have improved their support for HTTPS this has helped a very, very large number of other projects. Two other criterion that were commonly missed were involved analysis. One is at least when static analysis tool must be applied and all medium and high severity exploitable vulnerabilities must be fixed. The first one is really a lack of use by some projects of static analysis tools. Static analysis tools have their pros and cons. They certainly can't find everything. On the other hand using static analysis tools can help you find problems, defects of other kinds including some security vulnerabilities. And also by fixing these kinds of problems it means that other kinds of analysis including human analysis can be easier. So basically let the computer do some automated analysis and finally some problems ahead of time. I think number six was really in some folks a misunderstanding. Early versions of our BADJAP didn't allow NA and if you didn't have any vulnerabilities that were medium high severity people weren't sure what to do and I think we've cleared that up since. Another major area was knowing how to do secure development. Criterion number eight, the project must have at least one primary developer who knows how to design secure software. And number nine, at least one of the primary developers must know of common kinds of errors that lead to vulnerabilities in this kind of software. As well as at least one method to counter or mitigate each of them. We actually include in details of these criterion a very specific list of requirements. We're not expecting people to know everything. That's kind of ridiculous, but simply know for example how to design secure software. There are well-known standard principles. You should know what they are. We even list them for you. There are short lists of well-known common kinds of vulnerabilities. We point to two of the more commonly used lists. One issue that has been raised is maybe there might be a utility for some introductory course material or pointing off to some more, which is certainly something we're considering. But regardless, we think it's extremely important that software developers know the basics about how to design secure software. Most of the vulnerabilities in today's software are of a fairly common kind. And you should know what they are and where possibly you should know. And you should know how to design your software to avoid some of them so that these don't come to bite you later. Finally, number 10, documentation. The project must include reference documentation that describes its external interface. Some open source software projects have good documentation and some do not. I don't think it should be a terrible surprise that there are some open source software projects that do not have good documentation. This is probably not news to anybody. That said, I think it's important to note that although these were the most common kinds of issues for projects pursuing a badge, indeed a vast number of projects do meet each of these. None of these are a majority of problems for the ones that were close. Nevertheless, these are the areas that seem to be more common than some of the other criteria. And I think that's important and frankly interesting to find out. Indeed, the good news is that many, many criteria are widely met. For example, use of version control. You can still find a very few open source software projects that don't use version control software. And I think that what I have to say to those projects is shame on you. Shame, shame, shame. There are now several different version controls programs available. GET is extremely widely used, but we don't even require GET. Just use one. If you hate GET's user interface, fine. Pick a different one, but keep version control, keep track of your software. It just doesn't make any sense today to not use version control. The vast majority of software has some sort of process for submitting bug reports. And in general, at least for the projects that we were looking at, the ones that were pursuing badges, in general, they already were fixing vulnerabilities well, well, well before a 60 day limit if they had any at all. So far, I've been talking about what's called the passing level. This is the badge level that we had originally developed, but we had always had the notion of creating higher level badges. And so I'm happy to report that we've now developed some early draft criteria for higher level badges. We're currently calling them passing plus one and passing plus two. We'll worry later about bike shedding, what exactly to call them, maybe silver and gold or gold and platinum or whatever. But frankly, the naming is less important than that there are going to be these higher level badges. Currently, we're expecting that passing plus two is going to be significantly harder and not necessarily achievable by single projects. There are a number of criteria that are important or are valuable to do that small projects really can't do. We'd like to give credit for those projects that are doing those, fully acknowledging that some projects simply won't be able to do those kinds of things. We merge them from a wide variety of different sources and we do expect to drop an ad criteria due to feedback. And in fact, so announcing it's available for feedback, please, there's a URL, go take a look at our draft. I'm sure it will change. But we would really like to hear from people about the criteria just so you can see what's going on. Here are some examples of those passing plus one criteria. So for example, we want projects to clearly define and document its project governance model. How does it make decisions? What are the key roles? We don't have a mandate for a particular governance model. There are definitely people who have very strong opinions about which government governance models work. And yet when you step back, there are many different governance models that very successful open source projects use. So it's difficult to say one governance model is the one true way. But it is clear that having a way to make decisions when they're needed in their controversial is critical. So there needs to be a governance, a governance model for a project. Project must be able to continue with minimal interruption if any one person is incapacitated or killed. This may be done if you're a single person project simple with lockboxes and legal rights, but we don't want open source projects to end or be very, very difficult to switch over if any one person is incapacitated. Project must have floss automated test suites. And here we're going to have a statement coverage requirement. So at least there's some coverage requirement for testing. Now sure, there are pros and cons to this. But I think the notion here is that if you're not even testing most of the code, your tests clearly aren't good. That isn't enough to prove your testing is good, but if it is not meeting this, there's a problem. Project must automatically enforce its selecting coding styles. We don't require a particular coding style, but we do want to require that you pick one and then you use tools to enforce it. The project results must check all the inputs. This is really important for security. Having signed releases, monitor periodically checking their external dependencies. So these are the examples of the kinds of passing plus one criteria we're thinking of. Similarly, here are some ideas for proposed passing plus two criteria. So for example, the project must require two factor authentication for developers for changing central repository or accessing sensitive data. Project must have at least 50% of all modifications reviewed. This is current notion. Some people have commented that it's difficult to get every review, but surely we can get some level of review. Project must have a bus factor of at least two. This and the review are clearly challenging for single person projects, but this is also valuable to eventual consumers of projects. Knowing that there's multiple person review, knowing that other people can pick it right back up. Project must have a reproducible build. This is not just that you can rerun a build, but that you can rerun a build and produce the same bits. This is important for security, among other things. The project must apply at least one dynamic analysis tool. Dynamic analysis tools are very valuable for finding security vulnerabilities. That said, they can be a little bit more challenging because they require to apply in some circumstances because you must be able to execute the software. We think at the higher level we should be requiring that. It makes a great complement to the static analysis tools. Finally, the project must have formed a security review in the last five years. These are not final. These are just examples of the draft criteria we're talking about. If you lead an open source software project, I want to appeal to you because what you do matters. People depend on the software you create. In fact, you may really have no idea how many people depend on the software you create and not just directly because someone includes your software and something else, but indirect, those transitive dependencies that people create. And it's not even just within the software. People depend on services that are implemented on the software that in turn depends on the software that you write. What's more, the practices that you apply affect the result. They're going to affect the quality. Secure or more generally just quality software is not an accident. Nobody sits down and says I'm going to make the world's worst software that doesn't do what it's supposed to do. But it's really easy to make mistakes and saying I'm never going to make a mistake is just not really a practical approach. Instead, you step back and say what are the approaches that I can take so that as a whole I'm going to develop secure quality software. So if you are leading an open source software project, really if you're involved in an open source software project, please try to get a badge and show it when you have it so that you can show others your potential users. Hey, look, I'm doing well and so that you can also show others. Hey, there is an approach for showing that you're doing the right kinds of things. If you're considering using an open source software project, check on that project. Should you use it? One method you can use to help you inform you is seeing whether or not they've got a badge. And of course, we'd love your help in improving criteria, whether or not you're involved in an open source software project or not. But particularly if you're involved in an open source project, you'll understand what kind of criteria are important, what are not. We'd love your help. In conclusion, here's some key URLs. If you want to learn more about the CII, here's its URL. If you want to learn more about the CI Best Practices badge, see which projects have achieved a badge and or get your own badge, please go to bestpractices.coreinfrastructure.org. Go click on that big green button and start the process of getting a badge for the project you're part of. There's also the URL there for the draft passing plus one and passing plus two criteria. I'm sure it'll change, but it'll change only if people give us ideas on what it should change too. So please give us a hand. We want this to be a good list of criteria. And finally, if you want to just more participate more generally in the CI Best Practices badge project, there's URLs on GitHub. And although I'm speaking, I do want to make it clear that there are indeed a very large number of people who were reviewed, helped develop either the badging criteria or the actual software. I've got a list here of just some of the many, many people to whom I'm very grateful for helping to contribute to this process. So my thanks to them and my thanks to you for listening. Thank you very much.