 But he's going to give us a green light, or something? I don't know. I'm guessing. Oh, yeah. Mic is on. Perfect. Thank you. You ready? Yep. OK, cool. Well, hi, everyone. Thanks for being here. Today, we're going to be talking about a project by the OpenSSF called Scorecard. And we're going to walk you through what it does and how you can use it to assess the risks of open source projects. So just a little bit about ourselves. I'm Laurent. I work at Google as a security engineer. And I spend a bunch of my time working on the OpenSSF projects. I work on Scorecard, which is the topic of this talk, and on Salsa, which I think you know all about by now. And this is Navin. Hi, I'm Navin Srinivas, and I contribute to a few supply chain projects. I work for Endor, and I'm also maintaining along with Laurent on Scorecard. All right, so all of you know about the OpenSSF, so I'm going to skip that slide. If you're interested in joining the public weekly meetings or the Slack channels, just visit the openssf.org website. All right. Thank you. Thanks. So the agenda for today, it's a packed agenda. So we're going to start by talking about the attack vectors in supply chain in general. And then we'll introduce Scorecard, which we built to kind of have this holistic view about all the different kind of attacks and try to detect them. We'll show you a demo on how you can use Scorecard on your own repository by installing the GitHub Action. And then to make the talk a little bit more interactive, we have prepared a little quiz to show you the kind of stuff that Scorecard can detect. And for those of you who participate and answer, we have several OpenSSF t-shirts that we'll distribute at the end of the talk. So stay tuned. And then in the second part of the talk, we'll show you how you can use Scorecard on your dependencies to assess their risks. We'll show you the demo of the CLI, and we'll talk to you about the weekly scan that we have that we scan every week about one million project. And we have a publicly available data set that you can use to learn more about your dependencies. All right. So I don't think I need to motivate very much the problem with the supply chain. So real quick, supply chain attacks are on the rise. And people are realizing that it's a lot easier to compromise a dependency or attack the supply chain than it is to find a bug and exploit it in the software. And over the past few years, we've seen significant rise in supply chain attacks. Real quick, this is a short timeline about some of the prominent attacks that happened over the past couple of years. One of them is KotKov, which affected around 30,000 customers. And for about two months, the attackers had access to the CI CD system of customers. So if you use KotKov, say, in the same environment where you were maybe storing credentials or private keys, those might have been compromised. In this slide, we've tried to visualize the various components of the supply chain. And as you can see, there's a lot of squares and a lot of places where an attacker could try to target your project or your infrastructure, starting by sending malicious pull requests to your project. If you have an overprivileged CI CD system, it's possible that a malicious pull request could abuse the overprivileged CI and do something malicious with it. I will show a demo of this live later in this talk. Of course, someone could also try to compromise the repository itself where you store the code. This happened in the PHP attack a couple of years ago. The build pipeline also is a place where attackers might strike. I don't have to motivate this. It happened to SolarWind. Another place is taking over package manager's account. This was, again, demonstrated, I think, a couple of weeks ago. Some researchers showed that by scanning for expired domains, they could re-register them and then initiate a password reset and then take over the account and push malicious packages. And last but not least, the dependencies, which are the backbone of the supply chain, attackers tend to try to use this technique called typosquatting or package squatting. They simply register names that look alike genuine packages in the hope of tricking people into installing the malicious version of the package. And because of this large attack surface, this is kind of the reason why the OpenSSF scorecard project was created. To kind of give you this holistic view about the different signals and the different security risks of a project. So in a nutshell, a scorecard is an automated tool that assesses a number of important security risks or signals with projects and their software development cycle, starting with projects that are natively developed on GitHub. We started on GitHub, but we're trying to expand to other providers, Bitbucket and GitLab, but today we currently only support GitHub. So scorecard tries to detect good practices that are followed by a project. Things like, are you using fuzzing, static analysis and continuous testing? Those are important signals, because it tells you about the quality of the code and whether bugs are likely to be caught early on in the development cycle. Scorecard will look at other things such as, do you have a tool to manage and update your dependencies? And will warn you if you don't. Scorecard was way beyond that though and it also checks for some of your settings on GitHub to make sure you've enabled safe settings. For example, we will check for the so-called branch protection setting, which prevents an attacker from pushing directly to your repository if you have a malicious contributor, for example. We also check for, say, good configuration of webhooks. We verify that you have enabled authentication, for example. And we have about 18 checks today that you can look at the documentation online. Besides good practices, Scorecard also tries to flag anti-patterns. So let's say if you have a secret, some secrets available in a pull request, Scorecard will warn you. Scorecard will also warn you if you have over-privileged CI runs, which is really important because, as I said earlier, an attacker could try to abuse those permissions bypass code reviews and push to your repository. And we will see a demo of this later in the talk. There are two main use cases that we think Scorecard is useful for. First is for projects that you own. You can install it as a GitHub action and it will alert and flag all the sort of things that I just talked about. And then if you have dependencies, the second use case is to run Scorecard against your dependencies and learn kind of the risks that you're taking by using them. So for example, if you have a dependency that hasn't seen any activity for a couple of months, Scorecard may warn you that maybe your dependency is not maintained and is unlikely to receive security updates in the future. So with that, let's take a demo on the first use case, how you can install Scorecard on your own GitHub repository. So let me quickly switch here. You can see, yeah. So this is just an example repository. To install Scorecard is pretty simple. You can click on the GitHub action, sorry, the action tab at the top and then select, sorry, new workflow on the right. Scorecard is integrated with GitHub so you can search for it right in the UI and then simply click configure. That will automatically create a new file, scorecard.yaml, and its content is auto-populated. As you can see, here, Scorecard runs on a schedule every day and also on every push to the main branch. And as soon as Scorecard runs, it will start populating alerts in the code scanning dashboard. We'll take a look at this in a second. Something I wanna mention is that Scorecard takes an optional PAT token. You don't have to use it, but be aware that if you don't use a PAT token, some of the checks like the branch protection check will be disabled. And this is due to a current limitation of the GitHub APIs. We hope to resolve this soon by working with GitHub. So all you do at this point is just commit and push this to your repository. I'm going to cancel the changes because I have already installed Scorecard. And we can take a look at some of the results that have been populated. So let's look at this one, fuzzing. Not surprisingly, this one is just, Scorecard is complaining that the project is not being fuzzed as part of CI, which is correct. Something else you can see is branch protection. Here, Scorecard is simply complaining that we haven't enabled branch protection on the repository. And then we have all sorts of other checks that you can try to look into on your own. All right, so let's go back to the main slide. All right, so that's the part of the talk where you are starting to get involved. So we've set up a few, so that is quiz number one. It's a GitHub workflow. As you can see at the top, it's running on every push to the main branch. And it's checking out the repo, so basically cloning the repo, and then it uses this, an action called do slash some work. So I'll pause for 30 seconds and see if someone in the audience wants to get a shot. So does anyone have any ideas about what's wrong with this workflow? Yeah, over there. He's an insider. Go ahead, it's fine. Yeah, so what he said is that the permissions are not set. By the way, this is Varun. He's working with us on, he doesn't work on Scorecard, but he helps us, he has a tool to remediate those kinds of problems that we recommend as part of Scorecard. So like he said, the workflows by default on GitHub have right permissions, except pull requests. So all you need to know, all you need to do is to follow the least privilege principle is to add these permissions read all at the top. And then if any of those actions that you're calling in the workflow, one day it becomes malicious, then at least you'll be protected and they will not be able to do anything really terrible. All right, quiz number two. So this one is a little trickier. So let me walk you through what it's actually doing. It's working, it's running on pull request. Every time the pull request comes in, this code will run. And I'm skipping the thing in between, like a magician basically, trying to not show you what's happening. But basically what's happening, we're just echoing to the log the title of the pull request. So it looks pretty innocuous. So the question is why is this vulnerable? Yeah, yeah, so I repeat what he said. He basically said that there's a script injection vulnerability. So what's happening is when GitHub runs the script, it substitutes this entire variable, the one that's in between the dollar sign and the double bracket. It replaces that string with the value provided by the person, by the title of the pull request. And that basically allows someone who submits a pull request to change the code and achieve remote code execution. So the fix is to declare an environment variable and use it inside of your script. This way there's no substitution happening. Title is just resolved by the shell. So that's where some of you will come in again. So if you click on the link, it's a repository with the exact same workflow. It's vulnerable and if you exploit it by the end of this talk, we'll hand over another T-shirt. So yeah, good luck. Try to still pay attention to the talk. All right, so demo time. How many of you are aware of Devs at Dev? Cool. Devs at Dev is a cool tool. It gives you open source insights of what and all the open source packages that you have. It shows what it has dependencies and what are the other things that depends on that. Why am I talking about this? I will showcase a demo of utilizing this. Another critical thing that Devs at Dev provides also is scorecard data on the right side. You can see it gives all those checks and gives whether any of the open source packages whether or how do they score against using scorecard score. Like what Lauren mentioned, scorecard runs this for a million repositories every week and you don't have to, one of the easiest way if anybody can look at score for any of the open source packages is using Devs at Dev. So I'm gonna switch to a terminal and run something. In this demo, I'm gonna showcase how to use scorecard CLI. I'm gonna give shout out to Sasha where I'll build this package called demo. I'm not gonna type and talk because I'm gonna fumble. I'm gonna, so that's why I'm gonna feel like a superhero but I'm actually gonna be running the commands but not typing for that. How to install scorecard, scorecard CLI. If the easiest three ways are brew install, if you're a next shell fan like I am, you can do that. And the last one is go to scorecard, get a release page and you should be able to download and install. Now that I've installed scorecard, what's the best way to start? The best way to start is scorecard dash dash help. This will essentially give you all the things scorecard can do. It can be overwhelming initially to see all those things. The most important thing is the checks. Like what Lauren mentioned, there are 18 different checks that scorecard does. And all of this is using open data that's available there. No, we're not using any private APIs or any of that. All of this is samples as just openly publicly available. Scorecard action in turn uses the scorecard CLI engine for doing all of that. Everything is only based on this. So like I mentioned, actually those comments I need to type. A fumble and that's why I made that automated. So the easiest thing is now, let's use scorecard to check scorecard, especially on one check called branch prediction. We're gonna keep talking about branch prediction pretty often because that is one thing if you can walk out of this talk and say please go enable branch prediction, it'll probably solve a lot of problems. We'll talk about it, why it does. So right now it's running the branch prediction check and that comes back with the result. This is how it starts with the results says what's the aggregate score? The score of eight was a 10. The name, the reason why it gave that. And if you need any documentation, the mediation, all that is available in the scorecard repo. Okay, this is great, but why we didn't get 10? Can scorecard help us identify why we didn't get 10 or why we got eight? Yes. There's another flag that you append to the scorecard client dash dash show details. And it should essentially give you that result specifically. And this is that result which I tag along show details and comes back and clearly says, hey, all the results we perfect with either info or warning. So here right now it's coming back and giving the result saying, hey, everything is great, but you have only one required reviewer to merge domain. So that's a reason it's not be given 10 out of 10. Okay, this is cool. Can I get some JSON love? Because everybody wants JSON, right? So you tag along format output JSON. There you go. This should give you scorecard details in JSON format. One of the things JSON additionally gives you, you can see is gives you the repo name and the commit shot that it ran against. So you can essentially get a, you can store this data into wherever you want. And this data is where this data comes into the BigQuery which I'll demonstrate later, which will clearly show on every what commit shots that scorecard ran against those million repositories and you'll be able to utilize that information for what you need to. Next question, somebody asked me yesterday in another talk, does scorecard support non-github repos? Yes, but not with all the checks. Scorecard has an option called dash dash local and you can point to a local repository. It could be a GitLab, private repositories, any of the repositories and you should be able to run against those repositories. But it'll not be able to provide all the checks. It'll not be able to give you all the checks, can only run few checks on that. That's about scorecard fly to start up. The next thing that I wanna demonstrate right now is the BigQuery weekly table scans. How many of you are aware of BigQuery weekly table scans? Okay. The BigQuery, there's a scorecard once about a million repositories, checks for a million repositories and stores them in BigQuery. And so anybody should be, and it's public data, all this data is public data, anybody can go utilize that and run that. So it runs for a million repositories and stores that. Like I was demonstrating about depth to dev. Depth to dev also, depth to dev parses the dependencies and stores that in another BigQuery table. So in this demo, I'm gonna take the depth to dev data and just take the scorecard data, mishmash it and get some details out of that. I'm gonna cheat right now because running this query takes a little while. So this is the big query for getting dependencies of scorecard. So what I'm gonna do right now is I'm gonna get all of scorecard dependencies and use that dependencies to go run one check against the BigQuery for scorecard itself. So you'll see more over here right now. So I'm saying, hey, get me all scorecard dependencies from depth to dev and take that data and go run against BigQuery just for branch prediction, especially branch prediction that have force push enabled. How many of you know what is force push enabled? So I can force push to master without any checks. So scorecards has about 160 dependencies of that 124 of them have force push enabled. So that means any of scorecard dependencies can be compromised. So this is something that is not good. And all of this data is available in BigQuery. This is great. Wouldn't it be nice to know how were the dependencies doing in these last three months? The more I've been telling that the BigQuery has been running for a while, I'm not wrong, almost eight and about 10 to 12 months. So I can go back in time and say, hey, please give me how was the state three months before? Again, we're just doing for branch prediction and for force push enabled. It's not that things went good to bad. We didn't scan those additional repositories. That's why it was 54, went to 124. That's the demo. We wanna talk next. You don't have to be independent. Oh, thank you. Another, I completely forgot. Thanks, thanks, thanks, Lauren. One of the things that I also wanna talk about is, pin dependencies. How many of you are aware of what pin dependencies are? Great. So this is a GitHub action, it's a sample demo action. When I took the sample code from GitHub and ran, built this project. So in this specifically, you can see I am running my own action that I'm utilizing and I've tagged it to V1. What are the problem with V1 tagging? You can move git tags. So, sorry? Yes, it is mutable. So when I run this action, anytime I push, push, this is what once. It says, hello, Mona, the Octocat. That's what it, it's a sample, simple demo. So now I'm a malicious, a malicious player, which says all your data belongs to me. What can happen? This is a malicious user, force pushing a different branch with the same git tag. And let's see what happens. I'm gonna force push another branch called navin slash debased. I'm gonna force push that with a different, with the same tag and see what happens. So with that one pushed, now if I go back to my action, no, my action is running again. And as for us, everybody else, it was the same V1, nothing changed. Let it run, if people can see this, now I was able to push something to debased.com. Nothing else changed. So pinning your actions is gonna become critical. Scorecard specifically wants about those things. Good, okay, cool. Thanks, Lauren, for reminding me. You wanna switch over? Yeah. All right, thanks for the demo. So to summarize what we've learned so far, declare your permissions, follow the principle of least privilege in your CEI, pin your dependencies, and install Scorecard to catch things like script injection. So let's go back to the quiz number two and see if someone has managed to exploit the repository. I suppose someone might have, I haven't checked. Okay, no one has. So let me give you a demo. So as you see, the latest commit on this repository dates back to around six hours ago. So let's create a pull request. Oh, someone has a pull request. Yeah, did it run? Yeah, I think it ran, yeah. No, it ran, it ran, it ran. Yeah, so actually it has permissions to write to the repository. So let me, yeah. Let me, let me just show you the demo. Let's create another pull request. It takes a little bit of work to make it, you know, in 15 minutes it's not easy, so. So let's create a new pull request and then let's give it this weird looking title. So the title looks very similar to what Jonathan did. We are escaping the quote from the echo command and then we're just downloading a payload and piping it into bash, all right? So let's go and see the action that is running. So you see here without intervention from the maintainers, the CI run, like the pre-submit ran. And here we can see in the logs that it said, you know, open source summit at North America was here. Okay, so we managed to run, to have code execution and let's see what we achieved. Now there's a new commit to this repository from 36 seconds ago, which basically exploited the repository just changing a file, all right? If you had a branch protection enabled, it would have been protected. So that's why it's so important that you enable it and you also reduce the permission that you use in your workflow. All right, so yeah, you can visit and try it for yourself. All right, so before we conclude, I wanted to point out to a few success stories that we've had with a scorecard. Some very popular projects have installed it, such as Envoy Proxy, Flutter, TensorFlow, and Codecov. I'd like to point everyone to this OpenSSF project, which I think was blocked about yesterday. It's called the sos.dev reward program. And if you improve or fix some of the scorecard alerts in critical repositories, you can be financially rewarded with $505 or even more if you do more work. We have plenty of stuff coming up. We have scorecard badges that we're working on. We're hoping to release in the next few months. We have a website coming up, and then we are currently working on a project to improve our GitHub action. With everything that Navin talked about, very soon you'll be able to see the results of scorecard for your dependencies. Every time there's a new pull request that changes or add a new dependency to your project, you'll be able to see the results and hopefully make some informed decision about whether you want to proceed or not proceed. Last but not least, I'd like to give a shout out to the All-Star app, which is another OpenSSF project. It's a companion app to scorecard. It allows you to enforce policies on some of the scorecard results at scale across an entire organization at GitHub. And then again, the OpenSSF community works in the open. You're highly encouraged to join the conversation if you're interested. And that's the end of our talk. Thanks for your attention. Thank you. Any question people may have? Yeah, yep. Yeah, we're always happy to take more heuristics. That's what we have right now, but if you have some good ones, we're happy to consider them and add them. A lot of people have actually suggested and contributed new checks. So yeah, we're super happy if you want to add some support for new checks. Adding to what Lauren mentioned, Webhook was something, another external contributor came in and said this would be an amazing great addition. Did that whole pull request and got that done. So yes, we are. Thank you. Yeah, someone over there. Sorry, can you repeat the first question a little louder? So you're talking about, are you saying you want some... There's a microphone behind you, just so that. Oh, yes. One question would be an option to remove, to know well, proactively remove tests which are not irrelevant for scorecard. You show an example in the demo that for example, fuzzing give you a negative score, but you're not using the fuzzing in the CI. This is a question, one. Second question is, is there any plans to look at the results? Not just what tests you do, but what are the scores? So for example, you can get a 10 out of 10 for scorecard, but all the results of the tests are terrible. Like everything is a critical one, critical stuff like that. So question number one was, or can we ignore some of the, so right now you cannot do that. We're working on a project where we're gonna give the results without the score, and we're gonna make that data available publicly. So you'll be able to look at those without the score, and so I guess maybe what you're asking is your own project, your score is going down because it's spotting, is it a consumer on the consumer side, or is it you as a maintainer? You'd like to remove some, you'd like to configure a scorecard so that it doesn't bombard you with alerts. Consumer side. Right, so for maintainers, yeah, we'd like to improve and make it more configurable. We haven't done it so. It's just, we try to sleep at night, so we haven't had the time to do it. But yes, it's something that we want to do, and on the consumer side, I think the results without the score would allow you to, we're planning on having a more structured output that you can pass yourself and take a better decision without looking at the score and decide whether something matters for your use case or not. So we're working on this, but it's, I think this year we should have a prototype for it. Yep, yep, and also another one thing that like what Lauren mentioned, if you're a consumer you can, you don't have to pick all the checks. You can pick what checks are critical for you and decide which ones to run. Again, because not every repository requires a fuzzing check. So you can, on that specific, you can decide which ones you want to run against and get those results. And like what Lauren mentioned, then later with that, be able to determine, especially with the raw data, it's talking about the raw data without scores and able to determine what's the risk factor on consuming those dependencies. Yeah, we'd like also to support annotations. So if a developer says, I don't care, I don't have, you know, this check is failing, but this is the reason why we'd like the consumer to be able to see that. So that's something we're thinking about, but we don't have it implemented yet. So yeah, so I think for the second question, sorry, I forgot, what was it? Good results of fuzzing, per se, or? No, so right now the check of fuzzing just checks if you use fuzzing in your CI, let's say you have OSS fuzz or you have Golang built-in fuzzing, then we'll give you a score of 10. We say this is good enough. And we're adding support for more languages. We started with the generic OSS fuzz and CI fuzz, and now we have added support for Go, built-in fuzzing. I mean, I think they came out with a, Golang has support for fuzzing right now, so we can use it and we detect that. And we're planning on adding support for other languages. Yeah, which one? Yeah, so I think the question was, does scorecard differentiate between adversarial inputs today and not really? We don't. Yeah, so this is something that has been in the back of our mind, but we haven't really considered it that much. I think as a scorecard grows and gain popularity, those questions are gonna come up more and more. So I think we have to think more about it, but yeah, to answer your question, we don't have, I guess what you're asking is, you'd like us to be able to at least label some of the check and say, this can be the repository can change it or not change it. And scorecard also clearly says which are critical, which are high, which are low right now. So you can, because we think based on these data that these are really critical, like having dangerous workflow is a good compromise. So giving you an example on that. I think we gotta stop right now. Yeah, sorry about that. Sorry, we'll take that question offline. Yeah, thanks everyone. Oh yeah, we can have three shots. If you answered some of the questions, come and get your shot.