 All right, well, there is another challenge that we all have and collectively face on the internet and with the technology that impacts our daily lives. And there's a ton of hard working people working on it, and that is the matter of cybersecurity. I think many of you in this room, if not most of you in this room last year were affected by the Log4j vulnerability, people I talked to in cybersecurity groups and in enterprise technology continue to work on remediating this vulnerability that was so widespread and really compromise the systems of so many different systems. At that time, there was a call for collective action to work on open source security to improve the global software supply chain, in order to make it safer, secure, to in many ways sort of get ahead of this big problem that we all collectively face. And puts a lot of us at risk if a hospital is subject to a cyber hacking attempt through open source vulnerability, or if critical services get shut down. One of the people who really answered the call almost immediately to go all in and help solve this problem is our next speaker, Eric Brewer. Eric is the vice president of infrastructure and a fellow at Google. He's also a board member for our open source security foundation. He has a long history of being one of the builders of the internet. He is known for formulating the CAP theorem on distributed network applications in the late 90s. Today, Eric's going to talk to us about how we can collectively improve open source supply chain security. Please welcome Eric Brewer. We managed to hide the clicker. That's all right. Who needs slides? I'm happy to be here. I spoke to many of you in Austin on a topic of curation. And this is kind of a follow on to that talk about automation. You may not remember what curation is. I hear a thing coming. There we go. All right. So curation answers the problem of open source is as is. Comes as is. It says as is. And at the same time, it's used in important infrastructure worldwide. Almost all nations depend on it. 90% of enterprises use open source. So wildly successful, except with that success comes kind of top down mandates like, oh, you have to be secure. You have to have supply chain security. You have to have S-bombs. And more and more of these regulations are coming from many different countries. And top down regulations don't actually work very well with as is software, which by definition, it's on the consumer to figure out how to make those things happen. So curation is this thing in the middle which says the curator, which could be a maintainer or a company, Red Hat's a classic example of a curator. They provide security patches to their customers. So they are curating the open source so that it's not as is. It actually comes with some guarantees about security and support. So we need a layer of curation broadly speaking. And that's not going to go away. But you just dig into that. Well, how do we get there? And one of the important parts is automation. And that's what I'm going to talk about today. So the good news is because all these governments and companies have realized they have to solve this problem, I am seeing an increase in funding in various ways. It's not like it's immediately here. And economic downturn doesn't help with all this. But bottom line is I've met with the US government, UK, Germany, and several others. And they're all like, we depend on open source. It's a problem. How can we help? So they don't know how to help yet. And we'll work with them on that. But you can just see that there's interest in getting this solved. They're willing to pay for curation. They want to buy open source in some way that's supported. I think even pay maintainers, if they knew how to do that. It's not easy to pay maintainers because there's so many of them and they have all different locations in countries. But it is coming. So that's the good news. But we still need automation because it's really hard to get all this stuff to just work the right way. And in particular, you look at supply chain security. It's not like it's about fixing one bug. It's about fixing a bug in a dependency, fixing all the dependency chains up to the things that you care about, rebuilding all those intermediate artifacts, signing them with provenance that you can prove what's in them, and eventually getting a package or a container that you can use. And most of those steps right now are manual. If someone fixed log for J, someone else has to fix the thing above it and fix the thing above that. Fix the thing above that. And there's, by the way, millions of Java packages that indirectly use log for J. And all of them had to be fixed roughly manually. Very little automation in that process. And many people have stepped up and done that, but that is a heavy toll. So I kind of feel like automation is the only path to get to a secure supply chain where consumers can understand what they got and how it was built without saying, maintainers do more. That's not my message. My message is let's get automation working so maintainers do less and we can deliver more in an automated way. So it starts out with something simple, which is even the way you build something today tends to be manual. And I don't mean that you haven't figured out how to build your stuff in an automated way on your laptop. What I mean is consumers of your packages can't build it in an automated way. What does it take to build? Well, you go to the readme file and you figure out what to install, and maybe you make a VM image, and then you install some stuff, and it doesn't work the first several times, and eventually you get to the point where you can actually build the thing. But that's a lot of toil. If you're already maintainers, no big deal because you've already done that once, but if you want to build someone else's package quickly because it has a vulnerability and you depend on it, it is not so easy, right? And it might be by the way in a language you don't know. That happens too, right? So really I'd like to see us get to the point where any curator or consumer of a package or a jar file can actually know how to build it from source, you know? And if we were taking a step farther, I don't want to readme file. I want something that's much more automated. The best case would be here's a build system that knows how to build it and you can invoke it, right? And whatever needs to be installed in the VM to build that thing is already there because someone else set it up, right? If we can't get there, we can maybe get something which is more descriptive. Here's a YAML file that says how to build things, but it's not for humans to read, it's for machines to read, right? That would be a step up. By the way, I want consumers and curators to pay for this build, right? Who pays for the builds now? Is it on the maintainers to do builds and testing? Cover the OPEX for that? That doesn't seem fair, right? So I'll give some examples of this, but really a side effect of this is two things. We would actually have the ability for someone else to pay for builds that cares, including governments. We'd also start doing things that assigned output, like was it really built the right way or not? We don't really know that today, right? You could build it yourself and then you know, but that's, again, a high toil. So what does this mean? I really would like users to first know that the version they're running is actually the version you intend them to run, meaning what was built with the correct dependencies and reasonable tool chain, things like that. Automatic builds would be a huge improvement, but also the obvious next step is reproducible builds, right? Some groups like Dubu have done a good job with this, but like, suppose Google does the build for you, then you decide, do you trust Google's build process? Maybe you do, maybe you don't. If it's reproducible, someone else can build it and check it, right? And that gives us a level of transparency. It would be great. So that's a harder, it's much harder to get to reproducible builds, but that should be the goal. And because we're an environment where the provenance matters and things like S-bombs are coming, again, I don't really want maintainers worried about S-bomb generation. I want you to use a build system that generates S-bombs as a side effect of having an automated build, right? And by the way, the S-bomb standards will change. There'll be other standards coming or other regulations. I don't want you to worry about those either. I want those built into the build system and magically taken care of, right? Done well, my hope is that being maintained will be less work because you get more from your infrastructure and how it's built, how it's signed, where it's stored, how it's distributed, how your users know, which is the right one. So, signs that we'd be on the right track would be things like someone that wants to use a package and curate it for their use case, make sure it's secure, can actually build and test it themselves easily without having to know that much about the project. Again, most teams are using thousands of packages. They can't possibly understand them all in detail. It'd be really nice if I sent you a pull request that I could actually prove that it passed your tests. If I had automated builds and tests, I could actually get them signed and you'd say, oh, I'll take this pull request a bit more seriously because I know it already passes my tests. That's a little bit less work on me. And there are systems I've seen, Grapeus has this working with CircleCI, I believe. So it's not like this is totally novel but I think it's something we'd want to move towards. And again, letting others run their own tests at their own expense, absolutely reasonable. I can't really make a promise but I would say as a blanket statement, Google would be happy to run tests on critical open source packages. We're probably dependent on them anyway, directly or indirectly, right? We pay a huge amount on Fuzz testing as we get to in a second. So many groups should be willing to pay to do extensive testing on these kinds of critical packages at least. Not all packages but certainly the ones that are critical to society. Now, we do this already in a narrow case that's worth covering a little bit, which is OSS Fuzz. This is a fuzzing service that fuzzes open source packages about 1,000 plus critical open source packages today. This is paid for by Google and actually it's quite expensive because Fuzz testing is expensive. And so far it's fixed about 29,000 bugs of which from roughly almost 10,000 vulnerabilities. So what's interesting about this is it is doing automatic builds and tests for the narrow case of Fuzz testing. So when you have your package integrated in here, which is not that hard, what you're telling us is how to build your package and how to run tests on it so that we can again, automatically run tests even as you do updates. And once you have this automated testing, a huge bunch of things come out of it. The most interesting one is, on my mind at least, is bisection testing. So bisection testing is fuzzing found a bug which pull request introduced that bug and how do you find it? And the answer is you do binary search. So you just say it's between this point and this point. I'll see in the middle and I'll see if the bug's there and I can refine down to the exact pull request that introduced the thing. And of course, if you introduce a fix, I'll know the request that fixed it. So the reason that has a high bug fix rate is when Fuzz tells you we found a bug in your critical open source package, we tell you which pull request introduced it. It's very specific. So the fix rate on those bug reports are 90% way, way higher than normal bug report requests. Now part of this, these are critical packages in the community to responsive as well. But this is a working example of automated testing and building, making maintainers lives easier by giving them high quality bug reports tied to CLs or pull requests that are relatively easy to fix because of that precision. But I don't wanna do this just for Fuzz testing. I wanna do it for all kinds of testing and builds. And also I don't really wanna do it just for 1,000 packages. Now, even Google can't pay to do all the bills and tests for all the millions of packages, not realistic. But I do think we can enable a framework that would make it easier to do automated builds and tests and then as groups that care about curation and S-bombs and FedRAMP and all the regulations coming, that platform will enable them to pay for tests and to participate in a way that they can't participate today. So this is not an easy path. In fact, I'm not even telling you today, here's a build service you can use, although I hope many such build services will exist in the future. But I wanted to get it out there that this is the path we need to be on. And this is really the only way to get security to be pervasive. We can't have so much manual toil in so many packages that we depend on. It's not sustainable and it's not fun, frankly. Let's take all the unfun stuff out of development and put those into automated services as the best we can and go back to the fun stuff. That would be great. So what's this automation train? Well, automated builds is the easiest. In fact, if you look at the source of specifications, what it means to have a secure supply chain, having a trusted build service is kind of a core part of that. I really don't want critical packages built on somebody's laptop running unknown software. I'd really like it to be in a repeatable system, ideally reproducible, running on a build service that produces signed outputs. That's good development. We need that broadly speaking. Testing is not just fuzzing. I think we have lots of things we can apply in this space. Scorecards, which Tracy mentioned in the earlier talk is a fantastic thing. It's just really about the hygiene of your packages and the repositories, and it covers up wide number of things. OSV scanner and also other groups like Sneak, for example, do a great job of helping find vulnerabilities, but we actually need to automate those things as well. Go has done a good job with vulnerability check. You can actually forego. It'll tell you what are the vulnerabilities and all the packages that you're using. It's very easy to use. You should go check that out. Go user. And finally, when you're doing automated tests, let's prove that the tests pass because that again allows maintainers to not have to run those tests themselves. That's another way to reduce toil. And then finally, when you have built a correct and tested package, it'd be nice to actually automatically publish it as well. So we work with Python Foundation, PyPy, do trusted publishing. Let's make that a bit more integrated into that process. And that can again apply to a wide variety of languages. So this is kind of the beginning of this. I'm certainly not the first person to call for automation or reproducible build, but I think what's different here is saying that we need to do it as kind of a framework that everyone can participate in that focuses on enabling players like governments that have money to participate, at least in the OPEX, if not in even more direct payment of people, which I think is on the table too. So hopefully we want this change. Hopefully what it means is how do you do build? Like we don't have a spec general for how to do build, right? Readme file can't be the answer for how you do build. It's gotta be something that a computer can read. How to run tests, how to prove a test is complete. Those are things that I think we've, there's lots of tools that can help with parts of that, but we don't have a kind of a generalized way to do it. So last thing I wanna talk about, hopefully you saw the little tighten keys on your chairs. Those are a gift to encourage two-factor authentication and all developers, especially in physical packages, should be using two-factor authentication to sign their changes. But it's also ISRG's 10th anniversary. I just wanted to call out a success story here for less than crypto. 300 million certificates generated. You use these all the time. It means that the web is mostly HTTPS now instead of the insecure version. So let's pretend it's been a huge success and I wanna congratulate them for that. We've also worked with them on Rust and the Linux kernel to try to make that a big deal. So let me just stop with that and say thank you to them and congratulations on 10 great years. Thank you so much.