 Let's get started. So my name is Brian Fox. I'm the co-founder and CTO at Sonatype I'm also a member of the Apache Software Foundation and an open SSF governing board member as well. So I've been living all things supply chain for 15 years And I'll talk a bit about the view that I have on that and how I've seen it unfold during that time So a little bit of context Let's see First slide's always a tricky one. There we go. This is the Maven central repository download So Sonatype has always run the Maven central repository. That's Predominantly Java, but also Scala and some other things. So if you're Developing Java and in the report I saw this morning 50% or something like that financial firms said they were heavily using Java You probably all are you're doing so with open source You're getting that open source from our repository Not a lot of people know that and so that that repository and the consumption patterns gives us really unique Insights into what's going on at least inside of that ecosystem. So we're on track to About 700 billion downloads this year Java shows no sign of slowing down NPM same thing. This is a little bit of an older stat But their their growth looks about the same So if you were to look at Docker hub or anything like that the point being open source consumption is exploding The number of things that the developers have to choose from correspondingly exploding and I pulled these from the Report that Gabe announced just this morning. You guys might not have had a chance to see it yet This is the state of open source and financial services Supports the same thing open source consumption is double the number 48% from what it was just last year The number of people who are prohibiting it are very low getting even lower this year And consumption. Why are we doing this increases productivity, right? That's why open source Continues to be used more and more and more because we're sharing and able to build upon the shoulders of those that came before us the result of this in a modern system is that you're Pulling in components largely most of the modern package systems have binaries that are already built You're not cutting and pasting open source code You're actually pulling the binaries. What that means is you have a supply chain right and from my perspective not enough people are actually Recognizing that and using that pattern to their benefit So if we take a step back for a minute supply chains are everywhere, right? think about it think about your cars think about your food think about the planes and In the supply chain world Edward stemming is kind of like a superhero. He helped the Japanese auto industry rebuild after World War two and focused on a number of key things The important one here, you know is this last one continuously tracked a location of every part I mean, obviously we know we want to choose better parts and all that stuff But from my perspective so many organizations are simply failing to do this thing here Right and this is really important. So if we think about this in terms of real-world Implications I tell a couple stories here that I like this first one is the Chevy cobalt in 2014 They had a little bit of a fiasco Because what had happened was there was some some basically a bug in the ignition switch Where it was too easily turned from the run to the accessory mode And if people had a heavy keychain or something happened it shut the car off The worst part about that is about two seconds after the car was shut off the airbag stopped So not only did their engine stop they lost power steering and power brakes their safety systems turned off too people died The problem was the engineers at some point they fixed this problem and they didn't change the version number of the park Right. So anybody doing software recognizes this. This is the yeah It works on my machine not your machine kind of problem Like I have a different version of a jar than you do right tools like maven and pi pi and npm Have tried to help solve that problem by making versioning very obvious and discouraging this But this is a problem that happens in software and when it happens in the real world people die The opposite of that is back when Boeing 787 launch They had a potential similar fiasco with batteries that were catching on fire at the gate But they were pretty quickly able to zero in that they were all related to a single manufacturer in a single batch Because they did have supply chains. They were able to understand this and they were able to To to fix it very quickly as opposed to what happened to Chevy Because there were parts on the shelf with the same version number that were still broken even after they did the recall They were putting broken switches into cars thinking they were fixing them And then they had to do another recall and pull back all of those to fix it right again because of simple configuration management failures in physical supply chains and then there was a lesson the lettuce industry learned a couple years ago You might remember here in the US. There were a few years We had E. Coli outbreaks it was like two or three summers in a row and the last one the last big one that happened It occurred when the lettuce growing season shifted from Arizona to California when people started to get sick and so because the Lettuce wasn't labeled with the region of origin it made it impossible for them to figure out what was actually going on And so what did we do we threw out literally all of the romaine lettuce and like all of North America And that's an economic disaster. It's an ecologic disaster and a lot of a lot more people got sick because of it So what did they finally do up to three years of this? They started labeling the bags with the region at least if not the farm where it came from so these industries were making mistakes That led to real problems and and I'll tell you our industry is doing the same thing And so we need to start thinking about this in terms of a supply chain and we don't have to reinvent the wheel We can look at patterns from other other industries and and and kind of adopt the same practices So from my perspective, I looked at what's happening Over the time that I've been doing this and I see three distinct phases of what's been happening The first phase is really your old school exploiting open source vulnerability So these are bugs that existed in the code somebody found out how to make it do something It was unintended and then exploited the heck out of it You know when I first started we were talking to financial firms who would tell me things like well one We're not using open source Because they thought open source meant Linux and my sequel and Firefox when in fact those same people were downloading more components from our repository than anybody else in the world because the leaders didn't understand that 90% of their modern application was actually open source components and then they would tell me things like well I don't have to worry about that because I have a security team in a firewall Right, so that was the perspective of a lot of people when I started doing this and started talking about this But the first one that really raised a lot of attention was the first struts vulnerability. This was in 2013 And this is a financial conference. It really resonated within the financial industry We had a lot of customers and a lot of prospects really kind of flipping out about this one because anonymous was trying to make a name for themselves and A lot of banks took their websites offline for maintenance on the same day after this was released So draw your own conclusions from that But I know for a fact that the financial industry really started paying attention to this problem that I'm talking about here way back then Right, so I'm hopefully preaching a little bit to the converted here But but there's always something new for everybody In 2014 we saw shell shock and heart bleed. This was the rise of marketing around vulnerabilities and you started to see it on the news And and people paying attention to it right with outsized impact and and generally the conversation started to shift from I have a security team in a firewall. I don't have to worry about this to Maybe I should pay attention to my open source again small numbers, but it was starting to grow in 2015 Commons collection comments collections a very popular library from Apache that does things like link list and Hashtags and all that kind of stuff. It's so popular It's like log for J could be assumed to be present on the class path of any Java application Or at least the Java container that it's running in And this is a picture of Hollywood Presbyterian Hospital They were ransomware by an attack leveraging Commons collection a full year after this was disclosed at black hat and Subsequently patched right and so this hospital was shut down for a week They had no backups no way to restore it. They ended up paying the ransom in order to get this data back But I I bring this up because you know the old adages humans We don't act until somebody dies right now. You shut down a hospital outside a major metropolitan area It's going to have very real impact on lives There are studies that have said that if you have a heart attack or a stroke in New York or Boston around the marathon days You are significantly Have have significantly worse outcomes simply because the ambulance has to route around the the Marathon route and you know minutes matter in those two things and so, you know Mathematically you can say this very likely killed some people there have been unfortunately some stories since then one in Germany Where a woman had to be transferred for a hospital ransomware? She died because they couldn't get the stroke the blood clotting medicine to her and there was another one I think it was in Louisiana Alabama where a baby died because the ransomware shut off a monitor and they didn't realize a baby was Distra in distress. So these things have very real impact. We're not just talking about stealing of money We're talking about very real societal impact From frankly our collective failure to do the right thing here If we talk about log for shell, right? Who's not talking about this? These are the statistics. Wow, that's really blurry for some reason These are the I have a better slot of this later the early statistics of the adoption and you can see pretty quickly as an industry We got to about 60 percent of the consumption was of the patched versions And then they picked up all of those new ones but about 40 percent of the consumption pattern was still of the known vulnerable versions and that continued Shockingly until about last month. We finally crossed into the below 30 percent But it's taken us a whole year so for a whole year 30 percent of the consumption of this thing has been of the known vulnerable version that Everybody and their brother knew about and was writing about what the heck is going on, right? If there's not better evidence of a supply chain failure than this I don't know what there is and and there's a better picture that shows the full view view of this later So I raised this at this section because despite the wide impact of log for Jay This was a phase one. This was above This was a coincidence of two features frankly that existed in the code base that people figured out how to exploit This wasn't even one of the phase two or phase three types of attacks that I'm going to talk about next And yet we still struggled a bit to deal with this China has been exploiting the heck out of this this came out this summer They this came out in October national security agency has this at the top of the list of Chinese Attacks still all this time not a surprise given that people are still building stuff with it And I don't have a slide but last month they came out and said Iran Attackers were using the same exploit to go against government agencies, right? So some people say log for shell was a big bunch of noise about nothing Clearly that's not true if the government is acting upon this So we think about phase two what started in really 2017 Was was a new rise of it the attackers actually moving to Insert things into the supply chain where previously it was a race between they figured it out Could they exploit it before you could patch it? That's what I used to talk about then in 2017 There were a couple things that happened. This was a study that found that in the npm repository a majority of the things published here or the publishers had weak You know passwords either they were like password or they checked them in the github and it was easily compromised Coincidentally or not. I don't think so within a few weeks after that We saw actually the first typosquadding attacks So typosquadding using a confusingly similar name and underscore instead of a dash and confusing developers like you're probably all Familiar with this term at this point now But five years ago. I was out there telling people this is happening and I felt like you know I was the only one seeing this and what was really shocking about these two instances was that These things were stealing the open-source publisher credentials. They weren't trying to steal so security numbers or credit card numbers They were stealing open-source maintainer credentials and that was like a huge thing for me I'm looking around going am I the only one that sees this because this is really freaking scary that all of a sudden They're paying attention to the supply chain now and then not surprising Right after that we saw an evolution of these attacks back in 2017 2018 I was giving a version of this talk where I was kind of talking about this happened last week Did you hear about this? There is a tidal wave coming and so we have all of this documented at our supply chain report Now and you can see at the bottom lots of details But I felt like I was watching the attackers hone their craft in real time that everyone that came had elements of it It was like it was like COVID where each new variant had attributes of the old ones and new things that made it worse Is basically what was happening here? so the third phase that we're kind of still in now are these Attacking the developers and the development infrastructure itself specifically using those supply chain attacks So one of the first one this was a Jenkins that was unpatched. They were mining Bitcoin back in the day Might be worth less than that now. I'm not sure I should do the math and update the slide But it was a lot of money. It got a lot more and then probably came back down What happened here I lost my pointer and So last year there was another one where an attack got into vercata a camera company where they went in through the Development infrastructure and they were able to move laterally around the organization getting into the actual camera feeds, right? So attacks on the development infrastructure is not just isolated. These are the back doors that allow you to get into an organization and And they're using this through upstream types of attacks. So this was one Code Cove was another one. This was a popular tool that was breached And so everybody who had this tool in their infrastructure also was vulnerable and there were a number of attacks that were attributed to this one as well And so what we've seen is this evolution of additional types of attacks going on within the development infrastructure And if you think about inversion of not inversion Infrastructure is code that exists in many organizations now It means the development infrastructure potentially has the keys to the kingdom to the production kingdom Which is why the attackers are going there and so You know last year also there was a new attack It just started as white hat research saying I if I figure out the name of a NPM module that you're using internally and then I go to the public repo and I publish it with a very high version number Your tool that's looking for the latest can't tell the difference and will download the attacked one So this was initially researched that went out and he collected a bunch of bug bounties But we watched pretty quickly as first there was a huge flurry of of copycats going after bug bounties We saw it was seven thousand percent within the first week of those types of attacks But buried within those were malicious ones where they were taking even the proof of concept code But actually using it to steal credentials and go after actual dropping actual backdoors and that trend actually continues today The attackers are still focused on on this type of problem I have a slide later, but we've identified over a hundred and three thousand instances of this using some of the automated Mlai techniques that we have Because it was the only way for us to keep our customers safe You know these things are happening every single day still and they're getting more and more sophisticated Now why is this happening? Well, this study is a little bit old But still somewhat shocking way back in 2016 the worldwide Global drug trade as an industry was four hundred and thirty five billion dollars that same year with Cybercrime was already a bigger industry and if we think way back then we weren't talking about this, right? This seems less shocking now, but what is still a little bit shocking is that last year it grew to six trillion dollars and It's predicted to get to ten and a half trillion dollars just a couple more days now If you flip this around and you think about the motivations of the attackers This is the VC funds that are investing in them to attack our infrastructure They're getting this money from us. They're getting it from our insurance companies And they're using it to get more and more sophisticated against us because we're making it easy for them Right, so this is why these rise these rise of attacks are happening and why they're not going to go away And also the attackers are engineers So they're looking for the easy way and this was a study that looked at the npm Repository interconnection at the time they found that if you were able to get to just a few Maintainers you could affect nearly half of the components in the entire npm repository And if you flip it around you could target just a few packages and get millions of downloads, right? So of those hundred and three thousand Instances I talked about most of them are in npm a handful of them are in python So far none of them have been in Java and and we can have a side conversation about why that is it has to do with some Of the fundamentals of maven and some of the things we do to manage that repo But they're slim. They're easy pickings for the for the attackers right now And so, you know that growth of those attacks has really exploded in the first few years You know there were a handful like I said I used to be able to walk everybody through the progression when you know I could count them on two hands But then it exploded the 216 following year 430 percent year after that 650 percent and then When we look back over it, it's been an average of 742 percent year over year for the last three years Right, so this rise is not a coincidence with that rise of the money that they're collecting from us And then turning around and being incented to go after more Right, and so this is the the hundred and three thousand slide that I was talking about so We did some interesting stuff here you guys would all probably understand this we modeled our Approach after credit card fraud detection because every new every new type of attack was not exactly like the other So we couldn't we couldn't just build heuristics to recognize what was malicious But what we could do is recognize what was normal and identify the unusual types of things just like when I travel My credit card gets used at restaurants and all this stuff all over the world, but I don't go to Southern California I live in New Hampshire. I don't go to Southern California. I go to Walmart and buy a bunch of TVs That's unusual. That's that transaction is going to get blocked instantly so we do basically the same thing for new packages and and We're able to protect our customers that way and so it's it's a it's a pretty cool thing We could talk about it outside the conference if you if you want So what do we do about this in the wake of log for J? This was my perception of what was happening in the industry all of these articles were being written that were missing the point They were freaking out like oh my gosh these are free projects and we're dependent upon them and the the feds should get involved and You know, nobody's paying them their amateurs and look open source developers are not amateurs Most of them aren't even volunteers These are the people who want to do nothing else but write code all day every day and they do cool stuff And a lot of times their employers are paying for them But the outside world kind of figured out their thought that this was like a bunch of you know College kids writing code that didn't know what they were doing which really frustrated the heck out of me You know and and freaking out that oh my gosh, we're all dependent on this free software Let's get some policy right so the world kind of overrotated in my perspective towards all these things which I'll call classify as like fix open source educate them make better tools do all these kinds of things and and I think they're missing the point Imagine if remember the Takata airbag imagine if the manufacturer said you know what we're not going to do recalls We're going to pay Takata more and we're going to do a better job next time That's what all those things that the world is asking about open source and reality is they didn't really do that They said no we know exactly which cars are affected. We know when they need to get recalled I just had one in a truck replaced like six months ago because they knew that for the first three years or so It was safe So they were able to phase it out because it had detailed understanding of every part and everything and So this is that log for Jay year on view which kind of tells me a whole bunch of companies out there Don't have an understanding of where those parts are because I can't think of a logical reason why they wouldn't have updated Even if you could prove you weren't exploitable everybody was asking everybody else vendor Consultant type of relationships if they patched and how they mitigated so even if you knew it you were going to update Just so you didn't have to answer yes I'm still using it and then explain why that's okay to your banks and your insurance companies right and yet this is still where we are and so we did some some Studies this year again at that URL you can see the details We looked at the vulnerability the things that were being downloaded that were vulnerable at the time They were being downloaded from central across all vulnerabilities 96% of the time when somebody was pulling something from the repo there was already a fix Available right so all of that work on the left-hand side Why does this keep stopping on me all of that that work on the left-hand side? Is solving 4% of the problem right and so really we need to get organizations to make better choices Have an organizational view of all of the components that they're doing and then be able to manage that response And that's really the part that I've been focused on this whole time because from the download statistics way back When we recognized that that was actually the problem way back in 2011 we looked at it We saw that bouncy castle a Java crypto library the most popular version of it Happened to be the one that had a level 10 vulnerability found and fixed over a year later So even back then we were able to identify this same trend And and not enough people believed us I think but now now we're starting to get to that point so we've also looked at Some studies over the years a few years ago We got to the point where 50% had some process in place and 37% had automated tooling That's great because when we started surveying people these were in the teens, but I'm a little bit of a jaded person Would you accept those statistics for these things if only half of the parts in your car? We're supposed to be there no so so we should be happy that it's not teens anymore And it's 50% but as all of these statistics I've shown you we still have a long way to go And the good news is you know, we have examples where it really works This is an example of someone using Our tooling to help do it when log for shell happened. It was basically a nothing burger for them They had 80% of their 4000 portfolio over mediated in the first four days And they basically hit per hundred percent in weeks after that We've talked to other financial companies where they said yeah It took us a hundred days to understand which applications we had long for JN, right? So that's a big deal I don't know how it was for you guys, but maybe maybe somewhere in between these two things when we look at it across the board You know you can see Fast differences and companies that have tooling in place versus those that don't shouldn't be surprised, right? so So the point being you have a supply chain even if you don't manage it the attackers are using this against you You know if I told you about a new vulnerability today Literally sometimes I get that opportunity something breaks as I'm about to talk and I can announce something Could you tell me are you using this component anywhere in your organization so many places can't even do that basic thing? If you can do that, do you know exactly which applications it's in? Can you track the remediation of it, right? If you don't know what you have you can't even do this and how long would it take you to deploy an update because that's What you're racing against And so worse how would you avoid the next malicious release you have zero time to respond? The thing is taking action as soon as your developers download it They're dropping backdoors They're exfiltrating exfiltrating credentials and in a COVID work from home environment a lot of the Perimeter defenses that might have existed and many organizations don't so where a backdoor might not have actually been able to deliver The payload inside an office if they're at home on their home network or in a coffee shop that may not be true So the pandemic has accidentally made this quite a bit worse, right? And so you know the the point that I want to make here is that You know, we have a factory Deming principles and focusing on traditional application security things focusing on is this thing safe before I ship it and I Release it that makes better products. So in the auto manufacturer. It's making the cars better faster cheaper, right? But what is it not doing? It's not defending the factory against an intentional suicide bomber or whatever type of intentional sabotage That's what the attackers are doing to our development infrastructure in this phase three This is happening every single day, right? And they're they're using it to great effect to move around the organization and then finally a couple years ago We we did some other research and we looked at organizations and we said based on their survey response We kind of put them in the different categories, you know in the first category here Security is the most important thing even if it slows down development and then the polar opposite of them or whatever We're just going to ship stuff as fast as we can security be damned, right? So there's two polar opposites of that and then there are people in the middle that are kind of trying to do the sensible thing Now if I told you that the people that were focused on being fast You know paid a penalty and security and so it's probably worthwhile to slow down a little bit and do some security And that's the right thing after everything. I just told you I think everybody would go Yeah, that makes sense. Let's go a little bit slower because of security matters But what we actually found is when we put it all together is that the people that we're doing both We're both faster and more secure than the organizations who only cared about one dimension or the other And that seems kind of surprising But if you stop and think about it these people who only cared about shipping fast They didn't get a free pass when log for j happened They still had to deal with it, but they had no tooling or process in place to do it sensibly Right and and the people that have very draconian security practices. I mean, of course, they're going slower But because there's a little bit of mashing of gears and everybody's accepting risk and waving things and pretending like these don't matter Things don't get fixed. So it actually reduces their their security footprint, right? So the point being this is one of these classic win-win situations where you can actually be faster be more innovative And be more secure if you stop and think about this think about it like a mature Supply chain and so if you want to think about or see more of this stuff that I talked about Including the last eight-year reports. You can find them all here So pretty much all of the all of the charts that I showed came from one of these these years. So I think that's about my time Thank you for coming. Hopefully this was informative. I don't actually know what time I'm supposed to leave Do I have time for questions? Do you know I have two minutes? Okay? I could take one quick question if somebody wants No, okay. All right. I'm definitely worried about it changing, but I think the attacks will be different So real quick. There's two reasons why that is the case pi pi npm and some other ecosystems Default to updating the latest so every time there's a build unless you told it not to it's going to grab the latest for an attacker That's a ready-made audience. I put something out there. I got millions of downloads So it makes it a target for that reason the other reason is on Maven We have namespacing the group ID which correlates to the Java package Which is usually the company's domain so calm dot sonotype or calm dot fidelity or whatever, right? And so when when people publish the things we Validate that they control that namespace we validate that they own the domain or something else like that or a github project Can only use their github name so you can't show up to our repository and pretend to be Apache and publish something That is a typo away from struts. Nobody's going to do it So because Maven has that and because sonotype enforces it it makes it harder for them to do it But also because Maven doesn't prefer the latest by default it makes it less of a target So those are the reasons why it's happening over there. I am worried and we're doing some work and some other things I think that they will be the next generation type of attack where they figure out how to get nefarious code Into an actual popular project, which is a much harder problem to solve and isn't just a Package repository problem to solve actually so that's that's the reason why that's that's different All right, thank you everybody enjoy the rest of the conference