 Yeah, so I appreciate everybody coming today. My name is Jonah Castro. I am a Solutions Architect for Sonotype. Largely goal for today is to kind of be to talk to you about software supply chains, what we're seeing as far as those software supply chains, different types of attacks that we're seeing, and then to do a workshop where we'll actually go through the Struts 2 RCE that was what took down Equifax a few years ago. So with that, I think that one of the places that I really wanna start is, go ahead and click that. Why are we building software, right? And I think when you talk about organizations building software, it really comes down to two reasons. I would say it's largely innovation and value, right? Innovation for the organization to build new things, and then providing that value for customers or the organization itself. And I think in that goal, it's always the goal of the business to create those that innovation and value as fast as possible, again for their customers or for that organization. And I think that's why largely we've seen over the last few years, there be a large change in the way that we've gone about developing, right? So this is just a little cartoon that I pulled from XKCD that really shows somebody sitting around going, hey, I thought about that five years ago and I never got around to building it and this other company built it and I want a piece of that idea. And that's largely why I think the companies are trying to innovate and create code substantially faster is so that they're able to get these products and that value to their customers faster than other companies are, right? So again, like I said, I think that's largely the reason why we've seen this shift in the way that we're going about developing as we move forward, right? We're seeing things like at first it was going from waterfall development to agile and then moving from agile to DevOps and now we're starting to see DevSecOps so that we can kind of kill those silos, automate substantially more and again get that value to your customers substantially faster, right? So one of the ways that I think that we've largely seen this shift in development over the last 10 years is with open source software, right? Before 10 years, you know, prior to the last decade we really saw developers creating a lot of stuff on their own, it was all first party code and really you just had a lot of people recreating the wheel over and over and over again and we've seen a shift towards more developers pulling in open source libraries, packages, et cetera into their code largely to give them that ability to again, provide that value to their customers substantially faster, not have to go about recreating that wheel, et cetera and to that point, we've actually seen again, some really large shifts, you know, just in some of the research that we've done, we're seeing that most modern applications nowadays are built with almost 90% open source software which means that really your developers are just writing about 10% first party code, everything else is something they're pulling from a public repository or inner source or something like that. You know, in 2020, we saw about a trillion and a half open source downloads from Java, NPM, PyPy, RubyGems, et cetera which is a substantial large amount of open source especially when you're talking about, you know the entire pool of developers is somewhere around 25 and a half million so you're looking at about 50 or 60,000 open source packages that are being downloaded by your developers in a year. Just last year, you know, again, we saw almost 380,000 Java components downloaded by a normal average company, right? So just one company is downloading almost 380,000 packages from a single, you know, a single Java repository. So you can just kind of see the scale of the numbers that we're starting to talk about when we're talking about this open source eco-sphere and securing your software supply chain. Now the great thing about this is that it has made it substantially faster for us to build software. You know, it's much easier to go online, find these packages and bring them in. The downside is that most modern architecture looks like this picture that you're seeing here. It's kind of a Jenga block of, Jenga block stack of applications and different projects that are being pulled in. And most people don't realize that this is how their applications are coming together, right? Typically companies, you know, are planning to hire one Java developer. They don't realize that, you know, most of the time a singular package could potentially rely on 30 to 40 different types of open source dependencies being pulled in with that. So, you know, when you're looking for that one Java developer to bring in to write your code, you're not actually bringing in just one Java developer. You're bringing in, you know, 30 or 40 who are also maintaining all of these transitive dependencies, again, that are being brought in. So, again, you can see that, you know, it can definitely make some precarious situations when you have that one project or that one package or component that you're bringing in that's holding up all the rest of your application. And that component just so happens to be being maintained by some guy in the middle of the country as a second job and he's the only one that's maintaining it, right? So, again, while there's great, you know, there's good sides to being able to develop faster, bringing in this open source is really awesome, that definitely brings about some downsides. You know, we just published a report called the 2020 software, sorry, state of software supply chain report and one of the key numbers that really stuck out to me is that we're seeing almost a 650% year over year increase in software supply chain attacks. And largely that's because a lot of these repositories aren't really doing the necessary checks and stuff like that to verify that people aren't putting in, you know, dependency confusion packages or malware into these packages, et cetera, right? So you can see just some numbers here, you know, one in 10 open source components that have been downloaded contain some known security vulnerability, right? And when we're talking about the numbers that we were just referencing earlier where you potentially have a developer that's downloading 50 or 60,000 open source packages in a year, you know, one in 10 that are coming down that have software security vulnerabilities, you can start seeing the amount of security debt that you're building within an organization. You know, another number that it kind of sticks out is this one in five organizations that have experienced a breach due to open source software in the last 12 months. That's one in five companies that have openly admitted to that. I'm sure that that number is probably substantially higher just because most companies don't like to share the fact that they've been breached by something, right? So you can see just how widespread of an issue this can possibly be and understand why this is starting to become a substantially more larger concern for organizations. So just some examples that we've seen, you know, over the last few months, right? We had this CodeCove incident where somebody was able to get a password and because they got a password to the inner CICD systems, they were able to inject some malicious code into CodeCove, which then allowed that particular open source package to be dispersed amongst a number of different companies, which caused a large amount of breaches, right? And again, this attack vector is just only getting substantially more prevalent. Another one that we just recently saw was the attack on the Confluence servers. Again, they had an open source package that had a vulnerability in it. And because of that, a lot of Confluence customers ended up getting attacked, one of which dropped 500,000 Fortinet VPNs so that they could see all those passwords, right? So again, you can see that, you know, not only is the numbers of open source that we're dealing with staggering, but the, again, the vulnerability security debt that you're bringing in is extremely staggering as well and could cause not only issues for your organization, but really a downstream effect for all the organizations that you happen to work with as well. Another great example is there was a PIPI package that we just actually located not too long ago that was being downloaded by developers and once it was downloaded onto their machine was actually going and mining cryptocurrency on their machines. Another package from PIPI that was just recently removed as well was actually getting downloaded by developers and then was going through their machine to steal credit card numbers and network passwords, right? So you can see it's a really easy path for somebody to get into your system via a developer that downloaded something that they didn't realize was malicious and then really just make their way through the rest of your organization if need be. So because of all of the changes that we're seeing, you know, we just recently saw Biden issue an executive order, mainly to do a couple of things. One, to make sure that anything's being sold to the government is it now has an S-bomb or a software bill of material so that they can actually see exactly what is included in that particular piece of application and understand all the components that make it up. They're also gonna be starting to create some NIST guidelines so that we can have a better understanding of what needs to be done within organizations in order to secure this software supply chain, right? In the end, it all really comes down to this, right? Evil hackers, they're getting smarter and faster and with software supply chain attacks it's substantially easier for me as an attacker to poison the well of a single spot like an NPM library or a PiPi library and hope that that gets downloaded by hundreds of organizations and then use that as an attack vector into those organizations than it is for me to really spend six months, eight months doing a bunch of OSINT on a particular organization to find out what weaknesses they may have and try to infiltrate them that way, right? So it's really a low risk, high reward avenue for attackers which I think is one of the reasons why we're seeing this get substantially more prevalent. So this is really just a picture of what we consider the full spectrum software supply chain, right? All the different entry points or areas that potential things could happen. Again, with software supply chain attacks we're seeing a lot of stuff come into this public OSS repo so again PiPi, NPM, Maven, et cetera. There's definitely people just uploading some malicious packages there as well as just natural vulnerabilities that are being created by developers as they're building out these open source projects because they're not security experts in and of themselves so as they're building out some of this functionality, there's potential for them to be basically creating security debt for your organization as you're building out these applications, right? Couple of other places that we're seeing things, obviously first party code as you guys are creating your own code, you could be introducing your own security debt. But then even when we start talking about the packaging process, right? There's a lot more attacks that are starting to take place where again attackers are trying to get into your CICD systems, et cetera so that they can make those changes or inject malicious code during your build processes without you knowing before you even go to ship that particular product. And then lastly, as you're operating these applications, right? And they're in production. There's definitely often times where there may not be security risks before you put that in production but then as that code ages and as other people find a particular zero days or vulnerabilities, et cetera then that becomes an issue, right? A great example of this was the Struts 2 RCE that came out with Equifax, right? There was a lot of companies that were using that Struts 2 version at the time. It wasn't an issue before and then all of a sudden a zero day got published about that particular package and within days there was already a large amount of companies that were getting breached to quite a deep extent, right? Again, Equifax being one of them. So with that said, I'm gonna do a quick workshop walkthrough of what that Struts 2 RCE actually looks like and kind of show you guys what actually happened during that particular attack. This is being recorded so there's absolutely no need for you guys to try to keep up or anything like that and again, so if you wanna follow along on your own afterwards you can. There are a couple of requirements that you need to set up in order to have this RCE work. One, you'd need Python 3. In this particular example, we're using Tomcat 7 which does require Java 1.8 as far as a JDK so you may need to kind of bump your Java version behind and then any kind of Git account. It doesn't necessarily need to be GitHub, GitHub, GitLab, Bitbucket, Jardab, DevOps, again, anywhere that you can really fork over a repository so that you can copy this on your own and then I'm using Docker installed on my own local machine to kind of build out some containers to show this and then lastly, I'm using Visual Studio Code but again, you're more than welcome to use whatever ID you'd like. As far as the setup goes and kind of what we'll be doing, the very first thing that you would need to do is go ahead and fork this particular GitHub project. I've already gone ahead and done that and then from there, you would clone that project to your workstation. Again, I've already taken care of that process and then we'll go ahead and build and run the Docker image while we're here and then I'll walk you through what that exploit looks like and show you some of the damage that can be done with an exploit like this. So with that said, here's the kind of the instructions and what we're gonna basically go through and do. Very first step is we're gonna go ahead and package this particular application. Then we'll go ahead and create a Docker image and then we'll start running that Docker image. From there, we're gonna create a virtual environment within Python, download some requirements that are needed for this particular exploit and then from there, we can go ahead and run the exploit. Right, so sorry, let's go back and with that said, let me go ahead and pull up my virtual, my Visual Studio code, right? So again, I've already gone ahead and forked that GitHub repository. I've already cloned it to my machine. You can see basically the steps are in the readme. There's a number of files here that have things like your Docker file which will already automatically build out that particular Docker image. The POM file is what's gonna be used for us building this particular Java application and then there's a requirements text here for when we go ahead and pull down those those Python requirements for the actual exploit. So with that said, let's go ahead and get started. The first thing that I'm gonna do is this this Maven clean package and again, we're just basically gonna go here and build through if I can actually type correctly, bear with me. We're gonna build out this Java package. So basically this is gonna go through, it's gonna go out and reach out to these repositories, pull down all the necessary dependencies, all the different files that are needed to build out this and then it's gonna go ahead and build out a war. We've got our build successful, so we're good to go. Very next thing that I'm gonna go ahead and do is build this Docker image. So before I do, just to kind of show you what I'm running right now. So when I run a Docker PS, you can see I'm only running one container at the moment. This happens to be what I use is called Portainer. It's just a middleware that allows me to access my containers in a more user friendly way. But from here, I can go ahead and do a Docker build. We'll go ahead and build this box and we'll call it hack me. And again, we're just using the Docker file that's in this particular project to go ahead and build that out. That's gonna go through and that's gonna pull in all of the, again, Docker files, all the necessary images, and then go ahead and run some of those commands to get everything up and running. Once we're done from there, I can actually do a Docker run. We're gonna run this in the background and we're gonna change this to port 999 instead of 8080, just because I already have something running on 8080. And then we can go ahead and hit enter. And apparently I misspelled something. Oh, I gotta put the actual box, right? Tell it what box we wanna run. So we're gonna run that hack me box. And now that that's running, I can do a Docker PS, make sure that that's running right. And I can see my hack me box right here as called condescending Hoover, right? So we'll go ahead and clear that out. So now that I have that actually built out, I can start setting up my particular virtual environment. And I do apologize, I need to kinda jump back here and just grab some commands from here because I was hoping to have multiple screens, but it didn't work out that way. So we'll just go ahead and previous, right? So the first thing that we're gonna do is grab a virtual environment. So we're gonna create this in this VNV folder. And again, that's just gonna basically put us into a virtual environment. This might take a second. Once we've done that, the very next thing is we're gonna run this VNV scripts activate. Know that this is shell specific, right? So on my particular machine, this is what I need to do to run my shell. If you guys are running fish or some other operating system with another shell, that command may change. But again, it's largely just gonna be shell specific. So we'll go ahead and, let's try that again. Go ahead and buzz, bear with me. Scripts activate, love tab completion. And now that we're in our virtual environment, we can actually start doing things within here like download requirements, dependencies, et cetera, for that particular Python executable. So in this case, again, we're just gonna do a pip install. We're gonna grab that requirements.text. Try that again, there we go, requirements.text. This is gonna go through and again, grab all of those different dependencies, bring them down to allow us to actually run this particular exploit. I am not, not yet. I'm not in that Docker container yet. However, that Docker container is running, right? So if I come over here and I do, let's just jump into Google Chrome, right? And I can do a local host. And we set that to 999. And I can come over here to orders dash three, right? So this is that container as it's running, right? This is again, just basically an Apache struts webpage. I can come over here and go back to my orders. If I wanted to, I can add or delete things from the repository or from the database, et cetera. So now that I'm in my virtual environment, I've got all my Python requirements stored. At this point, I can actually start running that command, right? So you'll see the command here. It's gonna be Python exploit pi. And then we're gonna tell it where we wanna run that. And then from there, I can inject any command that I want to, right? So let's go ahead and run that. And then it's gonna be three. And then again, whatever it is that I wanna run. So maybe I wanna just see what is in this particular file at first. I can run this LS. And apparently I did not run that command, right? Let's, interesting, let's try that. Oh, I misspelled orders. There we go. Gotta love those typing errors, right? So in this particular case, right? All I did was run an LS. Basically I just wanna see what I have access to on this particular container, right? And you can see that it's giving me all of the different files that I have. I've got my building.txt, my contributing, my licensing, et cetera, right? So maybe let's go a little bit deeper, right? Like maybe I wanna show me what kind of machine I'm running, right? And now I can start seeing all of the actual files directly on my root, you know? And so now let's just do something like a who am I? And you can see that I'm root on this box, right? So with this exploit, I can basically do whatever I want to this container, largely because it's running this older version of Struts 2. Now one thing that I do wanna point out here is that this particular exploit doesn't run inside of your call flow, right? So one of the things that you'll see a lot of tools out there do is they'll help you prioritize using that call flow analysis. And in some instances, like for this particular vulnerability, it may miss that or give you a false negative or even prioritize this a little bit lower because again, this is not something that's gonna be run in particular call flow, but as you can see it's still exploitable despite the fact that this particular function's not called within whatever application it is that you're building, right? So it does leave you fairly open. So with that, you know, again, if there's any questions we'll have some time at the end, we can kind of discuss through what all just happened here. But again, you can kind of see the devastation that can happen when you have some of these libraries or open source packages that you're bringing in and basically leaving holes into your organization. So with that said, you know, here at Sonotype we have a couple of different ways that we can help out with this. One of the ways that we can help out with this is using our OSS index. This is a completely free tool that's open to all of the developer community. There's a number of different places that this can be put into. So you know, within your IDE, you can actually run these scans again directly in that IDE. A good example of this would be down here. You'll see if I expand out this Sonotype scan results, it's gonna list out all of the different packages that we have here. Again, it does multi-languages. So you'll see I've got some Maven packages, I've got some Python packages, et cetera. I can click on any one of these and it'll give me a list of what those vulnerabilities look like. So I can actually start looking into, all right, what vulnerability this particular package has, et cetera. So if I jump over to security, again, you'll see a list of these CVEs. I can expand these out and kind of drill down a little bit more and get some additional information on what CVEs this particular package may contain. So that's definitely one way that we can handle this. And again, you can install these tools within your build process, so in CICD, et cetera. A good example of this, and I'll show you guys momentarily, is we have a tool called Ahab that will actually do a scan on your Docker images as you're building them and actually will break the build if that Docker image is bringing in any kind of OS level or component vulnerabilities. So what is the OSN index? It's largely, again, just a scan that's gonna give you the ability to see a list of all of the vulnerabilities that are found within any of the packages that you're bringing in within an application or a container, et cetera. You'll see it's gonna give you things like your vulnerability score. It'll give you your CVE rating. It'll give you some information on that CVE. Again, like we mentioned, these tools are completely free and available to all of your developer community. These tools are built by Sonotypers, as well as additional Sonotype community members. And as far as languages go, you can see a list of the languages that are supported with these tools. Each one of them kind of runs a different language. The one that I'm gonna show you in a moment is Ahab, and Ahab really works on, like we talked about, that base OS. We're looking at, again, containers and any kind of flaws that may be being brought in by the packages that happen to be built into that particular container. If you're interested in any of these tools, you can grab them and play around with them on your own. There's a link down here, this ossindex.sonotype.org, slash integrations. And again, have at it and play around with them. If you have any questions, let us know. We're more than happy to kind of discuss them with you. And so like I had mentioned earlier, one of the tools that could have prevented this particular attack is called Ahab. And again, you can build these directly into your build rules, right? So as you're, again, setting up a Dockerfile or something like that, just put this scanner directly in there so that while it's going through that, excuse me, it can go ahead and do that scan and determine if there is anything wrong. So for instance, let's go back over here to this particular Dockerfile. I can go ahead and uncomment out these two particular lines. And then from here, let's go ahead and do a Docker, get back in here. Let's go do a Docker PS and then just wanna see what all we have available. And then from here, I will do a Docker kill and we'll kill that package. It looks like it's condescending. I love the names. The names are my favorite, right? So we'll go ahead and do I, I do love the names until you have the spelled amount. There we go. All right, so we went ahead and killed that box. If I go ahead and do a Docker PS, you'll see that box is no longer running. So let's go ahead and do another Docker build now that we have those lines uncommented. And you'll see it's gonna go through and build and why. Oh, I apologize. I forgot to save my file. Let's try it one more time. I apologize. So let's go ahead and do a Docker PS. It's not running. Let's do that Docker build again. T, it's gonna be hack me. There we go. So now you can see it's running that AHAB scanner and essentially because it found vulnerabilities, it went ahead and completely stopped and failed out that build, right? So you can see here, there's a lot of vulnerabilities, not just the particular struts one that I was showing you. But again, this is gonna give you a list of all of the different things that we found. It'll give you a description. It'll give you a CVSS score. It's also gonna kind of point you to a link that'll give you some additional information here. So you can see that, yeah, again, you can kind of build these tools into your workflow to prevent any of these particular large security issues from being brought in and kind of start failing these builds earlier on in your development life cycle. Sweet. So on top of our free tools, our OSS index tools, we also do offer some paid products. Again, full spectrum software, supply chain management and really the difference being is that in addition to us giving you the vulnerability information with our free tools, our paid tools will give you remediation information for those vulnerabilities. So now we'll start telling you things like, hey, if you jump to, you know, version XYZ, it's not only gonna fix this particular direct dependency that you're bringing in, but it's also gonna fix all of the transitive dependencies that may go along with it. We also give you things like licensing information. So you can start building policies around whether or not you want your developers bringing in things like AGPL licensing or GPL licensing, et cetera. So you can start banning or setting up different rules for, again, what licenses are allowed as well as different architectural information. So we actually start looking at different teams as they're building out these open source projects to determine how often are they fixing things? How often are they upgrading things? Is this particular package really old and maybe something that you don't wanna bring in because it's five, 10 years old and you want something newer? Again, we can start doing things within your software supply chain to block those, warn you of those items as well. So and like I was mentioning, again, with our Nexus Lifecycle, which is the paid version of that OSS index, we'll start giving you things like popularity, how popular is that package and how many other people are using that version of a package. Are there things like breaking changes? If you need to upgrade to a newer dependency, is that going to go ahead and break that change or are you gonna need to refactor the code? And then again, also giving you things like remediation advice where we'll tell you, hey, the next version you should really jump to is this version because it fixes all of those vulnerabilities, fixes all those licensing issues, et cetera, and will help you have a better product going forward, right? The other thing that I'll point out is obviously we understand fully that developers and ops people, they like to live in the tools that they use quite a bit, right? So we integrate with all of those tools. Like you just saw with Visual Studio, again, we have a number of different IDE plugins that we integrate with, both with our free tools and our paid tools. As far as build tools, we integrate directly with Jenkins and GitLab, Azure DevOps, et cetera. We also have a number of different, excuse me, source code management or SCM integrations where we can actually do things like run scans and then push pull requests directly into your GitHub repositories or your GitLab repositories so that your developers can see that remediation advice right where they live on a regular basis. So with that said, that's all I've got. I'm happy to open it up to questions. I know that was probably a little quicker than than I thought it was gonna be, but again, any questions, thoughts, things that you guys would like to know a little bit more about? Hopefully this resonated with everybody. Yeah, what's up? OSS index, the information that you want to be able to work on this, or if that's something that could be a deploy or it needs to be a SaaS model? Yeah, yeah, no, great question. So the OSS index tools, oh yeah. So the question was whether or not that OSS index, where it was getting its information and whether or not that was a SaaS product or something that's being done on-premise. So with the OSS index tools, our free developer tools, those are all free for you to kind of download and operate within your own environment. The actual vulnerability information that we're pulling is coming from our online database. And really all we're doing is passing along a hash. So we're going through and identifying each of these open source projects. We're giving them a fingerprint, and then we're literally just sending that fingerprint up to that database to match off and then send all that vulnerability data down. So we're not actually sending out your particular source code or anything like that up to this database to get those results back. Yeah, no problem, great question. Everybody else good to go? All right, sweet. Well hopefully, like I said, hopefully this resonated with you, hopefully you guys got some good information out of this. If there's anything else that I can help out with or any questions that I can answer as you guys are moving forward, let me know. Feel free to reach out to us and we're happy to help. Thanks much everybody, I appreciate it.