 Great, thanks. She said we're going to cover supply chain deep dive and one of the things I'm going to highlight is a dependency hijacking or dependency confusion vulnerability that discovered in Microsoft Teams. Just to talk about the software supply chain and the current gaps. Good. My name is Matt Austin. I'm the director of security research for contrast security. I've been doing security related stuff for about 11 years now and then also been at a few different development shops. So I've been doing development engineering and security for about 11 years. I'm also pretty active in bug bounty and things like that. And then Adam Shaw is going to be presenting with me. Hi, Adam Shaw here. I'm the director of enterprise security at contrast. I'm the founder of kernel con, a Midwest cyber security conference and my local con group. I enjoy homelabbing automation and I have about a decades experience, both making and breaking applications. And there's our Twitter handles. If you'd like to follow our follow our work or get in touch with us afterwards. So like I said, we do work at contrast security and a bit of our time is focused on our quite a bit of our research time is focused on software composition analysis and OSS product are, you know, part of what we do for the OSS product. So this has been a really interesting field of research for us and a really prevalent issue around the software supply chain and OSS. So today we're going to cover everything about supply chain attacks and what they mean to you. First, let's just break down what exactly supply chain is, right? It's a system of people, activities, information and resources involved in supplying a product or service to a consumer. So when you think about like examples in your everyday life, there's a couple that I mean, pretty much everything you can break down into a supply chain. Some of the more common ones that we broke down here, like when you get gas to drive your car, you know, you, you don't really think about it on a day to day basis, but there's oil pipelines. They have to drill for the oil and dig out oil, and then they have to transport it overseas to a refinery and the refinery, you know, refines a decreed oil and store it in these massive tanks that then they distribute to trucks and the trucks drive and distribute it to the gas stations. And then at the gas station, then you put it in your car finally, you know, and any, any disruption along that link can cause, you know, major issues. When you think about the food chain, whether it be, you know, animal product or wheat or grain, it, you know, it has to start somewhere, right? And in the example of grain, it starts from seeds, gets farmed and it gets put into grain silos and then it gets taken to a refinery and processed and then it gets taken to your grocery store where you have to go pick it up. And then it finally arrives on your dinner table. This is like a huge link, you know, from A to B to C. And like I said, anything along that line can cause significant issues. One major example, devastating food shortages, at least in my case, I'm a big fan, bacon prices have skyrocketed to record levels. And they might not be going down anytime soon. So this is, this is an example of something along the supply chain has caused bacon prices to, to go through the roof. And now it's harder to get bacon, especially affordable bacon. So, but we're not just here to talk about like supply chains in general, we're really here to focus on technology supply chains and how they, how they relate to you as industry technology experts, right? And there's, there's three major types of attacks. And we're going to cover each one of them in detail. First, let's talk about hardware supply chain attacks. This would be where modified hardware that would infect the normal use of the product. If you think about hardware, it's often manufactured overseas. And therefore, the control of the supply chain is extremely difficult. You know, you have hardware in your computer, your smart light bulb, your router, your dishwasher, etc. It goes on and on. And anything along that way can really affect the entire process. One of the more famous examples, despite the fact that this story has been mostly refuted at this point. In 2019, Bloomberg published a big article where they claimed that servers purchased from a Chinese company called Super Micro contained an embedded chip on the motherboard that allowed for remote execution and back doors into those servers. Again, this has been mostly proven false at this point. But the the story, it tells a good supply chain story, right, where a chip embedded on a motherboard in a manufacturing company was acting erratically, and able to allow people to remote into these servers. Another good example, researcher Mike Grover of the OMG cable fame, he actually created a Bitcoin stealing ledger in plant. So purchased a ledger online, cracked it open and replaced the SMT or STM 32 microcontroller with a malicious implant, packaged it back up, and theoretically could go back online and sell that. And they would give him access to the the person's cryptocurrency and allow him to essentially siphon off or steal that cryptocurrency at will. That's another good example of a hardware supply chain attack, right? If he he man in the middle of the attack, essentially bought it, modified it, resold it. Software is another software supply chain attacks are also really prevalent. That's compromising the software that's used to make up an application. This is often done through open source dependencies, which bring malicious code into your target environment. So your operating system, your local services, packages, your deployment dependencies, your desktop applications, all of this is software on, you know, at the base level. Some more famous attacks recently. There was a cryptocurrency launchpad that was hit with a $3 million supply chain attack. Basically, this attacker stole $3 million in Ethereum with a single GitHub commit. That commit changed the front end code to always use his wallet address during auction exchanges. So if somebody went and they went and attempted to win an auction, instead of sending the instead of sending the person that they were purchasing something from, it would send it to his address every time. And so he was able to steal $3 million in Ethereum doing that with a single commit. Another, I mean, this is the most famous one probably to date. Everyone on this calls probably heard about it. But so there wins an IP IT performance monitoring tool. They actually pushed out an update to their own software platform that went out to all of their customers that had an obfuscated backdoor. So what had happened was somebody made a malicious update to solar winds on their own environment. And then when they pushed that out, a malicious update then went to all of their clients, which is, you know, it's something that's near and dear to my heart as working for a vendor, right? That's something that concerns us. So this is one of the more famous examples. And another popular one recently was Kasiya. It's a remote management tool. It actually hit hundreds of companies worldwide with the ransomware attack once Kasiya was infected. And they used then the Kasiya's client lists basically to springboard and inject ransomware, hundreds of companies across the world. And another major type of attack is side channel. So that's compromising some piece in a supply chain that ultimately gives attackers to the main target. When you think about that, you can think about the people who built your building, the people who bring you mail, your employees. There's lots of different ways to weasel into an environment. One of the examples that I always like to talk about is this target point of sale hack. The attackers were, they broke into the HVAC vendor for target. And the HVAC vendor was just in to fix some refrigeration lines for target for their, you know, those display cases. They were just fixing some display cases, refrigeration lines. And while they were there, they plugged into their network and the attackers use that to pivot into target's internal network and target posted their POS, say their POS system on the internal network. So once the attackers were in the internal network, they could then attack and exploit the point of sale system and X fill credit card information, customer information. This is a huge deal. It was right after Black Friday. And so they were able to pull a ton of information. And it all came because of an HVAC vendor came in and plugged into the network. Another famous side channel supply chain attacks, the Stux Network. The Stux Network that brought down Iranians nuclear enrichment facilities was reportedly brought in through infected USB drives. So the Stux Network was a worm created that spread out through the internet. And it was just dormant until it reached a nuclear enrichment facility. And even though Iran had air gapped nuclear facilities, at some point, somebody had plugged a USB drive into a computer on the public internet. It was infected with Stuxnet. And then that person brought the USB drive into the air gap facility and plugged it in and then affected the actual facility, which spun down centrifuges and broke all kinds of stuff. And it was just such a devious like way to get the the worm inside of the facility was just to spread it everywhere and just hope that at some point somebody brought it into the facility. And this is another one that I thought was really interesting. Matt actually told me about this one. The Chinese hacking group corrupted the compiler inside of Microsoft Visual Studio. So it was essentially a supply chain shell game because they were hacking the tools used to build code. And then they could theoretically add malicious code without any of the developer's knowledge on, you know, in their own local environment. So as they talk about this, one thing is clear is these are just some examples. And the scope is the scope is too large for us to cover everything in 15 minutes. So Matt and I are going to focus specifically on software supply chain attacks. That's on your unmute. Matt, you're on mute. Yeah, we're going to talk about the software supply chain and how it can affect you and be broken down in a few different ways. Go ahead. So there's the the OS package management when you install dependencies. So all of the depend everything you install that you need on your server, whether it's, you know, web server software or mail software or like the example to the right here, FFM peg, you're installing that one software that maybe you trust or have some confidence in. But under the hood, that thing has a huge number of dependencies. This chart on the right is for FFM peg and all of its dependencies. And this is actually a super simplified version of the dependencies for FFM peg. I mean, there's a bunch of circular dependencies and then dependencies on separate versions of the same package and things like that. So the the sprawl for an individual project or an individual software package that you need is that the scope of it is always shockingly big and how much it spreads out. There's also, you know, disruptions that that can happen just because of bad commits into a software project. There's an interesting example where the researchers at the University of Minnesota were actually introducing intentional vulnerabilities just to test the response directly into things like the Linux kernel and were eventually banned from contributions. But it was just another interesting space you need to look at when you look at all of the when you install a package and then you look at all of its dependencies and libraries. You then also have to think that each one of those libraries also has a handful of committers and developers as part of that project. So the number of developers grows exponentially with the number of libraries. So that that tree that looks expansive and covers, you know, 100 libraries also covers x number of developers per library in many cases. So that's something we have to think about. And so the another place to introduce is directly into your code base when it when you're the developer of a library or when you're the developer of that software, what are your dependencies? So this is an example from Node, which is notoriously like large in the dependency space, but the stereo library is intended to be an HTML parser, which is relatively complex. But this again is a simplified version of the dependency tree. The full dependency tree has exactly 300 dependencies. So 300 different dependencies, 300 different projects with with a unique set of developers and each one committing back to the project. So yeah, I think it's really just shows the sprawl of how complex and how how much risk you're actually taking when you when you use a package or an individual dependency, you might review Cheerio and think it's fine or look at it, but you really have to consider the vast scope of dependencies that come in. So one of the specific type of attacks that we want to talk about is is typo squatting. And so this this package is actually an example that Adam made as part of CTF or capture the flag. It's like a hacker game, I suppose, where you try to find vulnerabilities and get points for them and stuff. But it usually impersonates like a realistic attack. And so one of the in the capture the flag example, this this actually the package that's included, instead of using the English spelling of color, it's using the the alternative UK spelling of color, and then the pickers missing an E. So it's a typo that could happen a common typo, or a an alternative spelling to a word. And instead of being the correct color picker, it's now the malicious persons code that comes in. So despite the fact that this was developed for a CTF and also has really, you know, clear instructions about this being just for a CTF and not a real package, it still manages to get a handful of downloads, weekly downloads, like, so it's a couple years old, completely unmaintained, just for the CTF actually does nothing, but still has active still has some number of active downloads. So realistically, you know, squatting something more popular project or getting a typo of a really common name that have a really drastic impact. So the second thing, the second type of supply chain or software supply chain of that can impact dependencies is is called dependency confusion. And so there's been some research recently around dependency confusion by Alex Bryson. And he actually talked about dependency confusion versus typo squatting. So typo squatting is just mistyping the name or having similarly named things. Dependency confusion is kind of different. So inside your package managers, you can have, you know, a lot of companies have public and private dependencies, you know, things that you developed in-house and things that you're that are private and not part of that you don't have public on the repository like Maven or NPM or anything else. And so dependency confusion works by basically putting up those private, if you know that a company has a private dependency, then you can actually take over, you can register that dependency as a public dependency. And then depending on how the either the proxy or the dependencies are pulled in, you can confuse the package manager into pulling in the public dependency instead of the internal private one. And we'll talk more about that in a second. But if you think about like all of the software that we talked about the software on the server, and we talked about what people develop, but all of that development goes somewhere. And there's a lot of it goes to web apps, but some of it definitely goes to the desktop. And so if you think about all of the software you installing your desktop and all of it has dependencies to just like these other software things. And, you know, a lot of these use public dependencies, no one wants to write all of the, all of the code themselves are rebuilt stuff that's already built. It's a really important part of the open source lifecycle is to reuse code and share and distribute and stuff. But it definitely opens up this idea of these, like the possibility for these attacks. At the end of the day, it's all software it's just, it's all software under the hood, all these desktop apps have, you know, this large sprawl of dependencies. So we'll talk about this dependency hijacking exploit in Microsoft Teams. Just a quick overview, Microsoft Teams is actually an Electron app. And Electron has a web renderer built on Chromium, and it's built with Node.js under the hood. So you're basically just writing a web app, but you have access to Node, and Node gives you system access, you can write files, read files, things like that, that normally aren't available on the web. And then it packages it up into a single executable that runs. And so, you know, there's definitely some benefits to building an Electron app. You only have to build it once. And it's the same code base can be shared between web and desktop. And then it's generally pretty easy to be multi-platform when using it. The important thing is under the hood it uses Node.js. And Node.js uses package render, like NPM or Yarn, to pull in its remote dependencies. So if you, I was looking at the Microsoft Teams app and pulled apart its package.json file. So this is actually the build materials or the package file. And it tells it what to consume. All of the dependencies that should be installed. And there's a few things I want to point out here that . So for one, these are all listed as optional dependencies. And I think that's kind of important because when you do an NPM install and it fails for whatever reason, like if they really needed these files, they wouldn't be optional dependencies. And some of them are optional because they're Windows only. And so the Mac build won't have them. And so if they're an optional dependency, if the dependency installation fails, the build won't fail. And so if there's a reason that it can't find the local dependency and the build doesn't fail and it continues on. So if the package doesn't exist on NPM and it doesn't exist locally, then it's fine. You still have a successful build. It won't disrupt the CI process. And then the other thing I highlighted and read these four specific modules. So you can see online 40 that one is actually namespace and correctly namespace underneath the Microsoft namespace. These other four aren't available in a public repo, but also aren't underneath the Microsoft namespace. So their potential target for this dependency confusion. And so to test out this theory, I created my own module called what would that was based on the name of one of the other modules, which is this MSFTWRM or WAM. And I said it to be the same version inside the package.json. So that means that anyone else doing a NPM install, like if a build server or a local developer does an NPM install and they're not connected to the proxy that's supposed to pull in the local dependency, it'll pull in, it'll try to reach out and grab it from the public repo. And so basically we're just confusing the code. This dependency confusion. We're confusing the package manager into installing our malicious third-party code instead of the private dependency. And so obviously don't want to cause a disruption and I'm not modifying anything once I get to the server. So just for this test NPM has a concept of a post install script and that just lets you write a little bit of code once it's done. Like once the install is done, does it need to clean up or do any build process. And I used that to just phone Tom to hit a URL so that I knew that a package was installed somewhere. And again this package never existed before so there's not other projects that might use this. It's hopefully only going to happen or come from one of these Microsoft products. For example, Microsoft Teams. So I threw this thing online and then I wait and see if anything happens. And then within a couple of hours, or within the hour actually, I started seeing a few installs, just these triggers that would phone Tom. And I went through and looked up the IP address and they're all coming from Microsoft corporate ASNs. So they're all coming from Microsoft IP addresses. And based on comparison of ASNs, it doesn't look like it's coming from just other random Azure servers. It looks like specific corporate ASNs. So we don't necessarily know if it's directly coming from Microsoft Teams or if it's part of a CI or just some other process. You know, for the proof of concept, I just wanted it to phone home. I didn't want to like pull in extra information about the server or you know, I just wanted to prove that that it was being required. But I think we can make some assumptions and talk about like a worst case scenario. And a theoretical worst case scenario would be that this is the actual Microsoft Teams build server. And there's obviously some confusion where it would really pull in the internal repository. But for some reason, the normal, the internal repository didn't have one of these files anymore. Maybe one of these was deprecated or this build server for some reason didn't have that local dependency. And now it's reaching out to me to pull that dependency or reaching out to NPM to pull my dependency in. And so my dependency again just phoned home, but it's now running with this post install script on the Microsoft on this Teams build server or whatever the build server for it. And instead of phoning home, it could maybe manipulate. So this is post check in and right before the build is being built and packaged into software, it could do whatever it wants, right? It could start injecting new code into this build pipeline or into that already built executable. It could start before it actually does the executable build. It could start rewriting code. And you're on the build server, so it's likely signed after this step. So your code signatures, things like that, you're at the step where it's putting in potentially injecting this malicious code. And then that malicious code could then be distributed to through normal auto updates from Microsoft Teams out to every Teams desktop. And so then you could start back over. One of these developers you have control over all these computers now. And one of them is a developer of some other software packets that then you can distribute more. Awesome. That was really cool to see live when you did all that work now. Now we're going to talk a little bit about mitigations, prevention and controls. How can we as industry leaders prevent this type of attack? So let's start a little bit about protecting your infrastructure. First, we recommend scanning your infrastructure. Again, some of these might be best practices, but once you have enough of this, I think that there provides a good level of detection and prevention. So scan your own network and have a well-defined remediation process. So check for vulnerabilities. Keep your server software up-to-date. Is there something that you can do if you have a portable package while you await a patch or can you scramble and get that patch up-to-date? The well-defined remediation process is important because I've been at places where I can tell you if you have something vulnerable and then it waits 130 days before it ever gets updated it's a really long time. So if you have a good way where you have a good standard in place that you can push remediation through, that's all the better. Also on your infrastructure, use well-known package managers instead of installing directly. Package managers like are going to provide extra protection like resource integrity, verifying signatures. They also inform and provide proper update mechanisms. So this is something that's going to allow for you to be better protected against potentially a malicious package or something that's not what you thought it was when you downloaded it. I'm sure we've all heard horror stories about downloading packages from like websites instead of using a proper package manager. There's also ways to protect your SDLC. Protect your deployment accounts, enable two-factor authentication, rotate your service keys, create a role-based access control on who can push artifacts. This is important because like that SolarWinds example I mentioned earlier, I mean if there are controls in place around their deployment, potentially could they have prevented deploying malicious code before distributing it to their customers and clients. You also want to continuously monitor dependencies. There's open source projects like Dependibot and MPMaudit. There's a bunch of them out there. I do want to preface this with if you are using a tool like that, make sure you understand it. Here's an example. Let's go back to Cheerio, the node example that Matt showed you with 300 different dependencies as a developer. As a developer, when I install Cheerio, I'm installing 300 different dependencies essentially on my local machine. However, Cheerio in production only uses about 40 of those packages. Make sure you understand what you're looking at when you are using a product like MPMaudit or Dependibot. If it's giving you a ton of warnings, make sure that you understand which ones to address first. You want to address, okay, when I get this off my machine, when I build this package for production, those are my most important dependencies and those are my most important ones. I want to say use a tool like that but also be mindful of what it actually means. There's also products and services that you could use. Contrast provides some of this. There's other tools and vendors. But the basics are you want to passively detect and remediate. You want to check all of your products. And then you want to detect attacks and prevent exploits. You want a tool that does this and implement it in your STLC. There's also other things that you can do to help mitigate including use sub-resource integrity checks on your dependencies. If we harken back to infrastructure stuff, we want to make sure that people use package managers in lieu of downloading directly. Popular package managers from software, MAVEN, MPM, GEMS, PIPI. There's all kinds of great package managers out there who do attempt to do the best that they can with making sure that this is the best way to do it. They try to give you updates and warnings if there's actual vulnerabilities, but you're deploying a project that has a vulnerable library. They try to give you warnings as well. Actually, code hygiene is very important. And there's like that could be a talk in itself. But the main points are lockdown your repos and place access control on who can commit and who has admin rights in a repo. That kind of thing. And also have pull request hygiene. Some suggested standards you could have require two approvals per pull request. This has a lot to do with you as a security practitioner, as an application security practitioner. And what you have to do is create well defined standards that your developers can adhere to because after all, you're trying to work with them to protect your company. They don't want to produce insecure code. You want them to have all the tools in place to actually produce solid, good quality code. And then speaking of code, let's talk about how to register namespaces on public dependency managers and then use those namespaces for internal repositories. Remember back to Matt's example where he showed the four packages in a red that were internal repositories didn't exist externally, but they also weren't using the at Microsoft scope. And had they been using that on MPM and actually had people install it had Microsoft teams install it somewhere. So that's important. Register namespaces and then actually use them. And then also treat your dependencies as static. Use a lock file. Now there are a lot of different lock files when it comes to package managers. There's gem locks, there's yarn.lock, there's package hyphen lock.json for node. These are the kind of things that it's important to you once you've evaluated a dependency. You think that you're okay with this dependency and its dependencies. That you are okay bringing this into your environment and the risk is acceptable and you're okay with that. So if you have a certain degree lock it at that version. Here's why. There's all kinds of nightmares that could have happened. If you say just update whenever there's the latest update out there. You are not actually controlling controlling when that updates or how it updates or what the updates are. Each update you should audit and figure out, hey, is this an update? Does it provide better code? Is there anything malicious in here? Is there often stuff that you need to figure out what's happening? Because those are all things that you want to approach first. And then one of the things that we also recommend is build a dependency checklist that's verified before bringing this into your application. So if you're going to be environment specific, your company is not my company, my company is not that person's company. But here are some suggested checklists. Is the age of the dependency consistent with your understanding of the package? For example, if you are installing a specific jQuery dependency, and when you search it, there's two packages. There's one that's been around for four years with four million installs. And there's one that's been around for two months with 30 installs. Make sure to lean towards the one that has more installs, more of a history and is consistent with what you expect. And then validate that history is legitimate. Again, using Matt's example when Matt created that package on MPM, he started it at the version that Microsoft said it was at. So his first version was 0.4.7, which is not really where you start when you're versioning a dependency. So there's something maybe a little suspicious there. Hey, where did the first four minor versions go? They started at 0.0 and started 0.1, started at 0.4.7. That's suspicious. And then look at the issues open. If it's on GitHub, there's, or BitBugget, there's a way to check what issues are open. Are there a lot of issues open? Are they getting closed at a regular cadence? Are there security issues open? Are there a lot of security issues open? I've seen dependencies that they fixed code stuff, but they weren't fixing security stuff, and it was enough to dissuade me from using that dependency. You have some open, you have three or four open security issues that haven't been resolved, and they're resolving code issues ahead of those. So that was something that could dissuade you as well. Is the package actively maintained? This is, again, you can come up with your own standard here, but you could say, has it been worked on in the last X number of months? The key here is you don't want a package that's stale or archived or long-term support because those are susceptible to takeovers. People can take those over or they can make commits that just fly through without getting any real looks because the original developer has long since forgotten about these projects. A lot of open source developers often churn out hundreds of these packages, and so when they're doing all that work, their early stuff, it's pretty easy for them to forget about it. And this is an interesting one right here. It's a little difficult, but does the package source code match the code on the package manager? Let me describe a scenario here. So there was a package called event stream. It was a JavaScript package that had 8 million weekly downloads and it was a very popular node library for managing streams. And it ended up with some malicious code and attacker asked to the owner if he could take it over. Again, this is something that was in long-term support. The owner had developed 600 different open source repositories and so he said absolutely please take this off my hands. And then the attacker took it. He added some malicious code, uploaded that to MPM but kept the GitHub repository clean. So if you were on MPM and you clicked through to the source code and you reviewed the source code you wouldn't have found anything because what was on MPM was different than what was on GitHub. And you can just do that like as an attacker you don't have to maintain good change management or good change control. So you can do that. And so this is difficult because sometimes things are transpiled sometimes things have happened but if you can verify that the source matches the code in the package manager you should take that extra step as well. And then lastly ask yourself if you really need this dependency do you need to install it? Can you code it yourself? One of the more famous examples here is left pad I'd be curious to how many people remember left pad but left pad was nine lines of code that was a dependency that added white space to the left of a string and it was nine lines of code that Babel depended on. Now Babel transpiles all modern JavaScript, React, Angular, Ember, etc. And they one day the developer who created left pad got angry with MPM and removed all of his packages left pad this nine lines of something brought down Babel and all modern JavaScript application shops at that time had major disruptions they could no longer transpile any of their code from these frameworks into vanilla JS and it caused a huge ripple effect and the reality is Babel probably didn't need to have that as a dependency they could have written those nine lines in the actual transpiler and not had to worry about that and so whenever I install a dependency I always think about left pad is this something I can code myself in a reasonable amount of time does it doesn't make sense to make this a dependency or can I copy this in can I actually use this here or create it myself so these are the kind of questions that you should ask and much of this can be automated tools definitely help including contrast many of these checks into our product there's a whole slew of tools that would help and really what the goal here is to help developers understand what they're looking for you want to build security champions and that's what's important developers and you you're a team and you're working together so you're trying to build security champions and there's a lot of great movement in this space and it's something that both Matt and I are excited about there's a lot of exciting new projects there's this Salsa project which is sponsored by lots of great groups including the Linux foundation it's a security framework basically giving everybody a common language for increasing levels of software and supply chain security and integrity I highly encourage you to check it out it's an interesting read it's slsa.dev and then there's also really what was this yesterday October 1st I think it was October 1st the secure open source or SOS rewards and this is ran by the Linux software foundation and sponsored by the Google open source security team but basically it's a project that's been a financially reward developers for enhancing enhancing the security of critical open source projects so this is another cool thing in the space where you're we're essentially giving out bounties to people to fix software that we all depend on which I think is just a great you know is just a great example for the industry this is something that I think it has great potential and then and it's not just these projects and and our tool our tools and other tools this morning I saw there was a dependency confusion tool being released at black hat Europe the arsenal section so there's lots of positive movement happening in the supply chain space but thank you for your time please feel free to follow up for questions Matt and I our Twitter handles are there feel free to reach out we're also I think we have a few minutes left here we can answer some questions we can answer some questions if they came in through the Q&A I did see one that came in from the Q&A and it says can you mention such tools as an example for the checklist and like Adam said there's the OSS part of our own product that does scoring but a lot of the other SEA bidders do as well so that generally the libraries and dependencies from some of these tools like ours or sneak or some of these other ones will give the dependency a score and that score is based on a lot of these a lot of the things in the checklist like the age of the dependency the number of open issues on GitHub and things like that so there's a few things you can automate some of it obviously you can't but I think it's important like there's some things that we do automatically that we come up with like a library or like a third party dependency score and that scoring is done by you know things that look suspicious like we'll lower the score if it started the version that's super high or inconsistent I mean there are valid reasons maybe it was a closer project that's now open source so there's a valid reason for it to have a version starting at 4 dot whatever but you know it lowers our confidence in it from just a automated review and it just gives you something more to look at I mean it's not saying don't use it but it's just saying maybe give this a second look I think about how it's getting how it's getting included so yeah we do that we do check out the get repo check out the activity on the get repo all of those things combined into our score and other other scoring there's different products that do different scoring but generally there are many of the items on those checklists are automated through these tooling awesome and then I saw another question I answered it in line but Matt and I will tweet out the slide deck and I will I will take care of that if you check us out we'll have the slides posted soon was mentioning a bacon along with the framed picture of a young pig behind Matt intentional not all that you take that was a coincidence I've had to pick up for a little while but it you know it fit the theme I suppose oh sadly now Matt lives in LA so he rotates like barnyard animals and pictures behind him where should someone start if they're maintaining a really old multi million line of code application lots of old libraries tools involved that that's a great one I think that's really hard and I think you know a lot of the issues that come into play are when it's really hard to update dependencies so if you can actively maintain and keep up to date with software that's really important because once you're a certain number of versions behind even if there's a known vulnerability it's really really hard to you know if the vulnerability wasn't fixed in your dependency because you're so many versions behind it's only maintained in like the current major releases then it's really really hard to fix those I think that was the case with Equifax that's right there's like virtual patching and other things that you can do there's software to help you do just that like just patch in the security events or the security issues that happen but yeah again you know there's some things to look at there's like how out of date is it what's most important for me to update big core things like if my framework has an update I want to update that first and maybe not some of the you know lower level dependencies that will knock them off so I think that's key and important and then also realize like Adam talked about before that when you do an npm audit or look at all of your dependencies and the stretch of them you can actually evaluate your dependency and say like this is part of my build process it's important to me but it's not directly going into production so you know so many I see all the time is like these high severity issues that are denial of service vulnerabilities because of a regex but it's just in a build tool that never touches like that never has user input into your app so I think I think you can spend some time looking at all your dependencies and try to prioritize which ones are most critical to try to get up to date and then work backwards from there yeah that's that's the same thing I that's one of the same things I'd say here is like first figure out what like assets you have like what exactly are using what reaches production what's on development machines and then move forward from there and you know pick off what you can at first but I think knowing what you have is crucial in a really old multi-million line of code application so thank you the other question was what trap in creating Microsoft Teams so it so it's a dependency confusion vulnerability and it happened because they were they didn't use a scoped module so they didn't use scoped modules which allow you to have a namespace that only your only your you can approve who can commit inside of that namespace or who can publish modules to that namespace so they weren't using that so that kind of opened it up for everyone and you can take over a namespace and then use that namespace for your private installs which is a decent idea even if you don't push stuff to the public repo if you push it under that same namespace still limits the opportunity for this confusion so that was the fact that they didn't register them on the public repo kind of blankly the fact that they weren't scoped was kind of the bigger issue and then that I think it also happened to be that because they were optional they probably weren't seeing errors for I think the reason this got installed was because maybe it wasn't used necessarily or wasn't pulled in correctly in a specific environment if it wasn't an optional dependency if they kind of coded around it being optional then I think they would have had install failures and it would have been clear that like it's reaching out but I think the most correct fix in their case would be to have the scoped dependency just to scope all of the dependencies you can still use the private repo if they want to or need to but there would be no way for me to take over this scope public repos right and and not every like build framework is going to necessarily check external before internal but that like happened in this case right so they had an internal dependency that was optional and but they checked NPM where Matt hosted a package and then was able to actually use that before before checking locally and there's a lot of different ways to do that Alex in his research he talked a little bit about it but like if a VPN is down or something now they can't reach to their internal network then it could look out and that would be a problem too so the other thing that does help in some of these cases is like the package lock because if the package lock should be checked into source control as well so the developer should decide the package that they want and then they should be pointing to the private repo the package lock would lock you into the URL to the private repo or the location of the private repo so in this case it looked like they probably maybe weren't using that or whatever service this was that made this call wasn't using it but the risk also happens to the desktop user right like if I bumped it up one version a local desktop user might be doing a clean npm install because they aren't using the package adjacent or trying to update and it's pulling in mine if they're offline so I think there's still a risk even with the package lock of attacking potentially the developers that are doing the install but if the developers are using those package locks and locking in that path to the URL then the deployment server or the CI environments things like that there's no ambiguity about where it's pulling the dependency from but that also helps good question thank you any last minute questions okay well thank you again everybody for your time Matt and I will we'll be tweeting the slide deck appreciate it and thank you for the Linux software foundation for hosting us it's been great awesome okay thank you so much Matt and Adam for your time today thank you to everyone for joining us as a reminder this recording will be up on the Linux foundation's YouTube page later today we hope you'll join us for future webinars thank you so much bye