 Hello, welcome to my talk. I'm Cheryl Biswas. I also go by encrypted and I really appreciate you taking the time to discover software supply chain attacks and the Chinese APTs who have been increasingly behind them, signed, sealed and delivered. So who am I? I work as a thread intel analyst with a major bank here in Toronto, Canada. I am a founding member of the Diana Initiative. We just had our this conference that was online. Much thanks and appreciation to everybody who was able to be part of this. We love supporting inclusion, diversity and just drawing a bigger circle about who we bring into this amazing field. I'm a member of the C3X College Student Cyber Simulation here in Toronto and it's an annual event because it's great to be able to give college students the opportunity to learn in a real life environment what it feels like to respond to an incident, how to communicate with management and just to be able to pay it forward because we all started somewhere. Okay, that's enough exciting things about me. I am a cat mom, but you don't get to see my cats. They are in the other room. All right. In this talk, I want to explain what software supply chain attacks are and the growing threats that they pose. So we're gonna take a look at code dependency, how that's a factor and how it's compounded by mistakes and misconfigurations. We're going to take a look at some of the attacks and the state-sponsored actors that are involved. And what this will do is highlight the prevalence of Chinese threat actors who continue to up their game and the number of victims. And then we'll wrap up with what we could do better and how we need to shift. Okay. I thought that this would be a great starting point. What a year it's been. Even before we got out of the game, the tone was set. SolarWinds was truly unprecedented. And it's the kind of event that we talk about for decades. We build and we learn off of it. But the thing is software supply chain attacks aren't something new. They've been around for many years. And we've been watching that check engine light for a long time, but not really addressing the issues. So I'm here to tell you there is a lot more going on than we realized. A supply chain attack is an abuse of trust. It's compromised right at the source. This can create access points into the networks of those customers, into the thousands, as was demonstrated by the attack against SolarWinds Orion. And as we learned, nobody is untouchable. Everyone is a potential target. According to MITRE attack, a software supply chain attack happens when hackers manipulate the code in third party software components in order to compromise the downstream applications that use them. Attackers leverage compromised software in order to steal data or to corrupt targeted systems or to gain access to other parts of a victim's network through lateral movement. And any other upstream part of an organization supply chain can then be targeted. Now that can include application developers or publishers of just off-the-shelf software like SolarWinds, but also think of API providers. And there are a lot more APIs by the day. And just the open source community in general. Quote, the attackers tamper with the development process of the software to inject a malicious component such as a remote access tool that will allow them to establish a foothold into the targeted organization or individual. Pretty powerful, very effective. Now, this is a timeline of events going from the end of 2020 into this year. And as you can see, it got busy. You might even recognize some items on there and we'll talk a bit about most of these as we go through. As said, attackers will manipulate software dependencies and development tools in order to compromise the data or systems before they reach the recipient. And as we've seen, they go after the source code. Now, in the case of SolarWinds, Microsoft said that attackers gained access to some of the source code for exchange, Azure and Intune. I don't have to tell you especially not after this March, just how many exchange boxes are in use out there or how many, many organizations have Azure up and running or in the process of getting them up and running with the mass migration to cloud environments. Having the source code, let's the attacker look for undiscovered vulnerabilities in order to exploit them first. And they like to go after certificates. And that's another thing. Software certificates, they're involved in so many of these attacks, pretty much all of them. Stolen code signing certificates allow the attackers to evade detection and to deliver malware payloads as though it comes from a legitimate source because of course with a certificate, why wouldn't it? Signed, sealed, delivered. This is what Chinese APT Barium did with the Azure's live update when they infected Azure users on mass. And certificate abuse is an ongoing component in all kinds of attacks. For example, two years ago in an NPM attack, developer accounts were targeted and build environments were compromised. So let's talk about the who and the why. This is definitely the purview of state-sponsored threat actors. Who have been in many cases identified as a group of Chinese cyber espionage groups known as APT 10, APT 17 and APT 41, the various other nicknames as well as some major Russian threat actors. Now China has actively targeted tech companies in Taiwan with supply chain attacks because they see them as major competition. And this is what where I do spread intelligence plays a key role because we're watching the geopolitics play out and we're reaching back for historical context on patterns of behavior because history repeats itself and actions have consequences. Ian Pratt, who is the global head of security for personal systems at HP, had this to say whether they are a direct target or a stepping stone to gain access to bigger targets. As we've seen with the upstream supply chain attack against SolarWinds, organizations of all sizes need to be cognizant of this risk. Now both cyber criminal and state-sponsored groups target the technology industry because these companies are relied on by many organizations and individuals and that can have a wide ranging impact. Attacks on tech companies can enable third party compromise of enterprise customers via software supply chain attacks and that's the beauty of a supply chain attack. You don't have to go directly after your target. You come at them sideways. Nick Weaver, the security researcher at UC Berkeley International Computer Science Institute shares this. Supply chain attacks are scary because they're really hard to deal with and because they make it clear that you are trusting a whole ecology. You are trusting every vendor whose code is on your machine and you're trusting every vendor's vendor with grappling with the growing sense of complexity in tech and it's not just what corporations deploy in their own environments, but how they're incorporating third party tech. As Windows Snyder says, the product assumes all the security risk of all the components that it incorporates. Need I say more? As I mentioned earlier, we are increasingly code dependent. Sorry, bad pun. The fact is that applications increasingly depend on external software to work. There's proprietary code, open source components, third party APIs. Modern apps are simply too big for just one developer to try and do on their own. So software reuse has become the norm. There's popular open source projects which are used as dependencies. These become attractive targets for an attacker who can then add malicious code to them and claim the users of those dependencies. Per GitHub security researcher, Maya Kazarowski, 85 to 97% of enterprise software code bases come from open source components. Yeah, that is a big number. What's the average project having 203 dependencies? That's a lot of trust involved. Now, given how sophisticated these attacks can get, any project that doesn't incorporate basic protections, like code signing, puts itself at considerable risk. Again, we're using other people's technology. This is about trust issues. So this is a timeline of software package repositories that have been involved in supply chain attacks. It's really easy to just get code from various projects online and then incorporate that into other software. But here's the risk. Some of those open source projects are widely used, but they're not well maintained. Some have even been abandoned. Code reuse can help to simplify and speed up application development, but it's at the cost of being vulnerable to compromised off-the-shelf components. Now for an attacker, compromising a software supply chain can be through the manipulation of the application source code or the manipulation of the update and distribution methods or by replacing compiled releases with modified versions. And targets can range from either a really specific and limited group to a very wide range. And that brings us here. The continuous integration and continuous delivery pipeline, CI CD. It's considered a best practice for DevOps because it helps them to deliver code changes frequently and reliably. It's a good thing. Now CI is a coding philosophy, a set of practices to help drive development teams to implement small changes and then check in code to version control repositories frequently. Again, this is a great practice. It's a consistent automated way to build package and test applications with one caveat. We're using other people's technology. Mistakes will be made. Yes. In McCarthy's blog from last July, Sikhar Sarokhai, and I hope I didn't mangle that, had this to talk about with regard to a source code leak and what we should have learned from that and how we need to protect our IPs. There was a SonarCube misconfiguration and it led to a massive leak of source code which affected 50 major companies. SonarCube is an open source tool that's used for static code analysis and to check for bugs before deployment. Now based on the attacks, developers and insecure development pipelines are ideal targets either for state backed APTs or highly resourced criminal groups. And let's discover what those adversarial inclinations look like. This is from a report called broken trust and it was released by the Atlantic Council earlier this year following SolarWinds. There was a lot to be processed. There have been 36 other cases of intruders successfully targeting software updates from 138 recorded supply chain attacks and vulnerability disclosures. Now of these 36, 15 had similar access to build or update the infrastructure. Think about that. That is a lot of control when you can tamper with and modify existing infrastructure. And half of those 15 could be attributed to nation states. On December the 14th 2020, the news hit. Now SolarWinds Orion is trusted network management and monitoring software. It's used by governments. Fortune 500s. Security companies like FireEye, major tech companies like Microsoft and nobody knew anything was wrong. What's distinctive here is the degree of stealth. The ability of the adversary to conceal their actions and the length of time it took for this to be discovered. This was an operation that took time and precision to do the necessary reconnaissance. The sophistication to tailor all the pieces and then the patience to just let it play out and avoid detection. They targeted and compromised the software build environment and code signing infrastructure for Orion code. There's that word code signing. They modified the source code to add a backdoor and they signed it. They leveraged the existing software release management system and they used stolen certificates certificates to laterally move through chains of trust. Signed, sealed, delivered. This attack is significant and I want to tell you a bit about it because it's something we all need to learn from and remember that things continue to evolve and this just presents opportunity to the attacker. Security researcher Alex Berson shone a bright light on a scary possibility. He took a hypothesis about a supply chain substitution attack where a software installer script is tricked into pulling malicious code files from a public repository instead of getting the intended file with the same name from an internal repository. And then he targeted Apple, Microsoft, Tesla and about 32 other companies to execute an authorized code inside their networks to prove that this would work. It did. However, somebody's watching somebody's always watching. And in this case that somebody thought that it would be a good idea to end without, let's say that permission. They targeted some other companies in March of this year. Again, Microsoft, but also Amazon, Slack, Lyft and Zillow to name a few. This premise prompted a study by researchers at the Red Hunt Labs. What did they find? Well, 93 repositories out of the top 1000 GitHub organizations are using a package that doesn't exist on a public package index. This can be claimed by an attacker to cause a supply chain attack. And 169 repositories were found to be installing dependencies from a host that isn't reachable over the internet. 126 repositories were installing packages owned by a GitHub or Git lab user that doesn't exist. Now of the top 1000 organizations that were scanned. 212 had at least one dependency confusion related misconfiguration in their code base. This is significant because according to the researchers much of the open source ecosystem depends on these giants. We know that these repositories have a lot of users. So it stands to reason that if any of their projects were to get affected. There is significant likelihood that millions of users could be at risk. X code spy. So we know that attackers are targeting developers and they're targeting the shared sites. The repositories where code is uploaded for use by others in March this year, a new malware variant was observed targeting iOS developers in a supply chain attack to install a backdoor on the developers computer. So X code spy is a malicious product project and it affects the free application development environment. It's common behavior by developers to share their projects online with other users. It's just good sense. It's collaborative and it's efficient. Well, threat actors behind this attack abuse this norm and they used a legitimate development environment created by Apple no less to fool victims into adding an online project to their applications that would compromise their system in a supply chain attack. Code Cove is an online platform used for hosting code testing reports and statistics. Now it provides developers with tools to help them quantify just how much source code gets executed during testing. And they serve over 29,000 customers globally. Many of these are enterprise level clients like GoDaddy Atlassian Royal Bank of Canada Procter and Gamble. But the impact went beyond this to thousands of public development projects like Kubernetes, PyTest, Ansible. Victims included Twilio and Rapid7 and the e-commerce platform Mercari. On April 1st of this year, Code Cove reported a supply chain attack that had occurred back in late January. What happened was this? Attackers leveraged an error in the process that creates Code Cove's Docker image. Now this allowed for them to extract credentials which protect the modification of the bash uploader script. This is a tool used by customers to send code coverage reports to the platform. The script was modified to deliver details from customer environments to a server outside of Code Cove. Attackers could export credentials, tokens or keys which pass through Code Cove's continuous integration environment, which we've talked about. They could then use these to access services, data stores or application code. In 2009, Operation Aurora served as a wake-up call when the Chinese state-sponsored group, APT-17s, targeted Google, Adobe and some other tech firms for their source code management systems. In order to alter that source code. We're going to talk a bit more about that later. In 2017, there was not Petscha, the elite Russian hacking team Sandworm, who's part of the GRU military intelligence service, compromised and took over the software updates of Medoc, which is accounting software used throughout the Ukraine. And they used this to distribute destructive malware known as not Petscha. This infected major companies like Maresk as unintended consequences and the costs amounted to approximately 10 billion globally. In January of 2019, there was a sophisticated supply chain attack that targeted the ASUS live update utility. And that's something that's pre-installed on pretty much all ASUS computers and an auto update feature. And we'll cover that a little bit later as well. This group also delivered something known as shadowpad malware to infect enterprise networks using a product known as NetSurang. NetSurang specializes in server management and security connectivity software. If this sounds familiar, it should. I think that just happened to silver ones. As promised, let's take a look at the Chinese APT groups who've been involved in software supply chain attacks. Now, all state sponsored adversaries. Chinese cyber espionage groups have been and will continue to be the biggest threat to tech conducting economic espionage and intellectual property theft. Technology companies are rich targets just on their own, but these groups leverage them to infect supply chains to go after their customers. And they are the leaders of the pack in terms of the number of attacks and their capabilities. These are some of the most well-known attacks of Chinese state sponsored threat actors. So I'll go through them briefly and highlight where attribution was able to be made some key lessons about the targeting intrusion and just the issue of trust. Something that really came across to me while I was doing the research on this was the overlap in terms of tactics and tools. My personal observation would be that these groups tend to work closely together and they probably were learning within one group then moving on to another and then they took what they had developed and learned to leverage it as needed because that's efficient and it's collaborative. And isn't that what we do when we're talking about open source development? Exactly. Why reinvent the wheel when you already have something perfectly useful when somebody's already covered background? These groups are united though in how they're serving their state and its mandate. Now, if you wanted to try mapping Chinese APTs to government and military. I found this work that was already in progress. It's by Anastosios Pinguos and I'll share the link on the next slide. It's very complex, but this is a great visual representation. Now, let me give you a moment to take a look at that so you can capture it. And if you're at all familiar with the crowd strike, they like to to group their adversaries in fun and colorful ways. All right, so let's start with 2009 and Operation Aurora. Like I said earlier, it really was a wake-up call. APT-17 had targeted Google, Adobe and other tech companies to go after their source code by tampering with their source code management systems. This led to Google implementing zero trust to track lateral movement and to implement better infrastructure. Those were their lessons learned. And what's interesting about this is that in the fall out of SolarWinds Orion, Google wasn't one of the companies mentioned. In 2017, an attack was discovered that targeted an administrative software package known as Evlog. This was made by Altair Technologies, who are a Canadian software company. This was a software supply chain intrusion, which targeted enterprise organizations globally, including military organizations and defense contractors, which is something China cares deeply about developing and surpassing the Westat. There were also banks and universities. And there were several major telecom providers. I get a little worried by targeted attacks against telcos for a few reasons. And that's not just because Akamai had a hiccup and went down, but we all feel the pain when we lose our online access for any degree of time. It's pretty much because we can't live without it. Now in this attack, there was a very impressive client list and you should be thinking about SolarWinds because that too was a very impressive client list. Several years later, we still don't know how many of these customers were and could still be compromised. Now, why FLOCK? Because the users were mostly system and domain admins and that offered excellent access to targeted networks after the initial compromise. Connoisseurs is really important. I think so many people may have already heard about this one, but C Cleaner. This has been linked to ABT-17 and more specifically to a subgroup known as Axiom, who have historically been engaged in supply chain attacks. The C Cleaner attack showcases technical knowledge, preparation, and patience. The timing was strategically advantageous because the original owner, Piraform, was in the process of selling to Avast. That creates a lot of distraction and confusion. So it's really easy to miss things like things in the network that don't belong there. The attackers took their time to move laterally in the network during off hours to avoid detection. Within a month, they installed a modified version of Shadowpad Backdoor Malware to escalate their privileges. Then they distributed a cryptographically signed version of a modified C Cleaner product. And no one suspected, signed, sealed, delivered. If we want to look at this a little more technically and just walk through the actual steps of the attacks, initial compromise came through unattended workstations from a C Cleaner developer. They were connected to the Piraform network and the attackers utilized TeamViewer. They also reused credentials that they found from previous data breaches in order to access that TeamViewer account. They delivered the malware using VBScript and they developed a malicious version of C Cleaner. They used RDP to open the back door on a second unattended but connected computer. There's some good lessons in here if you're taking notes. There they dropped the binary and a malicious payload of second stage malware and that was delivered to 40 C Cleaner users. Now they compiled a customized version of the Shadowpad Backdoor in order to allow for further malicious downloads and data theft in preparation for a third stage and then they installed their third stage payload. Now that malicious version had multi-stage malware in order to steal the data and send it back to the CNC. As far as action on objectives, very likely that stolen data has been put to great use in further espionage activities. If you wanted to try and go through this against MITRE's framework, there's a lot, I think, that would be applicable. For example, under Recon, you've got a lot of work that they did but supply chain compromise. Resource development, they compromised accounts. I believe they compromised the infrastructure. They certainly developed capabilities and they established accounts. For their initial access, thanks to those reused credentials, they found themselves a valid account to get in with. For execution, they leveraged some software deployment tools and for persistence, well, there was more account discovery and account manipulation to keep them in. For privilege escalation, exploitation for privilege escalation. For credential access, they had credentials I believe from password stores, discovery. Can I say they discovered everything that they was to discover? Lateral movement, as we talked about, they used shadow pad and remote services and then exfiltration over a C2 channel and the impact being some data manipulation and exfiltration. Shadow hammer. This is the ASUS attack. In January 2019, a sophisticated supply chain attack was discovered targeting the ASUS live update utility. So this is something that's pre-installed on most ASUS computers and it's to make life easier for the end user. It automatically updates components like the BIOS, UEFI, drivers and applications. According to Gartner in 2017, ASUS was the world's fifth largest PC vendor which would make it an extremely attractive target for APT groups that might want to take advantage of their user base and in this case, that APT was Chinese threat group, APT 17 or Barium, who were, as we've seen behind the attack on C-Cleaner. The attackers altered an older version of the ASUS live update utility software and then they distributed their modified version to the ASUS computers around the world. The software looked legitimate. It was signed with legitimate ASUS tech certificates and it was stored on official servers and it was even the same file size signed, sealed, delivered. And once it was planted, that backdoor program gave attackers control of the target computers through remote servers that let them install additional malware. In 2020, there was an attack known as Able Desktop by the APT group Lucky Mouse. It involved tool reuse by other actors that were not only Chinese groups and I thought that this was interesting because the shared use of tools and collaboration within Chinese state-sponsored groups affects our ability to do accurate attribution. They compromised the Able Desktop chat software that's used by Mongolian government agencies and then they hijacked updates of the software supply chain. This is a really great small picture of what could be a much larger, more devastating attack and again, the attackers didn't need to steal or forge an update signature because Able's updates were not signed. Another attack coming out of 2020 was SignSite. This is the Vietnamese government certificate authority and it was the target of a software supply chain intrusion which targeted a wide range of public and private entities. The attackers used its digital signature software that provides certificates of validation and software suites to handle digital document signatures. Now, this software is widely used throughout Vietnam and it's mandated in some cases and if you've seen, if you think of Napecha and how that tax software was mandatory and there are other cases, this is what attackers for looking through their lens are leveraging against us. This is truly an abuse of trust. So the key point, the abuse of trust here, leveraging a service oligopoly. We haven't got actual confirmation of who was behind it but it's definitely believed to be a Chinese state sponsored group possibly a group known as TA428 who have a track record of targeting East Asian countries like Mongolia, Vietnam and have possibly even gone into Russia for Intel gathering and that brings us to Golden Spy. Golden Spy malware was embedded in required tax payment software issued to corporations who wanted to conduct business operations in China. Now Chinese banks require businesses to install an intelligent tax to pay the local taxes. This intelligence tax software and it's produced by the Golden Tax Department of the Isino Corporation. So the malware installed a backdoor on systems which enable a remote threat actor to execute Windows commands or upload and execute any binary that could be ransomware, remote access, Trojans, anything and the malware provided system level privileges. So the capability to execute any command or any software on that system where it's installed and it was connected to a CNC that was distinct and separate from the tax software network infrastructure. You've probably heard of these guys. This year there was a very, very, very large breach and it's just coming to light the details around it. SETA is a global IT provider for 90% of the world's airline industry. The pieces of this when you put them together link back to show that Chinese state sponsored threat actor APT-41 was involved and it's potentially impacted over four and a half million passengers. SETA had announced the attack back in March and then soon after Singapore and Malaysia Airlines disclosed that their customers' personal data had been exposed. After that, Air India reported a major attack against its systems. Now Group IB took a closer look at this. They didn't believe what they saw at first and then they realized just the level of sophistication involved in what was an enormous level of attack that could only be attributed to a state sponsored actor. And that's how with some digging they were able to find artifacts through the malware, through overlap in terms of the code that they were seeing to trace it back to APT-41. APT-41 also known as look at spider, winty with ties to barium. A group that's been active since 2007 and has a track record in supply chain attacks. And that's why it's so hard. My attribution is so hard. All right. So let's talk about what we should and could learn from these attacks. We do have a new executive order and that's a great step in the right direction. But the picture is far bigger than we realize. These are some recommendations. We need to have prompt communication when something happens. We need to be able to share information effectively and we have to take action because time is a luxury we don't have and something that our adversaries have plenty of. Code signing is essential. Even though as we've seen certificate abuse can be rampant. We need to find ways to ensure that tech is secured by default establish an international norm with clear penalties because you can't have something without the other. If you're going to have policies you need to have enforcements. Eric Sheehan of Symantec who was also integral in helping investigate and decode Stuxnick says that this is like finding a needle in a haystack to have the right security telemetry and visibility at the right control points in your organization. And here's the shift. And so I'd like to end with this. I love podcasts and one of my go-tos is risky business. They had an excellent segment about Operation Aurora with Mark Rogers from Okta whom I've referenced a few times in here. So if you can gain a position of trust you can exploit it. Think service or non-person IDs accounts that nobody really scrutinizes with enough privilege to leverage to do e-discovery. And to find all the stuff that you need for an attack. Your chain is only as good as its weakest link and there are more ways to abuse the chain of trust than people realize. Thank you so very much. I hope that you got some good stuff to work with out of this talk and I really appreciate your time. My details are there. You can find me on Twitter. Thank you.