 Hi everyone, and thank you for joining my talk. Are we forever doomed by software supply chain risks? If you join this talk, then it means you care about software and you care about software security, which is great Most of all you probably are curious like I am how does open store software supply chain risks Impacts all of us you me everyone here So I wanted to begin this with interesting stories such as let's take a moment to reflect on this picture What you're seeing on your screen right now And what comes to your mind is it a futuristic outlook of the world like the robots or they Uprising and becoming a key part of our lives Perhaps it's other things like how much does the robot actually learns and uploads all of that to the cloud You know, where is it stored? Is it stored safely? What happens when someone can do something malicious with that data? Most of all what happens if someone hacks in and interacts with my child This is my kid if so to say and what happens when that interaction takes place when someone is able to compromise that So, yeah, we thought we're are getting started, you know These are some of the things that are keeping me up at night And today I would like to share with you some real-world stories of how developers play a Fundamental role in recent and growing security incidents and also now Why should you care about security software security and software supply chain risks? And also leave you off to think about who do you actually trust so in case you had a doubt We're seeing more and more open-source software being developed year after year right open-source software Repositories are growing with more and more open-source software and that is more software footprint The thing is the application that we build are ever growing in their dependency of open-source software Not just the fact that we're using open-source software, but applications themselves are using more and more of that as well So more of us basically software engineers are now accustomed to the way of how people work in open-source like Opening issues and becoming maintainers of open source more and more of us of us are Maintainers of open-source software and that growth of open-source software doesn't come without any risks Right. This is why we're all here because we care about it We are continuously witnessing the growth of open-source software security vulnerabilities in these ecosystems like NPM and Java and others And this could be anything from CVE based vulnerability reports that you know when you when you install a software You get that installation that said you know there are 1000 vulnerabilities there what happens when that takes place also the incidents themselves of malicious packages and different Targeted attack on developers because what because us as developers we rely on open-source software packages And so we are targeted as well and we'll get to that right now Let's rewind back first of all in time to get an early glimpse of how one developer perceived the risks of open-source software So in 1984 the Turing Award winning Ken Thompson wrote a short essay titled Reflections on trusting trust in which he describes how he added a backdoor to the Unix login C program and then he continued and added a backdoor to the C compiler And then he added a backdoor further on with his chains of attacks into the back into the compiler that compiles The C program that compiles the compiler right so in his paper here About reflections on trusting trust he actually explains how software can be taught to learn specific traits and pass them on So realistically it is very very hard to find the traces of things like Trojan horses and backdoors Unless you have actually written everything from scratch and I mean everything such as the compiler the linker the CPU The display that you're watching something the keyboard everything. It is very hard to trust an ecosystem So as we learned by Thompson's Trojan horse story, you know, it's dating back to 1984 Developers have been targeted as a vehicle to distribute Malicious backdoors and other sorts of malware for a very long time now. Let's explore some of these incidents I'm sure you've probably heard of some of them in 2018 They JavaScript ecosystem witnessed its first high-impact spearheaded targeted attack on maintainers and developers alike where they are working in open source in the ecosystems themselves and actually have been The attack target the duck vehicle as well to actually distribute malicious JavaScript code to all of us using those software and so The attack itself targeted also developers using open source and specifically a Bitcoin wallet application And this was the well-known event stream incident event stream existed on the npm registry since 2011 for a very long time now as you could see it's practically didn't receive any new releases in the last two years But gained millions of downloads per week out of the blue someone chimes in and say I want to help they get into the project They help they open a pull request as you know, you normally do in open source software One of those pull requests later has actually Introduced a dependency at that point in time. This was a trusted individual So they received a different kind of like access to the repository and publishing new packages and new versions of that package And so on and later on when they added that dependency now that dependency exists in a different state and they could just Add that back door to that new version that they released and now new versions of event stream that use that Dependency pull that the new version in and they get that dependency with the back door that now the original maintainer of event stream Did not know about they did not know that this is happening and they can't control it because this is how Software package managers work in specific ecosystems like npm like others So this actually resulted in two versions of of the copay Bitcoin wallet application Including the malware three months for three months until we found this. This is the event stream one now an academic research paper published in 2019 Investigated this, you know properties of language based software ecosystems and it's found that more than a half 61% of open source packages on npm could be considered abandoned because they did not Receive any release in the last 12 months exactly what happened with the event stream story And as if we didn't learn anything from this incident one year later a similar action comes into life Electronative notify any similar incident happening what happening in in short right this one point in time this existing this Electronative notify Package which is not malicious, but it's added as a dependency to a different project this easy-dex GUI thing Later on that the developer that owns the Electronative notify releases a new version that now includes a malware a backdoor to it What happens the Electronative notify project is now embedded into a different app this easy-dex things Which now pulls a new version of Electronative notify with the malicious version and releases new versions again sound similar Exactly what happened before how much thought are we actually giving to the security of our own development infrastructure? For example resources like cloud instances and your continuous integration tooling in January 2021 a Security researcher broke into Microsoft Visual Studio code github repository Essentially, this is providing him any capabilities like read and write access to modify the code of the you know Very well be beloved IDE VS code that many developers love But due to command injection that this researcher had been able to exploit It was able to give him and an attack vector to basically create code injection Now by opening a pull request all he had to do opening new code pull requests This researcher can now execute code into VS code's own CI Ecosystems without being authenticated or authorized all of this really just provides him the ability to create to Submit or if you would like to say spawn Remote reverse shells on the CI servers running code and you know from that being able to basically push and write access To the repository itself now fortunately for us so all of us here in this ecosystem This researcher Responsibly reported the flaw to Microsoft before you know Hopefully anyone else could have gained access to it, but you have to ask right? How much do we know about the current state of open-source security? What does it actually entail so in an effort to explore the security awareness of Python and JavaScript open-source communities a group of researchers investigated how maintainers work in the open-source community and you know How do they release fixes relating to security vulnerabilities? One of the research questions was how quickly do maintainers mitigate and newly published with security vulnerability once it's published Wonders a CVE out the research found out it takes almost 100 days on average for both JavaScript and Python maintainers to start mitigating a vulnerability you have to ask yourself if this is fast enough and This is a bit different because between the different communities because for for Python It has been remediated much sooner versus JavaScript Which took its time from 2018 until now to basically be a bit more mature and aware about about the security risks as A case study exactly of this we can refer to marked It's a it's a library marks own security vulnerabilities force from several years back Now what happened is this mark is a markdown library It's a parser for you know JavaScript and no JS and it's very popular getting millions of downloads. I know a week One day a security researcher just pops up open a code pull request to open to fix a vulnerability So this is you know All you have to do is merge it basically as you can see here But as it is with open source software, you know maintainers are really trying to do their best But you know they they're not contracted or obligated legally or something to support you So, you know me as a maintainer you everyone else We're really trying to do their our best and in time of what we can do but this vulnerability stayed out there for a year knowingly exact what's happening with this and this security vulnerability itself being public for more than a year When we're all so much very dependent on open source software We can't ignore the question of you know, where do we put our trust? You know, what are the mitigations and controls that we have to cope with the risks involved in 2017? a security researcher working with a no JS foundation Conducted a research which he wanted to explore and assess right the state of weekend PM credentials on the npm registry His he his work really revealed devastating truths about developers and their lack of security hygiene What I mean by that is this security researcher was able to gain published access to 14 percent of npm Ecosystem modules. This is a lot some of these modules were downloaded millions of time tens of millions of times a week And there are they wariness. They are still an essential key part of this You know thriving JavaScript ecosystem, which I am also a part of so this is you know very concerning for me as well The thing is the problem was rooted with insecure passwords chosen by well-known maintainer accounts such as literally the word Password for one of the accounts running millions of downloads for one of their packages So if it can happen to them it can happen to other packages as well if our code packages can reach Millions of developers shouldn't we have more protections in place as citizens of open source communities? Shouldn't we do better for like our account hygiene? So you would imagine so but in npm the largest registry of open source software even even though two factor authentication was enabled since 2017 only only 7.1% of npm package maintainers have enabled 2f a you'd think that we have done better so far But a year later by 2020 another interesting insight that we get into what what has what has been happening? Is that it has only grown by 2% only 9% of developer accounts on npm by 2020 have enabled 2f a we should do better as Cyber security expert Bruce Schneer says and had said and written in his book secrets and lies humans often represent the weakest link in the chain Think about it as you as you go and think about this term given enough eyeballs All bags are shallow as a term coined by as Linux's low back back by Eric Raymond in his work They cathedral and the bazar by now at 1999 He explored the differences of how software development takes, you know different shares between open source and enterprises And one of the things was that you know if everyone are looking at it, you know, we'll find the problems There are a lot of people looking at this code Really are we gonna find it because in January 2021 it was discovered that pseudo a common utility installed on many Linux distributions Had a security vulnerability Specifically it allowed any unproved users to gain root access simply by the default pseudo configuration Now this was so daunting as a vulnerability because it was hiding in plain sight This is from the you know from the disclosure itself for a whole 10 whole years. This was just showing up there now We can't deny being a part of this open source ecosystems as Consumers contributors or maybe even as maintainers for some of us We all play some part in a world where 90% of our application code is open source software components We reached the point where no we take open source for granted right open source registries are opening their nature Anyone can go in and publish your new packages even malicious ones, you know, you'll be surprised. Maybe it doesn't get caught We have been accustomed to this, you know opening an issue in a project source code or asking for help asking for a feature You know asking someone to get our bag fixed hopefully but we we import those open source into our projects And do we understand what's going on at that point in time because open source registries are Opening their nature, but they're not gonna be there all the time What happens when maintainers removed your packages and dependencies and this is a story that exactly happened and For a package that was very pivotal in the ecosystems But caused a widespread wreckage of continuous integration systems all around So at the very least it taught us that enterprises and organizations that were prompted is attack We're not understanding how to deal with open source software and registries themselves did not know either They were missing how within this could be possible and would it be possible? What kind of malicious activities and assets can we track back to open source software time after time? We're finding more malicious packages hitting the npm ecosystem type of squatting mistakes and others But they're not only a sole problem of the of the npm the JavaScript ecosystem because pi pi had also seen thousands of packages Just you know brute-forcingly Published in bulk to basically add malicious packages for to further show us how attackers can actually harness open source to their Advantage Alex Burshan published his research on in 20 in February 2021 about how he exploited design flaws in package managers and human errors and registries basically to Infiltrate into corporations like Apple Microsoft and others and so all of this is happening in open source And I would like to leave you off with some questions Are we going to have less or more software in the future? Are we going to use more open source software in the future? Who do you trust and what can you do about it? Thank you for coming to my talk My name is Iran talent my developer advocate at sneak where we build a security platform to help developers build securely with open source software Thank you all for joining my talk. Have a nice conference