 So hi guys! Welcome to my talk, attackers in open source supply chain, the new frontier. First of all, I am very happy to be here. Let me introduce myself. My name is Tsachi, Zack. I was the founder of a company called Dastyco, which was focused on detecting attackers in code packages. We started a couple of years back when software supply chain was just like a vague concept. Thank you Putin for Solowings. ‫הספר התפעילה נעשה מאוד קטע ‫בספר על זה, ‫אז אני רוצה להצלח על הרבה ‫שחקים ובספר על היחסים ‫שאנחנו רואים את הקטקרות ‫העבורת הנשיא. ‫אבל קצת, אני всегда רוצה ‫תספר על מה זה problem, ‫ואז לא אני עוד פעם ‫בספר, ‫כי שאין שכל מי שם ‫הם עוד פעם, ‫אז אני רוצה להפעיל. ‫אז פרס פרס, ‫אנחנו אתם כל מי ‫שחקים לספר, זה פרס, right? זה לא סינגל פונט שאתה צריכה להסתכל ולכתב על זה זה same in the automotive industry The car manufacturer doesn't manufacture the wheels or the AC system but he has a process in place to make sure that all the parts he is getting are at the right quality So for a very long time we tried to define the problem and I really want to thank for the guys who actually adopted the name of one of my favorite dishes Salsa So basically Salsa really helps me to talk about the problem in a defined way so I can talk about what I'm doing inside Salsa and what part I'm not actually dealing with right now and I really like Salsa because of course it came from the Linux Foundation and now it's been backed by every other major body out there So basically of course when we talk about the software supply chain and the set process we have the developer, he writes the code, stores in the source usually we have multiple developers and then we have the build process and the build process combines all of those codes together together with open source dependency and then we have a package or an application we can use internally or externally So again as I said before there is no one problem we need to solve there are many problems we need to solve and it will take us a lot of time to solve all those moving parts but realizing that and looking at the problem a couple of years back I quickly realized that those parts although there are a lot of different attacks organization can do something today organization can decide how do they want to secure their developer workstation or do they require their developer to use two factor authentication when they're submitting the code into GitHub where the build server is actually placed what controls do they have in place protecting their artifactory but the one point that we thought was one of the weakest link and that's my talk for today it's going to be around dependencies and open source packages because basically there's not a lot we can do to protect ourselves we can't define a standard that I now will require every open source contributor to stand through so we thought that would be a weak point and basically we were not that mistaken and open source is critical because basically everybody use open source right it's essential for modern application most of our code come from open source developer wants to use open source and to move fast the problem is that they don't necessarily understand the full impact of what they're doing for example a developer would like to use a package the package seems legit and well maintained and popular that's good enough not all developer realize that this package actually contains another package what we call transitive package and in many cases this isn't a one to one ratio it is something like this for example suppose I want to use a package called cncgs the package seems legit the package seems popular but what I'm actually getting is this so I don't know any developer it would go and vet anyone all of those packages and all of those contributors so you can't just say I'm going to use like one open source package and that enough because everything is connected it's an ecosystem it's an ecosystem like a jungle and the jungle is growing because we are seeing more and more open source packages being released every month npm alone has a million packages released every month so we automatically trust open source although we have all this complexity and we ask ourselves why is that why we automatically trust everything that comes from open source and developer doesn't feel the need to check or to do some kind of check and balances and basically it's because developer tells to himself mm I don't have time to do code review for open source packages I'm using but if it's open somebody will look at it somebody has spare time to look at every open source package that's being released out there and if there's an issue somebody will notice and if it's popular it must be okay like remember remind me of me as a small kid everybody's doing that and it gives us a trustworthy feeling as I can say we feel safe using popular packages and we are seeing attackers abusing those assumptions I will give a couple of examples of taking those assumptions about open source and trying to use those assumptions against us so a couple of examples guys this is phasel phasel is one of the good guys is an open source contributor he maintains a package called UA parser UA parser is well maintained he's actually been fixing and adding features and actually working with the community for 10 years and it's highly popular again used by millions including Facebook so would you use it and the answer would be yes if it's popular and everybody's doing that I would automatically trust everything coming from UA parser and there's a good chance that even if you didn't say I will use it you're actually using it as part of another package that you're using the jungle as I discussed and then we saw this last year we saw on a Russian underground forum somebody saying guys I've actually stole the identity of an open source contributor and I'm selling it to other criminals so they can make money out of it and you can see he's actually asking for several different prices so I don't know exactly how much this attacker got but I can tell you somebody paid and how can I know it because a couple of weeks later we saw this phased the rate online and said guys I'm really sorry somebody hijacked my account and published malicious versions of my package it wasn't me I'm totally sorry for that I'm sorry to everyone who was affected and that's exactly what happened the attacker and this isn't a vulnerability the attacker did a technique we call account takeover compromising a legit account and then using that account to poison the package so everybody down the stream using that package would be impacted and in this case what the attacker did he actually had a malicious what we call a password stealer and a crypto miner so I wish I could say this is a once in a lifetime event it will never happen again we can ignore this but two weeks later we saw a similar attack this time around packages called coa and rc I have to say different contributors right and again highly popular well maintained and we saw the same malicious code being injected into both of those packages so this isn't a vulnerability this isn't a logical flow that we actually discovered those are actually attackers targeting open source contributors in an attempt to poison the open source ecosystem and to be honest we are seeing more and more attacks on good packages being happening in a very expedited pace since then just I think a week and a half ago we saw this we saw a targeted attack targeting pi-pi contributors so the attacker just wasn't like poisoning everybody they were actually targeting python contributors and again this could be python on npm it doesn't really matter it's an attack against the entire open source ecosystem and they were trying to steal their identity and guess what they succeeded and they actually published a couple of malicious packages that were removed later again highly popular so together with Sentinel-1 my team because we really like to work together actually investigated this group and discovered that they were actually active for a couple of years so what we are seeing right now is actually like common cyber criminals realizing wow it's quite easy and fun to abuse the open source ecosystem to get more rich and again abusing the authentication is not a new thing we talked about it with with UA parser and we are seeing us as a community responding and actually starting to demand two-factor authentication so it would be harder for the attacker to steal our identity but like anything in cyber it's always the game of cat and mouse right now we are starting to add two-factor authentication and it takes a while you don't just like flip it overnight while we are preparing and implementing that this is what I saw last week we saw an attacker tools called Evil Proxy which is built as a phishing as a service and it just came out with a new feature and that feature add support to Github and PMPyPy with automatic multi-factor authentication bypass so we are moving one step forward and they are moving one step ahead so again and this is like one of those examples and it seems from the attacker point of view now they realize the potential it looks something like this so it could be either a duck season or a rabbit season and like every week we are finding new and newer attacks it's like the amount of attacks we covered in August was unprecedented so we like open source but as uncle Ben said with great power that give us open source come great responsibility and we ask ourselves whose responsibility? is it phasor's responsibility? is it NPM, PyPy, my developer, my CISO? who's taking responsibility to stop those attackers? and by the way this is an open question I don't have all the answer we are seeing what is happening and we are working together to try to fix the problem so account takeover is just like one of the example we saw this year but we saw other types of attacks for example guys meet Brandon so Brandon is an open source rockstar he actually maintains 41 open source projects which is a lot really kudos to him he's really spending a lot of time trying to make a positive impact and he has one project called Node IPC again when we look at a project we look at popularity usually how many downloads how well it's maintained so in this case million weekly downloads well maintained so most of us will probably yeah I would use this project and this year we actually saw something interesting this year we actually Brandon add a new functionality to his code so looking at the functionality it seemed a bit cryptic, right? let me demystify it for you basically what Brandon added is three functions to his code first function is asking his code to reach us to this website, right? IP Geolocation and we can all understand what IP Geolocation does, right? it's bringing you the location where the code is running it could be US, Britain, Israel, Dubai, whatever so why does Brandon care where his code is running? exactly second thing is doing, right? you want to start? if checking if his code is running in Russia or Belarus and if so now you don't need to be a developer to understand what come next delete, delete, delete so basically he turned his code into a wiper into a bomb, right? and to add insult to injury he actually added a small rt emoji after every file he deleted so a rhetoric question what happened this year with Russia and Belarus that got Brandon so upset and we know the answer, right? it's the conflict and before I'll move to the next slide is it the same tech I've shown you before of an attacker taking over a legitimate contributor account? this isn't the same case this is a different attack because we can see Brandon in his own world guys, you download my software for free so I'm allowed to wipe your computer this is all public, document, license and open source and he actually named it ProtoSphere so by the way I'm not sure he's mistaken with his claim but I can tell you that I think I think that the majority of the community thinks that mixing politic with open source is a really bad idea not that I am aware of by the way but I'll think about legal actions later, right? I'm not even sure that legally he did something wrong, right? attackers are not being caught, by the way and he actually documented everything so again, he named it ProtoSphere and the community really didn't like it he said, guys, and Brandon, don't become what you hate that's an abuse of power and thank you for teaching me not to take codes from others and I wouldn't like a Russian contributor doing the same thing so I think it's a bad thing but it is a thing we've actually tracked five more ProtoSphere since then and other acts of activism so it's a risk we must consider that it's a possibility that something like this will happen and of course I wouldn't use Node-IPC even if my computer aren't placed in Russia or Belarus but what do I feel about these other 40 projects? maybe I should lack the versions and look for replacements because I don't know what Brandon would like to do in the future so again, this raises a lot of questions and we know that a good reputation is a hard one and it is easily lost and as I said, those are like couple of those examples but we are seeing other attack techniques being evolved this year so we see attackers evolving so earlier this year my group actually tracked a group called Red Lily so what was unique about Red Lily is that they published 1500 malicious packages in one month which is a lot usually we see an attack with 515 malicious packages but they really ramp up the scale how are they able to do so much packages at one month and that's basically easy automation they invested the time and the effort to build automation to automatically publish those malicious packages and it wasn't just packages they even created a user account per package so we can't just block one user account and that's it so they were persistent in what they did more than that if you look at the code and try to block what we call C2 the command server they actually changed the command server so this is exactly what my group is doing and for a very long time we found them we report on them we go back we found them again and every time that we found them we report on them they keep trying to improve their payloads to hide their code to detect if they are running on our system to change everything but we still keep on finding them at one point they got a bit angry with us as you can see in the name in the package name they started publishing but we don't mind getting recognition even from attackers and to help the community to understand that there is a problem here we actually released a tracker site called redlily.info and in that site you can see all the information on every package when it was released what it was trying to do is this package still available or not because it can take a while once we report on something until it's being pulled down there is sometimes a delay so we actually put it as a way to educate developers that there are bad guys out there trying to do those stuff and basically you can look here and you can see the relationship between a user and a package and a C2 and learn more and as I said this was unique like a group putting a pre-source and doing an automated attack and it was unique for a month and a half because then we saw another attacker doing automation this time we named it Qtboy because he actually used the word Qt in the email E-laces he was using and he also built a different automation and he was smarter by the way because when he wanted to create a multiple accounts he actually used the commercial service called MLTM which actually helped him create all those accounts per user and again we found it we report on that we report on thousands of packages and we create a tracker site so it seems like trying to stop the tide so he actually stopped publishing for four days after all the hard work that we did and after four days he actually continued publishing new malicious packages but he noticed us and he modified the way that the malicious packages were written how do we know? because he basically referenced the malicious packages to our research still continuing but giving us credit for the work that we did so basically I encourage you to go to Red Lily and Qtboy and other sites again we are still reporting everything but it's a good way to raise awareness to tell people there are attackers in open source and we are talking about advanced persistent attackers that are always improving always trying more and more and I want to give you like one final example so it's been busy in the last couple of months for us all those the team has been really really busy working really hard so something good comes out of it we became really good at detecting the result and as I said before good reputation is hard one let me show you so I'm gonna talk about the attacker point of view so I have two packages here one is called Pompey one is called Pompeio this is an attack called type of squatting meaning the attacker registers a similar package name in the hopes that maybe a developer will misspell or wouldn't notice he's using the malicious package so I know what you're thinking if I'm a developer and I download the wrong package wouldn't I notice that? and the answer is sadly no because the attacker gave both packages or is malicious package the same code but and there's always a but now I can tell you that Pompey IO was the malicious package and it contained another bit of code when we look at the dependency we can see this code can you understand what's happening on the first line? can somebody figure out what he's trying to do? thank you very much come back later for a t-shirt I forgot to go in the room so basically it's trying to evade our scanner by reversing the URL and sending the developer SSH keys and other sensitive information to that website okay so we understand what the attacker is trying to do we understand this technique going back we always tell our developer check the reputation, right? so now we know that Pompey IO is malicious what's wrong with this package with this image both Pompey and both Pompey IO seems to share the same reputation, right? if my developer will look at the reputation it's not easy to understand that it's actually downloading the wrong package any ideas how the attacker was able to to get that reputation anybody? exactly in a minute so t-shirt so basically is it really that easy and usually when I'm talking to attackers to hackers they say he built a lot of bots and then he started collecting the stars and he used another botnet to do that he actually invested a lot of time to do that and sadly it's not that difficult so what I'm actually showing you now is from the attacker point of view how an attack looks like so this is a package lab package lab is our like metasploit for open source where my research team is actually trying and experimenting all those attacks so basically when you want to publish a malicious package basically what you need you need an account so of course there is no verification all you need is just an email account and you can just register an email account so that's not really a tough barrier the other thing that you need is actually package metadata so a package name in this case I'm going to do supply chain demo but again if I was doing typosquoting I would pick a popular name and just misspell it so it's not that difficult again as an attacker now I need to change to pick a version guys as an attackers never use version one developers that doesn't want to use version one so just pick a version and that was the interesting part now when I'm publishing the package into pi pi npm I need to declare what project do I belong to where my code came from so as an attacker I don't work for any project but I'm just going to go to github to the list of trending projects in this case I'm going to choose a project called the economist ebooks I have no idea what's inside that project I'll just declare this is my project wait for it what just happened so basically there is no vetting if we say we work with somebody we will automatically get his reputation so it's easier than you think in this case now the state that my researchers are writing all the malicious payloads ransomware, dynamic uploaders dynamic downloaders all those stuff different lecture maybe next year because we have a lot of interesting stuff there in this case this is a dynamic downloader and that's basically all you need is an email account a name a URL and you're free to go by the way we call this activity of claiming that you are with a project that you have no similarity to starjacking because basically you're taking somebody's stars and this is some of the traits that we are looking at and that's basically we'll wait a couple of seconds voila I just became a super developer in one single stroke so for developer it's really easy to understand that so why is that that easy because the ecosystem to be honest was never built for security right so we are now trying to fix that and we are doing a lot of work but there is no vetting of metadata in many cases of my website URL my description, my name the related give repository as I've shown you and it's difficult for developer to understand the truth and actually attackers are abusing that all the time so lack of vetting in typosquoting for example is my name similar to another package so this happened two weeks ago when was Defcon, three weeks ago when I was at Defcon we saw a typosquoting campaign targeting Python packages so basically they publish all of those packages again you can see the similar name right with the legitimate packages and just look at the number of downloads somebody would actually misspell that right it's a big numbers game and we found this typosquoting attack really interesting because it actually did a lot of interesting stuff for example they use Github as the stage of site so I'm gonna check the website that my package is accessing accessing Github who can actually block Github and it did other things like use legitimate service to profile the victim service where he's running from what is operating system adding root CA and they actually add something that it was I think first we've seen in this field which is called DGA so again this is DGA's domain generating algorithm meaning if I'm gonna block this Github website he actually add an algorithm there if this Github URL is blocked the algorithm will try to generate suggested other Github URLs the attacker is aware of so we actually saw in the algorithm inside so if we actually block the first Github URL he would actually try to go to anyone of those users who is still unregistered so it's like trying if you stop me here I have other possibilities waiting in line so this is like DGA and it's the first time we've seen it but again he was using Github right so after we as a C2 server and we can see what's actually written in Github so after we download the malware and reverse engineer it we were able to understand what were the commands the encoded commands he actually registered on Github and we saw this after a couple of days of course we reported it takes a while to get those sites down in the meantime we actually saw this we actually saw this cryptic message but we had the logic to reverse engineer it and we actually was able to understand he's actually doing a DDoS against a Russian Counter Strike server so understanding that it was obvious what we need to do right we actually challenged him for a match we actually opened an issue on his Github and then we are seeing what you're doing and let's do like a match a Counter Strike match where the winds takes the botnet so again it's a lot of fun a lot of funny things and of course it was removed but we are moving forward actually three hours ago my researcher called me and told me Zack I don't see any new typosquatting attempts on NPM and I said okay maybe the engine broke maybe something fixed because we always get a lot of similar packages we need to scribe and check what happened in the last couple of days so I don't know if somebody is here from NPM and it hasn't been announced yet but kudos to NPM in the last couple of days they actually had the ability if you try to register a similar package like BDbug they actually will stop you so we are moving forward starting to stop typosquatting and we are doing great progress again it's a cat and mouse we will never fix everything at once but we are moving forward and this ecosystem I said was never built for security so it can be things like account end-off so I'm an open source contributor a maintainer I don't have time to maintain the project all an attacker needs to do is just ask for permission to build the new maintainer and it actually happens so that's like normal behavior we can't fix everything so sure now you are the maintainer and actually Joseph my co-founder or CTO actually it's an expected behavior you maintain a project called Request somebody asked for to be the new maintainer so sure why not take it and again all the reputation all the things that you used to do people don't understand the maintainer change or we added the new maintainer we don't think about it so I call this the trust paradox meaning I hear CISO air all the day talking about zero trust zero trust, zero trust we don't trust our system we don't trust our people we don't trust our process everything needs to be examined but if the code is coming from GitHub no problem so if I were able to break into one of your companies as an example sit on your developer workstation write my own code inside your codebase there's a huge violation if I publish it to GitHub that's legit nobody's checking because it's open right so this is like the trust paradox where acres apply actually trying to get inside and I've talked about dependency but it's not just dependency guys there are a lot of moving parts across the software supply chain that we need to be aware of so I've talked about dependency who's vetting who's looking at the code of the IDE plug-ins actually updating or downloading who's actually looking at the code of the GitHub apps he's actually using on a daily basis and every update or the package you already used in the past or the build plug-ins so this is a huge tech surface and we need to a lot to take a lot of steps in the future to close the gaps because we started in a totally open system and now we are seeing that attackers are abusing that and I'm saying attackers because I think one of the first thing we need to do to address this is change the mindset I think this is the most important part that what we need to do what do I mean? we've been working and dealing with vulnerable code for many years and we need to improve that but when I'm talking about malicious it's not the same as vulnerable it's a different problem it will need a different attack set for example when we are talking about vulnerable code we are always reactive a vulnerability can sit in code for years until somebody will notice that and then we will fix that I don't think it's the same thing that we want malicious code sitting for years in our code and somebody will notice that so if we can live with vulnerability being what we call a reactive approach for malicious I think we need a more proactive approach go and hunt them look for them take steps we are really scared about managing vulnerabilities it's not the same thing with malicious you don't manage malicious it's like a virus when you find a virus remove that you don't say oh the malicious package is not in a publicly facing website so I'm just gonna let it sit there so it's a different mindset we are talking about tools, techniques, procedures of attackers and from that mindset we need to understand that first of all we need to share information right now we need the industry to start to understand that when I report on a malicious package it is just being deleted this is a bad behavior anybody coming from cyber from digital forensics don't delete the computer let me understand what the attacker was doing so I can better protect myself so instead of just deleting that we need to store that and share that to a central repository where other researchers and university can learn what the attackers are doing just like deleting that reminds me like 20 years ago when I had a virus on my computer the IT guy just formatted him problem solved now we understand we need digital forensics there is an attacker it wasn't a mistake we need to understand what the attacker is doing so we can basically stop them and one other thing when I'm saying this is not a vulnerability I really mean it Pompey and Pompey IO remember Pompey IO? you know what happened when I reported on Pompey IO which had I think 70,000 downloads it was deleted meaning all of those people who actually downloaded that in the past we'll never know and why is that because we don't track malicious as vulnerable so every company is starting to invest to invent their own standard I think it's a bad idea we need a new standard for malicious it's the same that we used to do for CVE because in many cases a CVE will not be open like something it will most of them it won't how do I know if a package I actually downloaded a week ago and it's not a verbal law if it's malicious where can I query that so again we need to work together this is not like one time you deleted it will not go away we need to tag it we need to research that we need to work together so we are actually working a lot with package managers I said Kudus to NPM he did that today couple of weeks ago they started vetting is your Gita account your real Gita account and that's great I think we need to do that more and if we're not doing that and that's okay at least say unverified because when you're displaying something to a developer and you're not saying unverified is normal behavior from other places it's been oh, if they say that somebody verified that so I'm okay not verifying that but just declaring what you're doing and what you're not doing because other it actually help us report by API again you can send an email when you find a malicious package it works great when you find five malicious packages a month we are finding thousands you don't want to look at my mailbox right and exactly I email exchange for every fine I need to see what actually did so we need to automate this process because the attackers are automating their process and when we so before when you remove a malicious package share the query from sample enter original metal data we publish a lot I'm saying it right now if you will ever need something that it was deleted and it's unavailable drop me an email I'll be more than happy to share the deleted sample so we can actually learn what the attackers are doing and better protect ourselves two weeks ago I found I think a thousand malicious packages on PIPI and NPM but how did I found them I found the malicious package doing a crypto mining on PIPI I took the URL from the package I searched the other packages that I'm actually searching I found similar packages on NPM so I reported on them so attackers aren't just stopping at one package repository so we need everybody to talk together to understand we need to share information so you take one lead and you actually start doing the threat hunting so this is exactly what my team is doing I have like two minutes I think so summary the basic question and there's a lot of open issues that is basically great power, great responsibility who's taking that responsibility we can't assume shared model when everybody's pointing at the other guys so I think if we're using open source and again everyone the company is using open source all the critical infrastructure basically it's our software our responsibility to make sure we get the funding we got the process we got the standard out there to protect that because this is I think the start of the trend and I think next year I'll have a completely different example because we see the barriers and we see where things are going so then don't take code from strangers without verifying and guys thank you very much so two things I have like three minutes and lunch is coming so I'll be even downstairs if you want to speak afterwards but in the time that we have yes, I think the White House published regulations based on Biden's executive order and basically it calls for self-attestation for the quality or safety how does it speak with your view specifically when it seems that you're doing the job of GitHub PIPI, NPM and others I'm not sure I'm doing their job everybody has their own job we need to work together that's the first thing and I think attestation is great but I can tell you that being from cyber we have a lot of signed malware so even if we'll ask people to sign there is no problem to sign the type of squatting package, right? so I think it's a different attack vector that we also need to address but from the attacker point of view it's not the main thing that we are seeing right now people hijacking the packages while it's being transmitted from PIPI and changing it on the fly we are just seeing them publishing it into PIPI so I think it's highly critical that we support projects like Seek Store and attestation and other great initiatives like NPM just took and started signing packages but in this example of attackers I don't think it will slow them down too much if that makes sense sorry babe, but okay מלישייס קורד סקנר שאתה יכולה להסתכל don't want to talk about it here you can come to the booth basically it's not a code scanner it's a different technology I came from the AV world you can think of yours as an EDR for malicious packages and just doing code scanning isn't enough I've shown you you can do a stager so the attack will happen in stages so we had to invent a whole lot of different technology to support those use cases so we are actually doing that reporting everything to the community sharing our findings sharing our samples but it's beyond just code scanning code scanning is just one engine of those engines we didn't think we will have so much, right? so we were really excited about Red Lily and then we did Qt Boy and I can tell you we're working on a new website wait for next week there's even a bigger attack that we are discovered but basically I have a GIT with the samples so I think the website is for awareness and we really love awareness we want to people understand the problem by the way, I actually you remember me showing you package lab the application we want to allow developers to play with it we actually I actually imported package lab into a VR everything to get a developer interested in understanding the threats so yeah, we didn't think we will do so much of those websites but that's the reality I got the request combine all the websites so yeah, we'll probably do that forward again it's like a way to think about it so guys, I really am getting the red signal to stop I'll be down, I'll be here happy to talk to you hope you learned something new hope you find it informative and see you next year