 Welcome to DevCon viewers 2021. I'm here today to talk to you about emerging threat targeting software ecosystem, the rise of software supply chain attacks. Here are few things about who I am and what I do. My name is Yadnil Kattalil. I'm Product Security Engineer at Red Hat. I drift around open source software, Linux and application security and I'm glad to be presenting here today. Let's jump into the topic. In March this year, the giant ever given container of Megaship blocked Swiss Canal in Egypt and the holding traffic costed 9 billion of trade, which demanded us find that how important supply chain has become in this global turbulence. So let's start by talking a little bit about what supply chain is and why we are so concerned about this attacks. Supply chain is anything that affects your product. Including all the component you would use. If you fail ice cream, then your supply chain may include physical store, list of ingredients, the packaging, production factory, but it's also more than that. It is something that let you build and deliver your product. In similar sense, we also have software supply chain that exists in software ecosystem. A software supply chain is anything that needed to build and deliver your software. It could be code, it could be library, module or collection of processes or even service required for delivery. But why do we still use this supply chain? Because supply chain makes things way more easier in some sense. Instead of worrying about writing things from scratch, you can reuse piece of library and focus on more meaningful tasks. But there is trade off comes with this and this is complexity. The extended attack service which makes finding vulnerability easier in supply chain than finding it in actual product. A supply chain attack is an attack strategy that targets an organization through vulnerability in its supply chain. This attacks aim to damage an organization by targeting less structural element in its network. These vulnerable areas are usually linked with third party vendors with poor security practices. A data breach through third party vendor is possible because vendor required access to sensitive data to integrate with internal system. When a vendor is compromised, the shared pool of data is breached. This is how software supply chain looks like. We have raw product which is using upstream dependencies which goes to CI CD and then the build system and the final product get distributed through CDN. In this example, the attack surface is the supply chain which is in the rectangle. In all of these attacks, the victim is not the ultimate target but tries at a stepping stone to access other network. And this is so called modern supply chain because it has something unusual than normal attack vectors. Where you can see there is no corrosion and there is targeting software but these are much more serious than that and it is hard to detect. So is it a new attack strategy? Probably not. The first practical attack was demonstrated in 1984 by Ken Thompson and you probably have recognized his name as inventor of Unix operating system. He wrote this interesting research paper demonstrating how he tried to put back door in Unix login panel first but later wrote a compiler and tried tempering with that. And 40 years later, this attack is more advanced now and moving away from traditional physical attack vector to the almost each and every OSI layer. One of early and interesting example of attack is from 2013 where 41 million scratch cards were stolen from target cooperation data bridge. But the interesting thing here is the attacker did not enter by tempering credit card software or payment gateway. Instead, they used some VPN connection which was open for their suppliers of heating and air conditioning. Then the attacker targeted wide open network to inspect every cash register at 1800 store of target. As I said, it is not only important to secure your product but it is very important to secure your perimeter which is supplies and services so this is interesting fact your software supplies in risk mostly are inherited from your dependencies or unpacked software. Today's software dependencies of privacy it's common for your project to use hundreds of open source dependencies this means the most of your application consist of code that you didn't write. So if there are vulnerabilities in your third party open source dependency, chances are your product is vulnerable too. What's scary here is that a dependency might change without you knowing it. We have seen such dependency confusion attack this year in February. Security researcher Alex Pearson has found a security vulnerability that allowed him to run code on server by simply replacing private packages with public ones. Let's see how it works in layman steps. Imagine you had a world document on your computer but when you open it your computer says hey there is another world document on internet with the same name. I will open that one instead. Now imagine the world document could then automatically make changes to your computer and it is not a great situation. He found same behavior for most common repository management tools like NPM, Pipi which is for Python and Ruby James which is for Ruby and many more. Let's see how this attack works for Ruby James. But before that here is a little background terminology of Ruby. There is something known as James which is so slippery that can take Ruby code. You can download and use those gems according to your needs. So if you wish to use gss functionality there are gems for that. If you wish to use website in Ruby there is popular gem called as a rel then comes gem.org which is official repository to publish gems. Then there is bundler which is command line tool which you can use to play around gem dependencies. Basically you can use bundler to install and update Ruby James. Now attacks work like this. We have three gems out of which rel and parent panda is parent gem and child panda is dependency of parent panda. The only difference between these gems are rel is hosted publicly on ruby jim.org and remaining both are hosted on private repository. Please note version number here is 4.0 for all three gems. Now what attacker have to do is he just have to register his malicious gem with the same name as a child gem but with higher version. So in our case attacker register child panda 4.1 on public registry. Now when you update your dependency next time and run bundle update instead of fetching child gem from your private repository bundler will fetch malicious gem which is publicly hosted by attacker. And this is because dependency confusion of package manager do. Now we have seen what those attacks are so let me give you a few tips on defensive security. Most of you are someone who already have their infrastructure in place and you are trying to fix things. So the first major and obvious step is to identify the supply chain of your product. Traditional way is if you are running hybrid cloud, genkin, build system, cdn or whatever you are doing you should have your software bill of material which should reflect who wants what. I do know this is one of the hardest step to do as you have to go through all your infrastructure but again having the list can help you to maintain transparency and diagnose your problem real quick. There is also one better way to do that that is by using NIST framework. National NIST work standard and technology NIST developed a framework for cyber supply chain risk management which talks about it's life cycle. This helps customers and vendors to identify where exactly we stand in the chain and they can later think on assessing each one of the phase and work on identifying potential risk involved in this area. So the supply chain life cycle has six phases design, development and production, distribution, deployment, maintenance and disposal. Please note this life cycle has applies to hardware and this disposal phase mostly refers to that. Our traditional software development life cycle SDLC which we mostly talk about fit right here in the first two phases of supply chain life cycle. Red Hat Enterprise Linux is one of such example which fits perfectly with this. Rail goes through design and development phase and once we are ready with rail as a product we release those on the content delivery network CDN and distribute those. Then our customer pool those images from CDN and deploy operating system on their systems. Then rail goes through maintenance phase and as I said disposal does not quite applies here. This life cycle is perfectly applicable for enterprise product but might not quite fit well with modern software as a service sales based application. Each phase of ICT supply chain life cycle software is at risk of malicious vulnerabilities. Here are some popular example of threats. In 2016 a foreign company designed software which was used by US cell phone manufacturer. A phone used to drift data that transmitted to the foreign server every 72 hours and it was a design flaw. Last year we saw solar brush which almost shook the industry after news from solar wind. In order to spread malware attacker compromised the company's build system and installed back door to access customer network. In 2012 a research found out counterfeit software was pre-installed on 20% of devices they have tested. Malware was installed in new laptops computers after they were shipped from factory to distributor and decilers. This was achieved by compromising distribution pipeline from the supply chain. In April this year threat actor had modified its bash uploader script of code curve which was online platform for hosting testing reports. In last year in the same solar wind attack thousands of public and private networks were compromised when threat actor used a routine update to deliver a malicious actor by compromising maintenance servers. And the last one is classic example of PII gathering where researchers bought hard drive and phones from trash and was able to get sensitive information like social security number, credit card and whatnot. Second tip I have is secure your development environment. If you have CI CD and build system instances running validate if those have unpaired security update to apply keep track of role based access control and identity access management permissions. And as we discussed most of attacks are inherited from dependencies so scanning and keeping track of known vulnerabilities early on is stepping stone. And last but not least it is very important to monitor and audit your infrastructure regularly. This can help to identify sophisticated threat and eliminate possibility of potential exploits. I found this funny Spider-Man meme on internet which you may refer to talk about current situation of the industry and which is certainly true. If you are a vendor or supplier you are probably consuming some of the products which might be I produce and I am probably using gears. So we are in this together and it's way more than we think it is. But we don't act like we are and that's how we need to change. So time to think on this more holistically. Thank you. Alright so that was some useful information about supply chain and I would like to invite this speaker on this stage. So Heyman, if you can see this speaker, if you can allow the access. Perfect. Perfect. Yes, we can perfectly hear you. So we have some amazing questions for you from our audience. So the first question is here, are there open source tools to manage supply chain life cycle? Yeah, I mean first thing we have to know is there is no standard way to remediate this type of attacks. These are not straightforward where you have buffer flow and you are fixing that buffer flow with address sanitizers. But there are ways, for example let's say there is repository management software like GitHub and GitLab. They have feature to configure security and analysis which will tell you dependency graph and stuff like this. There is second tool which I remember generating software block materials like TERN which I think made by CNCF folks which contribute to the CNCF Linux foundation. So this tool will help you to generate block material and there are tools, commercial tools for Opsys but I do not want to endorse that. So yeah, those are just the tools which I know. Thank you so much. So next question by Andrew is I work around containers and open shift. Is there any resources for securing supply chain for that? Good question. Yeah, yeah this is a good question. So that is first thing which comes to my mind is the NIST supply chain framework which talks about supply chain management which I also discuss in the slide. So that is the first way to go. And since you mentioned you work around open shift there is one good resource which is shared by Linux foundation only, shared by the CNCF. If you type CNCF supply chain management practices I think they released their version one practices guideline last year. So that is also a good point to start with anything related to container. So yeah those are the two resources which I want to share. Perfect. So another question from Ashutosh is he wanted to know about software consumers are they really truly helpless or do they have more control over this process than they think yeah. Yeah I think they do more have control. Mantra here is as I pointed out in the last slide trust but verify. So there is trust between publisher and user but obviously when you purchase so you kind of anticipate they have the base intention but still we do need to hold our vendors accountable so that even if we have software compromise the damage could possibly go to the environment which is already minimalized. So yeah. Great and Adil wants to know if like what Red Hat is doing about supply chain attacks and like whom should we contact in Red Hat for supply chain questions if you know anything about it. Yeah that's a good question because I have something in mind. So we are working internally to build a team dedicated to supply chain security we are committed to our customer security so we already have supply chain management there but we are trying to make sure like we also have steps towards supply chain security so that's the update I have on Red Hat site I do not have more information because we are still in progress and there are few exciting things coming up and to answer your second question yeah one good thing about Red Hat is like more than half of the Red Hat like you can say like 90-95% of Red Hat products are open source so whenever you find there is some there is something which need attention you can always go to the option product and open issue on the repository that is the one thing the second thing which you can do is you can open FAOR using Red Hat's support subscription if you are using Red Hat's enterprise product you can always open a support ticket so that our subject matter experts can help you out and last, not but least if you believe you have found security vulnerability you can send that report to Red Hat product security team at sec alert at redhat.com and may I surely help you there thank you so much for answering all the questions so neatly and that was really very informative presentation I must say so thank you all and the next session of the track will start in 2 minutes so stay tuned and once again thank you so much to our speaker Yathni Valak bye bye thank you Yathni