 Hi everyone, sorry about that. We had some technical difficulties. Classic Mac. But welcome to this talk. Thanks for coming to Security Talk. I know that can be depressing. I'll try my best to keep it light-hearted. But today I have the light and fluffy topic of why I think the internet is broken. And basically how the modern supply chain has essentially created some of these problems. So just to start with, a little bit about me. I'm from New Zealand. If you can't pick where my accent's from, I live now in the Netherlands. I work in DevRel for a company called GitGuardian, which is based out of Paris. You can find me anywhere under the handle's Advocate Mac. I'm also the host of a security podcast. My mom says it's her favorite podcast, and everyone here should subscribe. It's called a security repo. OK, so I want to start at an obvious place, which is 1960s cigarette commercial. But this is here from a company called Lucky Strikes. And basically what happened is research came out that undisputedly proved what everyone already knew. And that was that cigarettes were bad for you. So what did the cigarette companies do? Well, instead of trying to maybe make a better product, instead of trying to improve it, make it more healthy, they came up with clever marketing slogans. One of it was it's toasted. And basically what this implied was that Lucky Strikes cigarettes were toasted. Therefore, they were healthier for you. And you have all these kind of things that they protect you against irritation. And doctors recommend this because it's toasted. But has anyone here ever tried an untoasted cigarette? Anyone had wet tobacco before? They don't exist, a brave hand. So the point is, is that saying it's toasted was purely just a distraction away from what it was. And this was repeated in a lot of their campaigns. So what does this have to do with security and the internet? Is, well, basically, we've gotten to a point now where the internet is toasted. We create a whole bunch of marketing to distract us from the fact that we're really building everything on something that is quite insecure. And I want to run through really the software supply chain, how that's moved on to the internet, and what this has done to affect security and make everything really quite vulnerable. So the first question is exactly how is what is an example, a concrete example of how we've created this toasted environment for the internet? And we call it the cloud. Like what is the cloud? The cloud is essentially a server room, a data. It's exactly how we were doing it before we invented the term the cloud. But it makes it seem more secure. It makes it seem less physical. Doesn't mean, it makes it seem like it's not a server room that people have access to, that cleaners need to come into, that people maintain, that has all your data on it. No, it's the cloud. It's a way of making the internet toasted and really kind of moving away along some of the security concerns that we have and really how we build everything. So what exactly is this cloud? And it's exactly what it was before we invented it. These are server rooms. These are computers. These are managed by other people. So if we take statements that we often see on security data sheets or on FAQs, they'll say stuff like, we securely store our data in the cloud. But what does this actually mean? It basically means we outsource security to someone else's computer. That's essentially what this is saying. There's some formatting issues, but hopefully that won't be too bad. But that's how it, this is how we've become toasted, right? This is really what we're doing. But now we're moving everything into the cloud. So yes, for a long time, we've been doing web applications. But now we're moving our development environments to the cloud. We're moving our testing environments to the cloud. And maybe you could say that you have some of this as being self-hosted, but it's still remotely accessible. It's still in the cloud. And we're all okay with just kind of pretending that this is actually physical data that we don't even know where it is most of the time. But someone else is managing it for us. So a little bit of a brief introduction to Hone and my point, and then I want to get into some more concrete examples and stop talking about cigarettes. But if we look at like the history of the internet, 1965 was when it was first conceived. 1990 was when it started to become mainstream. Most of the security protocols that we build everything on today were invented in this period. So asymmetric encryption, basically the fundamentals of how we secure everything was built in this period. There is no way that they would have been able to imagine in 1990 how we would use the internet today. Well, perhaps some people did, some visionaries, but I think generally it has exploded to a point much bigger than what we anticipated. And it was in 2008 where this started to become a problem. And this was the first time cloud started to be using mainstream. Dell tried to clay a trade market. Now they actually failed in that, so they couldn't. But this is kind of really, you could say 2008 was the point where we all started talking about the mystical cloud that we have today. All right, so why does all this matter? Why does it matter that we don't really understand what the cloud is? What is the big deal? And for me, the big deal is the software supply chain. How we build, deliver, ship, and manage our software. So we're probably familiar with the supply chain in cars, and if you're not familiar with the software supply chain, it's quite relatable in a lot of areas. You have your raw components, so these may be your dependencies, your open source packages, your frameworks, what you build everything on. You have your assembly lines, and now we have our IDEs, our source controlled, we have our testing lines, maybe we're using Jenkins, CircleCI, other pipelines to test it, and then we're doing our shipping and packaging. The reality today is that all of this is gone onto the cloud, it's all remote, it's all interconnected, and it's really quite confusing. So it's not just the fact that we're storing our applications in this cloud, we're actually putting the whole software supply chain in there, and that has been the biggest gift that we could have given malicious actors and adversaries, because this has completely changed the economic game that they're playing in terms of how to try and attack you. So if we look at Hackonomics one-on-one, so how do the bad guys operate? So we think of them as hackers in basement, still hoodies on, maybe few people here, but they're more likely today, like organizations, like malicious organizations. They have overheads, they have employees, these employees have sick days and vacation days, they really operate like businesses now. And like all businesses, they have to operate within constraints of risk reward equations in economics. So we have a victim here, it's going to cost money to try and target that victim, to do reconnaissance on them, to try and fish their employees, to try and do physical security. That's going to cost money. At some point, if their security is good enough, then that might not be worth it, so they will abandon that. And that protected a lot of, especially smaller businesses from these malicious actors, that the risk reward wasn't quite there. But now with the supply chain all moving into the cloud, with everything based on the internet, we can actually target components in that supply chain. And that means that we can end up with hundreds, potentially thousands of victims by targeting one area, which of course means that we can invest much more resources into that. And this is why security and attacks and with how we operate in the internet, with the cloud, it's getting much more sophisticated, much more advanced and much harder to defend against. So if we look at a simplified version of the software supply chain, this is from Salsa, which is a project run by Google. And I think for developers, this is pretty understandable. We have a developer here, they write some stuff, we build it, we bring in some dependencies, they get shipped out and then someone uses it. That all makes sense and we can extrapolate this to a large company. All parts of this supply chain, because they're all interconnected, because they're available in the cloud, are attackable to this. So what I wanna do is I wanna run through, I don't have time to explain how everything is attackable, but I wanna focus on a couple of points and give real world concrete examples of how these have been attacked and why. So I'm gonna focus on the source, the build and dependencies. And we'll start here with our dependencies. So we have our application. The modern application, we use a lot of dependencies. Most of our application is written by other people in the open source community. This is really powerful and has enabled us to do things beyond what we could have imagined. So we have our application and then we have our open source dependencies. That makes sense. Then those dependencies also have dependencies and those dependencies have dependencies and I'll stop there because it will get too small, but you get the idea. We could go on for a long time here. In fact, sometimes we can go back to about 30 layers of dependencies that happen. There's another type of dependencies that's really predominant now that we often neglect and that's our third party services. So these are also part of our application's dependencies. We're using Stripe for credit card processing, Algolia for search, Okta to manage our authentication. All of these are part of our application and all of these ones in particular are interconnected. They connect to each other using secrets and they really give, that the attack is lots of opportunities to target our organization. Now this looks quite clear because I can see these are dependent to this and these are dependent to that. I can manage with that but the reality looks much more like this because we have dependencies that are dependent on each other. We have our third party services that are also dependent on our open source dependencies and it's impossible to try and figure out this web of what we call dependency hell. And then we have something that can happen like this little box down here has an open source dependencies. Let's just call this log4j or log4shell. And then you get downstream effects and effects in some of your dependencies over here and then your application has this massive vulnerability that you didn't even know you had. You never picked to use that. You never added that into your list. That kind of just appeared there through your dependencies and that's the reality with dependencies. So this is a statistic. I see, I don't actually know what the actual number is. I see some or 85, 90, 95. I just picked the biggest number to be honest. You know, 95% of our code isn't our own. It's written by these dependencies that we have these open source. But we have this thing called dependency bias where we trust it because we know that as developers we have some crappy coding practices. I may skip. I have secrets.txt on my desktop but surely the guys maintaining the open source dependencies, they don't do that. They do. They do. They're human, right? It's not that they're bad. They're just human and we all make mistakes. So, and this creates a problem because attackers now understand our software more than we do because they invest their time into understanding where these vulnerabilities are. So you've probably seen this before. I hate sharing it because it's been shared to death but it's so good that I can't not. So this is here, like the modern infrastructure. You have all this stuff and then you have down the bottom here a project some random person in Nebraska has been stanklessly maintaining since 2003. This is so accurate. In fact, that person in Nebraska actually has a name. There's under a hundred of them but one of them is this guy called Fischal Solman. We like Fischal. He represents a really important part of the open source community that we have and he has been thanklessly maintaining a really popular open source package for a long time. That package, what's called UA PASA. So it's on nodes. It does something really simple. It just lets you know what devices your users are interacting with your application with, right? So basically is it mobile or what operating system? So you can see this is perfect candidate for a dependent, an open source tool because I don't wanna have to write that myself. Fischal's done a great job. Now if we're looking, we're not silly. We're gonna make sure that we have good dependencies in our application. So what do we have? We've got 10 million weekly downloads. That's pretty good. That's a lot. 56 versions, good version history and lots of other things are dependent on it. So here we have an example of, there's no way this wouldn't pass a smell test. I would use this in the heartbeat. But that doesn't mean that it's completely secure because we trust these dependencies more than our own. This is essentially kind of a toasted situation. So then what ended up happening? So this got posted on a Russian hacker forum. It's basically, hey, I have an account. It's got seven million installations per week. It's on NPN. There's no 2FA on the account. We can change the password if we want and this is what it's kind of good for. And this is how much he wants bidding in 1K increments. So you probably figured out who's account this is referring to. And this was the account for that UA passer and this is what they added in. They added in some malicious lines. Someone bought access to this account and they turned this package malicious. What was it doing? Nothing terribly crazy. It was doing some crypto mining and then they was trying to steal some credentials as well on certain types of users. So this is just one example of how these dependencies can get made malicious. So how did this actually happen? Well, Fischal was phished. Didn't have 2FA. He was able to get back the account and is maintaining it again. And I want to stress this is absolutely nothing against Fischal because without people like him, this whole thing would fall apart. But it just goes to show how vulnerable we can be to something that really is completely out of our control because this is all interconnected because it so quickly happens. And people are still using the malicious versions of this package even today. There's another one. I won't go into too much. EventStream. This one was a little bit interesting because EventStream, again, like popular, had lots of dependencies, passes a smell check. What ended up happening is what often happens is that people create these dependencies, these open source projects because it's functioning as something for them in their work. And then their work may change and it's not. So then they stop maintaining it. So then what often happens is people of the community come in to take over that, which is great. But those members of the community aren't always best intentioned. So in this case, someone came along and said, I'll take over EventStream. It was a long-term maintainer of this project. But little did we all know that that was malicious from the start. He wasn't silly though. He or she wasn't silly because they knew that if they did something to EventStream, people would find out. People would say that this looks suspicious. So what he did, he or she did, added another dependency called FlatMapStream. And this is what they turned malicious in the end. So that it was obscured by another layer. So another area here of how our dependencies can really kind of cause trouble. So I want to move on now from away from dependencies to another fun topic. I want to talk about our source code. Because it used to be that we would write our source code on our laptops, would test it in our local production, but now our IDEs are increasingly going into the cloud as well. And we've shifted this whole process, or at least we're in the process of doing cloud IDEs and doing our coding with various online AI tools that are terrifying and that's a different talk. So source code is actually everywhere. A lot of people will think that I have private source code. It's behind authentication. So I don't need to worry about if my source code has credentials or other stuff in it. Source code is extremely leaky. It has a terrible habit of becoming out in public in lots of wonderful and weird ways. Tomorrow I'm giving a talk that's gonna focus more on source code specifically and all the wacky things that it can do. But if we just talk about one element of this for the sake of this talk, I want to talk about GitHub. Because GitHub is something that you could even say to me, GitHub's not part of my software supply chain. My company uses GitLab. We do not use GitHub at all. But your developers do. So even it's part of your supply chain kind of by default, which kind of can create quite a bit of a problem because it is a blind spot in your supply chain. So some stats on GitHub. Over a billion commits were made last year. 94 million active developers last year according to GitHub and 85 million new public repositories. This is just public information. So not looking at private stuff. So huge amounts of information are out there. And with huge amounts of information comes huge amounts of opportunities for adversaries. So the company that I work for at Git Guardian, we specialize in detecting secrets in source code, amongst other things. But we decided to do a project and scan all of these billion commits. Anything that was made public last year on GitHub, we scanned it for credentials. And we'll do a quick quiz to see how many trends. But this is, how we did that is the same way attackers do it. And it's not really a supply chain attack. It's more of an abuse because we're not, when we're attacking something, we're using it in ways that are unintended. But we can use GitHub in ways that are intended but still do it for malicious purposes. So this is how many credentials we found on GitHub last year. 10 million secrets, just in one year. And these are all public. And most of these we can validate. So if you leak an AWS key, we'll check with AWS to see if it's a valid key. So these aren't just random high-entry strings. So this is a huge amount of information that is just sitting out there in the cloud that is publicly available. Here's some of the credentials that we found. So data storage, so access to databases, cloud providers. This is about two million cloud provider keys that were active that we found last year in public spaces on GitHub. So this is really quite scary. So how do adversaries, how do we actually find this? Well, we're just using GitHub in an intended way. GitHub has an API, like lots of things have APIs and anything that has an API, you can potentially abuse it. Maybe not completely misuse it, but change it away. So if you go to this address, anyone can go to this address. It will give you a public ledger of everything that's happening in GitHub right now. And I'm talking about GitHub, but this is for lots and lots of different things when we've moved our supply chain onto the internet. And what we can do is we can monitor this in real time. So here's all the events that happen on this API and we can just filter out to come at them. So the public event, this is when a private repository goes public, bringing with it all of its history. So you did something a year ago and you've committed over it. It's out there in public now. And also the push events. So as an adversary, even if GitHub's not being used by your organization, I can monitor your employees. I can try and find if some of them is actually pushed out a secret. And I can use that then against you. So this is particularly scary because this is source code that you didn't even know was out there in public that could potentially contain information about your organization. And there's plenty of attacks that happened for this. I just wanna give one example very quickly. Toyota has a mobile application called T-Connect. It's kind of important, it can start your car. And to do this, they were working with a contractor. Contractor accidentally made this code public on GitHub. I don't know if Toyota uses GitHub or not. I'm just gonna pretend that they don't so that they were completely unaware that this was happening. And anyway, they wouldn't have been aware in any case that this code was public on GitHub. Inside that code were credentials to databases, at adversaries were able to find them and then gain access to the databases that contained all the information of those customers using T-Connect. And then they were then able to launch phishing campaigns against them and sell that data on to other people. So this is how your supply chain and the fact that we have everything out there in the internet, in the cloud, can be affected when we were just talking about source code. There's lots of other ways. I said at the start, source code is really bad at staying private. These are all the companies last year that had their source code involuntarily open sourced, I like to say. And if we take just a look at one of those Twitch, they had their entire code base leaked publicly on a torrent. We scanned it and found 6,000 secrets inside there, including 194 AWS keys. How did this happen? How did their source code get leaked? For less than a day, there was a misconfiguration on the Git servers where it allowed public access. Someone found it in that time and downloaded the whole thing. So just a very simple mistake. But that's what we're dealing with now. That's the kind of security implications that come with bringing everything onto the internet. Okay, so the very last one, I'm running out of time, so I'll have to go quickly. But the build process, right? This is now moved into the internet as well, which can create problems. So I want to talk about one of these. This is CodeCov. CodeCov sits in your CI CD pipelines, continuous integration, continuous deployment. It's basically where you test your application. CodeCov was a little app that did something very specific. It tests how much code coverage you have. It tests how much of your application is being tested. So it doesn't do anything critical. Not that we might think. But what ended up happening? Well, how CodeCov's application was run was via a Docker image. Inside that Docker image, there was a leaked credential. That credential gave access to, I think it was a Microsoft storage bucket that contained code that enabled the attackers to manipulate that code. And they did something quite clever. They turned CodeCov malicious by writing one line of code that said, anytime someone uses CodeCov, I want you to take all of the environment variables, all of the code that they're using, and I want you to send them to me. And so that's what they did. So the attackers were able then to move into different organizations that were using CodeCov. Now, those organizations did nothing wrong by using CodeCov, but now they're part of a breach. This is the line of code. It was line 525 of about 2,000, so it was pretty well hidden in there and just said, send us these environment variables. I don't pick on companies very often. They have breaches, but I'll pick on one very quickly. HashiCorp was captured in this. Lots of companies were, but HashiCorp was one of them. HashiCorp creates a product called Vault. It's the best secrets manager on the market. HashiCorp invented the term secret sprawl, and their whole premise of Vault is that with Vault, you never need to put secrets in your repositories. Because of this, the attackers got into their repositories and they had to rotate a bunch of secrets that they left in there. If HashiCorp has secrets in their code repositories, no one has any hope. And CircleCI is a more recent one. This follows a very similar path. A developer's machine was compromised, a session cookie was stolen, and then basically everyone's secrets were stolen from there. So these things are bad. So there's a little taste of how that build process can be compromised. And of course, we could go on further and further. There's plenty more examples, but we're limited by time here today. So can we make the software supply chain healthy? Can we build a healthy cigarette here? And the answer is yes. So a lot of doom and gloom. But we do have options here. We do have options. We don't just have to pretend that it is toasted, but it requires some fundamental shifts in kind of how we operate. So what can we do better? Some of it, we're already doing, and I'm positive in the direction. But we need to move away from secrets. A lot of everything, an elevation that attackers get is from leaked credentials. So moving away from using API keys, going into centralized trust, using dynamic secrets, better understanding the composition of our software to things like an S-bomb is a really good start. So we actually get visibility into the dependencies so when they do get leaked. Infrastructure should default to private. One of the things that frustrates me hell is it to make things easier on developers to get started faster, everything is kind of by default open. So Amazon S3 buckets recently changed their policy, but it used to be quite hard to make it private. So we need to move away from kind of making things just arbitrarily easy and making it more secure. MFA, this is moving in the right direction. GitHub's made good grounds on this. So has NPN and other areas. And security education needs to be a pillar of developer training. Now I'm out of time, I believe I'm gonna, and I wanna take some questions, but here are some very quick things that you can do if you want to harden your supply chain. If you wanna make it more healthy and get away from just toasting it. One thing is that in a lot of supply chain attacks, you don't know that you've been attacked. They sit in there for months, literally months waiting to do it. Honey pots and honey tokens are a great way to do this. I have a workshop tomorrow on how to make honey tokens completely yourself. These are really low cost, in fact probably no cost. And they're basically just a bait for attackers to try and exploit them. So if you wanna come to that, tomorrow I think at 10, we need to get better at managing secrets. There's lots of levels to this. These are the top level, but I always say that whatever gets you out of hard coding your credentials and get, let's start there and let's move our way up. Secret scanning, so GG Shield is a tool that I'm connected to, so I'm ridiculously biased, so just bear that in mind. But there are other tools that are open source to help you detect secrets. SCA for dependency scanning. Plenty of tools out here to software composition analysis to help you analyze vulnerabilities that you have in your dependencies. So these are things that we can do. These are getting better and better and also some open source tooling to help us start on our S-bombs. And then finally, some other areas. Zero trust, add additional layers of threat, segment your environments. And lastly, change your mindset to assume that you're gonna be breached. Know that your supply chain is toasted. Know that the internet is broken in its current form and when you make that mindset, then you can actually start implementing good policies. So I've gone over time, but hopefully I wasn't talking too fast. But yeah, thank you all and I'd be happy to take some questions if you have any now. That's a good question. So I'm always about the fact that we should absolutely, open source should be a job and we should fund it. I think organizations should be funding that and there should be dedication to go into there. So I think that's a good start. Getting more of the crowd sourcing platforms, places like GitHub have made a start, but I think we should do more on that. But if we can, there's a lot of people that want to work on open source full time that can't because obviously if you live in San Francisco, then it's not gonna pay the bills. But I think we definitely can improve on that. That's a really worthy investment for security because that means that they can actually focus on doing this. So I'm probably not the answer that you were looking for, but I think that's where I'm kind of passionate about is trying to get more funding into these areas and make this a really respected job that actually can pay so that we can. The whole, it seems crazy to me that all the internet is based on this and most of these people are just completely unpaid. Like everything's so strong to it. But anyway, a rant for another day. Any other questions? Well, thank you all for coming here and perhaps I'll see you at the party later today. Thanks.