 Good morning. Really excited to be here. Thank you all for coming. I'm going to talk a little bit about securing software supply chains, and I'm not the only person who's talking about this lately. It seems to be an incredibly popular topic. If any of you were here for the security conference on Monday, there are several talks about it and it's been coming up in the news more and more. There's a lot of buzz around this. And it's not surprising giving how many attacks we've seen in the wild. So rather than try to go over all of the things that you can see in another talk or look up yourself, I'm just going to give you a little bit of information very briefly about some of the challenges that my colleagues at Shopify have run into with this, some of the lessons that we've learned and why I think this is so important. Of course, I'm most familiar with Shopify's use case, but we all share operating systems, languages, libraries, and here in particular, a container orchestrator. So this is not a zero sum game. This is a collaborative effort. I think the best time for all of this focus would have been 20 years ago before we built most of our infrastructure on a tangled web of hidden dependencies and unmeasured risk. But the second best time is right now. Now, it's been a long time since I got to come to KubeCon in person. But when I was preparing for this talk, I was thinking back to my very first KubeCon in 2017 and I knew that if I wanted to give a talk on container images, I was going to need a lot of images of containers. So I'd like to take the simple task of how I would find those images and use it to show a software engineering process. It's not the waterfall with it, don't worry. So just like anything else that I might try to do when solving a problem, I'm going to go out and I'm going to look to see if somebody has already solved this problem for me and offered the solution for free. So I plug in my UB key, I fire up my work laptop and I get ready to YOLO click some risky links. Let's filter for only top quality safe containers. And if this isn't quite what we need, we can scroll down. Oh, this looks better. It's a magic container image toolbar. Let's give it a whirl. Okay, we can just download it. There's not even a charge for it and it's got great reviews. There's no way that those could be fake. Now, if it says it needs admin privileges, that's probably fine, right? It makes sense it needs to administer all of those sweet container images. So what do you think? Would you install this? Would you download this? Have we gone too far already? It looks suspicious, right? It has these impossible claims. It has clickbait. It's got a very questionable presentation. I think this looks suspicious because it looks like other things I've seen before that turned out to be malicious. So let's rule out the toolbar. Let's forget about the container images example for a little bit. Let's look at other sources we might use to try to get software. Here's one, NPM install fix error. This might look safe. This particular package fix error, it was a Trojan. It was caught by researchers before it did a lot of damage. But you definitely wouldn't have wanted to download that when it was first reported. So what about NPM in general? No offense to NPM. It's better, you know, it's not the toolbar. It's almost always safe. But we can't really be sure that your trusted NPM package hasn't been replaced by something that's just a little less obviously grotesque than my toolbar. How about this one? This is a really common pattern. A lot of applications that I like recommend that you install or run them this way. I call this pattern curl pipe bash. You pull whatever might be at that website right now and immediately pass it to a command interpreter. Now, this might seem like it's okay. You might have even checked the contents of that site, but it could change at any time. And it's even possible if the attacker controls the server to change what's being returned depending on whether or not it's being processed by a shell. So this information would allow the attacker to come back if you use this particular path, the infamous codecove bash uploader. They would exfiltrate all of your environment variables. They could attack your systems or even impersonate you using the secrets that were stored in your build environment. If we ignore the bash uploader and just look at NPM, you can see that there are still problems with this. If there is an upstream compromise, then there's no way for you to change it as soon as you run this code on your production systems, you're compromised. So let's move on to go get. And it actually doesn't need to be go get. You could be using import at the top of your go file, or if you're using another language, this could be an include statement or something else altogether. The point is you're taking somebody else's code and you're running it in your systems. There's a reason that this is the last one because we're going from the absolute worst, the toolbar, to something that's just potentially kind of a little bit bad. Now, this particular path is a common misspelling of a more popular package. During the time that this attack was happening, if you had used this, it would have exfiltrated a bunch of your data to somebody else. But even if you pin to a hash, you vendor every dependency, you could have been compromised at the first pull or any subsequent upgrade. So we have a broader problem of trust in our dependencies. Whatever the methods we're using here, when we import these, they become part of our software supply chain. We said the toolbar was suspicious because it looked like other things we'd seen before that were malicious. But all of these looked like they could be suspicious to me. So let's zoom out a little bit. We hear a lot about supply chains nowadays. I know it's why my washing machine was three months late and there are no new cars to drive off of a lot in my city. But when a manufacturing supply chain is disrupted, the manufacturer can't just go back and decide to eliminate the supply chain and build every single part in a single factory somewhere. They need raw materials and specialized labor that may not exist in every location. It might be tempting for us to simply reject outside dependencies and say that we're going to build all of our software ourselves, but it's not economical and it's not even safer. For instance, if you try to create your own cryptographic algorithms and you are not a cryptographer, you're almost certainly going to make the problem worse. As well, communities like ours are built on sharing ideas and learning from each other. Supply chains are so fragile that disruptions even happen by accident. There is an infamous case called left pad. You may have heard of it. There's no reason to believe the developer actually wanted to hurt anyone or cause real problems. They just wanted to take their ball and go home. They deleted, among some other packages, 17 lines of code in a package called left pad. It caused 200 failures per minute until NPM restored it and broke many, many packages that had no idea they were even using this. When supply chains are so fragile, it's no wonder that they're a target for attack. A few years ago, how many of you thought that you would be defending against nation-state attacks by hackers supported by governments? Not too many. But that's what SolarWinds was. It's widely believed to have had state-sponsored backing and it was one of the most prolific attacks we've ever seen. It even got attention from the White House and got an executive order encouraging us all to do better. And in 2020, how many of you really cared about the minutia of exactly how your log messages get passed from your application to the console? I actually really did, but that's just me. Most people don't. Most CEOs and CISOs don't. But somebody was paying close enough attention to this to realize that there was a remote code execution vulnerability and a ubiquitous dependency that did exactly that. So many businesses were impacted that the cost of mitigation and resolution is completely incalculable. You might even think this is a global vulnerability, but the ingenuity helicopter on Mars was also running log for J. So this is actually now an interplanetary problem. Congratulations. There are significant risks in the complex systems that power nearly every facet of our lives, from our finances to our utilities to how we shop and play. The risk is an inherent result of using other people's software and it can be managed. But our ability to do so today is limited by a poor understanding of how to measure the aggregate risk that comes from this system based on its individual components. In fact, right now, it can be challenging to even identify what those components are. When it comes to physical supply chains, manufacturers have experience evaluating this risk. We can't just stop using third-party code, but we can and we must learn how to use it safely. So even if you're not in security, if you're a developer or engineer or sysadmin, if you work with vendors or if you support any of those people, then you can have an influence too on securing supply chains. So I'm going to very quickly go through a few of the things that Shopify is working on in this space. I hope that it'll start a conversation. You need to start with secure development. You need to know who your developers are. You need to be able to trust that the people who are saying their committing code really are who they say they are. And you need to have effective reviews to make sure that they're doing what they say they will. You need to automate your builds. And something that was new for us is codifying the build steps. We're using Intodo. It's been very successful for us and it gives you the ability to document a recipe of what is going to be built and how it's going to be built. Now, Intodo won't stop you from doing something that's wildly foolish even, but it will tell you that the build happened exactly how you said it would. You can do vulnerability scans. There's lots out there, gripes a good one. We're using Trivi. I'd love to tell you exactly how to do reproducible builds, but I don't know. Please, somebody tell me. The illustrious S-bomb, our software bill of materials. I also wanted to tell you which the best format is, but we can't figure it out. We're using Cyclone DX right now. SPDX is also very popular. And the only reason we really favored Cyclone is because we were already using Trivi for vulnerability scanning. It also generates S-bombs, and it generates them more easily in Cyclone, so we're using that. Cosine, though. This is working really well. You need a way of guaranteeing your S-bomb. Wax seals are so three centuries ago, but cosine from the open SSF standardizes signatures and attestations. It's essentially solved the problem of storage and signing for us. This is misleading at the end here. There's not just two challenges. Those are just two that I want to highlight. One is that non-package software doesn't get detected by most of these generators. We're working on an internal project called Honsil that may solve this. I hope I can tell you more about that next time. And we're still trying to come up with processes for vulnerability triage. We've got all of this information in a new source that we didn't have before. And with some kind of other events and detections, we have years of experience coming up with heuristics, allow listing things, deciding which things need attention from a human and which can just be ignored every time by the computer. So coming up with ways of identifying deep vulnerabilities and dependencies of dependencies that we may not even be affected by is a problem for us that we're hoping to solve with better processes now. Once we have trust in our build system, we need to have a secure deployment. So we're using our cloud provider's binary authorization to enforce this, but we also have an open source project called Voucher. It already gives attestations for build provenance and CI checks. And what we're adding now is an extension of this to a test that there is an S-bomb attached from the builder. And then when we pair this with binary authorization, we can see that the Voucher attestation is there and refuse to deploy anything into our production environment if it does not have an S-bomb. We also take guidance from Salsa. That is the supply chain levels for software artifacts. I think the C is silent and invisible in that acronym. Now, there are a lot of other people here in this room who know more than I do about this. So talk to them. Our opinion of this is that it's an excellent guideline, especially the first three steps. The fourth one seems to be a little bit vague and hand-wavy right now. So I'd love to get more clarification about that or hear opinions on what others are doing there. So we have a great toolkit, but it doesn't matter if you have that toolkit, if you can't reason about the risk. If you look at my magic toolbar, you can see right away that it's probably suspicious. So if you see something like that, you might want to avoid it or at least have a trustee application security team review it. But there are other things you can do with less obvious examples. You can scan or review the code if it's open source. If it's not, you can use static analysis or take a look at the guarantees from whoever is offering it. Preferably, there will be some sort of a third party audit of the code and you can use that depending on whether you trust the organization, you can decide based on that organization's past experience and there's probably a lot more of it than the developer of the application you're using. You can also look at the reputation and the response to previous incidents of the maintainers. There was a really good talk yesterday about single maintainer dependencies and that's a real problem, not just for security, but for many other things as well. So soon, I hope that we can start to look at the components that are going into the supply chain as well. We can expect more from our suppliers. We can start to ask for receipts, ask for an S-bomb and ask what your software is made of. And some of the most important things to do for securing your software supply chain have nothing to do with software supply chain security. It's just things like updating your software, keeping it up to date, using least privilege. So if you see something that is massively overprivileged or a toolbar that is asking for administrator privileges, that's probably a sign that something is amiss and once you know exactly what your build environment is supposed to be doing, it makes it much easier to use a threat detection engine. We use Falco and you can write rules for this that will allow you to detect any anomalous behavior. If your build system starts reaching out to third-party servers that it's never connected to before, that may be a sign that something is happening and you need someone testing this stuff. So you can use external pen testers or a security bug bounty program. That's what we're doing. I'd like to talk about what you can do. You can talk to a SIG or a TAG or a working group about this. I'm not affiliated with any of those organizations. I just think that they're doing good work. You can find good information there. You can find open source tools, open a PR or an issue or just use them and give feedback, follow best practices, cloud providers and sys benchmarks. The CNCF released a great white paper that has some good information about this and the last few years, they've really underscored the grave concerns that we have and the supply chains we all depend on. Cyber attacks are wreaking havoc on our digital supply chains and more and more attackers are aware of these lucrative targets and they're working harder than ever to breach them. So please ask about your organization's supply chain. Think about the software that you're using and where it comes from when you pull a dependency. We have a chance to improve this process and to protect the physical and virtual infrastructure that depends on it. So please, help to make our supply chain secure for everyone. Thank you.