 Hello, everyone. Good morning, afternoon, good evening, depending on where you are in the world. On behalf of my colleagues here at CloudSmith and ChainGuard, I want to welcome you to our discussion entitled Everything You Wanted to Know about securing the software supply chain, but we're afraid to ask. So this is going to be an open discussion, a conversation, so feel free to hop in with questions if you want to. We do have a Q&A section specifically in Zoom for that. So it helps if you throw your questions in there because it will, definitely, everybody will get the benefit of the answer if we answer it on the call. In addition to that, of course, we're all here, and I want to welcome you, but I think over the last few years, software supply chain security specifically has become top of mind as one of these concerns for almost every organization that's producing and consuming software. So we as a tech community have to try and understand this, but if we're going to wrap our head around it, we really need to have conversations like these to improve our collective understanding of where we came from and where we're going. So hopefully by the end of this talk, well, we'll leave with not just a bunch of fear, but hopefully a little hope as well about the future of supply chain security and software. So let's go through the agenda real quick here. I will first off just do a quick little speaker introduction for all of us. Everybody will do their own direction, and then we'll give you a little background on what we do as organizations with CloudSmith and ChainGuard as well. And then we'll get into the meat of the talk where we're talking about some of the things that we want to bring up today as conversations. And again, this will be open, sort of like a fireside chat. We're just discussing stuff. And so feel free to ask questions as well. And all my speakers feel free to hop in at any point. We'll be touching on points like what got us here, specifically when we talk about previous tipping point events, attacks and other things that precipitated our big concern about the software supply chain. And then we'll touch on why this is such a hard problem to solve because there are definitely some challenges there. Each attack and each different compromise or issue that comes up really has different origins and different techniques and methods. So it can be quite tricky, sort of a moving target. And then we'll talk about specifically what's being done, where we'll highlight a few open source projects in this space, specifically around efforts that Dan Lawrence has done over at ChainGuard, but also it is time at Google and sort of with open source. So thanks for being here with us Dan. And then also we'll specifically talk a little bit at the end about what are we doing as organizations like CloudSmith and ChainGuard and also how you as an organization can help and contribute to the same efforts. Sound good? Okay, well without further ado, let's introduce ourselves a little bit. So I'm Adil Lagari. I'm a solution architect manager here at CloudSmith. I'm a sysadmin turned solution architect turned manager. I spent the better part of two decades here wrangling Windows boxes and Linux boxes. So over the past five or so years, I've become an active member of the PowerShell and DevOps automation community. And I'm a speaker and author and super passionate about PowerShell DevOps and automation. Over to you, Paddy. Hey everyone, I'm Paddy Kerri. I'm an engineer here at CloudSmith. And I've been an engineer for a long time starting in ad tech and moving into e-commerce and finance and eventually found myself deep in package manager land, which is how I ended up here at CloudSmith. And I've worked on everything here from core package management functionality, securing supply chains. And now our platform team, part of the responsibility there is securing our own supply chain. So hopefully that's some relevant experience for what we're going to talk about today. Thanks, Paddy. And yes, hi everyone. Yes, I'm Dan McKinney and I work in developer relations at CloudSmith and I've had a somewhat torturous path through to developer relations. So I've worked in systems engineering and manufacturing actually. So another type of supply chain as well. And yes, it's really my role to sort of engage with the community and educate the community why this is why this is a perfect forum to do this today. I'm also an absolutely crazy retro computing and video game collector. And I work as a DJ at the weekends. So that's a regular spot every Friday, Saturday night, every Friday and Saturday night, which is very different from supply chains. But yes, it brings a little bit of color to the proceedings. So nice. Thanks, Dan. Dan. Second Dan today. My name is Dan Lawrence. And yeah, currently I'm a founder and CEO at a new startup called ChainGuard. But before that I was at Google for about nine years where I got to spend the last five or six working almost completely in open source and started worrying about open source supply chains. So I've been involved in a bunch of different open source projects there and continue to be in products like Sigstore and the Salsa projects or SLSA. So we have to talk about some of those later today. Absolutely, Dan. Thank you. And thank you for on behalf of all of us. Thank you for your efforts with open source and keeping things open for us. Awesome. Okay. Well, let's talk a little bit about each other's organizations about CloudSmith. I'm going to let Dan McKinney complete this one. Yeah. Thanks, Adele. So CloudSmith is a cloud-native package management solution. So given how essential software artifacts and packages in general are to a software supply chain, it's very high on our agenda to secure that supply chain. So we have been around a lot longer than a lot of people think if they're only hearing of CloudSmith. We've been around now. This will be our sixth year. So it's a very exciting time for us because as you said earlier, Adele, you know, secure software supply chains are really at the forefront of people's minds at the moment. And it's so close to our hearts as well at CloudSmith. So it's a really exciting time to be part of CloudSmith. So that's who we are. I love this. I can Dan again today. Yeah. So yeah, we've not been around for very long. We've been around for about five months at ChainGuard here. We got started in October of 2021 and we were trying to build a bunch of tooling based in open source to help people make secure, make their software supply chain secure by default. We've seen time and time again that if you make developers do things differently, if you make them change the way they work, and then security is always an afterthought. It's not what you're thinking about when you're in that development loop. But we believe that if we do this the right way and if we take a couple steps back and think about the problem a little bit, we can actually make secure software development a better and easier way and have it work that way by default. So that's what we're focused on doing. We're still just getting started. So thanks a lot for having us here and giving me some time to talk to everybody. Yeah, for sure, Dan. And I must say like you're being very humble. You basically, you guys are slowly picking up the who's who of Kubernetes and other self-signed supply chain projects. So good for you over there. We eagerly await your announcements soon. All right, so let's hop in a little bit. And so now what I'm going to do is I'm going to take the slides off the screen because I think this is a discussion. So let's open it up a little bit and talk about what got us here. And when we specifically say what got us here, of course the scary elephant in the room is all the attacks and everything that went on. But I think there are a couple of events that will probably highlight that have helped shape the industry as far as software supply chain so much. So and I think one of the first ones, of course, is the SolarWinds attack, right? So that was December 2020. SolarWinds, if you don't know, you know, it was a very prominent active product company. They make software for network and infrastructure management. And so in December 2020, they announced that their Orion software had been compromised as a result of a software supply chain attack. And so this is funny because this attack very much was managed to burn the word software supply chain into our collective consciousness, right? Because everybody now knows that word, that phrase specifically. So because this malicious code was inserted into the Orion software early enough in that supply chain, it remained undetected, right? So what happened was it got pushed out with the other updates to the Orion software and slowly got out to consumers. And, you know, we're not talking about small consumers here. We're talking about big folks like Microsoft and Intel, etc. So little bit, they know that they were installing those malicious payloads along with the monthly updates. And so that's really serving the highlight part of our problem that these pipelines, we're largely up into this point as developers, we're largely ignoring these parts of the pipeline. And this is sort of a tipping point event for that sort of alertness. Now, now, Dan Lawrence, I know I've heard you talk about this before, but I think you were still in Google working on some open source projects at this point, right? And this is sort of one of those moments, those tipping points? Yeah. So this is kind of right after I switched from Google Cloud to Google's kind of core security group. So I'd finally been successful in convincing some people at Google that this was a big enough risk to take seriously as like an existential issue for the internet and for Google. And it was still kind of slow, though, until, you know, the kind of news broke about this attack and the amount of people affected and everything. So almost overnight we went from why does no one listening to us to why haven't we been doing this for years? And that was quickly followed up with some of the stuff I think you're going to chat about. And I'm in it with, you know, the U.S. government looking around too and executive orders and new regulations and new compliance and everything. So yeah, it's becoming an increasingly hot topic over the last couple of years, unfortunately, because of attacks and not because we were, as an industry, ahead of the game. But it tends to work that way, unfortunately. Yeah, it's not a problem until it's everybody's problem, right? Exactly. Yeah. So that, that, yeah, no, that definitely was one of those moments. I think as for everyone in the industry, I was working at the time, I believe at the chocolate software. And it was, you know, we kept on getting that question again and again, where suddenly folks were like, Hey, you know, we need to secure our software supply chains. And I'm like, Okay, that's great. I mean, we kind of been preaching this stuff for a little bit, but now suddenly we're getting all of this detention, right? So I think, I think a lot of folks perked up their ears here and, and sort of your, your C level staff, the higher folks who are normally not as concerned, suddenly started tuning in and asking these questions. Yeah, I've seen pictures of board decks now with this on a slide from every CISO out there. So it's something everybody's trying to rapidly come up with an answer to. I know, I know, over the past year, everyone's learned the acronym SPOM and it's kind of funny because it's like, really, we've been talking about this for a while. But yeah, no, very, very true. So, so I think, yeah, leading on to that timeline wise, right? So in February 2021, I think was one of these announcements where like, where Alex Burson, who's a security researcher, showed that he was able to compromise supply chains as well. And he was luckily a white hat hacker. So we didn't have the benefit of stress there, or sorry, the negative effects of it. But of course, they've been copied and taxed since then, right? And I want to highlight this one specifically because I think he did this basically by, you know, sort of subverting the logic in the actual process of the normal supply chain, right? So as developers, a lot of times we want to use dependence, right? Obviously, we don't want to reinvent the wheel every time. And so this is specifically referred to as dependency confusion, right? And it's similar to sort of typosplotting as another type of attack where, you know, essentially what he introduced public packages into widely used public registries that shared the name of private packages for a lot of big organizations. And we're talking 35 different companies, we're talking names like Microsoft, Apple, PayPal, Tesla, you know, like it wasn't a small attack, but basically he was able to very simply do that by creating packages on public registries with similar names. So it highlighted another thing where it's the functionality of the package registries and the package management systems by default are to trust the public registries and pull those down. But, you know, it's the whole who watches the watchmen, right? At that point, we're not sure. I mean, everybody had to overnight kind of figure out, wait a minute, we have to actually change the logic of how we do this stuff, right? Dan, I think McKinney, you were mentioning something. Yeah, yeah. I mean, I think the real problem until there was that, I mean, essentially this was a problem with like configuration, like the way people had configured their package manager clients to, you know, be able to hit public sources like in preference to their private internal repositories. But there was no red flags, like not, you know, it went ahead. It wasn't that it was, you know, in contrast to Solar Wins, which was a breach of their build system. This was normally running build systems that just were, you know, configured incorrectly, but didn't shout about it when something unexpected happened. So because pipelines are so automated now that that's going, that's what enabled that, you know, the automation kicked in packages get pulled in, things get built. And there was no point where something shouted to everybody. And why is, why was there, why was there no point when that happened? Well, it's kind of like Dan said earlier that it's not the default and it's not easy enough. It's not easy enough to have that default stance of, of the, you know, something unexpected or there's a problem or there's something we suspect may be a problem and maybe we should have eyes on this, the heavy automation sort of drove that. And look, I'm a huge fan of automation. I mean, obviously automation is what enables, you know, companies to move with a real, you know, high velocity and deliver software quickly. And that's fantastic as an industry, but it has to come with checks and balances because, you know, automation run amuck will lead to these kinds of problems. And that's, that's I think a core difference between dependency confusion and the Solar Wins attack. Both were supply chain breaches, right? And both resulted in malicious or potentially malicious software getting out into the hands of users. But there's a subtle difference there, you know, so, but yeah, the lack of visibility on that process and the lack of alerting and automatic red flags, there is what really caused the problem. Okay, so this is great because I'm going to pick on you a little bit. So I love to. So we have a Q&A question. So while you're at this talking about these specific software supply chain, Musa asks, starting from the basics, can you please define what is meant by software supply chain? So defining our terms, of course, where everybody wants to be on the same table. Okay, and I'll have a go with that then. Yes, I'll have a go with that. And well, look, I'm a big fan of doing this as well. So apologies to everybody, but I love to draw analogies, right? So and I worked for five years in manufacturing. There was a silicon wafer and fab actually. So I was very familiar with the term a bill of materials, right? And what goes into a final built product, that's very easy to see in manufacturing, right? Because you know, your suppliers, your raw materials, they come in, they go through a production line process. And at the other end comes a finished product. That's the same for software supply chains, right? You have raw materials that you ingest almost and those are the open source libraries and projects that you're building upon and you're using as components in your finished product. And unlike the manufacturing industry, the software industry, it sort of hasn't reached that level of maturity yet the manufacturing has. So for manufacturing, for raw materials and for supplies that you bring in, there are certificates of conformance. Everything goes through a quality analysis. You look at a company like Apple, they audit their suppliers heavily, right back to the mine that the the rare earth metals were mined out of and they have all of that traceability on the supply chain from raw materials right through to the finished product that you hold in your hand. And as an industry, the software industry kind of needs to get there. Now, a problem with that is that it depends so heavily on open source. Everybody uses open source and open source is fantastic. We're a great supporter of open source at CloudSmith. So we don't have those centralized industry bodies that do quality control and conformance checks like manufacturing does, which is exactly why we need what Dan said. For the software industry, we need zero trust because we don't have those centralized bodies and organizations to prove provenance of an open source library or to give that traceability all the way back to the source. So that's what a soft source supply chain is. It's everything that goes into making up your product and of course, dependency of dependencies of dependencies. It's just like the raw materials in a physical supply chain where a screw can be traced back all the way through the metal that was used to make it all the way back to the mine that the metal came from. And that's like the chain of dependencies and so on. Exactly. And we'll save some of the the problems with it for the later parts of this conversation because there are natural points that we can discuss that too. So but I mean like just summarized in a lot of what you're saying, which was awesome is the idea that, I mean, it's kind of around this bill of materials that people keep talking about software bill of materials as bomb stands for software bill of materials. And you're not you're not only trying to understand where code comes from, but it's sort of each step in that process. It's like what dependencies did I use, what image was this built on, what factory, if you want to say that, what repository was used, what commit specifically added this code and all these sorts of details. Kind of like, I know Adolfo from J&Guard says that sometimes they said for that packing slip on the side of the UPS package that details, this was checked by this person at this point went through these waypoints and came from this factory and all those details. So that's kind of where we're getting at as a software supply chain for. And a bill as well, even the environments, even the tooling that was used to build an artifact, that's part of that as well. It's the complete bill of materials, not just the raw materials themselves, but the environment that they went through, the versions of the tunes, even down to the build host that was used for a particular pipeline run. It's all relevant. Absolutely. So, okay, the last date I'm going to highlight real quick is May 12th, 2021. And the office of the president, of course, on this date issued executive order 1-4-0-2-8. There won't be a quiz later, so you don't need to know. But it was on improving the nation's cyber security specifically. And so this really highlighted that this was a real issue that the government was willing to throw its weight behind. And they really talked about removing barriers to the sharing of threat information and modernizing some of these practices in cybersecurity that had kind of fallen by the wayside. But not only that, actually establishing government run cyber safety review boards and details like that. So I think this was a big point for the industry for sure, Dan Lawrence. And I think you definitely were involved in some of this or your projects were referred to in some of this, right? Yeah, yeah, no, it was a kind of fast couple months from the attack to kind of the analysis mitigations covered from the breach earlier or later the year before out to getting this executive order out. The executive order, though, was kind of just the start of that road to the executive order, a directive on how to produce software. It was a directive to a bunch of other agencies within the government to start collecting industry feedback and use that industry feedback to shape some of recommendations that then get applied and purchasing decisions and that kind of thing. So we're still seeing kind of that process unfold. And actually two or three weeks ago, NIST, the National Institute for Standards and Biology in the US, finally published the not final version, they're going to keep iterating on it, but the, you know, a final version of the SSDF or the Secure Software Development Framework, which is really a great read, if you haven't seen it yet. But this is kind of the culmination of talking to people in the industry, people in government, people in the public sector, private sector, and soliciting what the best practices were today and then collating all of those into a checklist that you can run through and do an almost like a self-assessment for your organization. I'd highly recommend taking a look through it if you haven't yet. It's broken down into a couple different sessions. We've got something on the ChainGuard blog about it, too. But this is basically, you know, the industry set of best practices here for both consuming secure dependencies and then also operating build systems securely in your supply chain. There's some stuff in there that I think will surprise most people because, you know, there's a lot of stuff which is basics, you know, brush your teeth style hygiene that everybody knows they need to do. And then there's a couple things in there that are kind of pushing the envelope a bit. And I think we're going to catch people by surprise later this year if you don't get a handle on it now. What, do you want to mention any of those? Oh, man. Yeah, well, I mean, there's one funny one that I'm curious to see how it plays out. But it requires, you know, vendors to try to comply with the SSDF to also apply that recursively to their vendors, which makes perfect sense, because this is a supply chain. But yeah, it's going to be very slow until the first person tries to meet this. And then all of a sudden, it's going to, you know, work across the industry, because we're all in a supply chain together, most of us are somewhere in the middle, right? So it's not long before it starts creeping up on people. That's the tough part, right? Is it's provenance all the way down, right? You really need to get folks to buy into it. So I think if once large organizations start hopping on that bandwagon, I think the industry, you know, just by sheer momentum alone, it's going to be forced to to comply to a lot of this stuff. So as organizations, what you're going to see is if you're selling products and software, you're going to start seeing more and more security assessment, you're going to start seeing more and more from your customers and consumers wanting details about where, you know, the entire chain of trust all the way down to your dependency, even if they are open source, right? So I think not necessarily having to reveal the entire sausage and how it's made, but at least all of the different components that went into it. One of my other favorite parts, though, not, you know, easy or hard, just I love seeing it called out and you reminded me just talking about open source. The guidance kind of doesn't require, but it recommends reusing well tested, well developed components, including open source. Exactly. Right. We're talking about open source security. There's a lot of misconceptions that open source is worse than closed source or anything like that. And, you know, it's not open source software. It's just software. It's the same as closed source software. All software has bugs. And I think by and large, most people agree that, you know, in general, open source software is or can be more secure than proprietary software. It doesn't mean every single library on GitHub is, but I think on average, the stuff you're using, you're in a much better spot if you're using well tested, well reviewed, well maintained open source libraries. And so the business framework actually calls that out and recommends it as a best practice rather than everybody trying to rip all the open source out of their supply chain. That would be a mess and set things back even farther. So you bring up a really great point, Dan. And this is one of my little bug bears, too, is I keep hearing this in the industry. A lot of folks, and I've heard organizations even on sales calls or other places talk about you can be as secure as you want as an organization, but if you're consuming open source packages, you could be vulnerable. And I just feel like that's a little bit fear mongering, right? At the end of the day, it's really that the open source code is in every supply chain. Let's be real, right? There's pieces of it everywhere. And that's the tried and tested one. So why aren't we backing that as an industry and supporting those efforts more so? Because that's where we all, you know, collectively, as humans benefit a little bit more from the improvements and that, right, Dan? Dan and Dan? Absolutely. Absolutely. I couldn't agree more, really. In fact, I do think that frequently, you know, open source software effectively has more eyes on, you know, in the process of its development, you know, by its very nature, it's open to anyone to sort of audit and check. So, yeah, no, I totally agree with that. And I know even, well, Patty probably knows better than me, but, you know, within CloudSmith, we, of course, use open source software to build CloudSmith and we open source as much of our own stuff as we can and we contribute where we can. So, you know, I think that's something that we would definitely agree with. Patty, do you want to say anything better? I guess CloudSmith, like a lot of other companies, wouldn't exist for open source. You know, being able to trust that software, being able to trust the things we're building on top of is crucial. You know, it's also crucial for our customers. Our customers need to be able to trust that our supply chain is secure in the same way we need to be able to trust that the supply chains of our providers, whether it's cloud providers and even their hardware suppliers, it has to be secure the whole way down. And I guess we'll get there eventually, right? You know, it's going on a long, hard road to get there. But, you know, so long as we're in the right direction. That's exactly right. I mean, this is a journey, right? And we're all sort of on together and, you know, come reunified by our use of open source. And there is no, there is no like quick fix, you know, one size fits all solution here, but it's about figuring out what the bits of that solution look like, you know, and nobody can do it on their own. And I think that's where, then, Lawrence, you also mentioned, you know, this is security by default, right? There's no quick wins, but we need to make it easier and simpler for folks to be able to consume security as a part of the supply chain, right? We don't want it to be like this thing, like where it's like an afterthought or so. So for those speaking of journeys, for those who's following along with the actual agenda, we are now into the why is this a hard problem to solve, right? So I think a lot of this is, it comes from the challenge, like Kenny, you mentioned before about software supply chains and pipelines, right? We have complex pipelines now. So the more that we bring automation into it, which we want, we are also, you know, introducing probably possibly a little less eyes on some of our code with the automations and pipelines that kick off, and also some complex dependency chains, right? We depend on a software, which depends on software, depends on software all the way down. So that's where I think this can be a tricky, tricky problem, right? Yeah, absolutely can be. And you know, we've all seen this, you can pull in one dependency that maybe has a hundred dependencies of its own, maybe more sometimes it gets on pretty ridiculous at stages. And that's only one part of it really, you know, it's a hard problem on many fronts, it really is. Yeah, and I think on top of that, like, in addition to that, I mean, of course, every attack that we're about, right? SolarWinds to stay alive for sure, all of these ones that are out. So I think it was, was it the DevOps security report that mentioned that I think attacks have been up this past year by 600%. So there's definitely a lot of exploitation going out there in the software supply chain. And, you know, historically, Dan Lawrence, a lot of the stuff that's been put out there, I mean, I know I've worked on code before where I've tried to sign things, and different packages or even different pieces of code in PowerShell where it's historically very difficult to sign, get a signing key, or be able to sign anything, you know, and you either have to pay a lot of money or try and try and jump through a lot of hoops with the node servers and, you know, like signing mechanisms. So I think it's safe to say that these mechanisms used to be a little bit clunky, right? Yeah, absolutely. I think, you know, not for any real hard reason or anything, there just wasn't demand for it really, I think. And it takes, it's like a signing, it's like this, if you sign a package in the forest and nobody verified it, did you really sign it at all? So it's complicated to get started. Like, if nobody's asking you for signatures or anything like that, then you're not going to do them. And if nobody is used to asking, then nobody's going to provide them. And you kind of get stuck in the cycle. So you have to kind of break out of that somehow. And I think, yeah, the attacks we've seen have kind of proven the need for it. And so we're seeing some pretty rapid adoption in SigStore and a lot of these other projects. Just because of that now, everybody, it's aware of people are aware of it. Software producers are trying to do the right thing. And software consumers are trying to do the right thing now too. So it's kind of a great time to be in the field, trying to build tools to make that easier. Yeah, no, I agree. It's funny because I liken it to, and this is probably a poor analogy, but it was sort of like a Rube Goldberg game, sort of contraption to try and to sign things. You had to make sure a server was available to make sure you could sign things. And historically with co-signing, I know back when we were at Charlie, it's sort of you have to pay for some of these signing certificates. So I think this is good because the conversation is just naturally flowing this way. But here, let's talk about what's being done. Let's talk about some of these big open source projects. So I think a natural starting point to this is SigStore. Do you want to reference that, Dan? Yeah, sure. I can tell a quick story there. We're studying kind of co-signing and why it wasn't being adopted in different ecosystems for a while within the OpenSSF and at Google. And we saw kind of a similar thing play out across all the ecosystems where the tooling wasn't great, people proposed adding signatures to Python or NPM or Ruby or something like that. And it's just really complex when you start thinking about key distribution and who's going to verify what and what does the signature even mean. And these conversations just kind of stalled out because there wasn't much motivation, I think. But then on the flip side, there were all these kind of walled gardens and other commercial ecosystems were signing was happening all the time. If you publish an Android app or Windows drivers or something like that, you need to sign them. And so in those worlds, 100% of everything you sign is because you had to. And the way those kind of tool chains worked and everything was with kind of traditional certificate authority hierarchies, the same way TLS works in your browser. Or open source alternatives were most commonly using something like PGP and Web of Trust, which was kind of finicky and hard to get right. And so when we started looking into the commercial code signing stuff, it just kind of gave us all these flashbacks to what the internet used to be like, you know, five, six, eight years ago before Let's Encrypt came along. If you remember trying to set up like an Apache server or something and get the, you know, TLS, you'd have the green shield next to the green shield. Yeah, the green shield in your browser rather than the red X, right? You know, you'd have to find a CAA, you'd have to pay them, you know, a couple hundred dollars, they do some verification, you'd have to send them something on official letterhead for your company, right? Always kind of meeting the scoops and then email you something. Logo used to be in the corner of web pages, like the GeoTrust or Web of Trust. Yeah, encrypted. I can't just put that there. That's not how that works. But yeah. And so, yeah, you pay and they'd email you the certificate and you copied into the right directory and rename it and run these arcane commands. And if you got it all right, then you get the green shield and then you probably forgot all about it and a year later, the certificate expired and you'd have to remember how to do the whole thing again. Because everybody was complaining, right? And you forgot. Yeah. And then Let's Encrypt came along and kind of changed it completely, right? They set up a free certificate authority as a public betterment group and got accepted by the browsers and did all of this by automating everything, right? So instead of certificates being valid for a year with these kind of intrusive checks on your identity, they made a whole new standard called Acme where you could prove that you had control of a domain and, you know, the certificates were only good for a couple of weeks. So if something bad happened, it would automatically expire and they do these continuous checks. They got this going and then within a couple of years, it went from like 50% adoption of TLS out to like 99% of the web now is encrypted. And so they show that, yeah, people do really want to do the right thing if you put it in their tooling and you don't have to think about it anymore. I mean, most Kubernetes clusters, you can just put like, you know, TLS colon true somewhere in YAML and, you know, you're done. You don't have to worry about anything else anymore. So kind of a one line change versus what people had to do before. So we kind of tried to copy that model with Sigstore. So instead of having people buy things, manage keys, worry about losing them, we set up a free certificate authority. And this is all backed by transparency logs and some new technology we also copied from the web ecosystem. And now you don't have to worry about keys and you don't have to worry about, you know, validating each other's stuff in person. And you can just kind of get them and sign things on the fly and do it automatically inside of GitHub. And so we're seeing a huge amount of adoption in there, which is great because it actually gives us a way to kind of trace packages back to where they came from with cryptographic proofs and let people sign things with email addresses instead of having to, you know, keep things on smart cards and everything like field days. Yeah, for sure. I think Sigstore was a big turning point, right? And that's sort of one of the big projects that are coming out of this. And do you want to highlight coast sign at all? Because I think that's a big one for folks with Docker images. Sure. Yeah, we, you know, Sigstore supports all sorts of artifact types you can sign firm, where you can sign, you know, executables, drawers, all this stuff. But one of the first ones we started with was containers and the overall OCI ecosystem. Because yeah, there's all these language package managers. There's also been this movement to start using OCI registries to start storing all these other things with efforts like, you know, the artifacts project and stuff. So we said, hey, if we start here and more people start moving things into these registries, it's kind of a double win. Plus containers, you know, we're kind of critical. They're critical to most production infrastructure and they're kind of the last mile anyway. So the coast sign tool there is focused on container signing. But you could also use it to sign things like WASM and all these other ecosystems that have developed around storing other things in container registries. So coast sign is to github.com slash sigstore slash coast sign. You can download it. It's in most package managers and just kind of get started signing right away, whether it's on your laptop or in a github action pipeline or in whatever CICD system you happen to be using. Awesome. Actually, I'm sure in the next bit about what we are individually sort of doing to help this, I'm sure Pali has something to say about coast sign because it's on our agenda. So yeah, but no, I love the analogy there between like sigstore and let's encrypt because I'm old enough as well to remember the silly dance that you had to do to get a certificate and set it up. And I couldn't believe the first time I used let's encrypt, how simple it was. I mean, it was awesome. The DNS verification and the next thing everything was working. And I remember having a thought at that point that yeah, this is the way it should be. It should be this straightforward and that's where we need to get to with artifacts and packages. I mean, it absolutely is. And look, we already support on CloudSmith for artifacts that are stored in the CloudSmith repository, we already support signing them with the GPG key and things like that. But as Dan said again, that's all well and good. All the artifacts are signed, but it's when people come to consume them. Are they bothering to verify? Is anybody even looking at it? That's the other side of it too. Also, is it easy to do that? That's what I mean. It's all down to ease of use. And you make it so easy that it's automatic that people, why wouldn't you do it then is the question. Because it just, you make it difficult to not do, right? Rather than difficult to do and then people have to go out of their way to not do it. And we'll talk more about that in the next bit anyway. I mean, there's no official agenda, by the way. So for the record, we can flow back and forth. So I very much think we're now in what are we doing specifically as organizations to help and what every organization can do to support these efforts. So Patty, Matt, 30, please. I just wanted to add to the previous example that the thing that always sticks out in my head of how awkward things used to be. As a teenager, I was getting into politics, figured out how to build deviant packages, figured out how to sign them with my own DPG key, but nobody trusted them. And I remember I was 17, I had to travel 90 miles to a key signing party, show my government ID to everyone else in the group. And then they had to verify that I looked like my ID. And then maybe they would trust me, maybe enough people would trust me that suddenly my signatures would be signed or verified. And it was just this horrible, long-winded process. And I remember once, someone I knew personally for three or four years to time, refused to sign my GPG key because I didn't look like my ID. Some of these open source keys were really strict in how this stuff works. You were in a posture. Yeah, you were evil Patty. Yeah, it was just a difficult process. And by the time it was all done, I had no appetite to continue doing it anymore. Exactly. Yeah, exactly. That's a fantastic story, by the way. And you know, I've worked with Patty for a couple of years. I've never heard that story. That's awesome. That's a really good story. How do we know this is Patty now? Now I'm going to know. Have you got your ID? Can we see your ID, please? No, that's it. That's a great example. That's an awesome example. And I think this is also part of this whole thing about how not only us specifically as Cloudsmith and Shingar, but how all organizations can help these efforts. And I think a big part of this is now with SIGStore and Cosine gained popularity, Dan. And a lot of these languages now and package registries coming on board as well. We talked about Shopify's great open source efforts with RubyChance and stuff the work they're doing on there. I'd really want to give kudos to those folks because they're leading the charge. And you know, PyPy, Python folks, and then Maven Central over there. I mean, there's just a lot of like, I think there's, we've hit that, we've hit that like sort of, I keep referring to it as a tipping point, but it's sort of like momentum, right? We're pushing forward. Feels like a ball rolling down a hill and it's just going too fast to stop now. Yeah, now it's, now we've got the momentum. And the thing is that that is what we need as well. We need to have that momentum because of course, the last thing that anybody wants here is for these, and Dan mentioned it earlier with wall gardens and things. The last thing we want is for, you know, more proprietary stuff to come along. It will never get by in an adoption. We need the projects to be open, the tooling to be open. And of course, as vendors, I mean, CloudSmith is a vendor, a software vendor, we provide a service. But what we need to do is support those projects, integrate them with our offering. It's fine to be a vendor that builds value on top of those things in terms of analytics and reporting and all that's superb. But the core, you know, the core of it, Sigstore and Cosine, that needs to be widely adopted and to be widely adopted, it needs to be open, right? And everybody needs to have buy into it because we're very anti-vendor lock-in at CloudSmith anyway, you know? I mean, we are. We really are. For something as important as this, it needs to be open so that all the vendors can jump on board with it. And that's what's going to give it the adoption, right? You know, that's what's going to make it that secure by default, really, you know? This is very much my soapbox share at CloudSmith, for those who know me. I think a big part of what I see is this sort of shared ecosystem of Providence, right? And so that folks like, you know, our colleagues over at Sonotype with Made in Central or, you know, the folks at Shopify with the Ruby Jam stuff, this is the stuff I love to highlight because it's vendors, you know, that are proprietary and they're doing their own stuff, but they're working in the open source spaces, right? And they're developing sort of out loud and being open and being transparent about some of these, you know, different tooling and different organization, different projects that are coming through open source. And they're putting their weight behind that, right? And I think this is a challenge, right? I think a lot of folks, like Dan said, McKinney, you can build value on top of this stuff and I understand, you know, as an organization, you're beholden to your stakeholders, you have to make money, absolutely. But I mean, I'll be honest, I'm going to call out our industry a little bit here and say, I feel like, you know, the security by default thing isn't always adopted, right? It's often an additional feature that you have to buy on top. So I want to say that our commitment here at CloudSmith and something I'm going to continue to push for internally is that the mechanisms for security exist for the free plans, for the open source plans, you know, the communities that we're supporting, that comes in by default. You don't have to pay extra for that. You shouldn't have to. And we should support that ecosystem as vendors, right? And we should support those open source projects to continue doing it. Yeah, I mean, I'll get off my soapbox. Yeah, that's quite the soapbox. I don't like it. But no, I think you're right. And I agree that, you know, as obviously as a vendor, as a software business, that is what you should do. You should build value on top. And that's fine. And in fact, that's exactly the right way to go about it. But the basics really need to be open, open source software needs to be available to everyone, auditable by everyone needs to be out there in the community. It's the only way to get real widespread adoption. And of course, the attacks that happen shone a light on it, but those alone wouldn't really push that forward, you know, so it has to be adopted by the community in general. And I know, look, actually, Paddy definitely knows more about this than I do, right? Because I don't, I'm not an engineer in CloudSmith and Paddy is. But so, you know, Paddy, what do you think sort of coming down the pipeline in CloudSmith internally is worth highlighting here? I know our engineering teams are currently working on, I guess, in broad terms, meeting users where they are. So secure defaults and everything are great. But we need to make sure that the packaging ecosystems that our users are working in, whether that's containers or enterprises using Maven repositories or Python dependencies, whatever they're doing, whatever open source tooling they're using in the rest of the ecosystem, we need to make sure that all the signatures they generate, the attestations they generate, we can work with those. And yeah, we don't want, we don't want to create a wall yard. We need to make sure that whatever the ecosystem is doing that we support. So that includes supporting S-POM standards, like the Cyclone DX standard, you know, supporting S-PDX in the future. But it's about supporting all of these open standards, making sure that whatever, whatever way our customers or our users are working to secure their supply chains, that we're there to meet them when we're ready to do that. Not every user is and that's why the UX of these tools and the UX of these processes is so important. If we're going to convince these people to make them move and secure their own supply chains, we need to help them to do that. It's just about working with all those tools. So when RubyGems implements six-door support, we need to be there to meet them with that. When PyPI implements their support for signatures, when Conda supports the update framework, we need to be there and sort of integrating those things as they happen. Yeah. And I mean, Paddy, I know that we have, you know, support for cosine with our OCI registries coming down the line. And I know that kind of stuff, but it's not, it's not just that we're going to provide this kind of stuff for users of CloudSmith, right? We, like I'm sure on the platform team, you guys are eyeing this stuff up keenly internally as well for our own, sort of our own build pipelines. Absolutely. It's something that came up earlier. If we want our customers to have secure supply chains, we're part of their supply chain. Absolutely. We need to be secure and our supply chain needs to be secure. It's turtles all the way down. So we need to be able to trust the software that we're deploying so that when, when we make attestations about the software that our users are shipping via CloudSmith, that we know that those are secure. We need to be able to trust our cloud supplier, our cloud provider supply chain. They need to be able to trust their providers. So, you know, we can really only do the work for ourselves and push others to do that work. But so, yeah, it's really just about utilizing all of those tools internally as well and exposing the parts of those processes that allow the users to verify that we're doing the right thing as well. It's one thing for us to say that we're secure. We need to be able to prove to the users and the people building on top of our platform that we are actually secure. Yeah, absolutely. And I think that that's that's a great way to sum it up. I think I will say, Patty, I think hopefully we're going to see ourselves on the Friends of Sick Store repo soon. So that's the idea. Yeah, that's the idea. Absolutely. For sure. And I don't know if Dan Lawrence, you'd like to say anything about what ChainGuard is doing in this space, but I know that there's stuff that you were going to announce very soon, right? Oh, no, I don't remember. I'll just say, I love the way you talked about all the different ecosystems because you're using the word when and I think that's awesome. Instead of if you're saying when Sick Store comes to Ruby and when it goes to Pi Pi and when the update framework goes to Kanda. I just love that like in the last year, I don't think you could have used that word. I think you would have been like a maybe if future, maybe someday type of thing. But now it really is kind of like inevitable. So it's awesome. Yeah, it's funny how that like a year ago, you couldn't even imagine these conversations happening the same way. Yeah. So I will just add one thing. I know we've only got 10 minutes left and there's a few questions in the Q&A that we're probably going to want to address. Yeah, absolutely. I just wanted to throw this in because this is something again, not again, this is only the second time I've made an analogy in this discussion, right? So I will make an analogy again with the manufacturing industry because the entire focus of this talk has been on threats to the supply chain, right? And when you say that, a lot of people think of malicious packages, impersonated packages, you know, a bad guy, right? A bad person, you know, you know, corrupting your build process, but also don't lose sight of the fact that availability is a threat to your supply chain, right? Now, it's not often that a package becomes unavailable, but it has happened, right? Everybody remember the chaos when left pad was removed from MPM, right? You know, so there's, and this is absolutely not the focus of supply chain security, but it's in our aspect. And that is that in that analogy that I used in manufacturing, what manufacturers do, they build up inventory, they have a warehouse, right? So that's something that we do think about in CloudSmith, right? CloudSmith repositories, you can pull in packages from upstreams, you can cache them so that if the upstream source disappears, you have those local cached copies of packages, your build pipelines won't fail. Now, of course- Important difference though, Dan, I will mention is private is trusted first, not public. Yes, of course, of course private is trusted first. And again, this does not absolve you from all those packages need to be verified and signed. All of that good stuff that we've been spending the time talking about. But it's just worth, it's very easy to lose sight of the fact that your availability and the reliability is just a part of a secure software supply chain as well, just the way it's the part of a secure physical supply chain. If a supplier disappears, you need to have a plan in place for that. And it's just worth thinking about. But I absolutely agree. It's secondary to the threats that are being addressed by Sexto and Cosine and vendors like us adding support for those tools. Definitely secondary to those, but it's not something I hear people talk about much or bring up in these discussions. So I just wanted to put it out there before we lost sight of it. That was all. I think in general terms though, Dan, it's a good point that there's no silver bullet here. There's no one fix. There are a hundred things here. There are a hundred different parts of the supply chain, a hundred moving parts that all need care and attention. As we work to secure our supply chain, so signing alone, great. It's not enough. No. Caching dependencies from upstream so they don't disappear. It's great. It's not enough. All of these things to work together. We need the entire community to come together and work to address all these problems or knocking them down one by one. And we'll get more secure over time, but it's going to be a journey to get there. Definitely. Couldn't agree more. Absolutely. And I think that that's part of the challenge, right? I think people look at this problem and often security folks will come up and list all the different things that you need to have in order for something to be trusted as a package or any sort of artifact that you have up in the registry. And I think the challenge there, Patty, specifically, is I feel like, yes, absolutely, there's a lot of different things and you need to secure it all the way down, but we have to start somewhere, right? And I think that that's the important thing to not lose sight of. We have to make these micro steps along the way in order to get the whole thing secure. And I think you can't let a lot of folks get a little bit overwhelmed with all the different things to do and then just throw up their hands up in the air. And I think that that's, you know, hopefully projects like Six Four Cosines and other things like that and open source efforts and getting together as a community will help us to make these things not difficult anymore. Yeah. Well, I mean, the Salsa framework is a great example of that because level one in the Salsa framework is pretty easy to achieve. It's a pretty simple step to get to that. Obviously, getting up to level four, yeah, there's a lot more you need to do, but it's giving accessibility that you can get something in place and you can feel good about it and you can feel good that you've taken the first steps on that path. So you don't jump to Salsa level four from nothing, but that would be, you know, that would be a really monumental thing to try and do. And it would put you off if you tried to do that. So you start off, as Paddy says, little steps along this journey, but we need to do it together. Nobody can run off on their own on this. We need to do it together as an industry, you know. Yeah. And so I guess this is a good, like, transition naturally into one of the questions that was up, which I think then we're starting to answer, which is, you know, Moussa heard two terms on here, cosine and salsa. And he wanted a little bit, maybe just a little bit more definition and discussion around it. We can definitely provide links in the chat if that's helpful. But if you want to spend a little bit of time touching on cosine and salsa specific. Sure. Yeah, I'm dropping the links in here right now. Excellent. There you go. The system works. Cool. Yeah. And I'll just stress that salsa is a security level. I can never remember what it stands for home and security levels for software artifacts or something like that. That's my idea yet. Yeah. And it is a framework one levels one through four today and still kind of under heavy development as we iterate and people actually try to work up their way through that ladder. Like you said, level four is pretty hard. I'm not sure if anybody's really done it yet. So as we get to those higher levels, one was designed purposefully to be easy. So people kind of get on the train and get a badge set up in the repo and show what they're doing. The way I kind of think about it is a level one is like, don't do the build on a machine under your desk or your laptop or something like that. That's basically all you have to do is like use a build system. Level two is like prove that somehow. So publish some like Providence or something like that. So people can verify that you're using a build system. Level three is like use a secure build system. Maybe not just everything will qualify today. It'd be great if all these build system operators do increase that security. But and then level four is change your whole build process to make it hermetic and not pull dependencies from the internet and you know perfectly. So you know, kind of far out there, the most secure you can possibly be. So I think there's a pretty big jump from three to four today. That's kind of my take looking at it. So we might end up, you know, adding an intermediate one or something in there shifting some things around. But yeah, it's just going to be pretty easy to understand and you get value out of each level as you go. Awesome. Yeah, no, that's a great, great summary of it. I think, and so along those lines similarly, right? It's when we talk about six store and cosine, right? Cosine specifically, I think, so six store, you can think of it kind of like your umbrella, right? The way that just being able to sign software in different ways and cosine is specific for sort of OCI container image. Right. Is that a good way to understand it? Yeah. And we can drop the links in the chat as well. So there's one last thing I'll do is there's a meeting question here, which we kind of need to get to. And I think it's kind of one of those things, which is open-ended a little bit, because I think I'd like feedback from all of you on it. And I think it definitely, this is part of the stuff that we need to work together as an organization or as a community to do, right, as all of these organizations. So Varun Brun bring this up and says, many organizations nowadays rely on distributing their software from sandbox containers or images. What could be the best practice as a customer who's consuming these kinds of distributions from the vendor? They need to contain those containers and images or sandboxes are an entry point for supply chain compromise. Any good practices for the customers to conduct the test on these binaries? So please do suggest. So this is talking along the lines of, hey, as consumers of these products for these container images, what can we be doing as a best practices when we're when we're on the consumption step of this to make sure that we're checking the boxes? I'll let Paddy and Bam take out one. Yeah. So it's more a best practice sort of conversation. But yeah, go ahead, Dan. Yeah, I can start. I mean, there's probably like 37 different threat models probably wanting to worry about here, but I break them down into like two categories for you know, two high level ones. First one is, you know, you're consuming this kind of black box thing from a vendor. You know, it might have some vulnerabilities inside of it. And if you run it, then you become vulnerable. So that's kind of like, you know, if you're consuming bad software, it doesn't matter if it's signed to you exactly where it came from. And you have all the metadata, right? It's still bad software, not bad, bad, like, you know, it might be vulnerable, might be old CVs might have been found that kind of thing. And so you want to do your scans on it, you want to make sure the code is running an environment of least privilege, all that stuff. So if it is compromised, right, it's not running as root with, you know, network access and access to your database and all that stuff that can get actually traded. The second one is yet, did this actually get to me from the distributor correctly without being tampered with those kind of things. And so that's checking the signatures, if they provide them checking the hashes, if they provide masking them to, if they don't, depending on your relationship with the vendor and making sure that, you know, an attacker in the middle can't change it, that you're grabbing it from the correct place and that hasn't been tampered with along the way, or somebody could insert some kind of targeted malware. So at a high level, those are kind of the two big categories I look at and you kind of need to treat them a little bit differently. Good answer. Bad day, do you want to add to that at all? I think that covers it. Really, it's about provenance on the other side is just old fashioned traditional ops stuff. It's like, you know, if you're using, exactly, yes, if you're using this sort of tooling, right, I think as a consumer even, you know, build the expectation with your vendors, ask them to be able to support that. Like if they don't currently have a mechanism for having an S-bomb alongside an image, right, then ask them for it. But in the feature request, because that's this sort of, you know, desire or demand from the industry is going to what's going to push a lot of vendors that way. We've heard it a lot on this side and believe me, we hear you and we're working on that. And we want to continue to support those efforts. So yeah, that's what you can do to be a part of that, but have that build that expectation, especially with your cloud vendors, that it should be deep by default. You should make the stuff available to us. And it doesn't matter how much I'm paying you for it. Just, you know, like I'm a consumer of yours, you should be giving me this stuff by default and available to everyone, right? Democratize it sort of security, right? It's the rising tide lifts all boats, things, right? That's it. It gets better for open source, it gets better for everyone. That's right. That's exactly it. Great phrase. Absolutely. So I think we are running up against time a little bit. So I want to thank my colleagues here, Patty, Dan, and Dan for all of your wonderful insights. So I look forward to some of the stuff we're doing over here at Cloud Smith and also eagerly watching what's going on at J&Guard to see what's happening there in this space. So folks, again, thank you so much for all your time and joining us on this talk here today. And this doesn't happen without all of you and all of our efforts together. So remember that and thanks for sharing it. Hopefully you've left with a little bit more information around making this big issue and sort of tough issue a little bit more accessible for everyone. Thank you for having us. Yeah, absolutely. Thanks. Thanks. Thanks all of you. So with that, I'm going to throw back the word of Larissa at Linux Foundation. Awesome. Thank you so much, Adele and Patty, and Dan and Dan for your time today. And thank you again, everyone for joining us. Just a quick reminder that this recording will be up on the Linux Foundation's YouTube page later today. So we hope you'll join us for future webinars. Thank you so much again and have a wonderful day.