 Welcome back to Red Hat Summit here in Boston, Massachusetts. I'm Paul Gillan with my co-host Rob Stretch A. And big announcement at Red Hat Summit this week. The trusted software supply chain, Red Hat taking all of its expertise in building software and productizing it. And something we'd like to talk more with Vincent Dainan about. Vincent is vice president of Red Hat Product Security. Welcome, thanks for joining us on theCUBE. Thanks for having me on again. So why did Red Hat decide to get into the software supply chain business? I mean, fundamentally we had to scratch our own itch first, right, so as we were looking at modernizing the way that we were building and developing software, and looking at some of the different regulations and compliance frameworks, and you have things like the NIST SSDF and Salsa and whatnot, we were looking at modernizing our internal build system. And that was a genesis for what the trusted application pipeline is, is we wanted to build something that we could do ourselves. We also wanted to do it in a way that we could offer it to our customers for them to use, because it's not just us that has this problem. I mean, everybody has this problem. He's building software. One of the statistics that Red Hat called out in making this announcement is software supply chain attacks growing more than 700% a year over the last two years. Why now? Why is supply chain such a problem right now? A lot of it is the, I think, the number of people who are using open source, and it's just ripe for the picking. So if you look at that number, if you break down SonaType's report, a lot of that is the software repositories, so like your NPM, PyPy, RubyGems, that's where the significant number of those attacks are coming from, because a lot of people are building their own applications that are pulling that data down from these repositories. And if you can do typoconfusion or trick somebody into downloading a package that puts them back door, or is protestware, or what have you, that's where you're going to get a lot of kind of the bang for the buck, which is what attackers are looking for, like how easy can we exploit somebody, and that's a really easy way to do it. Security considerations that are inconsistent. You mentioned typo squatting. I mean, what are some of the techniques attackers use to intercept software in the supply chain? Well, it's not so much interception. I mean, there's the, you can attack and make changes to software, right? And then your downstream users are going to take that malicious software. But things like typo squatting and confusion attacks or basically you have a, you know, foo-bar as an example for a package name, you go and throw something like foo underscore bar, looks similar, somebody might accidentally take that other one. So it's more like opportunistic type of attacks. And I believe it was just, earlier this week or maybe it was last week, there was a whole bunch. PyPy had to shut down for a couple hours, because there was such an influx of these malicious packages that are just random names or confusing. They, you know, somebody types a J instead of a K, you know, you'll see packages with those names. And it's just opportunistic kind of drive by, we call them watering hole attacks, right? They're just sitting there waiting for somebody to grab them. Yeah, it would seem that it's, like you said, open source is one. So that's, you know, it's like, why do people rob banks because that's where the money is or was? And so now, okay, how are we going to get at that? Well, let's go through the open source route. And I think a lot of these, and Red Hat funds a lot of different projects. We were just talking two weeks ago up at the open source summit, which was, you know, again, a lot of community members as well as yourselves there. I think what has really been great is to see that, hey, you know, let's take this seriously. Do you see that? I mean, obviously you had to get your own house and water before you go and productize it. Have you seen that others have just been reaching out to try to get their houses and water even in the open source community? Well, I'll start from the customer perspective first, right? We have a lot of, you know, maybe consulting engagements and whatnot. Customers just asking like, hey, how do you guys do this? Because we want to be able to do it. And we have, if we go back, I mean, 20 years is a long time of developing software. And some of the stuff that we have for say something like RHEL, kind of the older software that we have, the way that we built it was very custom to Red Hat, the way that we would build things. Not necessarily tools that we would give to a customer or even necessarily open sourced, right? They may be built of open source components, but we'll call it maybe inner source, right? Just for use within Red Hat, just the way that we build things. So we're really looking at like, how do we answer these questions that customers have to enable them to do kind of what we do, maybe at a smaller scale, because nobody's going to be building thousands and thousands of components like we do. But even if you're looking at simple applications of dozens of components, like how do you make sure that you are pulling in the legitimate packages that you want? Is it the latest version? Has it been tampered with? How do we store it? How do we build it? How do we make sure that nobody can kind of come in and attack it while we're building it, throw in a back door that wasn't present upstream, but now it's introduced in the build system because that's weak, and then it goes out to, maybe not to their customers to install or use, but to their web application that's taking payments for pizzas or flowers or whatever, right? So we have a lot of customers who are asking for that, and so there was definitely a demand for it. And we would do these kind of, maybe some onesie twosie type things like for you and your environment, we're going to do it this way. You can use these tools as basically giving them a toolbox, but no assembly. Maybe some instructions or we can help you put it together. But we wanted something that was kind of useful for us, but consistent across the board because it makes it easier for us to support customers who are largely doing the same thing. One of the technology components of the Trusted Supply Chain is a project called Sigstore that was birthed at Red Hat and is now come to fruition. Tell us about what Sigstore is and why is it potentially a game changer in this area? Sigstore does a lot of signing for attestation for different components. So when you're building it and you have a piece of software kind of moving through the supply chain, it's being signed and verified as it goes. One of the things that Red Hat's done for ages is we've signed our packages, right? As a way that a customer can know that this package that I received was something that Red Hat intended to give to me, right? And that gives it a certain level of authenticity. There's no guarantees that it's free of vulnerabilities or anything like that, but it basically says you got it from us and not from some nefarious person over here. Sigstore is very similar in that respect, but it's broader in that anybody can use it for their own software development, for their own validation. So when it comes to something like the Trusted Application Pipeline, using Sigstore in there means that you don't have to stand up your own hardware signing module or an HSM. You don't have to do any funky GPG signing or PGP signing or different MIME signing or type of technologies. You could use Sigstore for that as well. And then it really makes it easy to verify and add these things to the public ledger so that anybody can verify it, kind of whatever they want. I think a big theme that's been going on this week here has been simplicity. And how do you make it simpler for the people to operate? And I think that the, you know, less yamble as everybody would say. And I think, is that what you're being asked for? Because I think security's very complex. I mean, everything you're talking about is, there are big corporations who can handle it, but when you start to get down out of that group of people, it becomes complex very fast for them. Interestingly, both, right? So from like a security implementation side, customers, even people like myself, we want it easy, right? Easier is better, it's harder to kind of mess things up. But on the other side, the simplicity is, if you look at the way you deal with, say, vulnerability remediation, right? The simple thing is to fix everything. But it's not really simple, right? So when a customer is asking like, hey, we want every single vulnerability fixed, it sounds simple, but you're looking at, okay, somebody has to build it, somebody has to test it, somebody has to ship it, and then customers have to do their own testing, their own deployment. And if you do that for everything, it starts to become really, really expensive. The level of effort is really, really high, particularly when not all of those vulnerabilities matter. And you don't necessarily have to fix everything, like we don't fix every single bug, and a lot of these security issues are just simple bugs. So there's a weird dichotomy between, we want security configuration and controls as simple as possible, but we're also willing to accept a large amount of complexity in other areas because we don't actually understand it. And that seems to be a weird balance that we're trying to tread. One concept that's been gaining traction is a software bill of materials. Biden administration recommended it. NIST is advising companies to use it. FDA, as part of your announcement this week, you're automating the process of creating these S-bombs. How effective are they really in preventing supply chain attacks? If you look at a coin, they're one side of the coin, right? The other side is more about vulnerability information. So S-bombs are fantastic for understanding what you have in your environment, right? You'll be able to see what version, what license, what components, and all of those things are there, which is crucial to understand, if there is a vulnerability, I need to know where it is. So you kind of coalesce all of these S-bombs because you're going to have different vendors, different places, and then you have your deployment where all of these things are in place. You can use the S-bombs as a discovery mechanism, but it's not really a remediation mechanism. And it doesn't also tell you if there's vulnerabilities in any of these things. It just tells you you have all the software in all these different places. The other side of that coin is kind of the vulnerability information, which is really kind of your risk reduction part. Once you have that vulnerability information, and there's a number of different ways you can go about it, we're using CSAF with VEX, the vulnerability exploitability exchange. Those documents there, once you kind of marry those two, then you kind of see there's a vulnerability in a component, and I also know where it is in my environment. So that's the benefit of the S-bomb is the discoverability part for the software. You need the other part to give you the discoverability information for the vulnerabilities that affect it in the first place. Is there a lot going on with saying, hey, because everybody has software deployed already, and they're trying to bring these secure pipelines into their companies and start. So it becomes a, okay, do we cut all the way over to this? Do we start here? How do you counsel customers on getting started with this? I mean, unless you're a startup, you can't really start doing this, right? So there is a transition period that's required. I kind of look at it from the, you need to understand what it is that you're doing, and that's where the S-bombs are super helpful, right? You have to understand all this information. I was talking with a fellow from Sonatype a few weeks back, and people are still downloading vulnerable versions of Log4J. People are still looking for Log4J in their environment, and it's been quite a while since that was highlighted. So why are all these downloads coming from that are vulnerable, right? It's because we don't know where it is. We don't know what version it is. Some automated system just keeps pulling maybe it's a pinned version down. Without the S-bomb, you don't have that, the ability to kind of inspect your environment without a lot of cost, right? It's very resource intensive to go hunting for all of this stuff, but that's what we've had to do to date. With the S-bomb that should, in theory, it hasn't fully been practiced out yet, but in theory, it should make the investigation a lot faster and smoother, and then you'll actually be able to find all of these places where you might have vulnerable software, or even software that violates the license that you've decided we can't have software with this license. Go find it for me. An S-bomb would be able to tell you that. On-going debate in the security world over open source in general. Is open source software inherently more secure? People say, well, there's a lot of eyeballs on it, but not every open source package has a lot of eyeballs on it. What's your take on that question? I look at it as a lot of these vulnerabilities in software are just that. They're vulnerabilities in software. Proprietary open source, they suffer from the same malady, right? Human beings writing code and making mistakes. Open source is that, I don't like saying it's inherently more secure. I certainly don't like hearing people say that it's less secure, right? That's kind of my biggest beef that I have is people saying that open source is less secure because there's more vulnerabilities. Those vulnerabilities are found because the amount of open source, I mean, the sheer volume of open source is huge. The sheer number of people contributing to it is also huge, but there's also that transparency in that you can't hide any vulnerabilities, right? If a one person finds a vulnerability and talks about it, they're very easy to find, they're very easy to share, and we kind of look at it from a transparency perspective as one of those true benefits of open source. The more information you have, even if you can't grab the latest version for whatever reason, your vendor decides not to fix it because it's a low vulnerability. At least you have the information. You know that vulnerability is there. You can compensate for it or mitigate it on your own, you know, with some other workaround. But with proprietary software, you have no idea what's there. If there's a vulnerability that they decide they're not going to fix, you don't know about it, you can't compensate accordingly, right? It's an unknown unknown, at least with open source. It is a known known vulnerability. Even if it's not fixed, you can do something about it, which isn't the same for proprietary. So many dimensions to this discussion. And unfortunately, we have, we have to wrap. But Vincent Dainan, fascinating discussion. Thanks so much for joining us here on theCUBE. Thank you for having me. We'll be back after a short break. This is Paul Gillan with Rob Stretch A at Red Hat Summit 2023 in Boston.