 Welcome to the special CUBE program. We're going to help you better understand how to manage risk by securing your digital supply chain. And we're going to first give you a high level preview of what's happening in the market. And with me is Ben Fisher, who's emerging security technology advocate at Red Hat. Yeah, so let's set it up. What can people expect to hear from this program? So today I'm going to start off and you're going to, we're going to have a conversation about some of the business challenges related to the software supply chain. And then the next video will be with Vincent Danden, Red Hat's VP of Product Security, and Lou Kynes, our security lead from the office of the CTO. And they're going to discuss more of the security aspects of the software supply chain. Thirdly, you'll hear some newcomer, Director of Hybrid Platform Security Product Management. We'll dig into some of the practices and the technologies. And that will be followed up by Andrea Hall and Andrew Block. Andrea is a specialist solution architect and Andrew is a distinguished architect. And they're going to cover some of the changing environments. There's a lot of changing environments related to regulations and different movements in the industry and organizations. And then lastly, we'll have a video from an interview you did with Lou Kynes discussing software signing tool called SigSor and how it can improve security supply chains. Excellent, thank you for that. Okay, so Ben, people hear the term software supply chain and they may think, no, that's an interesting name. But what do we mean by the term software supply chain, Ben? So it's a loaded term. Simply, it's just, it's the supply chain, but of software. And you kind of, people get think, oh, well, I just go to a store and I buy software and it comes packaged maybe in the old days. But these days, there's some open source software. So there's repositories and collaboration upstream where a lot of communities, people in the community contribute to all these different pieces of the software. It's kind of like when you go to a store, you go to a store and you just see this one piece. But that store carries lots of different products. And for each of those products, they have relationships with different vendors and different distributors to gather all those products into a store and it's pretty complex. So there's been this kind of curation of products and softwares that's kind of come about, kind of like a warehouse club. So like you would trust a warehouse club to be kind of a place to reduce the amount of shopping you might have or you can kind of go there and you trust that they have good products that you like and they fulfill most of your needs for your family and you can go there and you can kind of get most of your shopping out of one place versus having to drive all around town to go get a bunch of different products that are carried in different stores and then having to research all those products. Warehouse clubs or make that experience very simple. And so there's been kind of an upsurge of organizations like Red Hat that just helps simplify your choices and do that curation. And the value there is in trying to not just give you everything but also curate and try to make sure that what you have is secure, make sure what you have is up to date, kind of do all these kind of nuanced things. This software supply chain is kind of complex in that there's all these extra details that you need to be kind of aware of. And it's true, you could run around town and shop for every product you would like yourself just like in a software supply chain, you could go directly and get all the pieces of software and manage them and update them and do all the work yourself. But it's a lot of work. And it is, as the word implies, it's a chain. So it's not just one relationship, it's a whole chain of relationships and having a trusted entity as kind of a proxy that you could put your faith in and knowing that they're kind of doing some of that work for you makes life a lot easier just like in the warehouse club, right? You want to kind of go one place, get all your shopping done and be satisfied. And so just like you would in traditional times, a lot of, you know, before open source came out, there was a lot of proprietary software and you'd put your trust and faith into them that they would satisfy all of your needs and they would do service you entirely. But even proprietary software now is an open source software, so it comes into the same problem. So you need to have a trusted partner basically to help you understand and give you that level of trust in the software you're buying. Makes sense. Yeah, and Red Hat plays that critical role. So let's explain why all of a sudden this topic of digital supply chain, software supply chain is taken center stage. What Ben, what should people understand about the digital supply chain and how it impacts their respective businesses? Well, the digital supply chain is really, really critical. I mean, if nothing else, I mean, to bring up the kind of the COVID analogy, right? Everything changed with COVID and things just got accelerated because we realized that the old way of doing things in person in a lot of physical ways still things down. And so when we were trying to social distance and have space, the pressure for doing everything in a digital form and to make it easier to order your groceries and have them delivered to your door or do a trunk delivery of your pizza at the local pizza shop, all this became really critical. So, yeah, it's just honestly the COVID experience really accelerated the whole need for digital transformation. I'm not trying to necessarily go there, but that was part of the supply chain because all those companies also needed to have that digital experience with all their vendors and it's kind of accelerated in that respect. So the supply chain in general is something that's gotten a lot of attention. I think people actually understand maybe have an idea what the word means in the last over the last two years with all the incidents that have happened and kind of the power of having it as a digital electronic form really, really I think has hit home for a lot of people and it's critical because now I just don't feel like the world can ever really kind of go back from that. We're also dependent upon transacting in a digital form our business is relying on, we rely on a daily checking our phones, checking websites for information, doing everything. All this is run on software, right? And it's not just software that maybe one person wrote and can maintain for the rest of their lives and do it in a perfect form. At some point the software, almost all of it is using different parts of software that are open source and out there and available and the pieces that were already developed because there's no reason to recreate the wheel and they just kind of pulled in all these little open source components and then they didn't any programming it was the programming around that to kind of make that make that usable for their particular use case and everyone's just gotten very, very comfortable with this model of pulling software what we would say from the upstream down to the downstream and consume it and utilize it themselves. It's just pervasive everywhere. It's just open source, it's kind of eating the world and that's kind of where it's come from. Right. Yeah, and this is really a major issue for folks. We're seeing all kinds of new techniques and I mean, for example, just imagine you've got dozens or even hundreds of suppliers and the bad guys are targeting a victim and they might put a piece of malware in an individual, one of the suppliers, they'll get into one of the suppliers and that's a benign piece of code but when it gets actually through the victims, you know, the target's firewall, things will start to self-form in ways that we've really not seen before and so this is really a big issue. There's a lot of talk coming from policymakers. Of course, the POTUS has issued an executive order and it's putting pressure on businesses and technology companies to improve their security posture. I wish it were as easy as a sort of a swipe of a pen but what's behind these trends, Ben? So, there's so much behind there. So I think you're alluding to something really important. So in the security world, I mean, most of the issues in the security world is due to breaches, I should say, hacks are due to kind of unpatched vulnerabilities. So the problem with that is then the answer is, well, you should patch and patch regularly and that's absolutely true. You should patch as much as you can where it's not causing business disruptions. But when you get into a supply chain or digital supply chain issue, if you have a hacker who is able to penetrate into a vendor's software and they're able to place something that gets placed into their update mechanism and then gets pushed out to all of our customers, it can be catastrophic and it will spread very fast and all the customers that are doing the right thing normally by doing constant updates will get infected. This is kind of the scary thing. Obviously, it is the right thing to do and the right thing is for those vendors to secure their environments as much as possible and do everything they can to make that as tight as possible but also as in anything, it's really, we're in a world now where it's not if you're going to be breached or, it's going to be when like everybody in the world, especially in the United States, we've all had our had breaches with our confidential information exposed, right? It's kind of the world we live in. It's what we expect. So with that understanding, it comes more about how we react to that. If your credit card number gets exposed, it's not you just don't throw your hands up in the air. You go, okay, well, I need to put a credit freeze. I need to do certain diligent actions, actions, same thing in the industry. Something happens like that an organization needs to respond properly and fast to kind of share with the industry what has happened to stop those updates from continuing to perpetrate and provide guidance on what they can do. And this is one of the wonderful things I think about the security industry is it's actually the willingness and interest to share. You'd kind of think of people in the old days wanting to hide their security secrets, hide and protect what they do to make sure that to safeguard all their assets, to safeguard the company, their data, everything. And I'm not saying that everything is exposed, but there's a more willingness to share information on threats they're seeing and collaborate on fixes and work through very difficult issues in a collaborative way. Which is, I think it's really wonderful and it plays perfectly in my mind to kind of the open source mentality of doing things together out in the open across organizations. Right, so I mean, again, it's the very things that the good behavior we're supposed to be doing with patching and what everybody's advising us to do, you have to be really careful that can actually turn around and bite you. So how should we think about trust with software? What does that even mean today, Ben? It's becoming more important than ever before because before, I'll tell you way back when I was, long time ago, when I was quite young, you just download software and you would share it with friends and copy it and there was no such thing as antivirus and everybody was fine with that and you didn't even think of an issue. And then I remember the first antivirus or viruses came out and then you went down to your local computer software store and then I had no free disc antivirus fixes for that one particular issue. So you went down and you got it and you'd patch it up and that was that and you didn't really have any worries beyond that. These days, and that's because you trusted the store and you knew there was only one issue and nobody really, it was kind of a free environment where nobody thought that anything bad would really happen. Today though, we hear in the news constantly about cyber attacks, about breaches, about just endless numbers of things that are happening ransomware. There's so many different types of attacks and it's happening in so many different ways across every industry, every geography, it's everywhere. It's really, in my mind, kind of the world's largest industry, cyber crime. And that's just a scary thing and that's because it's profitable. And so when you think of it as that, as a kind of an evil industry, if you will, it puts things into a little bit of a perspective that, okay, their motives for the most part are money and they're trying to do this. So if that's the case, then you're just trying to create enough friction that it's just not profitable for them. And so it's not about doing everything in terms of security, it's about trying to do enough of the right things to mitigate the risk for the organization. And so getting back to your point about trust, how do you trust the software that you're given? You know, if you download a piece of software, you should be thinking about where did I, where's the software being downloaded from? There's lots of sites, there's lots and lots of ways to get it. There's absolutely millions of different pieces of open source code that's out there. And just because you download from a site, you don't know who posted it, you don't know a lot of these issues. So it can be scary. And as an organization, you can choose to take on all or part of that risk by trying to understand which locations are safe. You can try to understand which code is safe and in which code you can basically feel comfortable that there's a level of trust or simply you can shift that risk over to an organization that might do some of that work for you, like kind of in any business model. Red Hat is an entity and it focuses on open source software. So, you know, you can go out and you could download any bit of open source software that Red Hat sells and you can run it today. There's nothing stopping you and that's wonderful and we're happy that you're doing that. But Red Hat plays a particular role in that we're trying to kind of curate that software, we're trying to, we're not trying to pick the best piece of software that we feel we can trust. We have a lot of people in those communities working with the people who actually work on that software. We believe in the open source model partly because not only is it collaborative and just open and transparent, but in that transparency and in that collaboration, there is a view of all the code that gets submitted. So if you can go to upstream, the right upstream article repositories and you can work with those people, you have insight into what's happening and you can pull down the pieces and the components that you feel are best that you can package into a product that you feel can provide all, meet all the needs for your particular customers and you can do that in a particular way. And then having that close proximity to those communities, you also have an idea when there's updates and patches and you get to work on those and that allows you to bring, consume those faster and bring those to your customers faster. And so this is part of the trust element is having, it's a matter of do you want to do it yourself? Like, warehouse club analogy, do you want to go to 100 stores when you do a shopping list or 20, 30 stores driving around the whole day? I don't know, I don't want to do that on my Saturday. Or do you want to go to a warehouse club? Yeah, you might pay a little bit more, there's a premium there. You have to have that warehouse club membership and then you kind of go to one store and maybe get 80% of your shopping done there and that's really good. And maybe get the 20% from a couple of other stores down the street but you're down in a matter of a few hours versus the whole day. And so I would implore you in terms of trust, you need to think about what are the critical pieces of software that you have in your organization, right? What are the critical digital processes that your organization runs? Think about them and also not just think about what the risks are around them but also think about beyond them what the risks are to the people you're trusting. So whether it's Red Hat or whether it's a particular website you might be wanting to download that open source software fund from, you need to think about, it's a whole chain of things. So you want me to know that, okay, I have access to these things, I have this information and I have these risks. Now, if I extend that out one degree further then what risks are those folks are exposed to? What do they have knowledge of? And do that and then think about it and think about and evaluate who has the most information who has the most, where are the risks and think about what makes sense for organization in terms of mitigating those risks and giving you the best ability to respond when something does happen. I think you can reduce your risk exposure with some organization that curates open source or even closed source. But also you can also kind of reduce the blast radius, I think, because if they can get you those updates faster, respond faster than you could yourself, then that's hugely valuable too. Yeah, I mean, to your point about, it's very lucrative for the hackers. I mean, the criminal algorithm is actually pretty simple. It's about ROI for them, which is how much value can they extract and what does it cost them to extract that in a numerator, denominator. And so to the extent that you can increase the cost to the hacker, there's less value to them and they will go look somewhere else. So the question is what are the parameters of trust in software that can potentially help organizations increase that denominator? And how do you define trustworthy software? What are the attributes? Yeah, so there's a lot of attributes. Yeah, I mean, I come back to kind of warehouse club analogy. It's, you know, when you kind of go to the warehouse club, it's kind of already pre-picked for various use cases, kind of, you know, here's the, you know, here's the two brands of shavers and we have it in, you know, the disposal form and then you can replace with laid form and you just have the few options there. And it's, you know, nice simple selection and you look at it and, you know, you can see the price and you kind of, you know, you know the quantity and you have certain information. And if you did want to look at more information to see they're on the package or you pull your phone and get more information. In the open source world, you know, some things you want to look at, you want to see it's transparency. So everything in open source is very transparent. If you do want to go with the closed source provider that's fine too. But, you know, you do want to have as much transparency as possible so you want to build up a good relationship. Whether it's red hat, open source or a closed source vendor, you want to have that relationship to get insight. And if it's closed source, it's more important because you need to go deeper into that relationship to understand what's happened behind that veiled curtain accountability. So, you know, whether it is software that you're getting through another organization, you want to make sure, you know, who in that organization is accountable. You want to know how they're going to be accountable, how they're going to respond. If it's upstream, right now, one thing it's coming through is what they call S-bomb software bills and material, which has details about kind of an ingredient list, if you will, of that software. And that is something that will in the future make it a little bit easier for everybody. But also, if you're going to, you know, get software yourself directly, give you an understanding of maybe who is accountable, who actually wrote the software, made the patch or submitted the last update to a branch. That type of information is very useful because you kind of need, yeah, at some point you may need to know who did this to verify if something is trustworthy, if something was intentional or not, if you see something that might be curious or questionable in some nature. And traceability, you want to be able to have that ability to understand all the changes that have been done in that software, right? Software is, you know, it's highly versioned. So there's constantly new features or updates or patches and you want to be able to go through and know what's happened there. So not only for the benefit of understanding the things that have been added, the benefits that have been added to that software, but if something happened or you were trying to make sure nothing bad happened, you'd want to make sure maybe there has been no more malicious submissions into that code stream as well. And so by tracing that, that's good. And then the whole auditability of the, we go back and look at the software and having somebody understand what might have happened by kind of digging into all the records for that particular software. I'd also say risk management because you, as an organization, you really need to know what your risks are and you need to be able to to not just do that at the macro level, but now with the software supply chain, you need to bring that down to kind of the software level and really understand, if my business relies on a particular software component like OpenSSL for VPN software and site-to-site networking and whatnot, I need to make sure that if anything happens to this piece of software, which is a critical component for me operating my business, what am I going to do about it? Do I just terminate all my VPN connections and leave my remote workers stranded and disable site-to-site networking? So my different sites don't have direct networking connections. But you have to kind of think about what are the risks and what's my plan be? How would I possibly manage things? And it feels very overwhelming when you think about the number of components. And so this is where understanding this and trying to find ways to mitigate risk and manage it make things a little bit simpler so you can really focus on things that matter. I think are important. And then incident response, which is there's going to be something that happens sometimes some piece of software that your organization has. So how are you going to respond? How are you going to even find out? How are you going to know that something happens? How are you monitoring for vulnerabilities in the software? How are you connecting with the upstream communities and being aware that something kind of is happening wrong and there's a bunch of developers scrambling to try to fix something quick because maybe there's a no next way of some software out in the wild. So having that awareness and having that ability to respond really is probably one of the most critical things here. Ben, can you give us a sense of the scope of this problem or their metrics you can share to kind of frame the issue for the audience? I mean, yeah, so in terms of open source supply chain attacks some type of software vendor actually has reports every year and they've reported that there was a 650% increase in open source supply chain attacks in the past year. And this is on top of a 430% increase the prior year. So it's scary but it's basically literally exploding in terms of the threats happening in the supply chain attacks. And these are supply chain attacks are not new but they've become quite popular and the power of the supply chain how it's as an amplifying factor is starting to get exploited really well by the attackers these days. Okay, so let's kind of go to best practice. What are businesses doing about these problems today? What should they be doing that maybe they're not doing? So with the explosion you can understand that with the spike of the supply chain attacks organizations are honestly and understandably pretty caught off guard. So while organizations have been working on in their cybersecurity programs for some time now they're mostly trying to react and by react they're reacting with maybe not the most efficient of incident response plans yet. And these attacks kind of are spreading like wildfire but as an industry it's not really helping us get ahead. So it's kind of the unfortunate place where we're at. You mentioned that there's obviously there's some guidance from POTUS and other folks in the industry and various efforts in the industry to work on improving the supply chain work on improving different components that can help make things dramatically better for the industry but they're still kind of they're still pretty early stage there's still a lot of work to be done. So as far as kind of what we can be doing as an industry obviously I'll say collaboration again because by working together whether it's with the government or in an upstream organization setting standards these things are all really important. And especially within verticals I think it's really important to kind of get together because even if you have a general standard things can vary quite a bit within the verticals but besides that outwardly looking action looking inside and trying to understand really in a sense it's that kind of a simple thing it's a business process engineering question of okay what are your critical business processes? What do those business processes rely upon? What software components are there? And then okay for those pieces of software what are they also have different components? So even if you go to whether you go to an open source provider or a closed source provider there are open source components. So understanding the software that you use understanding where you get that software from and they're seeing the components in the software and how those are digested whether it's from an organization like Red Hat that's open source or maybe a closed source provider is really important to open the relationships so you have that bidirectional trust with those organizations that are running that critical software for your organization is really important. So it's a lot more about kind of a mapping and awareness type exercise because from there you can start asking a bunch of different questions and by having engaging in conversations about those questions you're gonna learn more and more and more and that will continue to lead forward. Eventually you'll get an understanding of I have these risks and you may not necessarily know everything but along the way you'll start developing awareness of risks and then you can ask yourself along the way okay as an organization let's come together and figure out how can we how can let's look at these risks and how can we think about mitigating these within our budget, within to meet our business needs, et cetera. But it's a hard question because there's so many so much software out there our businesses are so critical on so many ways to so much software and each software has so many different components it's a pretty overbearing problem so it's not trying to scare anybody but it's just important to just take some time and think about it and understand what you have and be diligent about walking through those business processes and start with the most critical ones and keep walking forward and as you're mitigating and think about do you want to have an organization help you with these or do you want to hire people and have them invest their time into doing the work that an outside organization might do for you. Hey Ben, I've taken a lot of your time really appreciate your insights and really great to have you on, thank you. Well thank you for having me David, appreciate it. And thank you for watching theCUBE this is Dave Vellante we are the leader in enterprise technology coverage.