 Hi, this is your host, Subhan Bhatia. And today we have two guests with us, Matt Jarvis, Director of Developer Relations at Sneak and Steve Hendrick once again, Vice President of Research at the Linux Foundation. Matt, Steve, it's great to have you both on the show. Thank you. Thanks for your watch, yeah. Pleasure to be here. Yeah, and today we are going to talk about Open Source Security Report. We are supposed to do this show at Open Source Summit, but some plans changed, so we are not able to do it there, but you know, that's the beauty of, you know, online work, we can do it whenever we want to do it. So before we get started about this specific report, I would like to hear from both of you because you know, you are here, is how has the landscape for security changed in the open source work? I do recall in the early days, it was, you know, open source is ultimately secure, then it was like open source is more secure, and now we're talking about, no, we have to do a lot of things to make sure that open source remains secure. So talk about this whole evolution or changing approach. Well, I think I would start by saying that Open Source Security is a journey that pretty much every organization is on. They've been on it for a number of years and they're going to be on it for a number of years more. It's something they're going to have to attend to as part of the software development lifecycle for as long as they're developing software. Yeah, I mean, I think, you know, Mark and Jason talked about software eating the world. You know, when we really look at that, at that quote, what's happened is open source software is eating the world, right? And I mean, open source software has somewhat become a victim of its own success, you know, this ubiquity has meant that, you know, it's become a target for bad actors, you know, attacking the software supply chain in open source ecosystems has become easier than looking for zero-day vulnerabilities in end-user applications. I recall my discussion early with Sam Ramji when he was at Cloud Frontier Analytics Foundation that he was like, you know, in five or six years from now, when we talk about software, we would even mention open source because by default, software wouldn't mean open source. So that's the direction we are heading into. So that also means that a lot of concern that come with, you know, any software that is in the open source as well. And the fact is that there's nothing different between open source or propriety, bugs become part of software development process. And then when you deploy software, human error, which is configuration, mixed configuration, that also lead to a lot of vulnerabilities. So this is not going to go away. And as Steve, you said, you know, it's a journey, it's a process, not a, security is never a product, it will always remain a process. Now, let's go back to this report. So first of all, talk about what is the goal behind this report? Well, let's see. We started down this path in March of this year. And in March, I did a number of one-on-one interviews with software maintainers and core contributors, subject matter experts trying to understand, you know, what direction we should take in the standpoint of the report. We went into, we did survey design and went into the field in April. We did analysis in May and a post-production of the report in June. So then the report was actually released in the middle of June at our open source summit in North America. So as far as the objective, there were a couple of things we wanted to accomplish with this report. The first really was to understand, you know, what is the state of open source software security inside of organizations today? We also wanted to know, and essentially how bad is the problem? Because I think we all will admit there's a problem. The second thing we wanted to accomplish was to get a read on where do organizations intend to go on their open source software security journey and how fast are they going to go there? Our third issue was what are the, what are best practices that are being followed by organizations in the course of dealing with open source software security? And finally, questions about sustainability and resourcing from the standpoint of how better to manage open source software security. So quite a large set of objectives for this particular survey was ended up being quite long and painful for some of the maintainers that were kindly, kind enough to contribute to their time, but very productive from the standpoint of what it told us. Matt, if I ask you from your perspective, first of all, I mean, of course, you folks, especially in security, but talk about what was your role in this report? How you folks were involved and for how long have you, because we did talk to Steve before the other service started. So we had a discussion, but I understand the sneaks participation here. Yeah, so I mean, Sneak have been doing our own state of open source security report pretty much every year since I think 2018. And obviously the additional insight that we can bring into very, very large back in database of scanned open source projects, scanned using Sneak. So we can actually provide data and insights into the makeup of many of these open source projects in terms of how many dependencies they have both direct and indirect, the amount of vulnerabilities that are in those pieces of software and things like how long it takes to fix. So as well as co-authoring the report with Steve and the Linux Foundation, Sneak brought that data to the report in terms of some of these statistics. Right. And as Steve was talking about some of the goals or kind of is behind the report from, if I look at it from a sneaks perspective where you folks have been doing your own report for a while, how do you see this report open source security report? How does it kind of compliment the efforts you are doing or like to help the whole ecosystem in general to improve their security posture? Yeah, so I mean, as an organization we're one of the members of the OpenSSF and they're clearly very engaged in the whole kind of journey towards a better open source security. As a company, our role is to help users have better security around their use of open source. I mean, the software composition analysis piece in terms of open source dependencies is really where Sneak started from. And so it's in our interest to help the ecosystem get better at this stuff. Since you brought OpenSSF and which is also, I'm kind of happy to see it's going on there. But interesting thing is that all the Linux Foundation projects, if you look at CNCF or any other product, they themselves have their own focus on security. They have security tag, special interest groups there as well. So talk about the role of OpenSSF across the spectrum or across the... Or I can, if I can rephrase the question is, what is the role of OpenSSF that you see there? Is it going to complement the efforts that are going on? Is it going to consolidate the efforts? Well, I think realistically, OpenSSF, Open Source Security Foundation is really the premier project of the Linux Foundation when it comes to looking at all things security. That is the sole focus of that particular program. And I think Brian Bellendorf is doing a fabulous job at being able to steer that particular project in a whole host of different directions. And his advice to me was, listen, we need a survey and we need to be able to understand how significant the problem is and how we're going to get solutions to it. Perfect. And Matt, I would like to hear your opinion as well, because you're also not only a member of Linux Foundation in our projects, but you're also kind of a vendor who is actually helping real users with real problems. So how do you see emergence of OpenSSF to help the whole ecosystem? Well, so I mean, I think there's been a growing realization over the last few years that security was going to become a very big issue in Open Source supply chains. We've seen more and more not just vulnerabilities, but actually malicious attacks in package ecosystems, whether that be typosquatting, whether that be malware and all that kind of stuff. And we've seen a dramatic increase in that over the last few years. And that's kind of compounded by the fact that we are both creating a huge amount more Open Source software. We talked right at the start about almost all software development in the world is now Open Source to some extent. And the amount of software that we're creating as a civilization is increasing absolutely exponentially. And so all these things kind of form this perfect storm of needing to look at Open Source security in a much more organized way. And I think there's obviously been a focus on that from the Linux Foundation before in terms of LFX security and that kind of stuff about providing security tools to projects under the Linux Foundation umbrella. And many of the foundations under the Linux Foundation have had separate kind of projects to try to improve, provide better security tooling to developers within their projects. So I think the Open SSF has become the kind of coalescence of all of those things that have been coming together in our industry and in our ecosystems over the last four or five years. And the goals of the Open SSF are obviously far reaching in terms of providing security tooling to I think the top 500 Open Source projects as well as a range of things available to all Open Source projects like the scorecard program and things like that. So I think it's got somewhat of an inevitability about it that we were going to need a project like Open SSF. We are creating a lot of open source as you two folks talked about. But it is also how it's being used. Of course we have SaaS where you can, but that's also security, AWS Azure, Oracle, IBM, everybody has to care about that. Then we also have software that is deployed locally. It's not running on any cloud provider servers. Then we are also writing a lot of software that can go on the edge. And that's why you folks also have CD Foundation where it's because cloud is not everything. There are a lot of software that is run locally. So, and all of those are consuming open source but the challenges are also different in every space because we talk about where does the bucket stop? We talk about SREs. So can you also talk about how this changing landscape where the way we are, it's not just about creating software, but the way we are deploying and using software has also diversified, which also means the challenges are different. So if I ask you both folks know, how do you look at this as a challenge itself, not just securing the code, but the deployment phase as well? Well, I think there's, I think the real takeaway here is that security needs to be a full life cycle activity. There's a lot of focus these days on security from the standpoint of CITD, but Swap, as you were saying, we are all connected by the worldwide web. So realistically, pretty much every, and in the focus today on building modern applications is to have them be web-based. So the reality is that security with only a small number of exceptions really does need to be a full life cycle activity for application development today. It just, you know, and whether it doesn't matter whether it's API-based or not, you still need to deal with security for every single one of those components. You have to be concerned about security that are part of your application portfolio. Yeah, I absolutely agree. I mean, it's, it has become more complex to understand where your risk kind of boundary is. You know, and you certainly, you know, you certainly write in terms of certain things, you know, it's your cloud service provider responsible for that or IU. And, you know, that can become a complex line to draw about where that problem is. But, you know, there's certainly risk associated with all those things. I mean, if we look at cloud platforms, I think people are only just coming to the realization that they have a potential risk from misconfigurations, right? I mean, almost every major breach of the last, you know, five, six years has been a combination of application level vulnerability and infrastructure misconfiguration. But if we go, if we went back three or four years, almost nobody was doing security scanning on their infrastructure as code, right? And we're only just starting to see the emergence of cloud security posture management tooling, you know, which is about scanning your deployed environments. So I do think there is a double whammy here, if you like, you know, the issues in understanding the issues in consuming open source, but also these rapidly changing models of how we develop and deploy software. Steve, if I may ask you, can you share, you know, what are some of the key findings, some data, some numbers, you know, that you collected through this report? Sure. Well, first, let me give you just a couple of numbers on where we're at from the standpoint of adoption of tooling and processes to deal with open source software security. First of all, when asked if the organization has an open source software security policy in place, we had 49% of the respondents said yes, 34% said no, and 17% did not answer the question. So perhaps they weren't in a role that gave them, you know, easy, ready access to the fact that the organization may or may not have a policy. So the 49% was a little disappointing, but, you know, it's, this is a work in progress. And if you actually factor out the don't know, not sure, as it ends up being a 60, 40 split with 60% having a policy and 40% not having a policy. It turned out that smaller organizations, you know, were more likely not to have a policy as you might expect. They just don't have the resources and the cycles to be able to deal with everything. From the standpoint of where we are over all the security, we asked another question which is how secure is your open source software today? And so we had a variety of responses, all mutually exclusive and collectively exhaustive. And when we actually numerically scale that information on a basis from zero to a hundred, the response was that we ended up with a 65, was the score across all the organizations in our sample. You know, if you think back to your time in school, 65 isn't a particularly good number. I didn't expect it to be a good number. We actually only had 16% of organizations say their open source software was highly secure and 43%. So it was somewhat secure and of course the rest, that's 59% total, the rest of them basically were some level of insecure. So 65% today, that number was based on how we asked them questions about how do you anticipate this is going to change for your organization. That number one increased by about 11% to be a 72, according to these organizations, by the end of this year, because we did the survey in April. So a significant improvement in just in less than a year. And that was going to improve from 72% to 77% by the end of 2023, a much smaller 7% increase. So that's one of the reasons why I said this is clearly a journey for organizations. You know, in the next year and a half, we're going to improve significantly, but it's likely and really not to be enough because security is one of these things can bring an organization to its knees. So anyway, that's where we're at. And we can go on and talk more about, if you like, about exactly what organizations are doing to address this. Yeah, I would love to. I just want to look at these numbers also from Matt's perspective, because as much as we are talking of shift left, but we are still things, there are still silos. There are teams, security teams. So when these kind of surveys are done, I mean, it kind of leads to two different, first of all, that the teams are themselves not aware what the security teams are doing. So there may be security practice already there, but they're not aware of that. Or second is the fact that, no, there is nothing. So if I ask you, because in either way, both things are not very encouraging. Yeah, and I think if you're asking developers whether they have a security policy in place and they don't know about it, well, that's as bad as having no security policy, right? I think what it exposes is that it can be challenging to address these things, even in large organizations, because looking at security around open source projects isn't just about looking at the code. You kind of have to have an understanding of the open source development model because things like the health of the community, the governance of that particular project all play into whether you should be using that project on a long-term basis to support your business, right? And I mean, we've seen examples over the last few months in the JavaScript community of these protest wear type applications, which had a single maintainer decided unilaterally to make changes to that particular application, to change its functionality, or to, I mean, luckily most of those weren't destructive. But I mean, what it shows is the risk that open source projects that don't have a healthy governance model can also be in terms of vulnerabilities and unexpected behaviors. So yeah, I think it can be a challenge for organizations to understand how they should be consuming open source. Right. Steve, do you have to add anything there or? Well, I think it's a perfect setup for what organizations are intending to do to help us sort out this problem and improve their security posture. So the first thing that organizations want to do, and this is by a significant amount, 59% of organizations in the sample said they were looking for added intelligence to existing software security tooling. Now, if you look at the market for security tools, software security tools, there are probably about 10 different categories of tools. And of course, we all know in this survey confirmed that SCA tools and SAST tools were the two leading categories, to no surprise. However, do note that those two tools are very focused on CICD. And so not as much focused on the tail and of course, much better to find problems before you go into production. That doesn't mean you don't want to scan software once it's in production. So that was the leading way that organizations are looking to improve their security. Now, note that this does not include changes in the process inside of the organization. This is essentially abdicating part of what they need to accomplish and pushing it into the hands of the vendor community. So our hope is that the vendor community is responsible, is attuned to these particular needs and does improve these tools. So that was the leading finding. The number two one, which I think is also really key here 52% of the sample were comprehensive best practices and certifications for secure software development. This is great to see. The best practices space when it comes to open source security or software security in general is fairly complex. There are a lot of different best practices that you can look to. And there are now frameworks emerging. Salsa.dev is a good example of one that's S-L-S-A.dev. So take a look at that if you're unfamiliar with the Salsa framework. Linux foundation has one too from the standpoint of how you should adopt best practices. We in fact have a free course and certification for secure software development. It's LFD 121, it's on the open SSF site. These best practices and there's about 150 in total. So this is not for the faint of heart but these best practices are absolutely fantastic in helping you understand the scope across the life cycle of what the security issues are and help you identify and confirm not only what you're already doing but also those things that you probably should be doing next. That's kind of the top two. I'll just mention the third ranked one at 49% was more automation to eliminate pathways to compromise security and reduce developer fatigue. And by the way, IAC tools, infrastructure as code was the number three used tool from the standpoint of addressing software security. And I'm sure it was all about automation across CITV because more automation means less manual touch points, less ways to basically do shortcuts that are sort of not security appropriate and allow you to have more loopholes to be able to be open to exploitation. Right, excellent. There are a couple of things that you said, number one, you're like a vendor's responsibility and then you know, developer fatigue, sneak is here. I want to understand from your perspective as well. Number one is that let's just talk about cloud native technology. They are already complicated. Let's not even talk about how complex Kubernetes can be. Plus, a lot of things are moving to developers pipelines. Earlier it was, hey, you write the application, you focus on adding business value. Now you have to also worry about not only maintaining it, keeping it running, also securing it. So a lot of things are moving into developers pipeline. We are not making things easier for developers. We are like, hey, you know, you have to do a lot of more things now and security is a specialized field. So that is one factor there. We're already seeing burnout is happening. We're already saying that retention has become a big challenge. Second is that automation he talked about. And then third is also cultural aspect. So even when we look, go back to the previous point of 65%, 49%, it's also, the tools are there, but how many, everybody loves to talk about security, but how much is it in practice? So if you can just look at these three problem areas and you can see that you do see a common pattern and how culturally it can be addressed. Yeah, I mean, I think you've identified some interesting points there if I can sort of unpack that a little bit. I mean, because in a way this problem isn't just about open source supply chain, right? Because at the same time as this, we've got this dramatic shift in the industry about how we need to do security. You know, as you pointed out, this move towards from security as a gatekeeping function to integrating security into our pipeline so that we can maintain developer velocity. And you know, so, and that developer first kind of movement requires different kinds of tools, right? Because we want to provide tools that give developers actionable insights in the places they work. And so, you know, successful kind of transformations in this way are about integrating security, tooling directly into your software development lifecycle at every point. So giving developers insights into our, in their IDE, in the source code management system, in your CI pipeline, and then in production. And it's a different kind of insight from, you know, the traditional big list of CVs that security tools use to produce because developers aren't security experts. So you have to, you know, action ability and remediation advice becomes a really key part of that kind of security transformation. And I think, you know, from the figures that Steve was referring to, you know, those really highlight the difficulties there are in these transformations. And because it's a very different way of working, you know, we sort of have to transition our security teams into being tool smiths and trusted advisors from being the kind of old gatekeeper function. But I think you also hit the nail on the head that, you know, the hardest problems in technology are always about people. So, you know, there's definitely challenges that we see working with organizations about cultural change around security as well. I think now we have very well-rounded and good discussion here, right? We not only talk about the report, but also we get deeper into, identify the problem areas and also kind of potential, not solution, but kind of solutions to that. So Steve, do you think it's good to wrap it up now? Yeah, yeah, no, I think we've hit on all the high points. I think I just wanted to build on comment Matt made about the tooling. I think one of the reasons, here's a great example of how vendors could provide more intelligence in the tooling to add value. Well, first of all, you know, increased automation in dealing with security issues is clearly the path organizations want to be on because this will be the easiest way to reduce the workload on the developers themselves. But also when it comes to SAST and SCA tools, there's a very strong need here to reduce the number of false positives. And if vendors can get the number of false positives down, that will generate, that means less noise and easier for developers to focus on what's really important. So that's just one simple example, but that's, I think, a really good reference to why improving tool intelligence and the security space is so important. Yeah, and I think that the other thing that's going to become increasingly important, and this is just starting to emerge is our ability to actually sort of tie the different viewpoints together, right? You know, we can look into cloud environments, see how those things are configured, see where your applications are actually deployed, and give much deeper insight into whether a particular vulnerability is actually exposed or not. And the same in terms of code parts, right? Looking at whether is that code part actually executed? And I think once we start as an industry to be able to link these things up and link these data sets together, you know, security tooling becomes much more of an intelligent viewpoint across all of your deployed applications. Yeah, I think one more thing to add here, though, on the tooling side, is that right now, on average, across those organizations in our sample for the survey, organization for using 2.4 tool categories. And as I said, there's about 10 or 12 different tool categories. The unfortunate aspect is that there are probably four important tool categories that, you know, top the list from the standpoint of penetration already to date. We've already talked about SCA and SAS tools. Those are both, you know, the leading tools been used. Behind that comes infrastructure as code and then the DAST tools. So whether it's web application scanning or fuss testing, there are different flavors of DAST. But those are four tool categories that it seems to me are instrumental to being able to being involved in addressing open source software security. And what this suggests is that since we're only using 2.4 of these categories, and this doesn't even address the remaining categories, since we're only at 2.4, although at the risk of alienating developers, I think we really need to focus on adding more tools into deal with the security issue that we have because we need more of a life cycle approach to all of this. So maybe we'll get some back because of what IAC can do. And then of course there's IAC scanners as well. But I just want to mention that because I think we really do need to use more tools in the security space, but we have to be respectful and find ways to do this in an efficient and protective way. Matt, Steve, thank you so much for taking time out today and talk about not only this report but it's going in depth to actually also, once again, suggest what is needed there. I think as you earlier said, you know, it's going to be a journey. So I do hope that by the time you come up with the next report, the number will jump from 49 to 99. That's what we're hoping for. But a lot of work has to be done in this space. So thanks for your, not only your insights today, but also the work you're doing there. And I look forward to our next discussion. Thank you.