 Okay, oh, I'm pretty loud Hello everyone Aloha What welcome to Austin is anybody having fun yet? I? Hope so I hope so if there's no point in coming if you're not gonna have fun. Like I said meet a friend Enjoy your time here, right? try to meet new people and Hopefully with my presentation today, you'll you'll have a little fun Try to make my presentations a little light-hearted All right people take things too seriously sometimes so hopefully you can have a laugh or two I'm not a comedian and I'm not witty, but hoping some of this stuff will resonate with you where you know You'll you'll laugh a little bit Okay, so my name is mobile Lopez and I am the senior technical staff member for supply chain security at IBM I lead the enterprise-wide mission for enabling product security specifically supply chain security I Have a couple of colleagues in the audience that also work on this with me, so thank you for supporting and I've been in security for about six seven years now have masters in cybersecurity Prior to that I was pretty much it Starting with virtualization and cloud solutions for IBM and their customers And that was about 15 some years ago, so So yeah, I've been in an industry for a while But security is my passion now Okay, so ready to learn about s-poms Okay, so this is a quick agenda Obviously, we need to know why we're doing s-poms. Why is it so important? And then I'll go over, you know common things people processing to technologies, right? What hiccups we ran into? What are the successes that we've learned? And then looking forward right not only from IBM's perspective, but from the community, right? What? Call to action from the community, you know am I looking forward? I think and I think the industry in itself is looking for and If you have questions throughout please don't wait till the end. I'm pretty bad with memory So when I have a question if I don't ask it immediately I forget so if you have a question feel free to raise your hand stop I want this to be as interactive as possible Okay, so why are we doing this anyone? Why are we doing s-poms? executive order Other reasons. What was that solar winds? Anything else to launch software? What else? vulnerability management that's pretty good one so You know devs hate security and my slides are kind of acting on their own One reason is devs hate security the Linux Foundation research arm actually did a survey and found that Only two percent on average of the time spent on security in the open-source community is on security And I love this quote. I find the enterprise of security is soul withering chore All right, so if devs hate security don't like security Then how can you trust the open-source packages that you are consuming? All right, and somebody mentioned solar winds, right? Solar winds happen ruined my Christmas vacation and many others 300,000 customers impacted by this right and this really was a catalyst for the White House to say hey You know something has to change. We can't keep up like this White House executive order as somebody mentioned, right? This is from the memorandum and it says Developers are responsible For all coding including open source, right? I need to use this pointer including open source and so people the developers need to know What they have in their software and produce a software bill of materials So I know a lot of folks have been saying, you know, this is coming down the pipeline, right? So if you are part of an open-source community and you are not providing this You can expect organizations to start asking for this, right? So you might as well get ahead of it Another reason is log4j, right? Voterability management was something that tossed it got tossed around but this is just yet another example of If things go wrong, it can go really wrong, right? There was millions of attempts made per hour to try to exploit this, right and this was a yet another my slides are doing something funky There was Another holiday ruined luckily not mine this time But other people that I know that they had to actually work through the holidays for this particular vulnerability So now that we have that foundation of the why we're doing S-bombs why it's so important to do this You know the VP of product security at IBM, you know asked me to take this on and You know, I said, you know challenge accepted. So, you know, let's make some S-bombs. How hard can this be? So, you know, let's think about this. Well, how many products do we have 1800 plus? Okay Not too bad. Well, we have sass. We have on-prem. We have mobile. We have consulting assets Basically code that's produced for a particular customer. Oh, and we have legacy. Okay. Well, how do they build their software? Well, this team does something different than this team. Okay. Well, where does that exist? Who maintains that? And Once I gather all this information, where's the product metadata? How do I know about the versions? How do I keep track of the versions, right? and Starting to think about the standards as PDX cyclin DX, you know, why are there more than one specification? Why are they different and why are they're not this was pre today? Why are there so many that's a lack of tools for creating S-bombs, right? To now today, there are so many different tools that claim that they can produce S-bombs. Well, why? right and Then starting to think about oh well, we have OEM code in some of our products Well, how do I get the software bill of materials for that to include into my software bill of materials? So all of these questions just started adding up adding up and eventually This was me Probably about late last year early this year, you know What what did I actually get myself into right people make S-bombs to be this very simple thing? Simple concept and it's and it's not maybe for smaller organizations maybe for organizations that don't have 1800 plethora of products and You know, it was quite overwhelming Not only for myself, but for some of my teammates so You know, I started to enlist Volunteer Army, right? No one person can do this on their own So little by little we started getting an army of folks developers Opportunity managers product managers, you know the security folks lawyers. Oh Sorry back security folks and What turned from one to five to ten to fifty? Volunteers across five different business units all virtual and by the way, I had no Relationships with any of these folks, right? So very much like the open-source community It was pretty spectacular that I was able to you know, try to get people to work together But after you know growing so much it was hard to do things in an agile way with everyone Right so started to look at you know open SSF. What are they doing? How are they modeling their working groups and bringing that internally? One of the most successful working groups we have internally is the S-bomb working group led by Matt Rukowski Right and that really deals with what type of data. Do we really want to be in our S-bombs? There are additional Metadata that you can include that are specific to your products or your company, right? So that in itself was a chore on its own And to date it's been very very successful working group with that working group below and has 50 plus members, right? Obviously not everything worked from the get-go. We had to iterate iterate iterate and sometimes we had to pivot because leadership said hey There's these new responsibilities or new priorities that you have to focus on we need to put things on pause And so that created kind of a challenge, right? These are volunteers. They have competing priorities And so as a result leadership realized the complexity of not only S-bombs, but supply chain security as a whole Right. It can't be a volunteer army. It has to be dedicated group of folks Focused on this problem. And so now we have a supply chain security org But even before that we had to educate and we continue to educate and what do I mean by educate? Well, a lot of people don't know what S-bombs are, right? We had to teach the lawyers, you know, here's an S-bomb Here's what it looks like. What can we or can't we disclose, right? Those are things that people typically don't think about There are things that unfortunately for a company as large as IBM you can't just share everything Right. So there's a field by one of the specifications that says author. Well, what if the author is an IBMmer? That's personal information We can't release that so now we have to alter that field to be a more generic field Maybe it's a division not a person, right? So working with the lawyers trying to understand that procurement Supply chain or supply risk management team, right that works with procurement, right things that you're buying Right. Can we change the contracts so that they can provide us an S-bomb? How long is it going to take to do that, right? But you have to educate all of these different types of personas to be able to express the why We need them to buy in and help us on this journey Any questions anyone work in data centers one person. Oh Come on ever Well anyone tell me what this is Yes, it is very much a mess I Actually used to work for system X in IBM and we actually had a lab that looked just like this it was a play lab, so it wasn't like an actual customer's lab and this was a mess and The reason why I bring up this picture is because you know, I envisioned a supply chain security pipeline But the reality is it everything looks like that inside IBM. There's legacy processes. There's new processes There's different be used different teams, right? And so how do we go from here? to here Right in my old lab. I just cut the cables. I literally cut cables and just redid everything Right because we could do that But I can't cut these cables for the be used and the product teams at IBM I think I would probably get fired So, you know, we have to basically untangle these webs little by little So how do we do that? For supply chain security pipeline, we have to enable policy and governance in the supply chain security pipeline We don't want devs architects release folks to be worried and my slides are going a little haywire Give me one second. We don't want The devs the devs or any product teams to be worried about security, right? Going through the memorandum going through the NIST frameworks the SSDs, right? All of those requirements take a lot of time to go through we don't want developers We don't want architects having to go through that unless they're security focused So let's build that into the pipeline so that they can do their day jobs and not have to worry about the Constant changing of new requirements new policies that they have to meet, right? And obvious extension would be CICD as a service Right, so not just supply chain security pipeline, which has a very specific purpose But why do we have all these different build systems, right? Why not create a service with some customization for all of the teams at IBM to You know leverage right they could still tweak it But at the very least the foundation is there as a developer Do you really want to be messing with your build infrastructure and maintaining it? No, right? You want somebody else to do that for you and Like I mentioned custom options, right an inner-source catalog for tecton Tecton tasks as an example And legacy and proprietary support we have a lot of teams that they don't use go Right. They don't use Java. They use an IBM proprietary language. So how do we support those folks, right? We need to create something custom for them and They can then plug this in back into the CICD as a service sorry, sorry, sorry these slides We also have kind of a consulting arm in supply chain security So let's say there is a legacy team, right? They don't know how to get started They don't know how to modernize their environment. They don't know what an S-bomb is They don't even know how to they don't know anything. So we actually do S-bombs for them analyze S-bombs for them We also do a colleague of mine is in the audience do Open-source software security vetting and license vetting, right? We don't want to have again the devs doing this so we provide this as a service and Then continuous compliance. This is actually really key Just because you've created an S-bomb does not mean you mean you're done and the keynote mentioned this earlier You have to do something with that S-bomb So let's give an example of your code does not change The S-bombs not going to change But what's going to happen tomorrow? Anyone there you go new vulnerability, right? Well Okay, your S-bomb didn't change your code didn't change But how do you know about that new vulnerability if you're not going to scan, right? Typically these pipelines trigger a new scan a new build to identify those new vulnerabilities Well, let's leverage the S-bomb Sorry, it must be using the timings of my my practice is my guess The I forgot what I was saying Thank you So analyzing the S-bomb instead of using like let's say a pipeline as an example You can use something off-site, right? That doesn't really use up a lot of resources Let's analyze the S-bombs for the packages compare that to NBDB or you know some other database of your choosing and Figure out what vulnerabilities might pop up new, right? Additionally, you could have an IBM team that we work with does do this where they have a continuous compliance pipeline that Daily builds a code and rescans it, right? So you can't just say okay S-bomb done. Nope You're not because just because you're done and your code doesn't change does not mean you're not vulnerable the next day So as I said earlier, I'm trying to make it easy for everyone to understand supply chain security Right and S-bombs in general when I was going through the NIST Cyber security risk management framework, which was just the final draft was released in May I Could understand it but visually I could not conceptualize what it was asking me to do And I'm a very visual person. I need to visually see what I'm being asked of to You know perform my my next task or whatnot So I've created this is a third iteration of this framework It literally takes you through the entire supply chain security process and the end based off personas, right? so if you think of You know your vendor or supplier coming in right they provide you an S-bomb Well, you need to take that into consideration in your design Well, what are you going to do with that S-bomb? Well, you're probably going to put it in some sort of pipeline to analyze Right or some sort of tool that might be down here for continuous compliance Additionally, you know upstream open-source communities that code that you're going to be injecting into your products, right? You need to be able to Assess what that risk is of that product or that package, right? And you know making sure you're doing all the compliance checks, right? A lot of the frameworks deal in this area development and build But they don't really go too deep into the other things well sure But what happens when you have a desk bomb, right? What do you do with that data? Well, should we have a data lake and a data warehouse to house this information and then apply machine learning to Identify maybe risks that we as humans are unable to see That is something that we're working on right now to understand not only vulnerability as a risk but a legal risk, right because Regardless if we like it or not the government tells us you cannot do business with these Countries and I know that there is a stance on well We don't want to do this as open-source, but we as IBM if we're going to consume open-source We have to understand it's 50% or more of this code developed outside of the US and if so where? So how do you assess a legal risk in? addition to vulnerability risk in addition to people risk, right? There's a Linux hypocrite, right? I think that was a couple years ago who Changed the code, right? Well, what happens if that user is? impacting 100% of your products How do you use this data lake or this warehouse to go back to identify that user to say? Hey, we really now need to look at all of these packages to identify risk that this user now poses to our products Right, so there's a bunch of different things that that happen in here. I know this text is small But this is the slides are available in online More than happy to go back with you to the booth the IBM booth there's a larger screen or I'm on the open SSF Slack channel and You can reach out to me. I'm more than happy to give you this presentation Right. The idea is is that this is supposed to be a framework for all and in my next iteration I plan to map this to salsa, right? So how does salsa help you achieve this, right? Any questions on the framework and thank you Eva for blessing me with your presence Okay, so technology Unfortunately, there's no Swiss Army knife and That is what I think the industry isn't really Telling all of us. You can't just do this with one tool You have to have a tool kit to enable you to do this as I mentioned earlier, you know, there's there's a ton of s-bomb tools, but Accuracy depends on when you run those tools and against what right? So if you think of An environment with co-located code, let's say a github repository with co-located code has dev test If you run an s-bomb against that repository Now you have accuracy issues because that's not necessarily in your product Because you've built it too soon However, it does do the shift left principle of telling the developers. Hey, you have a problem here You might want to consider fixing it, but it's not a customer issue So you don't want to give a customer that s-bomb because it's not accurate Well, you want to do this during build, right? Pull in the dependencies and the packages that you truly need for your product But then also analyzing after the fact So we're actually using a multitude of tools To And not only analyze the packages but validate, right? Did the initial s-bomb that was created actually have all packages? And if we think about Some of our products the source code repository doesn't have let's say an OEM package. That's somewhere else Right, and so we have to code for these types of situations Where we have to pull information about OEM or somewhere or you know a third-party Company that we're working with let's say red hat as an example. We don't have red hats code But there is a database elsewhere that says hey this product belongs with this product as a bundled package Well, we need to provide an s-bomb to our customers like that So it's not as easy and straightforward as people make it seem that it's just one tool to rule them all. Yes That it meets specification whether it be cyclone TX or spdx Really we can request them right now of our OEMs But unless it's in our contracts, they're not obliged to give them to us, right? And so we have to start having those conversations of we're happy that you Maintain your s-bombs. We don't want to maintain them for you, but we need to somehow link to them. Yes, go ahead Yeah, the idea would be and the question was you know from a process perspective What does it look like for you know? Getting an OEM s-bomb if I understood it correctly it would be automated. We don't want any manual, right? This is just too overwhelming some of our product teams release every two weeks We have 1800 products. So if 1800 products release every two weeks There's just no way we're gonna keep up with that scale, right? And so the idea would be that they would tell us hey or we would tell them put it in this repository here And then we would somehow link it, right? But we're still navigating that of how do we get our OEMs to give us the s-bomb because some are willing and some are not Right. Yeah, go ahead. Yes. So the question for the virtual audience is you know, do you do I think this will Drive a common framework or standard for for you know, this kind of interaction or relationship and the keynote mentioned it earlier today, right? S-bomb transport is a Topic of consideration and I suspect different folks will handle it differently And this is something that we've been thinking about in terms of like a trust center, right? If you go to IBM comm there's like a trust center page and that talks about all the product Security details, right? If you want to understand like, you know our ISO certifications or SOC certifications You can go and grab that data. Well, why can't we use that for s-bombs, right? Can we use our portal right for that kind of mechanism and somehow link it, right? Because we know the ordering systems as an example have a lot of this linkage. Let's leverage that But how do we leverage that? I don't know by the way IBM doesn't use one ordering system They use I think three different ones depending on you know the product so that makes it even more complicated But I do think eventually as the industry evolves as the industry starts to adopt s-bombs more and using them Then it will start that discussion of well How do we share this more easily where it's not everybody on their own, you know Architecting this thing and implementing it. I just don't think we're there yet Any other questions? Let's see So No, sorry, this is this only works as a laser What I can tell you is from Our our work that we're doing we're obviously using IBM clouds for our pipeline with red hat open shift We are Using tecton. We're trying to be conformant to not only spdx, but for cyclone DX Get hub we use github internally And we also have open source communities that are maintained by IBM that obviously use github And then I have the you know open SSF scorecard over there Like I mentioned earlier. We need to analyze different types of risks. It's not just about vulnerability It's also about the people right and so scorecard gives us that view of the people that community and what they are doing From a security best practices So looking forward Use a supply chain security framework, right? I'm more than open to suggestions and feedback on can you use this framework and how right? Can we improve it for you, right? It's not meant to be an IBM only thing I want people to actually use this to help them in their journey It takes a community not only internally in your organizations, but the open source community There's a lot of work that needs to be done and no one single person or company is an expert I don't care how much they tell you they are It's it's a community effort. We can't do this alone Again, no one tool to rule them all right people will promise. Hey, yeah, we can make an S bomb I call that a partial S bomb. It is not a true S bomb unless you've done analysis from different tools You're using containers. Okay. What does you know? What does the container scanning? Tool say right you're using open source packages. Okay. What does the source code? Analysis tool say right and during runtime. What does it say right? You need to be able to identify during the different parts of the development life cycle Whether the S bomb is accurate or not. So it's not just the one and done Iterate and educate you need to be able to speak the language of the folks that are going to be helping you with this You need to tell the stories from a legal perspective. Why do lawyers care? Why do product managers care? Why does procurement care? Right, you have to get the buy-in from the different personas in order to be able to Help you on this journey for IBM We are contributing to a variety of open SSF projects. I myself. I'm trying very hard To contribute more into salsa I am new to open source as of last year So kind of starting to get a little bit more comfortable and as I get more comfortable I'm able to contribute a little bit more But we have long been working on open source at IBM We have an entire printout with you know a little list and my battery is running out. Oh, look at that and We are also Again, like I mentioned trying to improve the consistency and accuracy of our S bombs as IBM We're a huge company with a hundred and eleven years of history and a multitude of products right in different technologies and some very very legacy and So it's not easy to create a single standard S bomb You know not specification, but a process in which you do that So we are working with a variety of teams to make our S bombs more accurate. I Mentioned earlier the machine learning and AI to assess risk Again, it's not just about vulnerabilities. You have to assess different type of risk and unfortunately IBM legal is part of that right so how do I assess the legal risk of Using a package and how do you bundle that up together to make a recommendation for let's say mergers and acquisition or a purchase of a product right so we need to be able to quickly identify the risk of using some sort of software So for the community Industry and open source community get your security people involved. We don't have enough. Oh I guess they don't like security That was a nice honk Get your security people involved. I am a security person. I don't do open source as a day-to-day job All right, I work under the CISO office But what we as security people can do is enable the devs who hate security Try to automate the security in their open source community or in their products, right? So try to get five percent of their time, right? I think as security focals we need to enable the entire industry the entire community and do our part in some way Incentivize devs to take security seriously gamify it Right if they find bunch of vulnerabilities and fix it whether it be open source internal Give them, you know, some prize or some money, right? Whatever you're choosing, but we need to be able to get devs to take security seriously. This is in Healthcare systems. This is in critical infrastructure. This is in government systems, right? It's not a game when somebody's health is at risk or You know when their Bank gets hacked or their credit card gets hacked, right? That's their software They need to take security a little bit more seriously and For the love of God, please validate your S-bomb against specification I don't know how many S-bombs I get That when I go run it through a validation tool, I get a ton of errors. I Can't do anything with that that that makes me spend a lot of cycles trying to figure out What's wrong with it and maybe yet can I tweak it? So before you give an S-bomb to someone at the very least use the tools that are available in the community to Validate those S-bombs like I mentioned earlier enable other personas, right? We can't just be talking to the devs We have to be talking to other people as well in their language So that they can help us create S-bombs help us improve supply chain security And this is a personal plug stay tuned there is a supply chain security paper coming out soon It's kind of doing its last revisions red edit myself and others at IBM Have contributed to it, and it's not your typical, you know framework. It's not a a brand push it truly is a perspective of supply chain security and what organizations need to know when they're embarking on this journey including the framework that I showed earlier and I'm also Doing blog series internally and hopefully externally soon based on personas giving a security checklist for developers for architects For lawyers for consultants, etc Because again, you have to speak their language, and if you can't speak their language, you're just not gonna buy it And there's a ton of IBMers here presenting. I don't know if you saw in the first day's keynote There's a huge list So check out our other presentations I know two are in the audience that have presentations Thursday, and I think maybe Friday and this is Slide on, you know what we're doing for supply chain security were very much heavily invested in open as a set and the different working groups and Then stopped by the code cafe. I actually did the This little training the space rover to you know do stuff, and I was stirred yesterday I don't know what place I am in today, but it's a pretty fun game And there's gonna be voodoo donuts, and there is barista made coffee So please stop by and it's again if you want to talk to me now or at the IBM booth by all means You know come see me any questions So I actually have one virtual question It says we've heard a lot about the value of s bombs But if we create them where should they be stored and then externally accessed Yeah, so that's a that's a great question One of the things that The paper tries to address is that kind of That kind of question right you do have to think about You know where internally are you gonna? Store it who's gonna access it, and this is kind of covered by the NIST Cyber should I think it's a cybersecurity risk management? No cyber supply chain risk management framework Or the SSDF I can't remember which one and they do talk about centralized storage They talk about locking it down doing logging right and so you have to take that into consideration from an internal perspective From an external perspective it depends right Do you want to be fully? Transparent with your customers, and you don't care who sees your s-bomb Then you can place it somewhere publicly But if there are contracts as an example that you are bound by that says thou can not Release an s-bomb without first notifying us as a you know big corporation You might have some constraints there, and so when considering that you might want to think about well Can you validate? NDAs as an example that could be a mechanism. Can you somehow link it to your entitlement? Process right a lot of customers are able to go to a website and see what they're entitled to maybe you can put the s-bomb there And as mentioned earlier today, there's not really a standard But there are a lot of things that you could potentially leverage. It's already existing in your you know ecosystem when it comes to ordering your products But it's kind of a gray area, and it really would depend. I know there's an another question up here. Yeah Is this working it is that my gun? Okay So how are you say if your customer is asking for an s-bomb to you today? Are you only sharing? The external dependencies or you're also sharing some of the internal commercial dependencies in your products And like what are the challenges that you're facing in that? So currently it depends on on the customer and on the request unfortunately The supply chain security team right now has a process because we don't have an automated way of pushing those things yet We're working on it We actually get the request which so far hasn't been a lot And we take a look at the why why are they asking this question for an s-bomb? Do they even know what to do with an s-bomb and a lot of times they're not even looking for an s-bomb They're looking for let's say vulnerability and have one more minute left and so I'm not sure if that helps or or not and I can connect with you. Yeah Yeah, you can connect with me afterwards since I'm running out of time and Ava A little bit yeah, and and that is also part of my colleagues presentation because It's not so much about identity. It's about is Quote-unquote 50% or More of the code being developed in a country that's sanctioned Doesn't mean that you're a citizen, but is it being developed there? And so that poses a legal risk unfortunately to a company like IBM Yes, yes, correct, correct And I am being told that my time is up, but I'm more than happy to take questions and