 Hello. I'm David Stewart. I'm a manager at Intel, and I'm here to talk to you about supply chain armoring, which has been a pretty hot topic in terms of supply chain at this conference. And I hope I have some things which are going to be useful and helpful for you in dealing with this area. I think it's important that I talk about the perspective I'm from and, you know, how do I approach this issue? The whole idea of supply chain is really pretty close to me because my experience, I've been in the industry for decades and almost all of it is creating software products or open source projects that I treat like products in the same sort of care. Sometimes it's working on other people's products. Almost all of these, all of this work has been in the area of system software or operating systems compilers, that sort of thing. I've also done work in client applications and in a variety of other different areas and really full time work and open source started in 2007. And so a lot of focus in that area in those years and then since 2018, my full time focus has been on the area of security. So I get to not only have the great experience of working on terrific products and projects, some of which are still even really popular today, which is a huge privilege, I think. But it also gives me the chance to now as a security person talk about, you know, how we can make these more secure. So that's the perspective I come from. And when we talk about the whole SolarWinds exploit, what happened in my journey here is that it was, you know, this came out public in December and over the holidays 2020-2021. And I got sort of interested in this because I would read the non technical press like the New York Times, while it was on holiday, and they would start describing the SolarWinds incident and how it was exploited and why do they call this a supply chain issue. And when I would talk to various security experts around, they would say, Well, yeah, this is this probably is has been exploited a bunch of other times as well. So suddenly I was like, Well, if somebody like myself can read this in the non technical press, I think we're going to get a lot of questions in the whole software world about hey, are we vulnerable to the same sort of thing. Is there anything that you're doing to try and address this and so I got took action. You know, really, when I came back to work in January of 2021 to say, let's put some things in place immediately. So it takes an actions across the entirety of our software world. And then in May of this year, we had some some actions which came out what they started to address as well. So that's the perspective I come from, and trying to really look at trying to take all of this this area much more seriously. So why this is such an urgent issue. I don't want to necessarily go through a bunch of details about the solar winds. Actually, I'll touch on it a little bit and give it kind of as an example why we need to take this whole issue very urgently. And some of the folks that I collaborate with have really helped me be more exposed to some of the challenges that we have. And then I'm going to use some specific tools to use. I'm going to give you an analysis tool to use to analyze your, your supply chain. And I'm going to help you understand where you need to, you know, apply a couple of other open source tools that I have for you. So this is what we're going to talk about the rest of the hour. So my assertion here this really is a significant issue. And one of the things, as I said, I started getting involved with this early on. And part of the conversation I've been having is with various collections of, whether it's, it's companies or others, or other folks from different organizations, nonprofits, etc. So we've been doing a lot of collaboration on this whole issue. And the realization is that there's a lot of concern about open source within a lot of these organizations, because they sort of feel like, well, how do we know that this was developed in a secure manner. Now, I argue with these folks look open source is often a way to get actually more secure code than in close source or proprietary sort of environment for the exact reason that you potentially have a lot of people looking at the source and can find security issues can find and fix these things, whereas with proprietary products, why you may only have a very small group of people looking at it. And in fact, this is well known as as the best way to get good good cryptography software is get the algorithms and the code out there for people to look at at least a year in the public before you actually use, you know, any kind of protocol in the crypto space. So, so even though the potential is there for better security and open source is where the reality is a lot of people because of the behaviors they use their consumption of open source software is actually kind of leading us to more problems. And that's where this perception comes from. So I think that the open source community really needs to take this seriously. So, the subject lines I looked at here was a study that had shown that like something like of all software being created these days something like 70 to 90% is from an open source repository of some sort of had its origin and open source. And so this this makes it something that I think we really do need to take very seriously. I'm kind of quoting myself here. I'm actually paraphrasing another longtime security expert who had a slightly different statement but but in terms of attribution I'm going to attribute this quote to myself. And let me tell you where I come from this I mean I as I said my history is that I've created a lot of software teams, you know, I'll set up a new team or, you know, as we're going to create a new project to some sort, and you assign roles and you bring in people to the team to work on various projects. And I will fully confess to you that that I don't necessarily always put the most senior person in the role of the build engineer. Now it's quite often that you will have a senior and very experienced person maybe architect the build system, but actually implementing it and maintaining it is is. I've been just as guilty as I think of folks and the rest of the whole software world is saying, not necessarily the build engineer is not necessarily the most senior person on the team. And so we've created this whole area of vulnerability and what has really happened is that the solar winds incident has shown us how easy it is to exploit this very weak area. This is the whole way that we provide software for folks. So, I often say from my sins I now have to go correct a lot of the problems that I may have created earlier in my career. But this is why we need to take this very seriously and in fact, of course, the details of the particular solar winds exploit was one in which the build server was exploited. And some extra software was slipped in into into updates that went into a bunch of government servers and other companies and allowed foreign actors to be able to, you know, do a lot of surveillance and espionage, all by exploiting the build. And our friends over in US government are taking action as well. To their credit, they say, hey, we can't go and insist that everybody doing software do it the right way. But what we can do the what the one lever they have is to control the software that the government buys. And so they've put out this what's called an executive order which basically means it's something that was signed by the president this to order over the next you know, 12 months to put in place work, particularly in this case regarding critical software. And so this is one of the things that if you have critical software that's going to have a lot of privilege on the system you definitely want to make because it was developed in a way that doesn't expose, you know, all these secrets so this is something which is is a present, you know, factor, and I think it really helps focus a lot of our efforts. Again, I think it's everybody's responsibility whether you're, you're going to be affected by this thing or not because guaranteed this is not just United States government I think world governments are all going to look at this initiative to try and improve things. And as I said the timeframe the timeline that we're looking at is is it's not instantaneous but but they gave themselves essentially 12 months from May of 21 to May of 22 to identify what all the bits of critical software are what are all the standards that you need to follow in to provide this critical software to the government. And they basically concluded that anything that's going to get affected by all of these orders are things that they would define as as critical under the focus of the executive order, which means, and this is a partial list and you would have security software. Well that makes sense. But then you have operating systems. Hmm. That's interesting. Now, for an open source operating system like for example Linux. If you have all those sources available. You wouldn't necessarily have any problems but if you're buying an operating system from operating system vendor and you know, getting delivered as a set of packages or something like that then then then you do need to be concerned about it. So here in the non open source space I think there's going to be a lot of a lot of concerns of people providing binary components into operating systems like drivers, and maybe this arguments everybody should be using an open source operating system but the reality is, there's a lot of impact on web browsers. Why it's because web browsers have password managers typically. So security software well that's that's kind of logical network control software anything that's monitoring or, you know, are managing systems right anything that has a high level of privilege. Now, what's interesting to me about this 12 month exercise is they didn't include firmware as critical software, which I think is kind of interesting because it says, Well, you know, do we really think that firmware doesn't have a lot of use of course, usually firmware, whether it's bias or microcode or little bits and bobs of stuff that's that's running on various chips in modern systems, while they typically a very high level of privilege and you know ought to be considered critical as well. But from the standpoint of at least the immediate exercise. That's, we can take a little bit of a past there but what I want to tell you is if you are involved in that area as far as, you know, creating firmware in that space that will be affected by this order, probably be on the first 12 months so something to pay attention to as well. And what, you know, what do you need to do as if you've got critical software that you're supplying. And there's a list of things they're, they're, they're kind of vague but one of these that I highlighted here administratively separate builds is something that's. And when I read that I was like, Well, it could mean everything from, you know, you put some controls on a on a specific build server, or maybe you have to have a completely isolated build server that has no ability to connect with the the network at all whatsoever and they're just doing a build. There are some environments that are like that, particularly in the payment card industry. And in ultra, you know, secure sensitive kinds of development they will do things like a completely secure build but I think trying to get to that point in the bulk of the software industry is going to be pretty challenging. I think there's a, and this build area, like I said has been something that's been very interesting to me so it's been more of a kind of a focus that I've had to try and understand what are the tools that we have from the open source space to help with this. Now, I understand that this kind of topic will often make people a bit grumpy. And one of the leaders in our company has has opined that software people tend to be lazy, and they tend to be religious, which I would hardly agree with that with what is his observation there, because I think if you've got something that's working. You're going to mess with it. And particularly if you've got a fairly complex build system that maybe you've been tweaking the performance of so you get it to, you know, you can rebuild your your massive millions of lines of code within an hour or so. This is awesome. And maybe you've also created a big problem for yourself as well. So, I'm hoping that actually you won't be as grumpy about this but that we can actually probably hopefully deal with some of the grumpiness that build the experts have to think about in space. So, the first set of things that I think to seriously address this topic is to do some analysis to use an analytic model. I'm a big fan of taking a methodology and applying it sort of rigorously and and convincing yourself and others that you've done a proper job and, you know, helping explore, hey, what else do I need to go and address so I think an analytic approach to this I think is a good one. So, the methodology that a lot of security people use very common practice in the industry today is to develop and use what's called a threat model. And if you if you go well I don't know how to address security potential security concerns, how do I address the right ones. Well, a threat model is a great way of doing it and at a very high level. These are the steps for a threat model. And this is super high level that what you try and do is understand your architecture write it down can you draw a picture what what we often call the gazintas and the gazoutas. What's going in and out of every box in your whatever you're analyzing and in this case, you know, your build system. You want to identify the assets and the assets, these are the things which are really important to you to protect. And so, for example, your source code and, and binary sign binaries etc these are typically assets. Not every Dotto or, or intermediate, you know, build product would necessarily be an asset so identify those assets and their interfaces, try and then identify the next step is to identify who are the attackers that you want to be aware of. And not every attack and attacker is as important as every other one. So you can be, it's important to understand that these need to be prioritized and think rationally it's like, okay, it's possible someone might want to do this and could how likely is it how hard is it. So that's part of the idea of using judgment to prioritize these things, and then go figure out what mitigations are maybe they're 100% mitigation maybe they're only a partial mitigation, but then prioritize them and go do implement them. So this is the whole idea of creating a threat model. So, how do you start. And then a great paper that I'll put the link to and make it available to you. And that's a software supply chain threat model by a couple of my associates at Intel, and they have a number of different charts like this where they basically said hey here are the major steps and this this is an example of one that I pulled out that's sort of a hybrid between waterfall and some sort of a continuous integration continuous deployment sort of model. And so as you can see this is a you know these are the various steps and so you could use something like this to analyze and understand, we're all the potential attack points of something like this. And that's exactly what these experts did. This is a, an example of a very exhaustive list of potential attacks. So if you're struggling with trying to figure out, well what are the potential attacks, you know that could happen in my build system. Well this is a pretty darn exhaustive list. Now, not every one of these is as important as the other ones. You know, at the concept level, right somebody you could have an insider that could somehow mess with the concept and insert, you know, something bad there. I mean it's possible and from an intellectual standpoint, I suppose you could say oh yeah it's possible so write it down. It's probably pretty unlikely. You know someone's going to exploit something it's much easier maybe to exploit, you know something in your, your actual build system in terms of injection of malicious code compromising build tools etc. So, this is probably more likely the kinds of things you want to protect against. Again, it gives you a sort of some ideas of the kinds of things that are possible. And, you know, although this is a really complete list I think these guys have done a phenomenal job I think that it can be a little bit daunting to see all of these in front of you. I want to give an alternative model which also is really helpful that you can use. And this is something you might have heard of called salsa or supply chain for software artifacts. This is an open source project that was started initially by Google. Although today there's a good collaboration between various different developers from different organizations and so I think this is really pretty exciting stuff, because you can look at this and you say okay this is exactly how my brain thinks in terms of maybe this model works really well for you thinking about how do I map what I'm doing against this. So let's take a look what you can see is that they have mapped out what are the assets. They've looked at how the gizintas and gizoutas of the various phases of the build process happen, and they've identified the places where you could have potential attacks. And it may not be 100% complete. It is at least a set of reasonable things to go analyze and prioritize. And so you look at A through H and you can say okay these are all potential attack areas that I can go do mitigations on if I care to. So I really like this way of doing things and of thinking about supply chain security. By the way there's another thing that salsa has that I think is very powerful. And that's this whole idea of levels. And so if you if you think well I can't mean how do I get started. What's is there a first step that I can get to and this is very nice to from that standpoint because you can say oh I could salsa has different levels and the font size maybe too small for you to see but I would comment on the fact that at least in salsa level one if you have a scripted build, right, you're not just, you know, typing make but you're actually have all the steps, you know, automated in a script, and you have the provenance of your components established and the provenance in this case means you, you know, maybe you have various different versions of software from, you know, npm or various, you know, get projects or various pipe I or what have you or maybe you have some code that you lifted out of something on stack overflow or something like that. Well if you have all the provenance available for all the components that goes in you that in a scripted builds it gets you to level one of salsa. It's pretty good. More things get layered on the different levels at salsa level four. We have a lot of really terrific things here with like the source you have to at least two people are reviewing every change can be a little bit challenging to to achieve that and it's not perfect right there's still possible for people to miss stuff and in a in a source code review. There are some studies based on that but it's it's a darn good practice and in terms of trying to identify things in the build area. We're talking about hermetic builds. This is a concept that says again this is this this comes out of a requirement from the payment card industry that all of the builds you do on the software have to be on systems that are completely disconnected from the network that all the services for the build dependencies source etc are all locally available. And some some projects out there if you search for hermetic hermetic builds you can find some ways of applying that reproducible builds tend to be kind of hard to accomplish this is the reason why they say it's required unless there's a justification. The challenge is, you know, in the Linux world at least and a lot of modern compilers will put a timestamp in a header, you know, coffee or something like that makes it extremely hard to compare to build a bite for bite. Identical. Now some people will say well, you eliminate those timestamp bites and then you can do a bite comparison you're, you know, you're good to go. And how reproducible builds help you is if you can, for example, SolarWinds this is a how they've publicly said that they've solved part of their issue how they've mitigated the supply chain attack is they actually build the Orion product with three different build servers, you know, one in the cloud one on prime one somewhere else, all three build servers have different users and authentication, and they do builds three ways and then they compare them and see if they're identical. Now, this does not eliminate the possibility of somebody exploiting their build but they've just made it like much harder. You could argue to yourself, well, it's not it's not 100% perfect mitigation. A lot of mitigations though are tend to be like this in which you can say hey, I've taken care of this by doing this this way. I've really made it tremendously harder for a bad guy to go do bad stuff. This is, it's pretty good, but it is hard. I remember back in the Unix days before, before Linux when, you know, controlling all of the different things that went into the build chain we actually could do bite for bite comparison on builds and prove to ourselves that they were identical, a bit harder now and some projects like Yachto project is an example of support reproducible builds which I think is and continue to try and support that so. So Salsa is a very useful analyst. So the to do here on your list is to, hey, I would suggest let's take the salsa map for example, map your, you know, build system against the salsa map, and then highlight, you know, the prior the higher priority threats and set priorities for mitigating these these attacks. And I have a quote here at the bottom the build system must be developed and maintained with the same security integrity and diligence as the source code and result and result the product itself. So, I think that's exactly the kind of requirement that that we will start seeing really make itself felt. I think it should just be a melt baseline mantra that we start telling ourselves as we're planning on software projects as we're staffing them as we're thinking about the care and diligence we're putting against these things. So the next step once you've analyzed your, your build system, the next tools we want to look at is saying okay, what are some good open source tools to try to mitigate some of these threats, and I'm offering some of these things as there's as I have learned a lot of the tools that are being developed and open source projects that are working in this space. There's there's a huge list of people working on various projects to try and address supply chain issues. And there's a lot of that are inactive development right now. So, I wish I could tell you there's a really complete set of things that you can go to start implementing right away. The reality is this is a little bit more of a scientific report to you of experimental results we have so far some are are better than others. So, we'll tell you what we've found so far what we've played with and some of the results you can make use of them immediately. So the first thing I think about in this is let's let's look at the dependency area. Now, as I've argued plenty of times and various different forums and you know I've talked about these different, you know, industry groups and collaborations between different companies and various organizations and nonprofits and communities. The dependency situation for open source is complicated, and it's complicated because there's there's there's somewhat of a reputation that open source software can sometimes have more security issues and close source codes. Why? Well, I think it's a there's there's fear that says well if everyone can see the source code, then people can find exploits. And what I've argued is hey, actually open source software can can be potentially much more secure than proprietary because you have more eyes, you know, on the code and and able to find the bugs and and and fix them, whereas with closed source code that's not always the case. The usual best practice in the area of cryptography code is if you've got a new crypto protocol, you want to make sure that the code for that is is published at least a year before you try and make use of it so that people have a chance to vet it properly. But there can be a problem with dependencies, particularly how do you do, and there's kind of a dilemma is the way I put it. So the thing about dependencies is do you freeze them or not and as a software developer, probably the fewer things that are going to move out from under you and cause your, your software to break the better. So there's this kind of strong Tennessee not to update stuff, because the update stuff man I got to go see if there are any bugs that weren't introduced by that update. Now not every open source some open source projects make an extremely strong priority not to break any, any code that depends on it. Not every project is as well behaved as that. That's a kind of a reality so if I'm using, I'm maybe writing something in Node.js and they use some code out of npm. What are, what am I dependent on there. There's a famous example where somebody was had something out of npm, and they had. They were going to pull, pull their code out of npm. And it was based on some naming dispute or something like that. Anyway, there was all of the software that dependent on this extremely popular bit of Node.js code code suddenly was just like broken. And if anybody had tried to build at that point that they wouldn't have been able to build. And so people want to protect against that sort of thing. Right. And there's, you know, somebody could maybe subvert one of these open source projects, you know, something else. And so you go, Oh, well, let me download the dependencies, let me vet them, and then only build out of my local repo, which is great. However, the dilemma is, what if somebody has discovered a security problem in that code. What if somebody has issued an update that has fixed the security issue, or has, you know, at least fixed bugs, right. So this is a, this is the dilemma. You have to keep your, your stuff frozen. Now, I mean, if you're developing against like the tip of the Linux tree or something like that, quite often, you will need to rebase your code pretty frequently I've had engineers trying, you know, gold to rebase stuff every week or something like that. But, but generally, what do you do about this and the suggestion that I have for you is, in this case, try to have automated CV checking integrated into your CI system, and try and make sure that you're every time you're, you know, if you do, like daily builds or builds every hour or however frequently you're doing it, that you're automatically scanning for security issues, logging the results and, and, you know, reviewing them as well to make sure that something hasn't popped up. You could spend a bunch of money on a commercial product for this and we do, we, you know, my company we spend money on these tools, but there are also some tools that are free, and that are free and open source. Here's an example the CV in CV Ben tool. I'm a, I'm a big fan of this one because it's developed and maintained by an Intel security engineer expert, Dr. Terriota. And so she has a longer presentation about this that she's given at PyCon before. So I'd highly recommend that you check this out. Now, as opposed to a commercial product that maybe has thousands of open source, you know, projects that can, it can scan against this one is, you know, has a limited number, maybe about 100. But because it's an open source project, people can contribute to this and add the other, you know, libraries that they're concerned with. And it really helps you identify the surprise dependencies. You know, in your code, and, and actually queries against a known vulnerability. So, so this is extremely powerful. Best of best of everything. It's also free and open source. And so it's extremely powerful. So this is one of the to do is I would suggest is absolutely make it make use of this tool because it's extremely helpful. So what can we address. In terms of this map, I really think builds are one of those areas we're going to have to just knuckle down and try and do something with. And, you know, even the best architected build can have problems. You know, somebody has a target directory that this world writable that that build artifacts are going into. Well, that's not necessarily something you might have documented in the build architecture but it's something that really be thinking about this stuff at a very, you know, a very careful level. Now, what are the tools available to us to help achieve a better result in terms of build security. We've been playing around with in Toto, which is another open source project when I read about this. I'm my other, you know, colleagues got very excited about it because it looks like it's just a really great solution to help make you know, in a lot of different areas that only people authorized to check in code are able to do to do those actions. And the only people able to do builds are authorized users. And as we've gone and played with this and we've done some work. We've had a we've had a bit of challenge one of them. And all this works is essentially there's a layout that gets defined for the project and all the steps in the supply chain are identified by who can take the steps and uses public key cryptography to validate that it's only authorized people are doing those steps. And the result can then be inspected and verified to make sure that only authorized people did what they were supposed to do. We've run into a few challenges and at the state of the project right now there's there's some additional, you know, learning curve that we had when we first started work on this we said hey baby we can manually create the layout. And we had we had a few problems trying to do that manually. And there are actually some folks that are working on some automatic layout generation tools. So that was. But whoever is going to create the layout for the build system for the for your project does have a learning curve to go up to try and address this so just be aware of that. And you know the documentation there is documentation it's not complete. It's going to be set for a lot of open source projects, but it's still under development there if they're making progress. And this comment that it's based on keys as as anybody who's done work on, you know, public key cryptography maybe use PGP or some other system like that. The system until you have it working really reliably. It can sometimes be challenging that everybody has all the access to the right keys and, and make all the stuff go so that's that's the based on keys it's this is both a challenge as well as a good, you know, feature of in total. The things though that are really positive about this and the thing that makes me optimistic that it will actually be very helpful is that it is it is a very good concept I think it really thinks through this whole area of how we, you know, secure and there are active maintainers that are, you know, willing to get on email and help and the metadata standard to the metadata format is a an open standard. So, with all these things I would say, yeah in total has some, you know, some challenges there's a there's a learning curve involved but I are experimental results to date are that will will continue to work on it and there's some development. But look, you're going to have to start someplace. And I think this is a really good place to start and, and to start opening up the box. By the way, it's like I suggested earlier, a lot of people once they get their build system set up maybe they've worked super hard to optimize it like crazy and are not really excited about re architecting the thing. And I certainly not right before a product release or something like that. But what I would say is this is going to be worth the effort, particularly when you have people who are asking questions about hey, how protected you are against a crook or criminal a nation state from getting in and violating your stuff so I think it's really worth the effort to climb the learning curve and try and address some of these things. So the in the to do here in hardening your build chain. And I, you know, first of all, like automating your bills know, have no human interaction required, log and retain everything. It's, it's true that the stuff takes up this space but I think this is there's plenty of storage out there and I think this is important to do. So automating dependency checking and check the results. Some people I know for example, when they automate tools like this they'll go and put in some rule based thing to basically say hey when when do we want to flag and fail the bill. You know, is there a new CV that pops up. By the way, the other thing about dependency checking. Let me riff on this just a second as well is that when you think about automated dependency checking. It's not just when you have issued. Okay, here's my binary that I product right. I think part of it's going to be. If you're supporting that that thing giving long term support of some sort, like it's going to be important to keep running CV checking to make sure that new CVs haven't popped up because that may imply that you need to, you know, update the product and fix some of these things. And finally in total, I think is a terrific concept. It has a little bit of work. But I think this is the best thing that I've seen so far in addressing that particular area of securing the build infrastructure. Here's the here's the summary. We really, as a open source community need to take the supply chain threat seriously. This is something we're going to. We have been asked and probably if you're attending the conference and attending these talks, I think this is something you really need to take seriously. My advice is to analyze your setup using salsa and create a threat model around the way you're building your software. And then, and then use not only using the threat model but prioritizing the kinds of things you want to address. And then we've offered a couple of tools CD been tool and in total that are great starting points for a couple of these areas. And there are more that are being developed as we find more things to experiment with and put more, you know, methods out there will will certainly be helping this whole area. So thank you very much for attending. I appreciate any questions you might have and certainly open to fielding them at this time.