 Good morning, everyone. Yeah, we're gonna get started now. Yeah, this is my fourth and then we're ready. I'm ready to go home fourth briefing at DEF CON. Let's do it Okay, so since since I guess we introduce ourselves right now. I'm Katie Trimble I am the section chief for vulnerability management and coordination at Department of Homeland Security So that means that I do all things vulnerability Related to cyber security things. I do not do threat. I do vulnerability. So under that you might have they are they are they're very different So under that you might have heard of a couple programs that we run. They're small programs. Just real small So MITRE CVE program. So I'm the program manager for that and I sit on the MITRE CVE board of directors So if you have problem, please let me know because I'm the I'm the buck stops with Katie A positive feedback is appreciated negative feedback. I'm happy to discuss Then the NIST NVD program. I'm also responsible for that the Carnegie Mellon cert CC program. I sponsor that And then you're welcome Happy to help and the ICS cert vulnerability Management program I sponsor that as well. So about nine federal employees three contractors and 50 consultants in seven states So we're a real small program We coordinate vulnerabilities for disclosure. So about 7708 vulnerabilities from January 1st to June of this year alone last year was 14,000 it vulnerabilities 800 ICS Vulnerabilities for disclosure. So that is what I do and art Just watch here. So yeah, as Katie mentioned, I'm part of the Carnegie Mellon cert coordination center DHS is our largest sponsor of that work. So thank you. Thank you DHS Yeah, Art Manion. I'm a vulnerability analysis technical manager. I have a small team of around 15 there One of the main things we do is the coordinated disclosure work that Katie describes there The cert coordination start center started in 1988 The first thing we published was a vulnerability advisory for bugs that the Morris worm exploited and I think this week We published some an advisory for bugs that are a problem. So You know in 30 years, we have not been able to produce software. That is bug free we want to my attitude is to accept that and As a result We need to fix bugs as they're found and there's really no other answer then you have to be able to receive bug reports Fix them and move on with your business So we're gonna cover a couple of things here CVD is the coordinated vulnerability disclosure part and I will let Katie Take the first chunk here. We're just gonna do this back-and-forth game this whole time So sorry, it's not it's not normally this disruptive, but we'll deal with it All right. So we look at vulnerabilities. We say like what is a vulnerability? There's two kinds of vulnerabilities in this world There's a vulnerability the noun which is like all things inclusive. So like you don't have CCTVs on your building That's a vulnerability in this big like noun sense. And then there's a vulnerabilities as an adjective So something describing something else and that's really where we work. We work at the micro level So we're talking about specific vulnerability CVEs in software hardware and digital services So when we're looking at the definition of what we consider a vulnerability We're saying that it is a flaw in the system that allows an adversary to do things that we're not intended by the producer of that system They typically come in three forms. You have your hardware your software and your digital services When we would look at this we would say like Flight software in the aviation world would be so like a flight simulator software or software that allows the the Ground to communicate with whatever that those are softwares very very software specific hardware would be the plane And then digital services are things like Flight booking systems things that you have access to from the internet So that's kind of how we define them up until fairly recently vulnerabilities in digital services We're not something that was covered by Department of Homeland Security We only look at product vulnerabilities We've had to change scope a little bit to adapt for kind of the changing environment of coordinated vulnerability disclosure So we are kind of getting into the realm of digital services. So that's new for us. We're in a learning curve, too Go the next. All right. So coordinated vulnerability disclosure what that is Researchers come to us. We work with the research community and we say, okay Bring us your vulnerabilities. Did you contact the vendor if there's a problem with the vendor if the vendor is not Able to fix it or we're not willing to fix it or there's a contentious relationship. Someone has lawyered up Or if the vendor has absolutely no idea sometimes we see those in We have vendors who don't realize their vendors, which is Weird, but it happened somebody whose mission was to do something else and then they suddenly started selling their platform as a service They're like, oh, I'm a news outlet, but now I'm a software as a service vendor Yes, you are if you're selling your platform, you are definitely a vendor now Yeah, they're good, right? It's really entertaining. They needed an alternate revenue stream So they just started selling their their app to other other other news outlets. I'm like, oh, no so Researchers come to us anything that comes to us is exempt from if you're familiar with what's called the vulnerabilities equities process Which is the government-led process that weighs national security against? Intelligence collection, right? So it's about 18 different agencies all sit down at the table federal government agencies and talk about vulnerabilities DHS is on the defensive side of that which means we want to release everything Any vulnerability that's brought to us from the public that was and I'll give you the exact quote for it from the Charter Anything that was discovered during the course of incident response or security research Which is intended to be disclosed does not make the threshold for VEP Which means anything that's brought to the Department of Homeland Security by an independent source that was not discovered With the intention to use to attack an adversary does not make the threshold for VEP We don't provide any of that information everything we do is about disclosing we hold nothing back We have no big database behind us. We don't We're all about closing tickets my jobs to close tickets my performance plan is based on closing tickets If I don't close tickets I get in trouble. I have a time window for closing tickets. It's 45 days 45 days to 90 days depending on what it is. We have some flexibility in that So when we talk about building a mitigation schedule, that's that's that flexibility So researcher notifies us that there's a problem. There's a vulnerability. They've tried to coordinate it with the vendor They got no love there We then take that vulnerability Department of Homeland Security does not find vulnerabilities ourselves We cannot do that because they're legal liability issues. They're involved there in select cases We may but that's only at the express request of the vendor So we typically don't we pass those vulnerabilities to the vendor We immediately contact the vendor and say hey, we've got this vulnerability Can you please validate this look at this tell us what this is Then we work between the vendor and the researcher to come to a conclusion to define a mitigation schedule to get a patch created We make sure all the patches work And then we everyone publishes at the same time. That's what coordinated vulnerability disclosure is It's the opportunity for everyone to be on the same page, but no one to spill anything So it's kind of like a standoff everyone holds guns on each other and nobody says anything till everyone says everything And we do that usually within minutes of each other. We'll put out a technical advisory They'll put out a Their research findings and the vendor will put out their security bulletins. So that's and then we close the ticket close ticket Next So just real quickly is anyone in the room involved in supplying Aviation components hardware software Okay, so your aviation vendors of some of some kind. Yeah, okay, and if you don't want to raise your hand That's fine. I do want to note My part of my point in this presentation is to actually speak to that audience So the any everyone's welcome of course to see the talk, but uh, I want to reinforce a couple things Katie said this very first Block to first two blocks here in that first arrow If that doesn't happen none of the rest of this process takes place what you have is things like Researcher who finds no who gets no love Drops zero day on the internet and publishes, right? Or it doesn't ever get reported and there's a latent vulnerability sitting around for years and years and years and that's yeah, so Critical critical critical step is that Suppliers and vendors have a way to receive reports and then the rest of this process takes some work as well But please please please be open to receiving these reports. It's not all that difficult You know there's some discussion about the timelines and how long it takes to patch safety critical embedded systems We understand that's different than updating my Android phone or your iPhone, you know every every so often There are differences there, but please try to open up the doors to be able to receive the reports Yeah, okay, we'll go the next slide All right So in the cases that we don't do it in 45 days or 90 days that does happen It usually happens in our industrial control system world Which aviation falls into industrial control systems in the industrial industrial control systems We're looking at safety systems So we will hold a vulnerability and not disclose that during the typical 45 90-day time window if the vendor is being responsive And actively trying to fix the thing We want to be the honest broker in this situation We have no desire to ruin anyone's day But as I have a responsibility to the taxpayer and to the citizen to be the honest broker and to be the unbiased unbiased arbiter of this situation, so I always tell people I don't work for the vendor I don't work for the researcher I work for the taxpayer and my job is to keep eyes on prize and make sure that that System admin in Newington, New Hampshire has the access to the information that they need to do their job and to secure their systems appropriately So when we say we disclose there is flexibility in that timeline and we want to make sure everybody's rights are advocated for which includes the vendor We want to make sure that the vendor has the opportunity to fix the vulnerability and to be on board It helps the vendor be able to disclose in an appropriate way where their message is actually getting out appropriately The last thing we want is to say something and then the vendor to come back and be like actually you're completely wrong And here's all the ways you're wrong. We don't want that to happen So we painstakingly go through With the researcher and with the vendor and make sure that everyone's needs are being represented equally Sometimes that means that if you're 80% happy you're 20% unhappy and that happens That's the nature of compromise But we do work very diligently to do that. So why do we think that we should disclose? We believe it's herd immunity We think that our mission is to make software safer and we do this by saying that each individual disclosure may be painful But we have to look at the overarching ecosystem here So our job is to make things safer to be that pull that pulls the community in the in the way of safe safer and More secure software hardware and digital services as the government. We're not making money We provide everything for free to the customer So we have a unique capability to do that to be that pull to be that push So that that's kind of why we feel that you should disclose we think that the the the Vulnerability disclosure in and of itself may be painful But as we do it more and more and more this things becomes routine And it makes everybody safer shining the light of day is the best medicine for any for any kind of ailment So that's that's really the reason why we disclose You hit the next one. All right. So this is the vulnerability ecosystem as we see it within Department of Homeland Security This is based on product vulnerabilities, not digital services vulnerabilities, but essentially it would work the same way So when we're looking at this As I said earlier the researchers to the top swim lane is all happening internally under embargoed status Nobody is saying anything until everyone says everything So the researcher comes to us After they have gone to the vendor or the asset owner They come to us we do either it goes if it's an IT system it goes to Carnegie Mellon if it's an industrial control system it goes to Idaho national laboratories We look at that information. We validate that information. We work with the vendor to validate it if we don't have the equipment to do it ourselves We do some analysis looking at it and deduplicating. We will reserve a CVE We are CVE naming authorities ourselves and we can populate populate and publish CVE's ourselves Once we're all ready to go patches have been developed security research is ready to go Everyone's on the same page. We then move into the bottom swim lane, which is the public facing our architecture So everyone publishes at the same time the public advisory goes out that pop that CVE that was pre-reserved is now populated It flows over into the NVD catalog, which is where enriching information and severity scores are applied the the Researcher publishes their results. We publish our results and the vendor publishes their results at the same time All of this happens within minutes of each other That is coordinated vulnerability disclosure One little quick note there. There's usually a lot of questions between CVE and NVD and what the differences are So if you look at CVE, CVE is the common vulnerability in exposures So it is the definition so think of it like a dictionary and then think of NVD like an encyclopedia It expands upon that word and provides you more enriching information than the original tag does Yep art So if you believe if you believe our pleas to please implement coordinated vulnerability disclosure and like our process and think it'll be great How do I go about this? We have some some pointers to how how to get started here? This is a I'm supposed to try to use more pictures in my slide So I took a picture of words and put them in my slides, so I'm not doing that well yet But um, this is from the ANSI as those American something national standards something ANSI will sell you resell you ISO standards, and this is a I didn't know there was a set This is both of the ISO standards. This is international We have countries throughout the world asking their vendors to implement this before you sell to their governments International standards on how to do this these are not without without giving away ANSI and ISOs funding stream I'm personally responsible for a lot of the words in both of these They are good. They are not super owners There are not a lot of very important shouts that are very hard to implement all the basic stuff Katie just covered is basically in there So it's it's fairly straightforward, but just to show there's there's ample ample Documentation on how to go about this. There's a bunch of free stuff too if you search around So don't don't feel you have to pay whatever Swiss francs it costs to get these things So also, you know so a little bit of how I might go about this a little more of the why? We are seeing and the medical device Sector in the US is a clear example of this the FDA medical device regulator in the US is telling medical device manufacturers that Coordinated disclosure is part of your pre-market Sorry post market right right post market. They have both out. Sorry the post market guidance. So part of Managing and maintaining and keeping tabs on my medical device Is that I will fix bugs and receive bug reports and that's the FDA's out ahead of this thing, but also proof that other other Embedded safety critical sectors can do this and have done it ICS has come a long way There's an ice does an entire ICS search and there wasn't so one was created and it works It's right. You remember when there was not one, right? Yeah, okay And here we have the regulator act taking an active active role the FDA's been out at Vegas all week doing stuff So they're over at the biohacking village, but it's possible. It's been done Not a lot of whole lot of argument as to why not anymore. So, okay Moving on other couple of our concepts again for the for the suppliers in the room software bill of materials Anyone heard this term recently at all. Yep. Okay, right. So do you know all the yeah? Did you talk to Alan Friedman recently? All right Does anyone you know all the software running on your stuff? I suspect some folks in this industry might I mean all the software how many DNS resolvers do you have? How many versions of open SSL do you have? Does your supplier and their supplier and their supplier know Using any use any Broadcom stuff? Do you have vx works running anywhere because those have vulnerabilities like everything and they're embedded in the supply chain You know who interpeak is and what ip net is because you need to know this it's part of your supply chain This is just a dumb example. We're gonna follow Maybe use open source gpl to code And you have a legal requirement to track that code and sort of pass the license along if I fly home today And I do the back of my infotainment system somewhere in the about menu is going to be a bunch of open source stuff If they're following the legal requirement You like high assurance components where they have not been tampered with and you have assurance that they're legit Yes, we want that into safety industry Um and overall lower cost better efficiency better higher quality suppliers fewer suppliers There's a bunch of dimming work on this The theory is that cleaner supply chain gets you security and lower cost two for one This is from a a very nice, uh, well a very good advisory from the researcher on some Broadcom Stack vulnerabilities and a very simple statement here at the time of the publication at least a year ago We don't know what's affected and today we know exactly nothing more than that It's very very difficult to track this Broadcom is actually not super forthcoming about they have like a proprietary driver that they probably fixed There's evidence of that and it's an open source driver with some patches And it's not clear that they fixed it no one will actually say anything And I don't know if anyone's actually actually tested horrible Also, if you read this one, there's a very very long disclosure timeline which illustrates katie's point earlier Blow by blow by blow the researchers path through the coordination process And I'm pretty sure that one of us is in there later on so, okay Uh More bill of material stuff, right? So wind river systems vx works actually intel about wind river systems Recently had oh, sorry armus armus published urgent 11 a couple of weeks ago And all the news was vx works vx works wind river systems And if you read a little more carefully and the wind river advisory is very clear on this But you have to read all the words not just a title They talk about interpeak software an entire company that they purchased in 2006 that before that date sold their vulnerable tcpi stack to other people like Integrity osc. I don't know who those systems are does anyone know familiar with those? Yeah Oh green hills is integrity. Okay, sure. So that's the linux c linux is green hills linux or is it a separate thing entirely? Okay Yeah, see what I know right so point being it's not the scope of this is not vx works It is broader than vx works and you have to go back up the supply chain to interpeak And then back out to everyone else who's affected. Do we know who's affected by this? No Here's what we're asking for Could we start with an ingredients list? This is a bad, you know bad zoom, but we have this for for food, right? It's not telling me do or don't eat this the answer is probably shouldn't eat it But you know And nothing against the read-os I love to read-os, but they're not really healthy for you So but you know if I if I am gluten-free I can read way in there or something right and no not to eat This so it's my it's the it's the consumer's choice right the the parts consumer's choice to decide Is this good for me or not? At least I know what's in there. So we want the transparency. So here's the Allen Friedman pitch There is currently a department of commerce in tia effort going on On software component transparency we call it s-bomb in the working groups, but transparency is the more politically correct term Meeting on september 5th in person wide open to the public, which is one of the ups and downs about these processes This has been going on for a year plus at this point and we are nearing Conclusion which will probably be I'm going to guess three sort of white papers come out But there's been a lot of thought going into What does a really interoperable global supply chain? S-bomb system look like it's actually a hard hard hard problem because you have to uniquely name all the pieces of software on the internet But this is going on you can there's a lot more on this website You just you can follow some of the links and find some of the works in progress and a bunch of references So going quickly next acronym secure over-the-air updates. We want secure updates. Remember All the software is vulnerable and we're going to accept that and that's okay But we have to patch it because it's vulnerable right in most cases isolate works mostly And okay, so does the nice acronym, but over the air really works best with your crappy iot and your phones Are already on the on the on the wireless network. That's fine I'm not suggesting that airplanes just grab ota updates from wherever but You can take out the ota part But we want secure updates is the very is the very important piece here, right? Secure is super super important even if over the air is not Downloading and running something without verifying that it's solid, you know cryptographically verifying that it's good Bad bad bad idea The cve dictionary is littered with examples of Insecure update in blank insecure update in blank people keep not doing these correctly It takes some work You have to build this into the the platform in your your architecture and your infrastructure You have to do key management of some kind and we have cases where You know a window of microsoft Following the pki one of their code signing keys gets popped and used by malware So it passes, you know microsoft's code signing checks. So you have centralized risk If all of the updates are signed by the keys in one location and there's one update server And somebody spends a lot of money to pop it like not pet you that cost 10 billion dollars is the wired estimate That's bad news So you have to have we want you want your secure updates, but you have to be careful with the update process itself We recognize i'm not a i'm not a safety critical systems expert But i've i'm able to read enough words that i understand that i have a certified system It's locked down. It's set a certain way. I made a change to it That may require a recertification of the whole thing of part of it That's an extra cost and a problem we have to consider I don't have an answer but i understand that it's not as simple as i'm going to accept my android update at some point Probably when i'm back in pittsburgh, uh just just in case And you know something went wrong right i i updated the thing and before i Put the equipment back into production i tested and something's not doesn't sent i'm getting an error code doesn't feel right We want a built-in way to roll back a separate partition Maybe two computers on the piece of hardware case one, you know, we're talking about redundancy anyway, so Turn that one off reboot go back to the known good known good configuration. There's tricks to do this I have a dumb pf since firewall not a critical system. It has that sort of fallback Capability built into it open source free firewall software um So how do I do this you can read about these two things, uh, i believe this is an nsf partly funded Bit of work All of the details that i glossed over about how to all the little devils in the details on how to do secure updates Read about the update framework Uptane is a update framework specific example for automobiles Uh automobile manufacturers are using up tame real ones in real life to do their updates. So it's entirely possible Okay, that was a lot of information to kind of thrown at you really really quickly and we're sorry about that But we only get so much time. Uh, we have about five minutes left I want to finish by saying like we understand that this is complicated And we're asking for a lot of things but the reason we titled this uh the ideas whose time has come Is because we genuinely believe that these things need to be adopted They need to have been adopted years ago and we're still not saying it So we're talking about safety critical things here. Uh, these are basic things that we can do We're not naive enough to believe that it's just a quick fix over the overnight and we can all get it Worked on we understand the software bill of sales is complicated and hard and we haven't worked through the Through the kinks on it. We get it. We totally do I think we're all intelligent people and everyone in this room is an intelligent person Who understands the the complexity and the nuance of what's actually going on here But we do believe very very strongly that some of these things need to happen Um, I will yeah, so I just want to finish up with this one quote Whether and then we'll we'll take a couple questions because I think we have a minute or two Um, I I tend to always when you hear me talk, I will always say these things I will always say one the opposite of love is not hate. It's indifference Um, and that means that if you have negative things that you want to say to me That is okay. Tell me those things because feedback is so important in this cycle I can take something being something negative as long as you're professional and not abusive Um, if when you stop talking to me, that's when I know there's really a problem because that means that tells me that you don't care anymore And that's the last thing we want to see the second is that I love this quote It's uh, vulnerability sounds like truth and feels like courage truth and courage are never are always comfortable But they're never weakness we often come to the table believing that if we just hide things or don't admit to things And they don't exist anymore. That's not true. Also when we come to the table, that's not admitting a place of weakness That's not saying you've lost ground. Um, these things are uncomfortable. We get it We totally do we deal with 16 critical infrastructure sectors. I've dealt with thousands of vulnerabilities. I told you how many when we started Um, this can be done. It has been done. We have several success stories and it's been going on for 30 years Um Are and then we got a question question question sir. Yeah Yes So, uh, I'll point to recently we released a disclosure, um an aviation disclosure From day one when we received that we brought in the faa and the aviation isaac and we coordinated with both of them On that disclosure well before we ever released it to the public. We don't want anyone caught off guard That's the point of coordinated voter vulnerability disclosure We want to bring in the people who have the opportunity to fix things We do have timelines that are one to two years They usually come in the form of things like nuclear power plants. Nobody wants to see that fail So you get a faulty patch and it's a bad bad day for hundreds and hundreds of millions of people Um, so we we do we do that and as I said as long as we have a positive Relationship with the vendor and the vendor is being responsive If the vendor is trying to stall or the researcher is trying to stall or there's just negativity or something going on there Then then we'll move forward But for the most part we have we do our very best to make sure that everyone's needs are being advocated for and we're Very very responsive to the fact that we understand things take longer The aviation timeline is 25 years Like we get it we truly do but what we don't want to see is the same flaws that exist right now built into the next harbor life cycle We realize, you know Ours our cert cc's is a 45 day soft deadline google project zero is 90 with an optional 14 We fully recognize there's not long enough, but forever is also too long So we need to you know years small number of years large number of months need to get it down So Were you paid to ask that question? Did somebody like Okay, awesome. Um, so the first step for me and my world is make sure if you are a vendor that there is a clear channel For researchers or public policy people to talk to you Um, there's this weird thing that has happened throughout the years here And uh, we've done all the social engineering training and now people are doing all the social engineering training Um, and so what happens is that uh in part of my mind. I'm like, oh, yeah, good job And the other part i'm like, oh no Because when I call hi, i'm katie from the department of homeland security And i'd like to talk to your product security team because we believe we found a vulnerability in your software They go, that's nice homeland security. Uh-huh click And we spend days going through the same 1 800 number that everyone else does So I would say the first step is to publicly Acknowledge and create a vulnerability disclosure program so that a researcher or a public policy member or somebody who needs To talk to that that particular group of people who can handle that Has the avenue to do that that saves every one time and effort and what happens is when when people can't get to the Right people then they go crazy and they just published on twitter and we don't need that Um, the it world has gotten really good at this and so we'd like to see that Sir you had one more. I think we were gonna Yes, absolutely. It's totally true and it doesn't have to happen We we've had several sectors when we first tried to start doing this medical was one of them Um, every single conversation was a conversation between lawyers We don't need to do that as this gets more routine. We it becomes less and less Argumentative. Yeah, I think we're being kicked off stage. Uh, we're we're able to answer questions. Thank you all for your time Yes, thank you Thank you aviation village. Well done this year