 All right. All right. All right. All right. What's up, y'all? All right, welcome to my talk IOT under the microscope here Virtually it's kind of disappointing. I'm looking at a screen here instead of all you in the in the audience But hopefully we'll get through this and we'll have a good time doing it I want to talk about vulnerability trends in the supply chain I've got some very interesting things I think that we found in our in our data set size So hopefully you guys will learn something here and we'll have a little fun while doing it. All right Okay So who am I? I'm Parker Wixel I'm born and raised here in Columbus, Ohio. I've got 25 years industry experience on cybersecurity on software development full full stack development Last 10 years of which have really been focused on cybersecurity research and product development I was a contributor and developer of open-source security projects like AFL unicorn It's a fuzzing framework for emulated binaries and then patchwork Which is a Compilation static compilation of Linux kernels or patching of Linux kernels for debugging purposes Like it was mentioned I'm a senior engineer at finite state. We're a IOT cybersecurity firm I'm Dealing with a lot of the topics that we're talking about here and all my data sets and stuff like that come from our finite state Repos and some of the products that we're working on here. I'm also a database lecturer over at the Ohio State University Looking to kick off yet another fall there and then I'm a composer and a musician Don't hold that against me. I realize early on that There's not a lot of money in music. So here I am on just another passion of mine computers All right, so why is this talk relevant to your interests? We're gonna be talking about supply chain trends vulnerable and not vulnerable vulnerability standards and reporting and then some firmware statistics and observations Probably about the first half of the talk is going to be developed delving into the background of what? Supply chains are some of the ways that we have to talk about vulnerabilities What the supply chain introduces as far as? vulnerabilities and Visibility into those and then probably the last half will be delving into the fund numbers Probably why you came to see this talk, but nevertheless, hopefully we learned something in both both parts Okay, so our data set that that I'm pulling from right now We do have partnerships with some private industry partners, so we do not include all of our private repos and stuff like that, but For this particular talk, I've got about 7 million files represents about 50,000 firmware images 10,000 distinct product lines and 150 different vendors This is different architectures different operating systems, you know a lot of them are Linux based a lot of some of them are our toss obviously And we're hitting all the different verticals that usually hear about in these talks medical devices critical infrastructure security devices home routers Alexa, you know, whatever We've got a bunch of different types of products in our data set So hopefully the statistics that we talk about here on the second half Keep this in mind. It's it's fun to be able to troll a data set of this size Okay, so let's take a step back. Let's talk about the supply chain. Let's talk about the problems That are introduced as a part of the supply chain and what the problems are maybe even go into some of the solutions All right. All right. So if you are a manufacturer XYZ of Security camera that security camera is running some sort of firmware on it There there's hardware and there's firmware the firmware is actually software that's written for that hardware device We like to think of it as firmware because it's kind of baked in It's usually not as fluid or as dynamic as software tends to be in say a PC or whatever But the camera still has a full Processor memory architecture. It's a full computer running in that thing So if you can take advantage of that or take over that product from a vulnerability standpoint You've accessed a whole computer's worth of resources You have hardware components that go in the thing you have drivers that talk to this hardware is operating systems libraries apps You name it. They're all just the same Except they're called firmware the problem is on a security camera like this or any kind of IOT device is you're going to have Multiple vendors. So if you're a company XYZ, you're making this product The those hardware components may come from various different vendors All right, and then there may be other vendors like vendor a who talks to your underlying camera optics and can put you know Some image recognition on top of it or whatever else like that vendor be maybe some support Some support libraries or whatever The real problem comes in is not only do you have to track vendor a vendor v vendor c and what all they're putting onto your device Which quite frankly is not always the case. There's not always full disclosure there But then vendor a and vendor B may rely on an open-source library somewhere vendor X that you don't even know about They may or may not even know that they're using it depending on if their developers have reported on it And the the thing that makes that even worse is that vendor X library say it's the same low-level image processing library It may be version 1.1 in vendor a and vendor B in their libraries that they're including on your device has 1.5 Maybe vendor a's 1.1 version of vendor X has a vulnerability in it and the 1.5 version doesn't How do you know what all component libraries you have in the simple camera? This is a full computer running on your network, right? So Donald rumsfeld when he was secretary of defense brought to this notion together that information can be divided into three categories He said at the time no knowns known unknowns Unknown unknowns. All right. And so we take that kind of approach towards IOT vulnerabilities Our known knowns our vulnerabilities that have explicitly been discovered through scanning and testing on our devices So we've tested it. We know that there's a vulnerability. We patch it or whatever like that. So that's our known knowns All right our known unknowns our newly created software versions or or Or even just upgraded versions or whatever that we've pulled in libraries Whatever that we don't have any kind of application testing behind yet, right? So who knows What is going on under the hood there? So we know the device. We just don't know if there's any vulnerabilities there and Then in the last categorization of the three unknown unknowns are vulnerabilities that are in your camera or Your device that you don't know and that nobody else knows and these are what we're calling zero days All right, so these zero days or not zero days because we haven't even We haven't even discovered them Like say ripple 20 ripple 20 was just discovered a month or two ago before then It was an unknown unknown. It was in all these different devices But nobody knew it was there, but the weakness was still there waiting to be discovered. All right so there's an awful lot of Work to be done in discovering zero days as a security researcher We know that trying to protect yourself and to get through all that is really Is really a challenge trying to find out what you're what is vulnerable or not? So For this talk, we're going to talk about known or unknown known This is a fourth dimension that we like to talk about and it's comprising that which we Intentionally refuse to acknowledge that we know or we don't like to know okay So that they're vulnerabilities that are known to exist But have not been associated with all the systems that are actually affected So we know all these CVs are against this open SSL library But we don't know if that SSL library is in our device. So we're just going to kind of ignore it for now. All right So this unknown known is where we can do an awful lot on our part as manufacturers or security researchers to ferret out and to discover and a patch before Other actors out there find those same vulnerabilities and test them against your same device. Okay So all of that can be done through a software bill of materials. So a software bill of materials or an SBOM is There's a bill of materials that usually comes in the manufacturing world that they're very used to of all the components that make up Certain devices. So if you're getting big printing presses or whatever else like that You know all the pieces that make up that printing press so you can plan maintenance, you know, all you know What the cost is going to be up front? So as a industry software industry iot industry We should have the same thing the software bill of materials, but we don't all right. So manufacturers don't know all their components all the different chips System on chips that are running inside their devices all kinds of vendors don't know all their suppliers And their suppliers suppliers because a lot of times vendors will Make a product, but then they'll ship it off to a manufacturer to actually make for them And so how do you validate that? You know that things are exactly as you designed them or whatever and then consumers those of us who put these devices in our critical infrastructure into our security systems into our monitoring software or our monitoring networks even in our homes We put these devices into our networks But we don't know all the software that's running on those devices. So the analogy is like if you found a laptop out in the parking lot Of of your company or whatever would you Bring that laptop in fire it up boot it up and plug it into your critical infrastructure security system network Just to poke around on the laptop and see what's running on it And I would hope that most of you or all of you would say no There's not a chance that you would ever do that Because we know That those are full computer systems with operating systems that can be compromised with viruses malicious software, etc But yet with that exact same mentality, we don't apply to iot devices We'll take a camera That we do not know all the supply and all the software that's running on and all the weaknesses that might be inherent on the Softwares and we'll take that camera and we'll plug that same camera Into and talk to it on our critical infrastructure network. All right, so we don't have We don't have any way to enumerate this. So How do we generate one of these software build materials reliably? How do we keep track of all those? Say it was even possible to keep track of all the components that are in there How do we validate one of our devices against one? So we as manufacturers may even develop a device and we might know exactly what we want on it But if we send it away and it comes back Who's to say that whoever made that device for us put our firmware on there exactly as it was intended And they didn't slip something else in The the other thing would be and this has happened Is if you as a consumer have a device and you want to update it to the latest update patch Say there's a security vulnerability and you want to patch it You go to the manufacturer or the vendor's site and you download The update for that firmware and you flash your device with that firmware. How do you know? That software wasn't compromised. We have seen in the industry places where upload servers have been compromised by malicious actors and custom firmware has been placed in there as updates And customers have downloaded malicious updates to their devices, which might have been perfectly fine in the first place But now are running malicious software. So not only generating the software build materials, but validating Against the build materials is critical and being able to do that. All right So let's shift away One thing I'd like to to mention about company commitment stuff like that is I just read Microsoft has a product azure sphere that they're developing. That's a secure iot chip and it's one way of approaching it Hey, let's control the let's control the ecosystem from the beginning and secure it down And they've made commitments to the software build materials So it would be nice if more companies could be able to control their environment such as Microsoft has the luxury to do Um, and I think these kind of commitments are going to be our way forward is following Practices like this generating our own software build materials taking a hard look at what's running on that. All right So let's switch to today We have these devices. We don't know what's running on them. Let's look at the cve cpe reporting mechanisms That we have for finding vulnerabilities in our software. Okay, so cve is common vulnerability exposures um, it's a system that Provides reference methods for publicly known information security vulnerabilities and exposures. Okay, so it's been around a while MITRE is the one who who came up with that and and help maintains that So these are all the vulnerabilities that are discovered and reported for public knowledge. All right So it's great. We have a central place to report vulnerabilities As well, we have a national vulnerability database with the u.s. Government that keeps track of cpes or common platform So this is a structure naming schemes for naming products. Okay, this is system softwares software packages, etc So there's a common way to put all this together The only problem is is that we have these these frameworks For um, enumerating these kinds of vulnerabilities or these products, but there's not a lot of adherence or, um, sorry Hard regulations around how we use this. It's a very flexible system and it works fairly well When treated well, but there's a lot of inconsistently inconsistency across the across the Whole space about how we use cvs how we report them how we link them to products that they Apply to etc. So let's look at some of these. All right Our cpes just complete products. Is it your whole camera system? Is that whole device? One product a cpe are the component systems in there You know your optics and stuff like that are the system on chips that are running on there Is that a cpe should that be have its own product entry into that database? Are there libraries within you're using open ssl? That's great But what about lib crypto that lives within it is is could that be separate from open ssl? Could that be its own product? So there's an awful lot of questions that we need to answer with all that So if we look at something like open ssl It's a you know, it's a commercial grade secure sockets layer Also has cryptographic libraries in it if you go to the nvd the database and look up open ssl Because you want to find out what product it relates to because maybe you found a vulnerability Open ssl brings back 405 results Now every single version of that software is going to be another result So 405 is not necessarily all the different types of open ssl But there are several so here's a few examples that come back with open ssl So the very top one is usually what we think of is open ssl. It's a cc plus plus library That is compiled into a lot of linux systems and stuff with open with open ssl included 0.9.7 was particularly vulnerable. There's a lot of different cvs on there is 0.9.8 1.0 0.1 etc. So there's a lot of different versions of that those star fields We'll just kind of gloss over for now But those are further ways that you can enumerate or specify what specific Product this is if there's a lot of different versions Betas alphas different platforms, etc But what are all these other cvs or cpes that we see out there? We see This calderon pi open ssl. So the first part is our vendor. The second part is the actual product So we have this pi open ssl I guess we can make a guess that that would be a python binding for open ssl or a python library Then we have lua open ssl on the next line. So we guess maybe that's lua script But then look at the next two we have node open ssl And if you go all the way down to the node.js, that's the target software that it's targeting not the language but the target software which is it's it's written for node j s right So it's a job the script module written for node But the very next line open ssl.js project Open ssl.js So you have node open ssl and then you have open ssl.js and you have node j s at the end So now you have some someone who's writing for node That has their own way of specifying but you have two different competing libraries Which one is which and then you go down to the last one open ssl project open ssl Well now that looks an awful lot like the first one open ssl So which one is that well the hard part is is we really don't know And if you dig into the metadata about this and you actually go to the the web page The title doesn't help you very much open ssl project open ssl You know, so we go into the references and in the changelog We see a github reference there and in there we have a rust open ssl And if you go in there and you look at these different things you have these keywords of rust Ah, that's for the rust language. Okay, so finding Prod a prod platform. Sorry cpes Can be a real challenge. The other thing that's a real challenge here is we don't know between cpes What relates to each other does um, does this rust language depend on open ssl of a certain version? The cc plus plus version of the first line to be a certain version, right? Does 092 map to 097 or 097 a Or 1.01 we don't have these kinds of interrelations. So not only is generating us an s-bomb difficult but developing a body of ground truth around an s-bomb is extremely difficult and it's it's um It's not a problem that you think of because you think it should be obvious But who owns that does the company is it the company's responsibility to To add their their platform To this cpe database so that if vulnerabilities are found against it that there's reliable data there Or is it the cvs who find the vulnerabilities? Is it their job to correctly find and identify the platforms if they're not there? So That's just a little background here on the cpe system They're good systems to start off with but we need more on top of that. We need some ground truth We need some ways to relate this. Um, by the way, there are some projects that are starting to put these together But it's it's hard. Do you scrape the apt get repose for what components are in open ssl or HTTP servers, you know, certain HTTP servers rely on open ssl Which versions go with which? Well, um, if you're installing it, you know, because you look at the readme for the HTTP Installer right and it'll tell you which versions you have But we don't have any way of systematically obtaining any of that information Okay So let's go to a specific example of a vulnerability and let's look at the supply chain here We'll go with an old example. Um, this is from four years ago I'm not going to try to butcher robert's last name But darkonius he released a write-up on a router back door that he had discovered originating from a version of open wrt and open source and you know router Operating system from at that point was 10 years ago was 10 years in the past. So There was this back door hash. That's down in the pseudocode below 1d 6 8d whatever if you entered in that on the command line you immediately got a root show and you can see the relevant code there The problem is is this was just in an open wrt version. Uh, that was 10 years old from I guess 2006 Right, but the the write-up that he had was this was not on an open wrt router This was on a commercial router That had included open wrt And it's libraries and we're using components such as this component script called logon.s.sh as a part of their um as a part of their Operating system. All right So darkonius found the back door in one or two sets of devices him and And if you read through the comments they different people were chiming in Oh, I just tested it against this and I found it on this and this so we found You know in the comments three four maybe five models of devices listed But how is darkonius supposed to know and go out and find is it his responsibility to go find Every single device that this relates to is it even his Obligation to go to The manufacturer of this device and tell him hey, I found this in there If you go and you look for that this cde You don't find this cde In the in the database What's if you go even further to the cpe the platform for this device that he he looked for You don't even find cpes that are related to this specific device So and this is not a small device. This is a this is a this is a device that made its its rounds and you can you can find it different places So whose responsibility is it? In our data set when we've looked at all of our firmware just sitting around We just did a string search for um that magic hash And lo and behold we found three thousand eight hundred and ten files That had this hash in it and it wasn't just the hash It was the full login script But notice the file name cli to factory boot.sh login.sh So the original login.sh was a part of the original package But we found the actual login.sh encapsulated in a binary call the command line interface the cli Or even two that was used by web servers and stuff on iot devices That still ran the same code from 2006 that nobody i'm sure nobody knows Is in there the the manufacturer doesn't even know it's there anymore um And the thing is is that three major vendor companies have this 44 different product models and 281 versions of this firmware. So this open Wrt login.sh this this hash has made its way into many many many different vendors This isn't just one person accidentally downloading it and putting it in this is the supply chain of people Taking from a who takes from x who takes from y and it multiplies So this is the heart of the problem. This is this is why we're in the bind that we're in Is the supply chain, okay? We'll do one more example before we get into the more juicy stuff the statistics that we found um This latest cbcp that that we're talking about is ripple 20. I'm sure you guys have heard of it 19 zero day vulnerabilities amplified by the supply chain. Its title says it all all right And this was two months ago that this was or so Was reported and according to their uh white paper on it This affects hundreds of millions of devices or more Includes multiple remote code execution vulnerabilities. All right So that's the worst kind of vulnerability that you would want to have um, many other major international vendors suspected of being vulnerable and medical transportation industrial control systems et cetera, so My previous example kind of laid the ground work for how something like ripple 20 exists All right, so let's just take one of the cbcs as an example that they reported. All right So this was published uh, june Two months ago june 17 2020 This the description of it the trekked tcpip stack before this version allows remote code execution Related to ipv for tunneling. Okay, so this is a vulnerability. They said it's bad There's a base score of 10 critical on here because the remote code execution um, and how easy it is to theoretically get access to this tcpip stack If we look at the cpe that is related to all cve's have all the cpe's that are related to them We find these four trek versions Are all rolled up underneath this trek stack cpe The problem is is when we look for this trek stack and we go back to or we click on The actual record in the cpe database. We find the quick info is that The cpe for this trek stack was created almost a month After the vulnerability was discovered All right, so there was no product for tc for trek tcp. We had to invent Not we but the the the vulnerability researchers had to invent the cpe that even belonged to the trek stack, right? And then we go back to our original problem If you have this trek library and you're trying to figure out where all it's Where all it's at in the world You've got to kind of do some detective work and they did some detective work because we don't have any cpe to cpe Relationships and even if we did the cpe didn't even exist. Nobody even knew that trek tcp exists So I kind of imagine Reading through the paper about this I almost imagine louise from ant-man 2 um when he's being Interviewed where is scott, you know, and I kind of imagine him talking about, you know, hey louise. Where is trekking? He goes, hey, man See that's complicated. See trek. There's some smart guys, but they they need some help, right? So they go to elmick systems and they're like, hey, we need some Developers and elmick systems like yeah, we think you got it going on So let's get together and so they started to working together for some but then something happened They're like nah homie. We're split. We're out of here So trek goes off on their own way and elmick systems go off on their own way and trek be like we got the trek of tcp Ip stack we're going to market all this in united states and elmick systems are like no homie We're no longer the elmick systems. We're zook and elmick and we got the casago tcp I know it sounds like yours, but it's better because it's been renamed and you think you got USA and you think you're all hot hmm homie Dang we got all of asia that we're going through meanwhile. You got security researchers in the middle of light going hey Can you just tell me where trick is and trek is like See it's complicated and they go to zook and elmick and and and and see they're just like drick. They're like Like like where is where's this casago at and we can't even figure it out because the legal system is cutting us out Whereas trek is back here going. Let's see. I got an uncle's girlfriend boyfriend Who uh, who had a relative named hp and uh, it's kind of like the like the rona virus like right like like where's the contact on this like tracing and and and their uncle's Second cousin once removed is named aruba and I think we were at a party to get and meanwhile zook and elmick is like And jay soft is just like where is this and zook and elmick and trek and all these people can't find out and they're like That's what i'm trying to say it's complicated Right. I'm sorry. I probably butchered that but I can imagine industrial control systems and industrial Plants that that have sip 13 Investigators or or People coming in I can see their first slide to the sip 13 auditors are going to be this picture of louise and two words It's complicated, right because it is trying to find out where this check tcp stack was According to this paper pretty nuts. Okay So anyways, it was developed somewhere in the in the 90s to 2000 It was included in a large quantity of firmwares for devices There's been various patches to it some of the ripple 20 cbs that were even disclosed were patched In versions as far back as 2009 So You're a vendor What what products of yours contain this tcp IP stack? How do you know? How do you update a device when a patch is present or even know? What version of the trek stack you used to have in your device how serious is the threat to your device, right? Maybe maybe it's not Maybe it's not that problematic. Maybe it was just included but never used So reproducing cvs are problematic the actual attack vectors are really hard to classify in these individual products You potentially waste time from vendors on trivial vulnerabilities that really aren't practical So really the bottom line for vendors are How much money do you spend trying to find out whether you have to develop firmware patch for software that you don't even know is in your device? Or how likely is the software that it's even going to be exploitable, right? So it's a hard hard question So i'll come back to the software bill of materials here in a little bit But just for right now, let's make a little shift over into our Database and let's look at some of the vulnerability statistics that we have here Hopefully you guys find this enlightening. I certainly did. This is just kind of represents a slice of The industry we have medical devices. We have information control systems gear we have All sorts of different sectors represented and verticals in our data section. So here we go All right. So etsy resolve popular name servers 127 001 not surprising is one of the top ones. We got some google addresses there 888 8844 and then the next three down there that you see 168 are all asian dns's And then you've got some strange noise in there 192 168 someone com a couple others corporate, but so um You know if you have all these devices and they're going to the us and they have asian Domain servers you're going to have a performance hit or vice versa How do you know where these products are going and which name servers are hitting? I saw some forum posts about 192 168 0.7. Maybe the people who Made resolve dot or this etsy resolve had it on test and they were just copying and pasting from a forum Into this file someone com that actually was one vendor But it's on a whole slew of firmware for all sorts of different products in their product line So you can see not only Is that illustrative that someone has just put that in as an example and they found that example somewhere But it's in all their etsy results and they just copy and paste that file Into all their other firmware, right? So where's the checks and balances on what's actually running in there? um Even stuff like corp dot ubnt com corp dot ubnt com doesn't exist But it's running on a device as a resolve So if someone decides to turn up that host name and then start serving requests on there You could have some interesting results, right? Here's another one tty count in etsy secure tty So if you look at the bottom, uh etsy secure tty file allows you to specify which tt devices A root user is allowed to log in on so Let's just make one thing clear once you have an iot device in production You probably shouldn't be logging into that device at all All right, so most normal operation shouldn't allow anybody from the outside to ever be tunneling into that device remotely um Certainly not a root user. Okay, so I want to highlight this 196 148 So what that means is that these counts are the number of shells or sorry the number of tty's Listed as approved tty's that root can log into so 1486 firmware Have file has an etsy secure tty in it with 196 Approved tty locations for root to log in on All right, so I tracked this one down because this this file is just like where is this anomaly from I found a yachto project Which is a way to programmatically or to easily generate a linux operating system custom build for your project And this particular file that I found was from 2013. So here is a supply chain Someone who's building and wants a secure tty They pull down this file that's pretty permissive probably for development purposes and that really supposed to be for production And yet here it makes its way into all these devices. So here's the supply chain in action All right number of valid login shells. Okay, so this is in etsy shells. This is a this is the number of um acceptable different types of shells that you can log in from and most have one so Again, whether you should have a shell that people can log into or not That's questionable, but at least a lot of them majority of them have only one but then we have this 10 Login shells and I guess what they want by this a little tongue-in-cheek is You know if their hackers are coming in and are used to have all their key bindings and their Environment files for bash. They want to make it easier for them So they're just going to throw in bash in there and if you're ash or or zsh We got you covered. You can log in however you want Again a little tongue-in-cheek, but that's what that number represents Um an additional thing here that was kind of concerning is is when I got delved into the actual Types of devices as this was it was over 60 of these or models of patient monitoring health systems Okay, so over 60 different models which represents however many thousands or hundreds of thousands of health systems patient monitors that are sitting bedside Have 10 or so Login shells that somebody could log in with all right. These are just configuration examples in the supply chain. I'm not saying these are actual Vulnerabilities, they're just interesting anomalies from the supply chain. Okay So here's a question. We'll configs. We're going to we're we're about out of time So we're just going to go through these quick firmware where the root user has a login shell 15 345 all right So 30% of all of our firmware that we looked at has some sort of login shell Whereas if you don't want the root to log in use sbin login guess how many over under One we found one firmware that actually had sbin no longer in there Okay firmware with keys and authorized keys now This is a bad back door because this basically says if someone SSH is into your box if your key matches their key In the authorized keys you get to log in without any password. So this is a classic back door whether it was purposefully put there or not What's the over under 175 175 firmware remember this is firmware however many devices this represents This is 12 different vendors with with these keys. All right And uh over 29 these were really interesting known hosts to see where these kind of came from and and the different medical facilities and different places that had known hosts coming from them clam av only form for firmware had any kind of a Open antivirus software firmware running htbd with mod auto index So this auto index is Is a way to auto index Files on your web browser or your web server when there isn't any kind of match for the html name So it's a great way to kind of troll through your directories. So these are again Things that are questionable that are in our configs firmware starting t Uh tty and init tab. So quite a few firmware are starting tty Um, but not that many distinct files. So here we're seeing amplification in the supply chain of Here 63 files, but these 63 files are found in 4,729 devices Here's a particularly bad one firmware with php to default to display errors 332 firmware so Using sequel injection or any kind of command injection I love When developers leave on Their display errors because when I made a mistake, it tells me exactly what's there The problem is is that php to by default allows display errors. You have to explicitly turn it off So maybe we should be Making software that doesn't have defaults that are vulnerable like this in their default configuration All right. Next part firmware with insecure default des encryption So if you don't specify the type of encryption for your passwords, you're going to use des which is a bit weaker 318 firmware 29 files But even worse there were firmware that specified the flag to turn on md5 as their password Crypt it was 12 firmware. Fortunately, it's small, but they're still 12 firmware have no idea where those are at But they're out there firmware with a master dot password with no password on root. So in the password file there is no over under on 100 devices 10 devices a million devices not only seven, but still one file made its way in with no password on the root My sequel binding to 000 000. So listen to the world and allow people to go right to your my sequel 26 devices Firmware with red is the same way default world bind. So if you don't specify a bind by default, it will allow you in Three, it's only one file, but still three firmware allow you into their reddest by default Average number of unsafe function calls for firmware. So this one is any unsafe function calls is Stuff like stir copy mem copy all those unbounded ones like ripple 20 found and stuff like malicious ways to use Standard allocation functions without checking lengths and stuff like that. How many do you think in per firmware? 1500 and some average unsafe function calls per firmware. Okay number of firmware with unsafe function calls 23 000 that's over 50 50 percent and those are the ones that we've named we haven't even Trolled through those with function hashing and stuff to find the unlabeled ones that also Call these all right Firmware and here's the last one firmware exporting nfs mounts to the world. All right over under on this one 42 there are four files that continue to 42 firmware Melt to the world such as slash root mount a star read write So you can as root write to root as user id zero So these are all questionable configs that have made it their way Just fun things that I found trolling through and finally what it would have talked like this be without talking about the common passwords and and all that so You see here amplified here again is the supply chain where you have the file counts Of all those are the light orange and then the dark orange is actually how many firmware use those files in them and The number one being admin it's only in five files in the password file, but those five password files are found in 470 Firmware which may represent hundreds of thousands of devices where we've cracked the password admin I'm going to give you 10 to 1 guess what the password is going to be for that admin user all right So anyways In conclusion, it's all about the software bill of materials drive towards generating and validating these This is the path for manufacturing's implementing software inventory systems being Developers being diligent and reporting all the components used in their systems consumers having a high demand Or having a high standard in transparency And favorite companies and buying from companies with that info as well policy makers who helped guide These unified standards and reporting in formats And our security community here developing tools to assist in the automation of the inventory If we're doing the same thing now that in five years, we're going behind we should be using machine learning We should be using tools and other things to help us Inventory and validate these things not just on the front and on the back end to validate patches And updates are what they say they are and they contain what they think they think they contain So thanks to all my team at finite state and a lot of good research that's been in the industry Finally Obviously the question and answer on this session are going to the discord But I've had a fun time talking to y'all Peace out. Hope you enjoy the rest of def con and iot village. Thank you so much for having me