 Today, I'm glad to have Andre Kostin here today, and he studied computer science in Bucharest and recently finished his PhD on security and embedded systems. And he's done some pretty cool hacks, maybe some of you have looked them up. And he's the author of the toolkit, My Fair Classical Universal Toolkit, which enables you to crack RFID chips, basically. And he will talk about methods on how to make a large-scale security analysis of embedded devices possible. So enjoy it and give a warm welcome to Andre. Welcome to my talk. Thank you, Christian, for introducing me. My name is Andre Kostin. I'm a recent PhD graduate from Uricom in south of France, and today I'll be presenting you basically the quick squeeze of my three-year thesis on security of embedded devices as firmware, how to do it fast and furious at large scale with some demos. So a very short QMI, I'll not go too much, but in principle, I'm interested in embedded devices as such, regardless of the shape they take. They might take the shape of a RFID card or a credit card, or they take shapes of printers or funky CCTV cameras or even airplanes. All these are kind of examples of embedded devices fit in different bodies, let's say. But inside they're embedded devices, and this is what makes them a little bit different than normal PCs, and interesting for me. So we know that embedded devices are everywhere. It's not something new, and we are seeing this. We use them every day. We see that embedded devices, we are getting more complex and smarter because we really want to do more useful stuff with them, like writing a script for them, or calling an API, or trying to run some piece of advanced software on them, whether it's a smart home or a smart TV. And of course, embedded devices, they're also getting on top of that, they're also getting interconnected in the new buzzword, Internet of Things, IOT. And these layers of complexity, basically the interconnection of devices which before were not supposed to be connected, combined with new complexity of software and stacks and APIs, makes it a little bit harder to actually understand all the security surfaces. So it requires a little bit of a systematic approach. All these embedded devices, they run software. And regardless of their shape, how they look like, their software is called firmware because of historical reasons, so your smart TV, your smart meters, your home router, and even your hard drives in your PC, or network cards in your PC, they run these pieces of firmware, which is kind of a smaller set software operating system or not running inside of it. So there are some observations at present that back in 2014, we've seen at least in a part of the Internet, which is just a small observable part, that there are hundreds of thousands of firmware packages, which we've been able to download and partially to unpack and analyze. So there might be many more thousands of firmware packages, but at least we know there are hundreds of thousands of them. And there's also these interesting predictions that by 2014, there were 14 billion interconnected objects by Cisco Internet of Things connection counter. And the prediction is that the well-known hype is that by 2020, there will be between 20 and 50 billion of interconnected devices. So this basically brings us to importance of embedded devices and their firmware because the embedded devices are ubiquitous, even though they are invisible, they are essential to our lives. Think about pacemakers, think about ECUs in the cars, they can operate for many years, for example, for smart infrastructure, if it's buried in concrete or it's put somewhere in out of rich area, they should operate and they could operate for years even without getting no updates or no security updates. And they have a large attack surface because those devices do not have normal mouse keyboards, input output systems like keyboard, mouse, and monitors. So they need some kind of web interfaces, networking interfaces, and debugging interfaces so that they're able to operate easily. And it's not news that embedded devices are insecure. We've been seeing this trend from I guess 2005 or 2006 when increasingly more every day there are examples of embedded devices being insecure, like a lot of routers being insecure, when printers shown to be insecure on large scale, then VoIP phones and VoIP systems used in offices, cars shown back and even in present shown to be remotely hackable and insecure, drones, fireworks, all kinds of PLC, SCADA, industrial control systems. And the thing can go on because there's actually no limit on the type or application of embedded devices. And we can go on the blogs and on the reports and the white papers and see every type of embedded device have some kind of a flow. But what is the observation here is that most of these white papers or research was done in a manual or not necessarily just in a manual, but it was like every research was done in an individual manner, like taking a particular device or a particular embedded firmware and analyzing it heads to end. And basically it involves a lot of tedious work, a lot of manual work, a lot of scripting. And basically it cannot scale. If you want to understand what are the security implications of connected devices or security flows in hundreds of thousands of firmware images, we need scalable approaches. And these approaches where we do a lot of things manually just does not scale. I'll try to give a very quick overview of how usually the manual analysis process of firmware takes place. And for most of you, maybe it's not something new and for sure it's not. But the idea is that there's a firmware package. You download it, you get it on UCD, you get it on your USB stick or you pay hundreds of euros and get it on a fancy SD card if it's a SCADA or ICS system. You unpack or decrypt. And if you cannot decrypt, it's not a problem, you keep it and someday somebody will dump the key and you will be able to decrypt it and go back and do the process as if the firmware was decrypted, unencrypted. Then there's very generic steps like detecting the CPU or the type of binary, the load addresses, and then applying some kind of static analysis on the code or the source code of interpreted files and dynamic analysis. And then looking for some kinds of patterns, either in form of strings or function names or symbols or just chunks of codes which people try to do now with machine learning and intermediate language. But the idea is to try to find these kind of backdoors or debug interfaces or UART consoles or all those wounds. This can be done statically or usually can be done also dynamically, but it requires more effort as we'll go through. Then if you didn't have a device, you have to buy the device. Some people do it otherwise, they have a device, they dump the firmware, and then they start as if we started with the firmware itself. But in general case, if you want to do analysis on some devices, you don't want to buy every device every now and then just to analyze. So you want to start as cheap as possible and the cheapest start as possible is getting the firmware free from the Internet. And then if you find some indication that there's vulnerabilities, you will want to buy the device, you'll have to set up the device, meaning all the cabling, all the network configurations, all the drum base, right? And then eventually if you need to go even deeper or some vulnerabilities might require some physical intervention, you might need to even go and disassemble. So clearly there are two parts of this manual analysis process and we can see that the part of buying the device, setting up the device and disassembling it to go to its JTAG or UART console or just hook up on the microprocessors or flash memory is hard to automate because you don't have so much budget to buy all the devices, just go on Amazon and buy random devices that you think might be vulnerable. Second is not very easy to set up all the devices in uniform unless you have a very smart robot, maybe in a few years we'll have them. And also not very easy to automatically disassemble the device and go some robots to connect to the JTAGs and UART and find them. There are some steps towards the direction to automate this with JTAGulators and some moving parts to try to detect these UARTs. Still it's like a hard problem, it's hard to automate. On the other side we see that the first part is more or less easy to automate because getting the firmware is nothing but going to a download link that eventually logging in if it requires a username and password, downloading it, then trying to unpack it. If it doesn't work with a simple zip, you can use the brute force method, you just use the brute force and I'll show you some examples. And then you just try to do some heuristic, you detect some things which you might want to optimize for your analysis and then you just do static and dynamic analysis. Of course there are some caveats for all of these steps, but clearly this as a process as a fault can be automated. So going from this point forward we have this idea to have a large scale automated analysis to better understand and classify and analyze firmware images without using the devices. That is very important because we don't actually need these devices because most of them can be with more or less effort emulated if we really need to confirm something or not. So there are some challenges like the large number of devices, the large number of firmware files. These are the challenges which have to be overcome. Also the thing that the IoT and the embedded devices, they are using different kinds of architecture. Of course, the most well-known are ARM and MIPS and PowerPC maybe. But there's myriads of other microcontrollers and microprocessors which you really don't know what they are and also not necessarily all of them use Linux. So you would need to know where to load that bytecode and so on. And basically we also have highly unstructured firmware data. Even though you can go and download all the files on the Internet, you need to know what is what because otherwise you end up in a very inefficient, very inefficient brute forcing process and you need to be really focused when you have a lot of data. So to overcome these challenges, we have some solutions, we propose some solutions such as for the large number of devices, the first thing we want to do, no devices analysis. So we try to do no devices. The large number of firmware files, basically we need to analyze a lot of data, okay, we need to devise scalable architectures. Because the systems are heterogeneous, we need to devise techniques which are generic enough, which do not depend on particular CPU architecture or hardware architecture. We want to be either on the intermediate level or have some specialized techniques from which we can pivot. Then there's a focus on web and network interfaces and APIs because all these devices they expose these network interfaces and APIs and in particular they expose the web interface as a mean to remote management or remote access to the device. Having highly unstructured firmware data, we propose that you need to do a large dataset classification, meaning that you use machine learning to classify the firmers and so on. And we'll show some examples of this. So first challenge, like firmware and device classification, why is that a challenge? We know that, again, there's hundreds of thousands of firmware packages and of course I don't see many volunteers in the room to manually triage and classify them. And we basically propose machine learning and we, in particular in our case, we use scikit-learn Python package and we try to evaluate our approach. The idea is that if you have firmware files and these firmware files usually are specific to devices, so if there is a device which has a particular number of megabytes for Flash, then the firmware file, if it's a full update, will have more or less 16 megabytes, right? So there are these kind of firmware properties like file size, file entropy, some strings and so on which makes those firmers particular to that device or that vendor. So these are kind of the properties of the files, firmware files which we use like file size, entropy value, how densities, the strings inside and unique strings in each category. And then we try to use these as features in machine learning algorithms, random forests and decision trees. And we did some sampling and we found that in our case what worked the best was the random forest with size, entropy, strings and strings unique features. And I know it sounds a little bit too much for this presentation but you can find the details in the papers which are at the end of the slides. How these features were chosen and how they are generated. But the idea is that if we, for example, include another feature of firmers which is not very good, then we have the classification of firmers or detection of firmers not very accurate. So in our case, we've been able to achieve more than 90% accurate detection of firmware by vendor or by vendor and product. So it helps us because we can basically say from this big number of firmers, let's put these in chunks and we have firmers from vendor A, vendor B, vendor C and then we can apply specific unpacking techniques or specific known exploits to test if those firmers are vulnerable or not because we know it's vendor A or B or C. So this is our local optimum where I list the feature is the same as on the slide. And the idea is that I can try to give you, I'll show a demo at the end of the presentation. So the next challenge is the automated static analysis. So the idea is that if you have a large number of firmers you want to do the analysis as quick as possible and automated as possible. So in the first instance, what we did, we took the files which we crawled, the several hundreds thousand firmers, we put them in a database. We developed some processing unpacking and analysis nodes. And basically you can scale it inside your own organization or you can scale it to Amazon Web Services, to whatever number of nodes you need. And basically these nodes, what they do, they just receive a firmware, they try to unpack using Beanwalk or using binary analysis toolkit. These are two very awesome tools. You can combine them or write your own and then you unpack the files, you get the file system if you're lucky. If not, you can write your own unpackers and plug them into Beanwalk and binary analysis toolkit and scale it to many numbers of nodes. Then we do some simple static analysis like collecting default or backdoor passwords or unusual passwords. Or we try to see if there are private certificates, private files, and which are in the firmware and should not be there. Then we also store this in a firmware analysis and report database and do some data enrichment, which I'll come down later. So what kinds of static analysis we perform in the first place? We do check for web server configuration. We didn't run any static code analysis because it was our first experiment. We didn't know what exactly to expect, what was the percentage of CPU architectures, what was the percentage of operating systems. And it's really hard to have a full set of tools, especially if you work open source or free software to tackle all these challenges. When we also look for credentials, weak default or hard coded in ETC password or we look for basically for strings in the firmware. And then we try to map them to hash, hashes in ETC password. And we found like dozens of them where the string, string inside the firmware is actually a password for an entry in ETC hash. We also use the versions and the keywords. For example, versions are useful if you want to test statically or dynamically if a particular open SSL library shipped in a firmware is vulnerable or not. The thing is that just getting the firmware version is not enough because you never know whether the library or the software package was compiled with particular set of options or was privately patched by the vendor and the vulnerability is not there. So if you have a CVE, it might not work just because of this simple reason because they privately patched it or they compile it in a way that it doesn't trigger. So version is just an indication of what you might be looking for, but it's not like a definite hit. And then we also use some correlation which I'll show some examples of them. We use the strings, we use the fuzzy hashes and private SSL keys. The fuzzy hash is an interesting thing, basically is a forensic technique which compares files by their similarity. They do not compare it one to one as in crypto hashes. So it gives you an idea if there is a common file in different firmware packages and there's some small modification between vendors and it makes it easy to find cross vendor bugs as I'll show in a second. So in one example, we had this firmware SD card, Wi-Fi firmware. So basically these are SD cards which have Wi-Fi chips in them and there was a known vulnerability in it like command injection and XSS. We know that this particular file in that firmware is vulnerable. Then when we do fuzzy hash analysis, we see that there's a few other like firmware 2 and firmware 3 and firmware 5 and firmware 4 which have similar files and we try to correlate them by fuzzy hash. And we see that some files are similar by fuzzy hash. So if this particular file is vulnerable to XSS or remote code injection, then the other two, because of their similarity, they might be as well. So this is a very good indication that you should pivot from firmware 1 to firmware 2 and firmware 4 and testing for exactly the same vulnerabilities or exploits. And the thing is that sometimes you get different vendors hit by the same bug just because it's a white label product and they buy an SDK or a development kit and they use the very same vulnerable software packages. Another example which we found interesting is that we had HTTPS certificates in the firmers and they had private keys along with them which is a mistake, but okay, whatever. You might think that maybe at the first boot they regenerate the SSL certificates. We look it up in our database, we see that this particular SSL certificate corresponds to a CCTV vendor A and we actually found some vulnerabilities in this vendor A. Going forward, we take ZIMAP outputs and they have the HTTPS ecosystem scan. What we do, we try to correlate the same fingerprint of the certificates in the IPv4 fingerprints and we find that actually the same fingerprint which is from the certificate in a firmware file is found online and by random sampling, we found that some of these online devices which use the HTTPS certificate, they actually come from a vendor B and the conclusion is that most likely these two vendors are affected by the same vulnerability we found in the first place by analyzing the firmware and you can go and pivot from vendor A and basically go and exploit vendor B devices which are online and one example of this is, I'll just show you, this particular example, so you can see that it is a prominent sentry, it's a CCTV camera and you can see that it's a prominent sentry, right? And that particular example I was talking about, but when you see the SSL certificate, it's Brickon and I'll show you dynamic analysis of Brickon device and how we do it. So the idea is that if you have these kind of situations where you have devices labeled by different vendors but have vulnerabilities which you don't know, you will not know if you don't do it at a large scale, you'll just focus on the Brickon devices and you'll think only Brickon is vulnerable. And to give you an idea is that I have a virtual machine, you can see that I have the same IP but different ports, 3080, 4080 and 2080, so I started with a Brickon and you see that in that virtual machine I'm running an emulated firmware which I'll explain a little bit down later, but what I want to say is that once we have vendor A, let's say Brickon firmware and we can do dynamic analysis, we can emulate it and find vulnerabilities inside it, we can easily find automated exploits and use them against vendor B and we can do admin-admin and you can see that this is basically an automated emulated system which emulates automatically this firmware from this particular CCTV vendor and you can do pretty much, in our case we focused on the web interfaces and I'll go into details a little bit later, but I want to show you that it's possible to do dynamic analysis as well, not just looking for SSL certification, do this in an automated manner and you can see that the web interface is pretty responsive and you can do, yeah, you see? I mean, this is really awesome, you find bugs like that, right? And that's the thing that you can start penetration testing tools like Arachni, like W3AF or your preferred commercial tool and you will easily find the XSSs and remote code injections and remote command injections and SQLs, whatever that web interface might have. Okay, so it was a small parenthesis. So in that case of these Brickon cameras, we found one certificate and we found one vulnerability which was important and we correlated to two vendors and correlated to a total of 35,000 online devices and we have, at that time back in 2014, we had 109 private RSA keys which were without passphrase and we can skim the fingerprints and go look around on the internet using Shodan or Senseys, whatever your system is. So as a first result from that very simple, not very complex static analysis like no code static analysis, we found 38 new vulnerabilities like XSSs, backdoors, non-trivial default or hard-coded passwords which affected around 700 firmware images and we correlated some of these vulnerabilities to around 140 online devices using Shodan and Google Dorgs. So the last challenge we had or we tried to tackle is the automated dynamic analysis like a sneak peek what I have shown you on the firmware emulation of the Brickon cameras. So the idea is that we have these unpacked firmware sources or you might have tons of them unpacked, let's say. Then you want to select some of them to do dynamic analysis. Your selection criteria might vary. In our case we selected the ones which had clearly a web interface inside and we could start the web interface because usually the web interface is the first thing which ends up on the internet when you plug your device to your router or to your direct connection and this is the thing you usually want to access to manage your device remotely unless you use some smartphone application but which basically does the same through the cloud and piggybacks on some HTTP connection to your device. So we selected these. We do some file system preparation because if you did a lot of firmware unpacking you'll know how messy it is and especially when you have hundreds of thousands of files or very diverse then automatically unpacking and extracting you'll get a lot of false positive, a lot of junk data, a lot of broken files just because the unpacker was brute forcing too much or maybe just because they used a slightly different settings of the packer for the firmware for example for SquashFS or for JFFS2 they use Big Indian, Little Indian with some specific options and then the unpackers, the default unpackers do not work out of the book. So the file systems sometimes are broken and you need some heuristics to broke the SIM links, you need some heuristics to find that some files are not properly there and try to make them as good as possible so that your emulation will be successful. Then what you do, you also try to do it scalable like the static analysis you want to do the dynamic analysis as well. It's a little bit slower because it involves both emulation on a much slower emulator and then also involves dynamic analysis tools basically which are fuzzers or penetration testing tools which send a lot of input. And we perform this dynamic analysis. Using QEMU, we collect the results and we use it for machine learning and for improving the analysis or analyzing why the particular emulation or failed maybe there was a kernel problem or maybe there are missing kernel modules and this might happen a lot because not all firmware packages are full firmware packages. You must know that most of the firmers are kind of partial updates. Most of the vendors, they just send you a diff or the updates of the impacted files. They do not send you every time the whole image of the file system and that's why sometimes you need to know why the emulation or analysis is failing. But the idea is that the challenging thing is how do you scalably emulate heterogeneous architectures or in general, how you automatically emulate because if you do it by hand, it's funny but you get tired after half a day. Then we need to choose an emulator and there are lots of options how to do the emulation or the analysis. So there would be the ideal emulator but the problem, it doesn't exist. And when I'm talking about ideal emulation, is the emulator which you give it a file and it will automatically know the architecture and will be able to emulate the CPU, will automatically know the operating system and will know how to boot it and will know at which addresses basically to load all these files. But from what we know publicly such thing is not available. Then generic system emulators, it's the QEMU and there are two options to use it with original firmware and the original kernel or with original firmware and generic kernel. The problem is that not many firmware come with a kernel update. Most of them, if you look at them, they're on kernel 2.6 to 10 years ago. So they do not update very often the kernels and you will not get them very often in the firmware update. So this means that you'll have to either extract it by JTAG or use some manual process which doesn't scale. And in that case, the generic kernel approach where you have a generic kernel built with all the possible modules and options would be a better idea. And then there are some other options where you can just take the application you want to test and you don't test it in an emulator but you load it in a test environment which is different from the emulated environment. And this works very well for interpreted languages like PHP, Perl, and things like that. Basically, where there is a virtualization environment and you can replicate the virtualization environment without putting all the QEMU kitchen soup in there. Again, we had to make some choices. Perfect emulator doesn't exist, so we cannot use it. Not many firmers have kernels inside, so we cannot use it. And this was too experimental. The details about architectural CH route are in the paper. I'll not bore you now with this, but you can look it up what exactly it is. So we used these two approaches using the original firmware with a generic kernel and in cases where it applies to basically strip the whole document route if it's a Perl or PHP and put it in a X86 test environment and run it there under Apache or Light HDPD. Now, the idea is that you have the QEMU which runs inside the normal machine or it can run inside the VM. In our case, it runs inside the VM. We run on top of Dibian. These are technical details, but then we throw the firmware inside the QEMU and we CH route. The CH route is the procedure where basically you change the root file system, the perceived root file system of the environment you are working on, and you can basically say to the QEMU that from now on, the root of the firmware file is your root file system and you can basically use the environment from the firmware as if it was being provided by QEMU. So, in this way, you kind of can easily emulate. There are cavits with that, but it works pretty well for many firmers. Then you just, because you are in a CH route environment, you just bootstrap everything with Vinit, Linux, Rc, and so on and leave it run. Of course, some of them will fail because there's no device attached to QEMU because you need to write some plugins for QEMU in order to emulate the flash memory, the EEPROM memory, the serial port, the ethernet. Luckily, the networking is pretty okay done in QEMU and it works fine, but for the others, you would need to basically write a good set of plugin libraries which would be able to accommodate most of your devices, and once you have that, you can basically overcome those failures. And then you just fire the web server whether it's a BOA, LightHTPD, T-HTPD, and so on, or it's a standalone binary, and then you just interested in that part. In our case, we're interested in the web server part, and then we just plug the Arakhani Zap W3AF penetration testing tools. We use TCP dump in order to recover all the communication to be able to roll back and find what input trigger a particular exploit or a particular crash or a particular command injection. You can use Nmap, you can use Metasploit and Nessus, and basically fire all your arsenal of tools on the emulated systems on the ports you want to work on. In our case, we focused on the web interfaces, so we use these tools. And as a result, we had around 225 high severity vulnerabilities, meaning command injection, XSS, and CSRF, impacting around 10% of the original dataset we've started from, and these vulnerabilities impacted 25% of the vendors. The idea is that we didn't develop this framework to find all the vulnerabilities, but if you have such a framework and you can find the load hanging from very fast in one day and kill basically or close 10% of your vulnerabilities if you are a vendor, then it's a very useful tool, or if you are a pentester, you want to find all these things quickly. You don't want to spend all your days just going manually and try to emulate manually. Just leave it overnight. You come in the morning. You have 225 reports on 200 firmers, and your job is done for the day, right? No. Okay, so I showed you part of this demo. I mean, this is the BrickCom camera where some things you can play with, but what I wanted to show is that the process of starting from the firmware and going to the correlation, and then finding the interesting things. So, for example, if you have the analysis on these files, for example, you can correlate by string, and you can see that some of the firmers, in particular HP, for example, they had some firmware which they pulled out from the Internet for, I don't know, what reason, which contained this error, failed to open BackdoorNetChannel. I don't know what it means, but I think it's very not very funny. So, if you have this kind of, you can search by keywords, and you will be amazed how many firmers actually hit on the keyword Backdoor. Some of them, of course, are these snort or IDS rules which look for keyword Backdoor, and these rules basically give you a hit. But sometimes you actually find real Backdoors by searching for the keyword Backdoor, right? Now, I mean, they understood that this is very obvious, and if you know the story of Juniper, they started to use cryptograms. But I mean, if you have this kind of cryptograms, you can also start looking around at these firmware files, and then you look at the firmware details and what's the name, what's the version, and you can pivot from there. You can also look for those dual EC keys if they are present in this firmware or other firmers. But to give you an example of the fuzzy hash thing, as I mentioned you in the slides, somewhere here. Okay, so this is the fuzzy hash where I mentioned the SD cards and correlations of files. And the idea is that I can, I'll keep it shorter. We know this file is vulnerable in one of the SD cards. It was very well known blog post. We can look it by fuzzy hash, and we can actually see that this file is present in these firmers. Okay, there's some of them, and there's nothing particular because they're just incremental versions of the same device firmware. And we can also see that it's 100% similarity, so it means this file didn't change across versions. But the interesting thing is that you see that for some entries, the vendor change, and you can easily automate to trigger an alert or to basically pivot automatically when you have a similarity hit for one file across vendors. When it goes across vendors, you know that something is fishy. It means that the other vendor must be vulnerable. And you see that the similarity is not 100%, it's 46%. This is because these vendors of SD cards, they take the white label or SDKs or development kits, and they put their logo and their copyright text. So there's some change in the output or in the files. But the idea is that the core functionality stays the same, they never touch the core functionality of a web server or the kernel. They just update their copyrights, and that's it. That's why there is not full similarity, but there is some similarity. So if you look at this file, it's a PQI Air, and it's version 147, but the idea is that, okay, once you have this knowledge that this file is vulnerable and you have two vendors, you can just try to emulate both of them and see whether they are vulnerable. And in one case, you see it's the same IP, it's the same virtual machine, it's transcend Wi-Fi SD. I can kill it like that. The idea is that with another parenthesis is that having this kind of emulation, you can also find very quickly bugged like unauthorized access, right? So you see that it asks me for a password, but if I go to this page, it's automatic like this. So you see that you don't need a very sophisticated thing, and you can use these pentesting tools or plugins for pentesting tools to find these kind of vulnerabilities very easily and basically generates CV reports every minute. Okay, we can log in. Sometimes the thing is the interfaces are broken because you see there is no MAC address, right? Because it tries to call some IOCTL which is not supported by QEMU, and you need to write a plugin in order to support all these kind of IOCTL. But the idea is that you can still find the vulnerabilities if you go, see, there's Arachne name. Arachne name is because now the Arachne is pentesting this machine. Okay, so it used to pentest this machine. I can do a very quick thing like that. So basically you see that using the emulator you don't need to buy that expensive 40, 50 euro card. You can find the vulnerability. And the same way, it's just a very quick example, but you can find also we found the same command injection which took some time for the guy to find them or write the write up, but automatically it's easier to find. Now we go to Kcard, which is basically the PQI air and we can go to Wifi setup and try to do the same. I mean, it's basically the same page in the back because it's a parallel, but the output is different. That's why I told you the similarity is different because if you look at the source, no, not this one, frame source, you see that there's this Kcard everywhere. So basically they use this Kcard SDK thing. And you see that it's basically the same. The idea is that you can find exactly the same vulnerability exacting the same files, Kcard edit, whatever it name it is. Very simple by just triggering the tools. And basically the pentesting tools will do this for you because it files for everything, for XSS, for command injection, for path traversals, for file inclusions, and so on. So basically that's the idea. I guess I'll conclude now and we'll leave some minutes for questions, but the idea is that I can assure you there are plenty of Latin vulnerabilities in embedded firmware. Why it means Latin? Because they're there, they're just way to be discovered and many of the firmers are old as hell. Or even if they get updated, we see cases where they ship in a 2014, 2015 firmware update, 10 year old busy box or Linux kernel version. I mean, firmware security analysis is absolutely necessary is another conclusion and we see it by every day because the day that Juniper backdoor popped out, everybody started analyzing the firmware of Juniper OS, right? So we think and we also agree that the firmware security analysis is absolutely necessary and a broader view on the firmers, not just very particular view on these set of firmers is not just beneficial but is necessary because you can see that you can relate the same vulnerabilities or different vendors and you can actually discover the big surface, the big impact. You never know when we'll find those dual easy keys in other firmers and you'll be surprised that actually not just Juniper OS was affected by that backdoor or whatever. This is just an example. But of course, it involves many untrivial steps and challenges. We have to overcome them in some ways. Sometimes we take shortcuts and we don't get the 100% results but as long as we get some vulnerabilities, it's important and that we also see that for many vendors, in particular the white labels but not just white labels, it's also the big corporates who buy the white labels that security is clearly not a priority, that it's a trade-off with cost and time to market and they prefer not to invest a lot and actually be thirst to market and they completely ignore the security. So these are my references. You are more welcome to read them, share or retweet or cite them in your papers or studies or give me a sign if you need any question or any advice or you have an idea. I'll be around or you'll reach me on my email. I'd like to acknowledge my colleagues with whom I've done parts of his research and work. Jonah Zadak, Aurelian Francilion, Davide Balzerot and Apostoli Zarras and thank you. If you have any questions, I'm ready to take them. Thank you. Thanks for the great talk and we're taking questions now for 10 minutes and if you absolutely have to leave the room, please do so as quietly as possible. Okay, so just walk up to one of the microphones on my left or on the right and ask away. We start with the microphone to my right, please. Hi, thank you. Hello. Hi, thank you for the great talk. I was wondering if because you don't always actually purchase a device and actually test as exploits directly on a device if you have trouble with vendors acknowledging or being willing to receive vulnerabilities that haven't actually been tested against the physical device. Okay, so I'll repeat the question, whether the fact that we don't always buy the devices and we don't test on actual devices, whether the vendors are willing to take those reports in and address those vulnerabilities, this? Yeah. Yeah, so actually the talk today, the plan was to drop some zero days in some of the routers but because vendors actually listen or started to listen, we'll have to delay some of those. So the short answer is yes, most of the vendors start right now looking at these reports even though they're not tested on real devices, they look at these reports and we provide as much detail as possible regarding how we found them and what is they triggering input or what is the exploitation methodology and if they're not willing to take it in a responsible disclosure manner, then it's becoming full disclosure after the timeout and okay, otherwise I don't see, but yeah, to confirm is that vendors started looking at these reports, at least in our case, and it's good if they don't have a zero day, maybe they have time to fix it. Great, thank you. Okay, another question from the microphone to my father. Congratulations on a great talk. Thank you. Willa, did you actually write QMU devices and if so, will you make them public? That's a very good question. I'm not a very QMU guy. Dr. Jonas Zadak, my colleague, he's a very good QMU guy who is actually working on binary dynamic translation and analysis and he actually writes those plugins to map those devices and I only recommend to contact him if you have any question regarding this thing. In our methodology, we just used bare metal QMU but the idea is that once you have a library of QMU plugins or devices, it's just easy breezy to plug them in and basically broaden your testing surface. Thank you. So we have one question from the internet, if I'm correct. Yes. So Janix asks, are all the features you mentioned today available in firmware, Dolary? Or, and also is the project open source already available for download? Okay, so there are two questions, whether the firmers are available to be downloaded and whether it's open source. First, we can share some of the metadata and information about the firmware but we cannot share the firmers themselves because of the copyright. I mean, we are not distributors of software and usually this fires back, if you start putting them on your side, this fires back with legal claims that you're distributing illegally software which you're not entitled to and this is kind of a tricky situation but to address this, if anybody wants to run a specific test or analysis on the data we have, just drop me an email, I'll try to accommodate, I'll run your script or I'll try to give you access on our unpack environment where you can run it. So this is the first question. And second, the project is, well, it's not intended to be released as open source, it's more intended to be run as a service where we can give back to the community as much information as we can. And at the same time, try to improve our things. Basically, it's not rocket science what we've implemented and it's well described technically in the paper. It's just a distribution of jobs and so on. We sometimes also contribute commits or bugs or improvements to some of these tools like BINWALK or Binary Analysis Toolkit. So in that sense, yes, we contribute also to the open source community. Okay, we'll take two more questions, so please to my left. Hi, thanks for a great talk. And actually I have two questions and the first short question is that, how long does it take usually for the dynamic analysis on the formers? Okay, sure. Find vulnerabilities. Okay, so to emulate it, it takes, okay, we take some measure of precautions, we're having some dead times where we do weights or slips just to make sure that all the setup is fine because you cannot expect a consistent or constant time from QEMU running in different virtual machines, running on different hardware across the globe. So you don't have a consistent time, but in general is to start a QEMU for a particular image, takes around, I don't know, up to five minutes, depending what it has to load and what devices it can or cannot find or all kinds of faults. And to pentest it, it also depends on, okay, what kind of dynamic analysis? In our case, we run the pentesting and we use the XSS, CSRF and OS command injection modules from Arachne. And they take quite some time. They generate around 10,000, 20,000 fuzzing inputs per each class. And basically it takes around half an hour, one hour to test. Can I have one more question? Yeah. Wait, I think we should take the ones waiting over there for a long time, so. Okay, first of all, thank you for the beautiful talk. I have many questions, but I have selected one that is most for me interesting. How would it go about scaling the firmware retrieval part? Because from some of your slides, I took, I thought that your current sample size is about 2,000 firmware images, is that right? And actually, how do you go about loading and processing all of the firmware from different routers which are not publicly available? And it seems that it involves manually Googling for the website to download the firmware. And so this was the first one. And the second one, what is your current sample size? Okay, so current sample size, we have the collected sample size which we used for the first study and it was estimated to around 170,000 firmware images. Okay, 170,000 firmware images which we crawled using the Google custom searches or using FTP spiders and FTP indexing sites. And we just took a seed of 500 FTPs and vendor sites and start crawling everything. And then we just filtered out the width, basically the PDFs, documentation, drivers and so on. And we ended up with 170,000 files. Now we did the static analysis on 32,000 unpacked files because academic deadlines do not usually meet with our desires. So we had to cut some lines and do on a shorter scale the static analysis and the dynamic analysis as well because it takes much more time to emulate and pen test all this. We did it on 2,000 firmware images. We tried to emulate and find web vulnerabilities in around 2,000. Okay, so this is basically the breakdown of the sample size. Now how do we do about the files we cannot get? Of course, there will be always files behind the paywall, a registration wall or which are not available if you are not registered as a customer or as a reseller and so on. That's why we have this service where people can draw them and as we move forward, we'll try to perform machine learning as well to classify them and also to analyze them and give results back, not just on the files we have. And from the public interface, we collected around 2,200 files, around 10 gigs of data, so around 50 megs per file. But there are some bogus files as well. People just drop some movies and MP3s. Okay, thank you. Unfortunately, we're out of time so we cannot take any more questions. Thank you for being here this morning and enjoy the rest of the Congress.