 Welcome to the last session of today in this tent I have the pleasure to welcome Dennis Giese, a security researcher from Boston who will talk to you about privacy leaks in smart devices extracting data from used smart home devices. The stage is yours. Yeah, so thank you very much for the introduction. Thank you very much for you to be here and greetings to everyone on live stream So today I want to talk about privacy leaks in smart devices and in particular how to extract data from used smart home devices The outline for the talk will be the following first. I start with the motivation Then I will talk about which kind of data you can expect on IOT devices. How is it stored? What kind of reset states you can expect if you have used devices and Then I will talk about Extraction methods. How do you extract this data? In the last one in the last point? I have like some example cases for some devices Which I analyzed Some information about me. I'm a PhD students right now at the North Eastern University in US So I left just yesterday from Boston. I'm working there with professor on the beer But I'm also a grad student at the TU Darmstadt. I'm working there with Matthias Hollig from the Zemo lab and my main interests are reverse engineering of interesting devices and this can be everything from IOT smart locks, but also like for example physical locks Some side notes about the talk. I know I have given a lot of Xiaomi talks but this talk won't be Xiaomi bashing talk because the Issues apply to all the vendors Most of the vendors which I present here. I'm already known and why they use but most of the time for it for different use cases For legal and also ethical reasons. I had to censor most of the data So you also see like a lot that I kind of blanked a lot of data If you want to use the methods when it's on your own risk So if you destroy use a smart home device by unsorting the BGH chip and pulling pulling the pads then it's your fault Not mine And at some point I have to simplify a lot of things for example, and in particular is way more complex But I have to simplify it there then the thing is how To interpret or to reassemble NAND data is also out of scope of the stock and also how to route device specific How to do device specific routing methods are also out of scope So let's talk about your motivation In general data on used hard drives is a very traditional problem. I mean technically it existed forever But this problem increased with platforms like eBay where people started to sell their used computers or used hard drives And most of the times computers contain still a lot of data like personal information emails pictures other media or sensitive documents and In the beginning of the 2000s there have been like some awareness rise by by researchers in this particular case There's like a paper where people bought like 150 drives I believe and then analyzed what kind of information was stolen drives and tried different methods to erase them Um This same problem more or less affects also Matic function Matic function printers or lab instruments and at some point in this which is like the standardization organization in US Recognize that as a problem and created the standard how to securely erase devices And one of the solution was for example to write the hard drives Or if it's not possible to just sell the devices without any hard drives and for example shred the devices like here in one picture However, there's still like a lot of remaining partners for example There's like a lack of knowledge or awareness Not every user knows like what kind of data is stored or how do you delete the data correctly? Also, there's like people who say like I don't have anything to hide I have no Important information so they totally careless and they just don't care and the third case is if your device is broken And you cannot delete it securely anymore Then you have a problem the data is still there Um You would think that this problem is not nowadays Not a big issue, but it has shown like in this this year actually in a study That's still 42 percent of the hard drives which you can buy from eBay are still containing some kind of Confidential information or sensitive information After hard drives the next interesting thing was Smartphones because smartphones are more a sensitive data to go and the phones most of the time contained sometimes even more Sensitive information when hard drives for example pictures measures messages account credentials or call list and What many people don't know? The device storages were not encrypted by default for a very long time. So for example, I Apple introduced in 2014 of iOS 8 the default encryption on this on the iOS devices and Android was only in 2015 with Android version 6 Where they enabled the encryption so for a very long time it was not enabled Another problem is that factory set is not done properly and we have in like a paper in 2015 Which analyzed like this for Android devices and we figured out that many many devices are still Don't still containing a lot of sensitive information and are not wiped correctly and again here The list addressed that with a standard in 2014 baby is that like okay need to do particular steps for smartphones So the next generation of problems is IoT and IoT is everywhere The problem of IoT in particular is in contrast to like smartphones or PCs You don't have a real good user interface or you have no user interface at all So you don't can access the information on the device directly. You have no idea what kind of information is on the device What information is collected in the first place so you cannot access it directly? Which means obviously for factory sets if you do a factory set of IoT device you cannot fully verify that it's actually empty and The implementation on how this factory set is done as models and depending on vendor to vendor On version to version of like firmware even for the same model. So it's like absolutely unclear so about the motivation about this particular talk why I have Looked into that topic was basically when I was doing my master thesis at Zemo and analyzed the security of IoT devices in general my goal was all the time to Route to get root access at some point on the devices So what I had to do is like I had to do like some attacks when I had to reset the devices again And then had to do the attacks again And one thing which I figured out every time I do reset there's still a lot of traces of data left on the device and Sometimes even all of the data was still available, which was at some point even for my research like a problem So the other question is like okay where you would find use devices and the thing it's kind of kind of easy you just go to eBay or to Amazon warehouse deals And you can order them directly there. Sometimes if you go to some flea markets Then you find like bunches of like old Alexa echo dots like first-generation or something or just look in the trash Sometimes your neighbors if they have like broken light bulbs or something. They just throw it in a trash could be also interesting There's also like a tradition. I think in many families that they give like old devices to like Families and friends And here you get again like use devices and at some point obviously you have use devices in your home. Why is this important? Well, it shows actually that in Germany at the moment there's like a trend for the German police that they want to get Also access to this data. So basically everything which we talk about now like how to access data or what kind of data is existing It's obviously also interesting for other parties. So you should keep that in mind All right, so let's talk about the data which is on a device The actual data of the types of data which you have on the device is more or less depending on what kind of device you have Which is kind of kind of obvious. However, all the IOT devices have one thing in common We need bifur credentials to obviously connect to the bifur We need some kind of cloud credentials to be able to connect to the cloud and sometimes We create also like cloud bindings what we kind of connected to a particular account and as a rule of thumb What I found is that the more performance functions or storage a device has the more data will store no matter what So lock files, whatever So let's take a look in particular devices or the particular types of devices If you have vacuum cleaners and they got a lot of vacuum cleaners over the time Also different models Most of the time what you find there is you have connection locks For example, when the vacuum cleaner connected to your wifi access point or when smartphones are provisioning the device for the first time you have maps of your apartment, for example, you have cleaning clocks and Quite often you have also the user ID in there because you need to kind of create like a binding between the device and the user If you take a look at other devices, for example, smart home gateways, which connect different sensors for you, for example They again contain connection lock files But in addition to that they connect also the sensor and actuators bindings and lock files So for example, if you open the door Most of the time this device actually somewhere rises in the lock that's received something from the sensor If you interconnect multiple devices multiple gateways Then you can find also key material which can give you like also access to devices and again here user IDs Another thing which is kind of popular especially like in us But I think it starts also like in Germany are cameras and like the smart double cameras And some of the cameras are actually caching snapshots of video clips You find sometimes also recorded video which we store locally We find event logs you find again user IDs and the other thing in particular for cameras Which are special as you might find to close a cloud storage credentials where the camera is uploading its camera its video streams For example, and sometimes it's also possible to access it again One thing which is not directly smart home But it's also important are routers and here the information which you can find for example Is the acp leases so you get like a binding between Mac address a p address and time stamp So you know like if someone was connecting before kind of devices to this phone choose router Some users Do fiber configurations so you find also the fiber configurations if you have like a very Powerful router like the avm routers, which you have in Germany You might also store media files there and again lock files for connections dns queries filters some parents filter the internet traffic for their kids and some other things and if you have for example like Like dynamic dns sometimes you find also the credentials there and people tend to use the credentials also in other places Um, media players are also very very often used in homes. I think that's probably one of the most used devices Um, again here you find connection work files. You find media libraries. You find play playlists of like played movies audios Um, this device is also cash a lot of content and some of the devices they run for example an android operation system They can contain also like a browser So you might have like a browser history and again other credentials for example for google play store If it's android device or network shares if you have some kind of kodi which is connected to some network attached storage um, there's one particular problem with this particular device and I have unfortunately to for ethical reasons I have to skip this particular device because I've got some devices which contain a lot of adult movies of a very special taste And so I was told I should maybe not talk about that um Last but not least, um, there's also like toys And many people don't think but this devices can contain also a lot of data Configuration settings might be not that interesting, but some devices can collect audio and video streams or video data And again here you can also get like users usage logs So you can figure out how often was this device used at which points was the device used at which time All right, so let's talk about how device how data in general is stored on this iot devices um, in general, um, you need obviously to you to use some storage and the storage which is used on iot devices You can more or less put in two groups. Um, one group is the raw flash, which is um, sometimes known also in two Like subgroups like a serial flash spi or a raw Nan flash which is like connected parallel and the other big group is our block devices. Um, in this case, it's emmc emcp or sd cards and depending on the On the choice which you vendor made for like using one or another It affects also like the selection of like file systems later. So if you use one kind of storage type and It has multiple consequences for the for the other design choices of the device So let's talk about a raw nan flash There's one thing like spi flash and you find spi flash usually in typical sizes smaller than 64 megabytes So if you even know like with your iot devices less than 64 megabyte or 64 megabyte of flash when it's most of the time spi flash Come in different packages, but it's kind of kind of easy. So it's always like something like eight eight pins um, the other group, um, raw nan is typically coming in sizes between 128 megabytes up to 4 gigabyte It can be also higher, but this is like the more or less the typical one and they come in bigger packages So they have like um, typical packages like tso 48, which has 48 pins or a tso of 56 56 pins or also like some pga variants um The reason why this is a flash is used so often is basically because it's cheap and fast storage On the other side because it has bit errors all the time You need to take care of some some measurements and in this particular case It's so that your host processor in the operation system needs to take care of particular things. For example, we're leveling ecc, um error correction code and bad block management If you use linux, um, that's more or less done by the mtd the memory technology devices subsystem Which takes care of like the um All this kind of things and what this uh, the subsystem is basically doing is it takes a character device Which nan is and converts it like in a block device so you can access it like more easily um in general to um Nan has particular properties and one of the properties is that this organized in blocks and pages So a block contains multiple pages in this case and if you want to erase Like data when you have to erase the whole block This becomes important later and the reason means basically you just set all the zero all the all the bits to one To give you like an idea how the sizes are between blocks and pages A typical block size is like between 60 and 512 kilobytes And a typical page size is between like half a kilobyte to two kilobyte Um, one of the things what you can do is basically you can do programming on a page level But if you want to delete data, you have to do it in the block level Um, there's an addition to the actual data area and this is called out of band data Which is there for a management or for ecc purposes Because the vendors know that the flash most of the time contains bad blocks What they do is basically they add additional spare blocks So um more than I think like two percent of the blocks are more less spare blocks So if you have like bad blocks you can always replace them often there and here um the Aerocorrection code is more most of the time computed by the host CPU or the operation system And the important thing information here is that the um computation can be sometimes Vendor specific so if you use a particular uh chip it can be um completely different than if you use like a number chip So it's like vendor specific All right, so Let's talk about ware leveling for raw flash and why why do we do that? And the problem for for nflash particular is that the individual flash cells have like very limited Lifetime for for writes so you can write to two flash or maybe one thousand times and after that it's broken and if you use file systems like x 234 On on the in those devices you can basically destroy or corrupt the data Because this this file system is not um ware leveling aware or not flash aware even So what's the solution for that the solution for that is that you use flash aware file systems Or like you put an additional layer into that if you want to use like a flash aware file system Most of the time what you what do you use is like jeff's or gffs2 or gffs um The alternative way would be you use like an additional layer between the flash and the operation system Which would be in this case ub or ub in addition to ubfs And these both things are doing more or less taking care of the bad block management and Ware leveling which is when happening in the operation system So what's the general idea of this ware leveling? Well, you um if you delete data then basically You don't delete the actual block. So you just mark it as like dirty um And the information which you change is basically copied to your new block And at some point if you run out of space for example, then the garbage collector will come through and hopefully erase at some point this block To demonstrate that in a very very easy way. Um, this is super simplified. It's way more complicated But just to give you an idea of what's happening You have a logical layer which is basically your data at the physical layer where the data is more restored And let's say we want to change this particular block. We want to change some information by for example, wifi credentials Then what happens is um, this block is read Then the information is changed in the memory and written to a new block And then there's the created new link to the new block And the old block is marked as a dirty block The problem here is the data is to present. So the data remains there until the garbage collector is coming through at some point So for us To do forensic analysis on that there's like some interesting properties and one of the interesting properties is you might have multiple copies of Of the data every time you change data You have you write in it into a new block And the old data is not a release as long as the whole block is not released and The aggregate is more or less try to avoid that as as if it's possible So the sizes of the copies which you have from like some files is typically bigger than two kilobytes So you have multiple copies of a particular file, which is bigger than two kilobytes And if you change data more often for example lock files or bifur credentials if you change it like often then the more copies you have Which means if you have bifur credentials Then at some point you have a history of all the bifur credentials if you set up the device the first time You use like one password when you figure it out Oh, it might be not a good idea to you to connect this device to Like your secure wifi and you create a new iot bifur you change the credentials in this device But it still remembers the old ones Um, there's some recommended material about this whole topic, um, which I don't want to go too deep into that There's a blackout blackhead talk from 2014 about reverse engineering flash memory for fun and benefit uh by matt oh and he gives like a short introduction about the Communication protocol which is nand using he gives some tips about soldering unsoldering of nand flash And in particular how to reverse engineer a raw nand formats So I put the links there and another there's another article like a block article from nand ships to files From john michael pico and also bears the link As a side note about this this flash stuff, um, basically even vendors are not aware of this Properties for example for jfs 2 and have you like a sample from my talk from the defcon 26 last year where the vendors actually forgot their developer credentials in the jfs 2 image, which we copied on millions of devices So the space was still this good. There's a history available and the vendors didn't delete that so Um, if you can't delete it when the vendor can also vendors forget about bad things Let's talk about the second group, uh, the block devices and this block devices are mostly known as management nand There's different standards like there's emc 4 standards, uh emc 5 standards depends on the standard that has additional features In general there's like a distinction between emm c and emc p emm c is itself It's just flash with an integrated controller. So you have that raw nand flash and you have an additional controller And this is a one chip and the typical size or the typical packages that you find various like fbga 153 um In contrast to that emc p is more or the same like an emm c but it has also additional dram On the chip and the advantage why a lot of vendors are using this particular, um package is basically You may have only one chip, um, which contains ram and flash In one device and it makes it way easier to For the supply chain for example And the typical packages which you find various fbga 162 and 221 um If you look under linux on these devices then basically you see them as normal block storage devices which support fully like x2 x3 x4 And um the chips itself will take care of the wear leveling ecc bad block management So you have like a layer inside of this chip which is taking care of that and the operation system has no idea about that So the big question is well how we can access deleted data on on this chip and Usually the emm c controller doesn't allow you raw access for this data as soon as it's deleted some emm c's Even support like trim commands, so we just delete whole blocks of data However, the thing here is again the emm c Um are using raw nand internally and if you somehow can bypass the emm c controller and directly attach to the nand Then you can access the data again The only challenge then is basically have to understand what kind of data format the the emm c controller Has used there's a very recommended talk there about emm c chips data recovery beyond the controller from a forensics company called resort and Basically their their summary is that even if the emm c controller emm c chip is completely erased from the operation system And wiped the data still present on the nand flash But this this is a little bit out of scope properly for for for most people Okay, let's talk about the typical reset states which you would find if you Get some used device in your hand well The most of the time the reset state is more or more or less dependent on the previous owner So depends on what the previous owner did or knew or wasn't able to do Um a lot of times you find devices which are not resettled at all So basically they had they contained all the um device information They contained all the data. They contained the configuration everything And the most proper because here is for example that uh, there's like knowledge missing for the user how to erase Actually data or that the device was broken Last week I bought like five amazon echo dots Where the usb connectors were broken and I just restored the connectors and the devices were still like provisioned. So That's one of the cases which can happen. The other case which can happen is um that the the device was by far resetted so that The by far information was deleted, but the device still contains all the data And um many devices only support as one particular mode So if you find a device which has a reset button and you press the reset button most of the time what it does is only Why for a set so it doesn't lead to data really inside of it? The third kind of case what can happen is like the device was wiped. Um if the device for example, um supported some kind of Factory set where it just wipes all the information and resets the factory state of the firmware Then that's a potentially good thing However, even in this case there might be still um traces of data available on the device And even not all devices support this this kind of thing Um to summarize more just both reset types. Um There are some actually some devices which support both things. So um if you do a by far reset There's usually like a particular marker for the by far reset where you press the button and even you can do that And the device vibe you have usually initiated by a special button combination or you have to do it all by the app The general idea why the vendors don't do all the time device vibe is the is the following Well, every time let's say you want as a user to reconnect the device to a new wi-fi You want to change the wi-fi credentials or whatever you don't want to Erase the whole configuration of the device. So basically if you press the button It must be available directly so that you can reprovision that again. So um And you want also to have the most most of the settings to remain in contrast to that the device web should just Delete everything which is maybe not the most favorite favorite Kind of outcome for the for users if we want to reuse the devices themselves all the time All right, so let's talk about uh data extraction methods The main idea of data extraction methods obviously for iot devices is to extract all available data Which we can get from the device and there are more or less like three groups of methods Which you can use to can try to access the data by a software. For example, if you get root access um, you can dump the data From the flash without the soldering which is probably the most favorable thing for most people Because otherwise you can still be dangerous dangerous or you can dump the flash contents by the soldering chip Which is also called the chip off method um For the software method for many devices even for for like particular versions of the amazon aqua dot They are like routing methods available publicly. So you just for example, install like a custom firmware Uh, you x or your access to the device over usb or urad and then you just dump the whole data Some devices also especially the cameras. They have like special If you put a special file on the sd card when it boots like an Operation system from the external media so you can use that to boot like your own system And then as soon as you have access to to a shell on the system You just use dd to copy all the information from the flash One particular thing where you want to use dd and not like nant dump or something Because the dd is not aware of flash and it will just copy everything Which is exactly the use case which we want. We want to get all the data including the out-of-band information Um, and this is very useful for our case um And one way to extract like data um from the devices for example We use like external media if you have another sd card or if you use the same sd card or you can use ssh or netcat This method works great, especially for systems with jffs 2 or ubfs file system Um, the disadvantage here is it depends a little bit on the kernel. What kind of interface the kernel gives you So, um, the low level access on the flash might be potentially limited, but most of the time it works Works pretty fine Um So the next thing which you would do if you can access Like over the the flash over over software would be to try to dump the firmware without disordering and this works great for spi flash or mmc flash and it would kind of requires for the device allows in system programming So, uh, one thing what you do is like basically follow the traces and you figure out where the um, where you can access the flash data lines over test pins and you need to make sure somehow that the processor is not interfering with that process One thing which you can do is you just ground for example the crystal or you Pull down the um, reset pin of the processor. So you just need to somehow the processor to be to stop The advantage here is obviously while if you don't disorder bg air chips when it's it's you have a real reduced risk of destroying the actual hardware Um, this advantage is here You need to figure out where the test pins are and you need to somehow stop the uh, the processor Especially like if you have like, um, pcbs with four layers where the data lines sometimes are in between the layers that gets a little more difficult Um, the last method is obviously disordering chip and this works literally for all the flash chips And one thing which you want to do about the soldering is this shouldn't be like a disordering session But uh, in general you want to preheat the whole pcb because most of the time you have like a huge ground plane And if you didn't do that then it will just pull the pins Um, if you have uh devices with uh, like flash chips with accessible pins What you can do is like you can try to create like a low temperature alloy of the soldering So you just use like a 130 degree Celsius uh, soldering paste and then just mix it with the Existing soldering paste which makes it easier to remove the chip for bga chips Well, unfortunately you need like hot air infrared or overflow soldering stations. Um, you cannot use like a usual soldering iron for that Well the disadvantage in general if you disorder bga chips If you want to reuse the device at some point again, then you need to reball the devices and uh, Reball the chip which requires additional tools um And if you um disorder like an end flash then to read it out You need a particular special adapter because of the pin count. You cannot just connect it to normal devices Um, so let's talk about the tools in general. What kind of tools do you need? Um, if you want to read sbi flash you can literally use any device which supports any kind of bit banging on gpaos You can use a rest paper. You can you can use an adrino You can use a bus pirate One tool which i use is like flash cat usb because i use it also for for some some other stuff where you can just Use like an adapter you put the chip in and you can read it out um for emmcs it's it's similar simple, um even though but you have to be a little bit careful about emmc low So emmc is the flow of voltage So before you do apply any power to it you should read the datasheet if it requires for example 1.8 volt And um, there have been some very very cheap methods for example with the expertise The emmc adapter for 10 dollars where you can just sort of the Cables to the lines the data lines of the chip for example and connect it to the sd card reader and you can just dump the Most of the content the difficulty here is if you have bgh chips obviously you need to have very good eyes or do you need to have a microscope and If you use an sd card adapter then most of the time you don't have access to all partitions Which doesn't really matter for us Um because you need like to use a special protocol to access the boot partitions So but because we want only to get the user data. We don't care too much about that Um, there are also some cheap Chinese tools available for that which many many smartphone repair drops are using So there's like the ufie box slide where you have like multiple adapters For bgh chips and what you do is like basically you put the bgh chips on that and you close the lid and it has some small Needles which are connecting everything for you and this kind of device also supports supports dual voltage It has some disadvantages because You need to find the correct position where to place the bga and one thing which you shouldn't use is actually the windows software Of this particular product because it's detected as malware and the typical recommendation by the vendor is like Yeah, it's fine. It's because of our security technology is just the activator virus scanner Maybe you shouldn't use that Um for run and it gets a little bit more complicated because we have like a you need to connect a lot of pins for that and Also this kind of device Usually requires some sort of nan controller and again here There's this other talk which I can recommend the reverse engineering flash memory for fun and benefit If you want to use an adapter for that well, um, there's again the flash card thing where you can Buy like a particular adapter where you can even read bga chips and this device supports more like lots of different kinds of flash devices However, the important thing here is because the ecc and out of band data is dependent on the on the sock which you use Um, it doesn't really interpret that for you But it doesn't really matter for us if you want to extract the data But it might matter for you if you want to do some reverse engineering of these devices Another method which is sometimes possible is you just get your Development kit for this one particular CPU which you have for example on the device and the idea here is you just take the Trip which you just remove from the upper board and you just solder it on and then you should be able to access it directly This advantage here is um, you cannot always buy this development boards For all the trips which are available And the other thing is they sometimes very very expensive So it's kind of like if you have access to that then it might be an option, but usually you don't So if if you have to dump at some point then you can use typical thing What many many people for recent during years is obviously binwalk You just run binwalk and will detect like when the partitions or some data You can use also like a normal hex editor to just find like if you're looking for Wi-Fi credentials You just look for ssid and you find all the time something um, if you want to further disassemble Um NAND dumps when there's some tools available Like the dump flash tool from that. Oh, there's the NAND dump tool But in general the problem with this kind of tools is that they're sometimes exerting exotic out of band sizes Or ecc data, which are not working for this particular tools If you at some point has like I have like a UB FS image There's like UB FS dumper if you have like JFS 2 there's jfsm Which is also most of the time I think used by binwalk So you can you have a lot of tools available which you can use to to analyze this times Okay, so let's start with some example Devices which are analyzed to give you an idea of what's going on The typical method what they use this disassembly devices I uh dumped the flash for example, I'm soldering or just in system programming Then later on I resorted the flash again. I powered on the device a router device if possible And at some point I also try to connect the device again to the app or to the original like If it uses particular app just to connect it to like my own account um, and to get like an idea of what's happening if you reset the device or resetted it and Was just comparing between the data which was available before the reset and which was available after the reset So the first device which I have is the ecowax depot 900 Which is I think kind of popular in Germany right now I got this device in 2019 and the previous owner told me that it was resetted actually And this kind of device uses like linux platform We use a rock chip quad core arm quad core, which is kind of interesting for a vacuum cleaner again um, they're using a raw nand flash 128 megabyte and this device has also like 128 megabyte of ddrm What I did here is basically I was unsoldered the ddr the nand flash and um Dumped it and then tried later on to connect over uart Um after I dumped the flash this was some information which I found I wanted to verify that this device was actually factory set And that I don't like what the vendor did not idea what he did and one thing which I found is for example I found the actual lock file was set at this point when before I got to the device where this device was factory set complete um, I found also like a lot of other fragments about lock files key maps, um, and even bifur credentials In this particular case what you see there is um in the bottom, um the lock files of the usage of the brushes of the device Um, the particular problem what I had with this device is I didn't knew what kind of out of band Computation they use or how the out of band data is kind of Built so I couldn't really reassemble the image again Um, next step what I did is like a connector to a uart which is unfortunately they close it down So it's only read only but it have given me like a lot of useful information For example, I know that rock trip is using a very particular very specific nand driver for their For their devices and it does like custom ware leveling so it doesn't use the ware living of linux And they use quarter fs for the root partition and x4 for the data partition Speaking of credentials, um, like I mentioned earlier you find like a history of credentials and what you see here on the right side is basically In the beginning the factory settings were only with the factory wi-fi of the of the vendor like it was like echo wax underscore factory And at some point the vendor started to add more and more wi-fi credentials So this device to connect it to the home wi-fi so at some point I had like two different credentials for the wi-fi Um Another interesting file was the connection lock file which told me some information about the user for example when this device was connected to the wi-fi Um, or for example, which kind of device was used to initially provision it for example The owner the previous owner had a samsung galaxy phone where which was used to the provision but The question is can we figure out where the user is living or where did the user is coming from And one thing which you potentially can use and what's what your smartphone is doing all the time You can use the google geolocation rp And the input for this geolocation rp is basically you give it two mac addresses and the signal strength which you receive It will give you back like a location With some accuracy rating the problem for this particular device as I found in the lock file only one mac address So I couldn't use that so it was a little bit sad However, there's another um search engine where you can look for ss id's and the particular ss id's which have been used there where some kind of device specific from the vendor from the um From o2 so they have like some random string at the end and um, basically just run it against the search engine and found like For both wipers exactly the same position Which was kind of useful So as a summary for this one particular device while most of the user data was still existing on the device You could see the x mmp network locks because this device is using x mmp for controlling You can see the maps you can see credentials However, I couldn't really extract a map format or the maps because I didn't figure out how the map format is to What I have to do to reassemble it Later on I tried to reset the device three times with the factory set which was in the manual and the data fragments were still where One interesting aspect also I found the factory locks of the devices of this device So I could see what kind of tests they did in the factory what kind of bypass we use in the factory What kind of so I could see everything when when this device was actually born um And with this method of the lock files, I could also track down the previous owner the good news In this case was that the app didn't leak the previous maps. So if you connect it like to To the app it doesn't show you like the previous cleaning clocks The And this particular like behavior is also like but the data is still there It's not not only for this one particular vendor So I found it like for multiple other vendors for example, there's like biome vacuum cleaner robot version two Which is exactly exactly the same hardware more or less All right The next device which I found where from 2018 It was in an unclear condition because it has water damage and the xiaomi vacuum cleaners run Ubuntu 14.04 they have like quad core all-winner s16 processor and they use emmc before gigabyte for the flash And this device is also 512 megabyte to give you just an idea How powerful this device is and what I did here is basically because I rooted this kind of device like a long long time ago I Dumped the partitions over uart and connected the device later on to the cloud The good thing for this device was that the rooting method exists because I found it like two years ago on the cows communication congress Um, so you can get a root shell over uart or if you just push like a custom firmware Um, you can then if you have like the root access you can just extract all the data over ssh And this is a great method To avoid removing the emmc flash which is bga. It's not that nice If you take a look at the emmc layout of the of the device There's um, you will see that the operation system is existing multiple copies But we are not carrying in this case too much about the operation system We're caring about this one partition which is at the end uh u disk which contains all the user data So we interested in this one particular one so one one interesting thing about this device is that it supports both vifre reset and factory reset and If you do the vifre reset then only the vifre credential file is deleted However, if you do the factory reset it requires a very specific Specific procedure where you have to press three buttons at the same time. It's mentioned somewhere in the manual But how many people are actually reading the manual? What this factory set is basically doing is it restores all the operation system from recovery And it's formats also the data partition. However, it doesn't wipe it. So it's just to like a mkfs x3 I think x4 sorry Um Also, another thing is the usage data. So how much how often this vacuum cleaner was used? How is the how the how many hours and how what kind of area does the vacuum cleaner that clean is not erased at all times? So it's basically still there So what kind of what kind of information I found for this one particular broken vacuum cleaner which I bought out of ebay Well, I figured out after I provisioned it with a new account But um the previous data is still visible in the app So I could get all the the cleaning records I could see the map of the of the of this person's apartment. I could go down. I could go back to a thing Yeah, so for a long time I could see all the kind of runs six eight runs in total Um, one thing which I figured out is what the data is re-uploaded Re-uploaded to the cloud. So even if you reset the device and at some point change the user It's the vacuum cleaner will happily re-upload everything again to the cloud and you can access the log files locally on the device Um one good thing is if you do the factory set properly with a special button combination Then at least you can't see the map files anymore, but they're still there somewhere Um, so here I tried to again again to locate the former owner And the good thing here is that when I was looking in the log files I actually found two MAC addresses of the Wi-Fi access points because the owner had like Two Wi-Fi access points in the house apparently and you could see like in the log files two MAC addresses and here the google geolocation RP returned actually the core coordinates and it was somewhere in Hano in Germany Interestingly after I did that I figured out that the Wi-Fi credentials actually will be a part of the address So it was like the house number and like some part of the street name in the in the In the Wi-Fi credentials and also the person apparently used some personal information for the Wi-Fi password Um, one cool thing which you can do is you have to have the user ID And you find the user ID in one of the files Then you can search the user ID in your Um smart home app and then you can just for example try to Reveal the user's name at some point which is kind of nice So we just enter like the user ID and then shows you like the profile picture and the name and ask you if for example Want to share device with him and this is kind of kind of interesting So as a summary for this particular device well All the data was still available because the user didn't reset it probably because it was broken The data was wiped instead. Um, Wi-Fi reset was done. Maybe done at some point Um, the reset button on this particular device is a little bit misleading. So if you press reset basically does only Wi-Fi reset Not the factory reset. Um, however the procedure in this case is documented in the menu But again, not many people read that and with this information, which I had as I could track down the previous owner Due to the lock files and this device also creates a lot of lock files. So you find always something Um, one example from For for like toys, um, like a very short example I have like small just as much as small drones which I bought and I have also some some new ones And the children toys basically very powerful device. So this is like a 50 dollar device a thing It's like super small and it runs android has like a think quad core, I believe the 4 gigabyte emmc Memory and a 512 megabit of RAM The school thing about that is it has two cameras for one for navigation and the other one for like video recording And this kind of device you can access over a serial if you connect the UR data lines or over adb after you do it I have like a custom frame of it and here the thing is this device records, um, um Stores the recorded videos which you can record with this device on internal memory and Obviously if you if your kids play with this kind of device and crash it to the next wall and destroy the device When you have no chance to delete this data Even deleting the data if the device is working it's still very difficult to give an idea of what kind of images it looks like So it is like just a picture which I make in the lab. So I was flying around and This device was recording Again at center to data the actual original data which was on the device because it might be a little bit embarrassing Um, another quick example door belts There's many many models which are used maybe most of the time use exactly the same design. It's like high 35 18 This device is usually have spi flash for something with 8 megabyte And these devices also use jfs 2 or ubfs again Or you can expect all the bifurc credentials the set part of about this particular door belts, which I found where They you can usually insert a sd card which is then used for storage of the video So I didn't really found video data on it because it usually uses external sd cards Um, however, not everything is bad actually So I found like one particular vacuum cleaning model which I'm rooting right now Which I don't want to disclose yet Which does us a little bit different we use trust trust zone for the key storage. So basically all the user partitions are encrypted for flux and the keys are Managed by the trust execution engine And the device specific and every time this device is boot It basically unlocks the configuration at the user data One interesting thing about the factory set is basically just leads the key and recreate the partition Which is I think typically the best practice how you would handle that kind of situation So it doesn't mean that everything is totally bad, but some people know about the risks As a conclusion Um, well the device remembers, right? So you have like a history of bifurc credentials You can't be sure as soon as you've provisioned something on a device that will be gone at some point And secure and correct factories that is very difficult to implement If you use nand flash in particular like if you use round nand flash when the full vibe is like very very difficult And you have no way to ensure that the device have been corrected wiped correctly by Also many vendors actually don't know that there is user generated data So you have to use that data which remains you have lock files which remain on the device If you do by first that the bifurc configuration might be overwritten But you can't find the same configuration in many other places like lock files And one of the biggest problems is still the missing knowledge of the user So if users are not aware that they have actually to wipe the devices or that this device might contain data Um, then we have a big problem So as recommendations, well do not sell or throw away your devices Especially if you expect that it might contain some sensitive information And especially also if you cannot verify that you have done a full vibe Try to physically destroy the device and for example, if you have like a broken device, well If it's broken why not break it more, right? So just disassemble it try to just practice a little bit soldering The other thing is Use for your it devices a separate Wi-Fi if some some of your neighbors for example If your Wi-Fi credentials they can access your network, obviously If you use a separate iot network, then it's a little bit more difficult However, it doesn't prevent leakage. It's still like The attacker might have limited access to your to your network, but still they might have access All right, uh, that's more or less Nearly the end of my presentation I want to thank a few people who supported me to do that research Professor Nabeer and Professor Manfred Daly from the Norfison University But I also want to thank the Zemo labs attitude Darmstadt for supporting me in my master thesis And now I'm happy happily here for any further questions and Yeah, any further questions. Thank you Okay, thank you Dennis. I think we have time for a few quick questions. There's a microphone angel over there So signal him or just walk over to him I can't see very well. So you have to make eye contact with him There's a question right behind you microphone. Oh, okay, you go first My mic is uh You mentioned one device that has uh that did it pro properly securely Can you tell us which device was this and if there do you have a list of devices that do this good? This properly Um, well for this one particular vacuum cleaner I can tell you yet because I'm at the moment in the rooting process and at some point I want to publish that and I don't want to You know scare the vendor yet so that they do changes would prevent me from doing that Um, but there are devices out there which which obviously do that. It's um Not trivial to find and the other problem is also that the vendors are actually not kind of You know advertise the feature that does some kind of secure factory set In general like, um devices which do it very good are like obviously smartphones because they learned out of the history um, but other other from that it's like, um, I'm I can't think too much about devices which there are not many devices out there which do it correctly Unfortunately, I can't give you too much information about that Okay one over there uh, uh, can you give a sort of Sliding scale on what would be I have to ship an iot device and I'm trying to figure out where the What what would be the Sorry Can you give a sliding scale of what should and should not be implemented in sort of in data cleaning process that kind of thing? Um, well as a vendor what you can do at some point is um, That you can just start in the beginning and just not collecting lock files at all I mean this would be like would be helpful for a lot of things In general Wi-Fi credentials are always like a sensitive thing because they have to be at some point in plain text If you want to connect to the Wi-Fi But there's also like methods for example where you Can encrypt them or like put them in hardware somewhere like there's like trust zone Which you can use for that but not all devices obviously support trust so it was bad like a good question to you But that was great. So did it answer your question? Thank you Okay, okay That was a question. Okay Yeah, besides destroying and not buying it. Um, is there any best practice that you could recommend for a typical user that is not into deeper IT knowledge how to handle such a device after usage Yeah, I mean that's a that's a difficult question So the thing is my head of discussion earlier today of someone who said like yeah Well, I mean you cannot just destroy any device which you have like smart TVs and everything because it's like from the you know from the economical perspective and From the environmental perspective. It's not a not a good thing However, if you have the idea that your device is like has contains some some information If it's a very very cheap device for example like a media box for 30 dollars and just take a hammer and you know Hammer it down. It's the same thing if you have like a broken hard drive, right? It's like a broken hard drive the typical recommendation is if you if it doesn't run anymore If you cannot wipe it then just open it and drill some holes into it. It's just destroyed some more physically I know it many people won't do that for like a expensive TV, but it's it's a very It's a hard to answer question because it's um It's it's such a huge problem and it can't imagine like a good solution for it at the moment to be honest Thank you Over there In mr. Robot, we see elio destroying chips with the microwave Is this a good way or what would be a good way to destroy flash chips? um I think from they for microwaving passwords. We know that that can sometimes cause a fire so I wouldn't recommend to the microwave flash chips or like chips at all I Think usually breaking them somehow like I mean if even if you if you have a like for example Let's decard and you break it in two parts where people can reassemble the two parts and can still read out the data But the question is like how important are you that people are actually trying that? um, so I think it's totally fine if you just like I don't know like Physically with a screwdriver just remove the flash chip and throw it away somewhere and If no one can find the chip that should be fine or if you just break it Um depends depends if you see I want to get any information from you. So uh, so what's your what's your enemy? So we are slowly but steadily running out of time. So, uh, let's thank denis again for his talk