 NAND flash chips, a little different than standard flash chips. It'll be a little more problematic. So we're going to talk about that and kind of step through with a number of demos. So great. Let's go ahead and get started. So thanks to everyone this evening or this afternoon based on where you're at in the United States, presentations, NAND chips, introduction and NAND chip extraction and data reconstruction. This is an intro, intro primer. We're going to cover a number of different topics and we have like three different demos. I think you'll get a lot out of this. So my name is Daryl Highland. I'm the principal security researcher for IoT at Rapid 7. So the section one, we're going to just give a basic introduction and look at structure. So what is an NAND flip, a NAND flash memory? Simple. You can read it. It's non-volatile storage technology that doesn't require power to retain the data. Pretty straightforward. See those references there? If you're interested in learning more and the nitty gritty, take a screenshot of that real quick because that is great references. I go back to these quite regularly to kind of refresh myself because some of the stuff can get pretty deep as you're trying to reconstruct data in a number of cases. So go ahead and get a screenshot of that. I'm getting ready to jump off of it now. So a couple of things. When we start thinking about NAND flash memory chips, you need to go to the datasheet. You really need to understand how things are laid out. This will play a part in recovery a number of times. So this is the memory array structure that lays out the page, the blocks and the planes. And in this case, the page is 2K in size or 2048 bytes. And then there's what's referred to as an OOB area, a 64 byte area at the end. And this is critical. So what is OOB? It's out of band. It's a spare area. It's used for containing error correction code information. A bad block information and it exists in every page. So what does that mean? Generally, what it means is you have all your data jammed full of potential garbage when you're trying to parse it out and rebuild the data. And this could be impactful and can cause problems in some cases. So we're going to talk about how we're going to fix that. And we're going to show that in a couple of examples. The OOB area, the out of band area is also laid out in two formats. One is known as adjacent and one is known as separate. If it's laid out, the OOB is laid out adjacent. So a 2048 byte page would be laid out with a 16K or 16 byte of OOB after every 512 bytes. This would be an adjacent layout. The most common I see is what's known as separate. A separate area is the 64 bytes of data exists at the end of the 2048 byte page. This is the most common. But often you need to verify that as you're working through this and trying to remove the data or correct the problem. Another thing that comes in handy and this is in reference to a number of the tools that we're going to use today. Your chip ID code. An example, anytime you use a flash memory reader, so you pull the chip out, drop it into a memory chip reader, often it will, when you set it up for that chip, it associates that with the ID. And it will read the chip, compare the ID on the chip to make sure it matches before it tries to read the data. Now, a lot of applications out there have a use for this type of chip ID data. This is out of the data sheet also. And in this example here, it's showing the data sheet for a S34 MSO2G1 chip, which is a 512 megabyte NAND flash. And you can see at the bottom, it shows what that chip ID is for that chip. And you'll see that we'll use this later on and we'll probably come back and look at it in the data sheet again when we get ready to do that. So at this point, are there any questions? So Jonathan, is anyone posted any questions or anyone not having any questions about some of these basic concepts? Let's see here. So far, it looks like no questions posted here. One quick question that is actually coming up here that I'm seeing. Whenever you're pulling firmware from NAND flash, do you have a consistent philosophy that you follow, such as always first see if you can first pull, maybe if you can log into it via serial port through UART, or do you typically go straight for attempting to dump to NAND flash content? Yeah, usually the first way, if I want to get the data, the most most effective way is get the data when it's in a constructed fashion. That would mean if I can get a console on the device with the entire operating system running and the entire partitions for all the data laid out, that's easier because then I can easily DD the partitions off the off the chip in their constructed fashion. And that would avoid me almost always having to do what we're going to talk about here. What we're going to talk about here is in that case where you actually have to read the data raw right off the flash chip, either chip off where you remove the chip and drop it into a reader or some other methods where you may be able to clip on or hook onto the device and read the raw data out of the chip. And that's when you get this structure that we're going to be dealing with today and how to clean it up and a few methods of recovering that. So I shout out to everyone. If you're interested in interacting or asking questions during this, join the Zoom chat, the Zoom channel that we have set up for the webcast. Once you're into there, you'll be able to post questions up during the presentation and hopefully we'll have time and we'll be able to answer them. So please do that if you can. It makes this way much more interactive. Instead of just being a data dump, we have a chance to actually interact with the audience and the people and learn more in this process. So I'll go ahead and move on. So section two is a general chip readers. Since I talked about chip off from the question Jonathan gave us, we get into chip readers. So when you're interacting with a NAN flash chip, here are the signals you're typically going to need to interact with. And they go everything from the 8 bit data channel, the bidirectional channel, all the way down to input signal, output signal structure and some other various pieces. So if you end up buying off the shelf chip reader, this kind of solves all these problems. But there was some work done out there by this gentleman here. I would recommend taking a look at this write up. If you're interested in building your own chip reader, this one actually works against a TSOP 48 NAN flash chip. And he uses a technique known as bit banging using this FTDI board that he has here. He's able to bit bang the data off of the actual chip. So I would look at looking at what bit banging is because I'm not going to explain it here. And look at how he's set this thing up. If you're at the hacker mentality and you like the idea of building things your own, I would check this out. It's kind of cool. If you're like me, and you just like to throw money at it, these are chip readers are reasonably inexpensive. The TI 866 plus, you get all the sockets with it. It's like 130 bucks. This will read a number of NAN chips. Pretty good. And it uses a TSOP 48. Right below it is a socket that doesn't come with the set that you need to buy. This socket is for from TSOP 32 to 48. It's known as a Nando 8. This is for larger memory chips. So anytime you encounter larger NAN flash memory chips in the TSOP 48, the one that comes with a kit will not work and only work with smaller memory blocks. So you need to get the socket for that. The Proman isn't made. This model wasn't made anymore, but they do make a similar version. So if you search for Proman, you'll find a similar version out there. The socket below that is designed for 1.8 volts. The Proman is typically a 3.3 volt device. And we're starting to see way more chips in the 1.8 volt arena. So it's good to have that add-on. Most of these add-ons are $8, $9, $10 on both of these chip readers. So they're fairly inexpensive. The thing I like about the Proman is if I can't identify a TSOP 48, I can't find the data sheets or not. Well, let's say I do have the data sheets, and I can't find any information on none of my chip readers. They don't identify it. They can't detect it. This will let you hard set page OB block sizes into the actual tool that comes with the application that comes with this, so that you can attempt to read it using that kind of raw information. And it's been successful several times with me, making it possible for me to read a NAND flash chip where I had the data sheet to identify that structure, but none of my chip readers would recognize the chip. And the last one's RT809, and that one's a little more pricey if you buy it off Amazon, but I think you can track that one down on AliExpress cheaper. I like this one. It's not a bad chip reader, and it has, you can buy a lot of different sockets for it. Most of the sockets that come with the TL866 plus, the one on the left, will work on this, all but the TSOP48 sockets, because the TSOP48 sockets have a circuitry built into them, so they won't work on this reader. You have to have a straight pin through. And the one I have below shows that TSOP48 socket that's a straight pin through. I also recently had to pull a NAND flash that was a BGA ballgurt array, 63 balls on the bottom of it, and I was able to buy this off AliExpress for the RT809 for about 40 bucks, which isn't bad. If you get into the high-end chip readers that run, start off at like $1,600, $1,700 and go up for them, the sockets for them start off at $100, $200 and go up into the thousands for BGA sockets. So these things are very cost-effective. The sockets are not high quality, so you have to take care of them. But overall, they're pretty good products. And I use them quite regularly in my lab without issues, working with NAND chips. So any questions in reference to the chip readers and chip reading process? One question here. What are the most challenging NAND packages or pitches to work with that you've encountered? Is there any package or pitch you would not attempt messing with? I would never say there's nothing I wouldn't attempt messing with. I would literally mess with everything. Probably some of the biggest problem is, obviously, if you're using a chip reader to try to read this, the BGAs or ballgurt array devices are the more problematic, because there may not be a socket available for some of these devices. In cases like that, if you have the data sheets, you can always go to the process of dead bugging. That is turning the ballgurt array chip on its backside and soldering small 40 gauge wire to the actual balls, and then feed that into a connector or a chip reader or a device for doing bit banging to actually be able to do that. I have been successful doing that a couple times, but it's a pain in the butt. It's way much easier to pull the chip with hot air or IR if it's a BGA, clean it up, drop it in some device, close the lid and have it give you all the data. But typically, if I need the data off of it, and I can't get it any other way other than chip off, I will chip off from the board. And if necessarily, I will dead bug one. I mean, me and you, Jonathan, we worked together at one time where we dead bugged that one weird device. We had a 0.25 millimeter pitch BGA, and we had to solder 40 wires to it or 20 wires to it to get it hooked up. So yeah, we're going to try anything. I'm not going to pass up any potential target. If it has data on it that I want make sense, and you're definitely bringing back some crazy memories with that. And actually, you partially answered the next question as well. The next question was what hot air tools do you recommend for BGA removal? Actually, I have both hot air and I have a IR oven. I like the IR oven. Once I got it, it's a small device. It'll take a board that's probably eight by nine, eight by 10 inches or something like that. To me, that's the sweetest thing in the world. I usually just stick it in the oven, run it through the cycle that I brings it up to the temperature, and I take a suction tool, I pull the chip off of it, and then I let the whole board cool down. Because it heats up the whole board, you have to watch out for things like plastic components that are on the board. So you may have to remove those prior to putting them in the oven, but you can usually pick up one of these IR ovens. If I remember what was the name of the one I got? It spelled P-U-H-U-I. It's a Chinese made product or Asian made product oven that I have. I think it's a Model 2 962 is what it is. Small oven, I think I only paid like $130, $140 for it somewhere around there. And there's a number of products out there like that, but an infrared is way much better, in my opinion, than hot air. So another question. Sorry, Darryl. Elvis Cousin asks, is it hard to reball BGA without killing it to resolder? Yes, it's not that hard. It's just the process of learning how to do it. Not to drag this off, but I had a project where I wanted to learn how to do this, because I wanted to get root access on a device. It had 153 ball BGA embedded multimedia controller on the device. So I used hot air. I pulled it off. And I had a couple of devices because I wanted, in case I killed one of them, the first one, when I put the balls back on, there's a couple ways to do it. You can use screens with paste or you can use just the balls. In my case, I just used the balls. I floated on there correctly, but I overheated the chip and it looked like a chocolate brownie when I was done. That's when I learned that you don't want to go on these BGA chips. You don't want to go over about 210. Well, melting temperature is about 183 degrees Celsius, if I remember right. And you don't want to go over more than 25, 27 on the temperature in Celsius. If you hit 215 or higher, it starts causing the fiberglass substrate of the chips to start breaking down. But if you keep it within those range and you don't shock it by overheating it too quick or cooling it too fast, it'll work. And I was successful. The first one I burned up and I failed. The second one was perfect. Since then, I've never had a problem ever working with BGAs, either pulling them or putting them back on without damage. And once I learned those temperature parameters. Okay, so the next section we want to move into is going to be demo. So we're going to get off into the demo area and see how that works out. Hopefully, we won't have any technical problems. So what we're going to do here is, we have a flash memory that's been pulled. This has been pulled from, let's go ahead and look at it. I got a couple of them here. It's pulled from an Arlo device, which is a doorbell, if I remember rightly. It's actually kind of like a ring doorbell type product. Its memory chip was a W25N01GV. So typically when you pull this thing, what's the first thing you want to do is bin walk. And again, bin walk works many times, but sometimes it does not. So we'll give us a try and see what happens here. Hopefully, we won't run into any real technical problems. And this is kind of the common flow. We see we hit a squash file system. We should have at least two squash file systems on there if it's coming off a standard embedded device. And there's a second one. Then it kind of hangs right there. If we set here, this thing would set here in idle for hours and not produce anything. So we're going to stop that. So if we change over to the extraction from bin walk, we can see that it opened up a bunch of stuff, but then there's a squash file system. So as we notice, there's nothing in there. And truthfully, because it went past the squash file system file, it should have put something in there. So as we can see, there's literally nothing in there. So for the most part, this didn't work. This is actually a fresh rebuild of bin walk. And also I have a bin walk pro account. The results are pretty much the same in this particular case, because there's some things that have to be dealt with here. So let's go ahead and remove this structure just so I don't start running into space issues by blowing all this stuff out of here. So what do we want to do? Well, as we talked earlier, we talked about the actual OOB area. So we want to be able to remove that area. So we have a tool in here called NAND dump. And at the end of this presentation, I'm going to have links for everything, which will cover all this stuff. So here we can actually do input files, output files. Remember that ID I mentioned to, chip ID? We can use chip ID or we can specify actual pay size, OOBs in the layout. Remember adjacent, separate. So in this particular case, we are going to, well, before we go ahead and separate the OOB out of here, let's go ahead and look at the data sheet. So as we look at the data sheet, we see that it is 2048 bytes by 64. So it is in a separate format. That means the 64 bytes as at the end of the page. That's a good sign. That tells us the structure. You always want to go to the data sheet. But typically I always like to dig a little deeper and make sure that the chip hasn't been kind of messed with. And why I say that? Because just because the data sheet says this is how the chip is going to be laid out, the manufacturer that actually used the chip, I've seen them do some strange things to say the least to the data. Meaning that, you know, you go and remove the OOB area thinking you know where it's at and literally have it not be laid out the same way. So we're going to look at this with hex edit. So we're going to look at the raw data of the actual device so we can see what it looks like. So at this point we should be able to go to 2048, which would be 800 in hex. And it should be the beginning of the second block. So we can see that there's something here. It looks a little different, but it may not be. We're not quite sure. It doesn't really tell us a lot. So we could go to the next block. So for example, if it's hex is 800 and we want to, of course we want to go, remember is 20, gosh we get it right here. So if we go 2048, we'll get it cleared here in a second. Try it again. 2048, which would be 800. And we want to go up to the next beginning, which should be 2,112 bytes. Remember 2048 plus the 64. And we look at this. It should be at 1040. So if we go to 1040, we still see some interesting things. And this may tell us that this is correct. But I've seen a lot of devices where the actual OOB area was nothing but all Fs and wasn't actually configured. So what we're going to do is let's look a little deeper. So we can see some patterns because I was a big fan of spotting patterns. So we come over here and get it going. And we're going to search for strings so we can get some data. So here we are. So we're in an area where there's a lot of strings and let's scroll down until we see some patterns. And quickly we see a pattern right here. So got a pattern right there. So if that is, so we go hold up 2112 and go, so be 840. So we go 840 plus E880, E880 equals F0 C0. So the next pattern should be an F0 C0. And we can see that we're starting to see that pattern there also. So as we can see more than likely we are witnessing the 64 bytes at the end of the 2048 as the OOB area is showing up. And again if we look over in the string area there's no common words because I would not expect that to be in the area. So let's scroll down to the next one. There's another thing I like to do sometimes, especially if I'm having problems positively identifying this location. So this will involve a little bit of math. So I like to use a calculator for that. So if we come over here and we put this in here like that and we turn it to decimal. And then because the programming calculator does not round off or have decimal points we want to do this basic. So what do we know about the overall structure? The first OOB area or OOB area showed up after 2048. So we had the first 2048 and then to get to the next leading ones we had 2112. So we go ahead and subtract and hopefully my math works here and doesn't mess it up. 2048 and then divide it by 2112 and hopefully we'll get an even number and we did. If you do not get an even number then this is not mathematically correct for an OOB area. So what we're seeing here is 0500F900 is actually the 39,749th page of data in this particular device. So now that I've punished you with some math we're going to go ahead and strip that out of it. So we have this command called NAN dump PI. We're going to use .i arlo for the input. The layout's separate. Page size happens to be 2048. The OOB size is 64 and then the output will be arlo no OOB. Hopefully this runs and it finished up. So there once we finished up we have a file that's a little smaller as you can see where the OOB's been stripped out of it and let's go ahead and run this and see if this works for us. The first squash system you see the pattern's different now and then we also see that there's an UBI structure also starting to show up. Often you'll see inside an operating system on embedded device you'll see squash file systems which are typically read only. UBI file systems allow read and write so it may be an area that's used dynamically that's being created that can use to read and write data into the device. Okay so that one actually finished. So let's go ahead and jump over here and see if it worked for us. A little different there we see we got three folders we have an UBI root system. So we go to squash root and there you go we were actually able to extract this data by effectively removing the out-of-band data within the NAND flash dump. So do we have any questions? Yeah we got a couple of questions here T-dub so Elvis Cousin asks I have two BGAs on the daughter board is it likely I can read each individually or will I need to reassemble them somehow to try to get a squash FS and they're also both Intel-based flash. Oh you're saying you have two of flash memory chips on the device really comes down to how they how they lay those out. I would not expect them to stripe across separate chips like that I've never seen that. Often when I've seen two chips done up like that each one will contain a different partition or they may be redundant of each other but typically they'll be partitioned separately on embedded that's the way I've seen it and I've seen everything from where one chip actually contained the the boot U-boot and a kernel and the other one contained a file system as an example. So the only way is to pull them off examine the data and make a determination from there to see if you can do it. Of course the best solution is if you can get a root console on the device somehow then obviously you could figure that out more quickly and DD the partitions off that way which is also the the sweetest way to doing it but if you have to chip off yeah you still may have to do some analysis of them. Yeah you kind of pivoted off that said both are connected to a common connector but I don't have a pinout to try to recover cross-connector by soldering something. Hmm yeah it's kind of hard to say if you have some kind of pictures or images of the data and the chip or not the data the chips and the layout and the board shoot them to me sometime and and maybe I can take a look at it and come up with a help you come up with a plan of attack. Like I said anyone's feel free to reach out to me anytime I may or may not have an answer for you but I'd be more than glad to give you a hand if I can. One final quick question here that I'm seeing so in this particular case it looks like you're able to find the data sheet have you ever encountered a NAND chip that you wanted to dump but couldn't identify it or could not locate a data sheet for if so how did you approach that? Yes guessing often often if if if I can't find a data sheet actually for that I've actually gone out and looked at other chips that may have data sheets that are in that same series are produced by that manufacturer and figure out how they do their layout even though I may not be able to quickly identify the chip ID there are ways to actually enumerate those and it usually typically needed data sheet to do that but guessing does work and crazy as can be I've actually dumped a lot of chips by going hey here's a manufacturer and it seems like most of their chips or large portion of them within this structure range are laid out this way they have this page structure this size often the naming conventions like like here on the screen W25N01GV that's one gigabit chip and often in the name you can figure out what the size of the chip is a lot of times that would be 8 bits what 128 k bytes or whatever so it's doable just looking at the name sometimes to pull some typical information out of it and then guessing on some of the other stuff but again I've also like smoked chips by doing weird things to them it obviously didn't work so gotta take some caution and take some risk but amazing enough it can be successful one of the things to do also is kind of look at if you can get a UART console onto the main processor let's say the main processor has a console even though you can't get root access to it capture all the debug data from the boot up that can also be very revealing sometimes okay time to move on so we've kind of done that we're able to get the data off that I'm gonna go ahead and clean this up because we have a couple of exercises to go to and I don't want to run out of time so okay so the next one we want to look at I'm also gonna R M R low so the next one we want to look at is the nest this actually came from a friend of mine who had his nest thermostat crash out it just died he threw it my way said hey can you get any interesting data off of it so and I struggled with getting data off of this but it was eventually successful where we're gonna show some of this there was no and confidential data actually extracted from the device so there's no problem with us using it here but so what we want to do is what we're gonna use is a process where we're gonna create a NAND SIM so what does that mean we're actually gonna simulate a NAND chip in memory we're gonna use like mod probe if you haven't used mod probe mod probe will let you load kernel modules onto the system so we're gonna build a block device using mod probe kernel modules we're gonna load those up and then we're gonna write the data out to the chip we're actually gonna tell the data with NAND write to use the OOB data to reconstruct any error corrections and that type of stuff and then we'll bound it up so this particular one I actually ran Benwalk on this and it never recovered anything I actually shot it out to good friends at that do the Benwalk Pro and unfortunately I didn't get back to the system for a couple of weeks and it churned out there so I apologize to them again for eating up several terabyte of data so this thing was constructively a pain in the butt and running Benwalk was potentially catastrophic it would eat up all your memory and produce nothing and I'm not gonna run it here but if you run it every page or every header for a GIFs2 file system or every header for one of those it would create a folder for it put nothing in it and literally eat up terabytes of data in a couple days quickly so let's go ahead and get going here so we're gonna load mod probe mtd mod probe block this will help us set up a block device and then we're gonna go ahead and we're gonna load this up here so the way this works mod probe nansim bch is error correction functionality this was the chip it uses a bit structure there's a bch16 for a 16 bit structure if that's actually being used and this here happens to be the chip id so if we go over to this particular chip that is not the one I want so this is the actual chip and I believe we go to page 25 if I have this memorized okay and this would be the read id the id of the actual chip we know this chip is a two gig chip it's organized in eight bit structure it is a 1.8 volt bit chip so zero one is the first one AA9015 four four isn't used in this case but it deals with error corrections and multi plane information multi plane information just to make note here if you have a chip that has multi planes on it and you'll see it in the data sheet it may or may not stripe across the two planes meaning it puts one block on one plane and one block on the next plane versus putting them contiguously on the same plane I have not encountered this but I have read about it so it's something to be noted so at this point we're going to go ahead and load this up so now we've loaded up that particular module a kernel module and now let's go ahead and just look at some information MTD info and this will tell us a little bit about what we just created so we created and simulated it's how many bytes 256 megabyte chip 2048 sub page size 64 bytes pretty straightforward and then we go ahead and this one may show some errors sometimes it does sometimes it doesn't and we're going to look at this within a d-message and if we come down here we can see that it took the manufacturer ID chip ID and information and actually have identified the actual chip that is the chip ID that does that and helps lay this out so the next thing is we want to be able to write into it so we're going to use nanrite which is another tool I hope you don't mind me not typing it saves you guys a lot of headaches so cancel out of that nanrite I want to see the command so here's all the functions so what we're going to do is we're going to input contains OOB data so we're telling it to utilize the OOB data and we're going to go ahead and let this run and hopefully it'll run through clean boom there we wrote all of the blocks out onto the device so the next step is we'll come over here so you see this we go LS and we can see there's nothing there and then what we're going to do is cut and paste so I don't have to type we're going to mount this up on block zero change directory to GIFs and voila we have data so if we look through here it starts looking pretty good hey we've recovered all this data the sad thing is most of this stuff is system logs that we recovered but if we get up here toward the top we find some weird things here's bin it's a file ETC is a file well truthfully this is bull crap we know that's not a file so if we cat ETC wow look what we get now I wouldn't expect to see that in ETC so we recovered a lot of it but some of it's corrupted in a weird fashion so how did I recover from this I experimented and we're going to go over that a minute but before we go forward I actually want to see if there are any questions Jonathan any questions out there yep they're going to look now no updates here checking Twitch real quick yes here's a question in IoT devices you've examined do you ever come across NAND storing non-file system data if so do you just dump and then use bin lock and other usual tools and methods for poking around yeah I've had them storing non-file system data on NAND flash so yeah typically typically I get into that when I get those and there's no structure to them I start off with throwing things like strings to the thing with strings can we identify any known structure in there I will run bin walk just to see what it'll do sometimes it'll point out some structure headers that I can more quickly find so I think that plays a big role but yeah sometimes you'll have them find storing things like blocks of configuration data and I've seen them just blocks of encrypted data which may be a file system structure but they're not in they're not framed as such so it's just encrypted blocks of useless information so yeah that's not all that in common well most of the time when I get an NAND ship it's a file system but you will encounter this to be honest so right now before we move on to the next section I want to remove the kernel modules so they don't conflict with what we're doing okay and we are down to we've still got some time doing pretty good so the next section I want to get into we are going to use MTD RAM to actually do this so we're going to take that same system that same same dump that we just did where we mounted up the file system did not come out exactly the way we want it and we're going to mess with that so we're going to make a make directory and we're actually going to create our own block structure and then we'll go and then we're going to make a node if I learn how to spell it I should just copy this over make node dev slash MTD block and we're going to create zero we're going to create it as a block device we're going to give it a master number or primary number and a minor number of zero so we're going to create that device in there and then what we're going to do is we're going to use mod probe and we're going to bring in a kernel module called MTD RAM so this kernel modules MTD RAM we're setting the total size to 274432 that is bigger than a 256 megabyte RAM area 256 megabyte RAM area would be 262.144 and that would not contain OOB data in this case here we're bringing in the OOB data also because if I bring in without the OOB data I noticed that we get the same results as the other ones so experimenting and I encourage you to experiment when you're dealing with these NAND ships it can be quite fascinating what you can find and figure out and make work to get you data out of this system I'm currently working on one it has me slightly befuddled but I will master it eventually where the actual entire data on the chip is shifted 10 bytes they even shifted all the OOB data 10 bytes and it took looking at it manually to figure this all out and doing that math and looking at the data in the structure to determine that they had also shifted that because I was getting block errors saying that block is 10 bytes longer than it's supposed to be type thing that's when we found out that everything was shifted and was able to confirm that the OOB was shifted to but I digress so what we're going to do is we're going to load this up and it's supposed to contain the entire thing and then we want to load one more module MTD block for a kernel module and then we're going to DD excuse me and DD is a bit copy I use this all the time matter of fact if you get access to almost every embedded IoT device you'll find DD is actually installed on it because most of the time DD is actually used for doing firmware upgrades you'll have duplicate root file structures duplicate kernels on the device and what happens is one is primary running so what'll happen is when the firmware comes down it'll DD it over the secondary and then it'll change the U-boot or secure boot arguments to make the new one primary in the old one secondary that way if you have a failed firmware update it does not break the device it just goes back to the original one which was not overridden so again input file here is the nest file and then output is to our block device that we created and if I did everything right this should start writing data should be fairly quick and then you see we're able to write out 277 megabytes of information fairly quick because it's all internal to the system so from here we're going to go ahead and mount this up same way we did on the other one and we're going to mount MTD block zero up now if someone wants to explain to me why this worked because I really not sure why like I said often dealing with NAND chips little quirky things have worked for me so I'm not going to speak bad of them now we can see ETC and bin are actual files we do have some corruptions in here so there was a few files actually corrupted within this but for the most part a large portion of them were actually able to be recovered properly and so these are methods that I use quite regularly for recovering when you start working with UBI file systems if I ever get some good methods, methodologies hammered out for working with some of the strange UBI file systems like the one with the 10 bytes that shifted I did one on an engagement here earlier this year where again nothing our large portions of it were not laid out contiguously and trying to build good solid methodologies around UBI I'll probably come back around and put together a training session presentation for that down the road that's what I'm currently working on trying to work out those kinks and bugs and build that knowledge level to build to work with those now probably half the chips that are UBI file systems I'm successful at recovering them using various techniques and UBI tools that are available out there but typically GIFs squash file systems things like that can easily be quickly recovered from NAND flash chips using any one of these three methods that we covered today in this presentation so hopefully there's some questions out there yeah I got a couple here so one question here asks have you encountered an IoT device that was encrypting the data it was storing in the NAND flash no I haven't which is kind of amazing to me I mean the only time I've ever seen IoT technology using any kind of encryption if it was running an Android operating system and by nature Android can support the data section being completely encrypted so that's quite common I do have some other tech that I consider IoT but it's more enterprise IoT and not necessarily the normal consumer grade which actually is very much encrypted in most of the not all the file system but most of the file system that contains user data similar to the Android file system is also encrypted so typically if I see it it's an Android it's usually 50-50 if it is a standard phone style Android operating system dumped on there then most always it has encrypted but yet if it's an older Android style that's put on there there was no requirement or typically that's not always encrypted so it's a mix and match but it's pretty rare to see encrypted at least from a lot of the IoT I've looked at unless you get into more enterprise enterprise level technology Anonymous attendee asks hi I am new to this sorry you already have the binary file how did you extract it from the hardware device in the cases of these both of them were chip off so early in a presentation and so you can rewatch this presentation or in the presentation earlier today where we talked about building an IoT lab and we showed a number of devices I have a number of chip readers so I actually use those other ways are often if the if the chips if the chip that's being used has some kind of debug function you can put it in debug and pull the data off of it also you may be able to check connect in with JTAG or Surawar debug and actually read it off the device but if those fail I often rely on just desoldering the chip and dropping it into reader and extract the data from there if I need to put the device back into functionality then I'll go through either resoldering the chip back on or reballing the chip and reflow it back on in an IR oven which are fairly cumbersome at least the reballing BGAs but it's still doable and it's not all that complex I did a presentation last year but the year before last at Bo Nuita hack I did it there I also did it gosh I did it in Perth, Australia and I also did it at Derby con and it was I'm trying to remember the name of it basically it was how to get root access the hard way and it talks about slashing smashing cutting the device apart rebuilding it to get root access or something to that effect so you can look at that and it goes through some of the concepts am I learning process on learning how to manually reball BGA chips Elvis cousin asks these are all variants of Linux on a Windows CE device might it still use Squash FS or some other Windows thing ooh probably some other Windows thing I have I have not looked at Windows CE device I have been looking at some Chrome devices in those particular in those particular cases they have this has an SD drive solid state drive where you're able to get the data off of it and just mount it up on your system I'm not sure if they have NAND chips is it going to be the Linux file system probably not it's probably going to be some variation of Windows but I have not tore into any Windows or Windows CE type devices to be able to give you a solid answer on that let me know if you find out any other questions nope I'll cut up all right well I think we're what time we start six o'clock so we're probably right on time so let me jump over here let's do a final stop to share so let's go ahead and ask question one more time anyone have any questions any requests for questions or information if not you'll probably catch me on Discord either here in a little bit after I've eaten or late tonight or tomorrow feel free to hit me up for any questions also oh gosh yeah I was promising that hold on before y'all drop off there let's get this up so I had promised slide show play from start so I want to be able to get some of these things captured in the video so here's the NANDSIM example that I use the mod probe commands the NAND write commands and the mount to be able to mount this particular GIF file system up and here is the commands that I use for MTD DRAM example quick note on some of the commands we use there is the link for NAND dump tool concept of modpro make node NAND write DD and then here is the reference so that gets captured in here all of this is kind of cool material there's a lot of great work that's been done out there on NAND chips like I said I'm constantly reviewing and reading and consuming this material on a regular basis because every time I encounter NAND chip that wants to fight with me I have to go out and start rethinking okay what am I missing what do I need to learn expand my knowledge how do we kind of move forward from here and there's been a lot of work done out there and of course we shared much of the information that these people provided in this demonstration today also if you're interested in connecting up with me on Twitter my handles are %x so it would be P-E-R-C-E-N-T underscore X and feel free to connect with me out there or reach out to me and ask questions out there awesome Darrell thank you so much oh man sure enough Sam had a blast yeah and I mean honestly if there was any if there are any last questions we still have a little bit of time so was there anything else that you saw there Jonathan yeah another quick question just popped up it says this is some very epic stuff extracting flash from NAND chips I have to ask did you find anything epic while performing post exploitation on the extracted firmware yeah yeah I mean you know any you know remote shells no unfortunately but but again it's information that we could feedback to manufacturers to help improve processes you know they get into hard-coded stuff mainly a lot of this lately was purely an effort to go hey what happens when IoT is thrown in the trash type thing you know is there data is like the guy with the nest thermostat what happens when something dies if you throw it away does it contain PII information so this was kind of a lot of it was driven out of that and some paid assessments and also you know like we tested a lot of devices from hey if you go ahead and flush the thing back to factory does data stay on it kind of start checking for things like that it's all about a privacy risk and issue because we've gotten we've gotten really big on the old kind idea that hey we don't want to throw our hard drives away but in this modern world these chips are our hard drives and we need to probably start thinking about the potential risk there so that was kind of the main purpose of a lot of this and the fact that I had been working on some engagements that involve Dan chips that were problematic in those cases we we're working on things like reconstructing keys to gain access so a lot of this stuff ties into engagements also where we're looking for data since we're testing an entire ecosystem that isn't just the hardware but it may be mobile apps cloud services and other devices what can we get off this chip that could allow us to attack everything else or attack somebody else using the same product because of key reuse and key session keys and password reuse and things like that so a lot of it's within that direction another question here this individual asks does it seem like it is possible to be able to create a module so that a chip reader would automatically be able to to detect those out of the OOB areas of the flash memory oh current number of chip readers will let you identify someone will allow you to extract the chip the data without the OOB so yeah it's not uncommon you can often manipulate data from the from the chip reader and include or not include OOB on some chip readers obviously you get more control as the price of the chip reader goes up and I don't necessarily have a $2,500 chip reader in here but I think one of these actually had some OOB settings but for some reason I didn't necessarily trust it because because I told it not to include OOB and then when I visually looked at the data the OOB data was there so I kind of lost trust I have a tendency to also like to map it out not let the chip reader make the determination for me that this data is structured a certain way because like I said it's not uncommon with vendors the futz with it so that it may be incorrect or that the OOB shifted or is not used in the same place and they've just written a new controller to read the data the way they made it this way it allows me to go hey this is correct or isn't correct and then I can easily manually remove it or modify code or scripts that are out there to take it out the way I want it removed so that's kind of my take on it that makes sense I think that we're at the bottom of the list here for the questions okay well I think that kind of concludes it if anyone has any questions catch up with me later or reach out through me on Twitter and again I'll be on Discord probably tomorrow and maybe later tonight so I'll catch everyone later have a good night