 give it up for Brad Dixon. Thank you. So I'm Brad Dixon. I work with CARV systems. We have a lot of fun working with systems that involve embedded or IoT devices, you know, trying to do security assessments and penetration testing on the whole system. And this is going to be a presentation, a quick one, about an attack technique that we found repeating itself as useful more times than we thought it would. And I'm going to describe this as being useful because it works, it's easy, it's pretty dramatic, and it provides a teachable moment about designing more secure systems. But it is a novelty. And it's a novelty because it's risky, it's crude, and it's perhaps redundant to a lot of other great techniques that are out there. But it sure is fun. And when you do this, you do this device, like you can demo this attack to your mom and she'll just be like, oh, I get it. So let's just quick get to the demo. So to set the stage here, we've used this on a lot of devices, but this one in particular is an embedded Linux device with UBoot as the bootloader. There's no JTAG that's actually been turned off with microfuses inside the CPU. So that's gone, can't find it. JTAGulator was great, but couldn't find JTAG at all. And because this device had had some challenges before, let's say, it had a homegrown secure bootloader added. So getting to the demo here, this is the system booting up. And you'll see why we call this pin to pound in a moment. This is my trusty like 1995 multimeter just jamming into the flash chips. And I'll show you where in just a moment. But as the system is booting, it's doing a cryptographic checksum across the flash image. And when that checksum doesn't match what it's expecting, it does a fallback to a secondary partition. And it's going to start that in just a moment. And as you can see, using a serial console on these devices, you can track what's going on. I didn't put the clip in there, but I'm again poking the flash device to get the second partition to fail as well. And they designed this device to not respond to any serial input. But what happens is they've misconfigured it so that if it fails twice in a row, the primary and the secondary partition, I get a U-boot prompt. Excellent. That gets me in. So the primary misconfiguration was not checking their failure paths. That happens as I found out more often than you would think. And there's also going to be two kinds of flash devices you'll see in these demos. This is a parallel flash device. It comes into standardized package, typically a TSOP 48 package. You can actually attack it from the left or the right side. I'm attacking on the right side between two of the data lines that feed data out of the flash device when it's called by the CPU. And as you saw, it's pretty low tech. You just poke it. But there's been a lot of prior work on this. Glitching as a form of attack has been done numerous times and just some really amazing stuff, like extracting cryptographic keys with transient glitches induced by a number of manners. There's also been other blog posts about this precise thing using, you know, a transient electrical fault to get a failure mode that's advantageous to doing a penetration test. And so there's some of the links you can check out. But what I want to do today is just provide more detail so that when you sit down, pull out some devices from the closet, try this, you might be surprised as I was how often this works. I have to warn you, well first of all this is GROG, GROG is our mascot. Sometimes, you know, you try elegant techniques and they work. Sometimes you just beat stuff with rocks and you get what you want that way. But you can definitely break your hardware doing this. I haven't yet destroyed anything. But you can. And the way you'll do that is by exceeding, there's an absolute maximum current that can go out of each IC device. And if you exceed that for too long, then that device might just be, you know, kaput. The black smoke comes out. You can also temporarily cause a device to fail. They have sometimes protection circuitry so that if you exceed the operating conditions, that pin just shuts off. Usually power cycling will fix that. I've had that happen a few times. And of course, depending on what access you have to the device, like if you have JTAG, you don't need to do this. If you have other means to get what you need, use the safer means, certainly. It will, you know, prevent you from breaking something. If there are any time travelers in the audience, go back and listen to Joe Grand and Joe Fitzpatrick's 101 ways to break your hardware. I think I'm adding a new manner to do so. So let's get to the details about how can you mount this attack yourself. This is a general architecture diagram for the kinds of devices that we work with. CPU devices, 32 or 64 bit processors running Linux using typically a boot loader like U-boot. And you're trying to interrupt the communications between the external flash device and that CPU. These can either be serial or parallel flash devices. The reason that this works is that systems boot in stages. And what's being shown right here is the activity on the flash bus. You don't need to, you know, looking at trying to decrypt the details of this by zooming in is not helpful for this, but you want to get an idea of like the wall clock timing on this. And figure out, well, when is a boot loader being loaded, like it's shown on the left. When is the kernel being loaded? What's the duration where the device is booting? And then where does the user space and knit process kick off? And you can actually attack in two different places. The most successful for me has been interrupting the loading of the kernel so that you fail to a U-boot prompt. But I'll show you an example in a second of something that was actually much more surprising to me. So in this example, this device had actually been pretty well secured. This was a device where JTAG wasn't going to be an option, you know, based on how they designed the hardware. But I found a point later during the knit process where poking the serial flash device caused the knit process to fail and give me exactly what I wanted out of this. A root shell on the device. Yeah. So this kind of misconfiguration is much more rare. Forgetting to have a useful failure mode for U-boot, that's pretty common. I've seen that a lot. But this one was more rare and the reason for it is this embedded Linux system had been really cleanly set up. It was great work. Um, and they had, but they had left something in there to help the developers out that when the primary application started up, it would grab all the serial ports, throw up an authentication prompt. It wasn't Getty, it was something that was built into their application. Um, and then, um, but then if their application for whatever reason failed, the next step in the knit process would be just run a shell. Now another thing that happened, another characteristic of the system was important to this. Uh, this system was using busybox. Busybox is like a, uh, what's called a multi-call binary. It's like a Swiss Army knife that does all the things that a typical Linux system needs, but does it with just one binary. And so since most of the pages for that had already loaded before the application we caused to fail had started, when that application failed, even though I was screwing up flash for it, busybox was already resident memory. I think it might have been different if it had to go load other pages because I wouldn't have been able to time that attack very elegantly. This is also, uh, an example of a serial flash device. These typically have an 8 pin, uh, pin out, very standardized, much bigger device, you know, so, uh, you can use a multi meter probe to, you know, to poke at this. And I was, uh, poking between the chip select, which says, hey, flash device, read me out some of that data. Uh, and the actual data output on that. So here's another example, this was an LTE router. Um, on this device, um, as you can see up in the, your top left, uh, there's a little dim looking, uh, CPU module. The flash device was on the underside of that dim device, so that's outlined in, in the red box below. So to get set up for this, I had to pop that device out, flip it over, put my pin where I wanted, throw a little blue tape, uh, tape on it, and then pull that pin out, just a fraction, you know, a couple millimeters, so that it'd be ready for the attack. I'm working on the left side of a parallel flash device, another TSOP 48 pin out. It's a standardized pin out, so you can find this pretty frequently. And I'm probing, uh, the chip select, uh, sorry, the, uh, command latch and address latch lines, shorting those two out. And what those do is when that, when that command latch line is toggled, it says, here's a command to make sure you read this in, do what I'm asking. And then here comes an address. And just by screwing with the logic on that, you got this flash chip confused enough to do what I wanted. So this is a quicker demo on this device, just a little, boop, on the pin. And, uh, we're gonna end up back at our U-boot prompt, which is where we wanted to get on this. So, doing your own pin to pwn attacks, uh, pretty straightforward. You need to survey the hardware, pop that thing open, you don't need to take anything apart at the PCB level, but just, the key things you want to do is find all the flash storage devices, figure out how you can access them. You need some way to monitor the boot process. The serial console great, because that's probably the access that you're gonna get. The developers will turn the serial port on when something goes wrong. Um, but, you know, there's other ways that you could do that, perhaps. You know, monitoring with a logic analyzer and just getting a wall clock time on it. Um, you need some data sheets to figure out on the flash storage devices that you have. Um, where can you probe successfully? There's some trial and error on this. Not everywhere I tried worked on each device. So you're gonna have to take a few cracks at it. And you need a way to be able to understand when the device fails. Maybe it's LEDs, maybe it's something on the serial console. There'll be something. Because the last thing a developer wants is for something to go wrong in the lab and for their device to be bricked. They left a door. Find it. Um, you got, after you selected your pins, you're gonna, gonna start poking at that. This took me a try. Usually about six or eight attempts per device, just to get the timing right on it. Um, but, you know, it's something you can work through pretty quick. Uh, make sure you power off between each test. If some of that, uh, protection circuitry in the IC gets engaged, power cycling gets gonna help you out. And then monitor for a different operational state. One that's not the normal one. Uh, getting U-boot prompts, uh, pretty common. But some devices have different failure modes like, uh, enabling networking ports or failing to like a USB, uh, device firmware upgrade. You'll need to find those. And you need a little bit of luck on this too. So, we pulled out all the devices in our closet and we said, hey, let's go for this. Let's see what other devices can do this. And, you know, about, that little bit over 50% of the time, we were able to get some failure mode that was helpful to what we're doing. You know, getting root on one of these devices is really just the start that's kind of like day one of our project. It just helps us do the rest of the, rest of the work. Um, but it's really cool to be able to demo to people, to say, watch this. Boop. And you end up with what you want. Um, it's, it's a great demo. I highly recommend it. But let's talk about some of the places where I was unable to get, uh, to get root. The devices that have thoughtful consideration of how to fail, those are pretty resistant. The best ones, uh, were the ones that would reset if, if they were unable to read flash on it. From a consumer perspective, that device is bricked. That's probably maybe a bad thing for business. Um, but from a security perspective, that's a proper reaction to it. Ways that you can improve your design or, uh, you know, give recommendations to others. You know, if you can't all turn on, uh, watch dog timers early in the init process at the boot loader level and then start servicing them in user space so that something interrupts in between, you're going to confound an attacker to a degree. Um, you know, who's trying to get at this device. Um, and just be very cautious, you know, about shipping failed to debug mode systems. Uh, those are exploitable. You just got to find the way to get in. The other one is really a design, um, decision. Uh, hide your pins, hide your traces. Um, BGA or ball grid array devices. Uh, they have an IC with little balls underneath that mounts directly on the PCB and the PCB designer can actually take signals and route them immediately through via to an inner layer of the printed circuit board. When you're at an inner layer, it's, it's hard to get at them. Um, you know, traces that are on the, uh, outer layers of the PCB, you can actually scrape off the solder mask and get at those. Um, but you're going to have less options. And so it just makes it harder for an attacker. So, BGA devices, uh, some security conscious PCB design and PCB routing, uh, can make a big difference. And, and the last which I, I think, you know, everybody should be doing when they lay out these devices to help improve security is just be very terse about what you're doing. If you have a serial console, certainly don't accept input on it unless you really want that for some reason. Um, and boot that thing fast. Uh, you can get like embedded Linux systems to boot, you know, well under a second. And it just makes it hard for a hand timed attack like this. You could do something more elegant and crafty of course. Um, but you know, that's, that's, that's more working attacker. Maybe they'll move elsewhere. So that's the last bit of this. Um, Max is here. Somewhere. Max is right there. We're going to be, uh, outside to take some questions. And if you ask some good questions, we want to have something for you. But, uh, I'm Brad Dixon. Uh, thanks so much for your attention. Appreciate it.