 Thank you guys for coming. My name is Joe Grand. I am a Portland-based electrical engineer, hardware hacker, product designer. I love electronics. My name is Joe Fitzpatrick. I am a Portland-based electrical engineer, hardware hacker. I love electronics. Yeah, so sometimes it's hard to tell us apart. We ran into each other a little while back and realized that we had a little bit in common. Yeah. Like we both have brown hair. That's right. He has a much better looking beard though than I do. So we've been designing electronics for a long time and we've run into all sorts of problems and failures with usually unintentionally but sometimes intentionally of breaking electronics. So we thought it would be good as like a DC 101 talk for people who want to get involved in electronics and get involved in hardware, sort of to learn from our mistakes and we can sort of share our pain with you guys hopefully so you don't have to go through the same thing. Who has ever bricked something before? Yeah. You're the right audience. So you guys could all come up here too. You should be teaching us. We'll take turns. Who wants to go first? Share your stories. So yeah, we sort of broke this down into 101 different sections. 101. Oh, sorry 101. 101, whatever. Binary. Yeah. That would be a long talk. We have fool dealers, only five or whatever. What is this? Yeah. This is 101. So we're sort of cheating. We're not starting at zero. We cheated. So yeah, so we have a few different, you know, kind of broke it down into different sections. So first we got to define what a brick is. So the authoritative source of all this is what? Of course, Urban Dictionary. Brooke, a pound or kilogram of any drug item requires clarification from speaker as to the mountain tended. Yeah. So that's what a brick is. So we're going to talk about 100. No. Well, we got to do the, I get my dope straight off a brick. Oh, there you go. Yeah. Give the example of it. I got a lot of bricks at home to get dope off of it. So, Brooke, to brick something. All right. So yeah, this is the action of rendering any small, medium sized electronic device useless. This can happen while changing firmware, soldering or other practices involving hardware software. This was actually in Urban Dictionary. Yeah. Someone added that in. Yeah. It was Jules Verne. Awesome. So it's an actual real verb now. So now it's like beyond Def Con. It's a real thing. I bricked my mobile phone when I tried to install Linux on it. That guy's a noob. Okay. So we have two different types of bricks. There's the soft brick. So the soft bricks kind of easy, you know, like you did something. It doesn't work. It doesn't turn on. But it does have signs of life. Right. This is when you get like this little message on those, those, those Jesus phones that like, oh, you got to plug it into iTunes phone home or on an Android, you get the little Android with the little, little belly virus thing sticking out. Operate on me. So yeah, but this, you know, soft brick, that's a software problem. Let's talk about hard bricks. This is what's real. Who, who, who loves the hard bricking? Yeah. Oh yeah. Yeah. Bricks are awesome. Yeah. So yeah, these are the things that actually require some sort of hardware hacking, um, modification or fix or something. Usually if you can get it unbricked again. So there is this sort of variation. We are focused pretty much exclusively on hard bricks. Yeah. And the great thing about hard bricks is it's, it's a hard brick. Like you're not going to brick it harder, right? Right. You're done. Yeah. So. You can only undo it. So yeah, we'll go through a bunch of different sections. Starting with probably the most common and the most obvious is, is bricking through messing up firmware. Yeah. So we have lots of, you know, crazy examples. I should mention, so these are examples of our actual bricks. And we have, you know, a lot, a lot of them here that we're sharing, um, the same things could happen on your stuff, right? So you, you might accidentally go through these same problems, just not with the same products. Starting with the DEF CON 18 badge. How many of you guys have a DEF CON 18 badge? Like five people. Are you serious? All you, all you guys are newer since then? No, no, no, no. Apologies. Cool. That's awesome. Okay. So no one has it, which is better because you probably didn't run into this issue, but welcome to DEF CON. Um, so the DEF CON 18 badge was one that I designed. It was the last one I designed. And, um, this particular one had an MC, uh, 56 F 8006. It was a free scale based digital signal controller, microcontroller that had lots of good hardware functionality for sort of DSP types of functions, but in a microcontroller. And, um, I had a bootloader in there that you could load through USB, um, new code in to try to make it easier for people to hack on their badges without needing JTAG hardware and debug hardware and all that stuff. So just through USB. But if you mess up during the linking process, like with your compiler, and if you point your code in the wrong spot and if you don't include the bootloader back in and you reprogram it and you screw something up, then the badge isn't going to work. So it's sort of a lesson in, in proper compiler configuration. Uh, and the only failure. So at that point it would not work at all. And the only way to fix it is to use the JTAG interface, the standard development tools to reload everything and JTAG being an industry standard debug interface that is useful, but more of a pain in the ass because now you need the tools to connect up to it. You got a solder on a connector and do all of that. So that's sort of a standard thing is you mess up a bootloader, save it with JTAG. And that's something we see a lot with people hacking routers and phones and things like that is they can always recover it usually through JTAG. Um, so not, not necessarily. And like most devices start their life, you know, as a non-functional block of something that it's programmed by a manufacturer. So, you know, there's got to be a way to get something that doesn't have any code on it to get code on it. And JTAG is usually that way. And usually you would put JTAG to load the bootloader and then the bootloader to load your code. But if you brick the bootloader, then you got to start all over again. Or you just buy a new one. Yeah. Or get someone else to buy a new one. So, wiping critical sections. So this is a Chromebook firmware. Who's ever used a Chromebook? They're kind of cool. So they do some fancy bio stuff. They're all the ones that are based on Linux platforms. They have what's called a manageability engine. And you see the difference between these two histograms right here. So there's this big block that's up here. And, you know, it's got a lot of stuff right there. So this is a histogram. It's a tool called BinWalk. It analyzes binary files, firmware images. And it's missing in this other one. So what happened is if you go and you take your Chromebook and you're like, oh, I'm going to tamper with this thing. So let me get a backup first. You take a backup from software. You get this. You get this big bunch of zeros right here, right? If you go then and you get the heart, oh, I bricked it. It doesn't work. I need to open it up. I need to use my bus pirate or something else. Or I need to plug pins on there and reflash the chip. I'll just flash this on there. You'll totally brick it. Because this block of code is a block of code used for the manageability engine on the Intel PCs. If it's not there, the thing doesn't boot. So it kind of sucks. So you get a software dump. It's different from your hardware dump. And you flash it back via hardware. And you bricked it. So the lesson being, if you can get hardware access to get code out, that's probably a safer thing to do. Yeah. If you happen to have a backup of it before you mess up. So another one on the Chromebooks. I do a lot of poking at these Chromebooks because I have a lot of them. So you can mount the read-only file system as read-write. Okay. That makes sense. Then you can make changes and you can reboot. Okay. Let you do that. That's cool. Now the kernel verifies the root of S before it mounts it and it doesn't match. Okay. So it checks the signature. It checks all this crypto stuff that the math people can figure out. But all that matters is that mismatch causes the error. You made a simple change that shouldn't have bothered anything, but you tampered with the whole chain of trust. And now you have a brick. Chrome OS is missing or damaged. Please insert a recovery USB stick or SD card. Note the blue USB port will not work for recovery. So backup. Backup. Backup before you tinker. Hardware backups. The only way. So then... Oh, that's a good one. Oh, yeah. Who's done this? Who's done this? That thing. Okay. Yeah. So DD, like copy blocks of stuff. And you know, you got to copy to this USB flash drive and you got to copy a hundred of them. And you're like, okay, so do a DD interface, install that ISO, OF equals DEV SDA. So SDA is generally the first serial disk in your system. So that's probably the drive you're booting off of. But you have to do it as root because otherwise you can't access the low level block devices. So you just erased everything on your system. Who's done that before? Back everything up. So many people are admitting it too. Acceptance is the first step. Are we taking pictures of all of them? So here's the other thing. Now I've got a laptop and it's got an ex, EM, what's it called? NVMe, non-violetable memory express. So it, storage drops in and connects to a piece of express. So in Linux, it's DEV NVMe, what? NVMe, NVMe. And that's great, but except I plug a USB drive in and now I need to put something on it. It shows up as DEV SDA. So I do this all the time now. And if I ever get a new laptop, I'm going to wipe it on a daily basis. That's just evil. So make backups. So, yeah, so unbreaking these types of firmware issues. If you have a backup that's good, you know, if you're going to hack on something, try to get a good known image before you start messing with stuff. Yeah, yeah. I don't know, directly rewrite the storage media. He wrote that one. I did. Yeah. Oh, yeah. If you really want a backup, don't trust your operating system. Don't trust your CPU. Just wait to the device and read it, right? If you have a chip of some sort, read it with a programmer. Don't read it with software. Yep. And other hardware things too. It's funny because you can unbrick your, you know, firmware using hardware. Swap out the flash device, memory device, whatever you've bricked, if you have another backup, or maybe you take one from a product that is good, that has the same content, take one off one board, put on another, or copy one the raw dump, put it in another one, and then use the debug interface. If it exists, it usually does. Either it's JTAG or a vendor-specific interface of some sort that will let you reload new code back in, assuming you took a good backup in the first place. So yeah, if you've got those backup, it's great. You know, if you don't, you might have to buy a new device, and that's sometimes expensive. So swapping out the physical flash device, whatever the device is that you actually broke, sometimes it's a lot cheaper than replacing the whole system, right? So if you, if you've worked a flash chip, you've worked something else like that, just replace that chip. You're good. And then just return the one you just bought. Yeah. I don't condone that. People are shaking their heads. No. Cash only. Just stop working. Never mind. Actually, we'll tell a story about that with this... I've never done that. I've never done that. No, me neither. Never. All right. So the next... Actually, can anybody in the audience identify what's wrong with this PCB? Oh, yeah. It's a little quiz. It's a tough one. If you're new to hardware, it might be difficult. Yeah. Okay, so now we're getting into some actual physical destruction. Breaking PCBs. A main concern when people get into hardware hacking is, am I going to damage the board? Am I going to damage the chip? I don't know how to solder. Normally, it takes a lot. Circuit boards are pretty robust to heat. Chips are designed to withstand a decent amount of heat when they go through a reflow oven, when they're being soldered. And typically the failure modes are thermal cycling if you're removing a part, putting it back on the board, removing it, putting it back on the board. But sometimes you get a little overzealous and other problems happen, like we'll talk about here. So yeah, who's ever tinkered with like a wireless router or something like that? Okay, they're fun. They're fun to program. And you open them up and you can find all sorts of neat headers on them. So if you're poking around, can I get over there? I can't get over there. You have to describe it with words. I have to describe it with... I don't know words. So what we've got is a bunch of pins on here. And this is just a bare header that's sitting there on the motherboard. And we need to get that solder out of there to get a header in there so we can use a JTAG adapter. So it's unpopulated. And, you know, sometimes you just want to get something done and you're sitting there and it's like, okay, you crank the iron all the way up. Come on, come on. Melt solder, melt solder. You really don't have patience and you need patience because too much heat, sloppy work. I basically completely peeled out the through-hole lining of each of these. I usually do a better job soldering but that's kind of like, that happens sometimes. This is a great job soldering when you're talking about it. I've seen worse jobs. I've done worse jobs this week. This week? And the reason this happened is the way that circuit boards are manufactured is you have a conductive layer and then it's basically glued down to a non-conductive layer. And that glue will get softened with a lot of heat and get pulled right off the board. Yeah, so you can pull the copper off of the board. You can pull the layers of the board apart. And yeah, it gets really messy. So yeah, patience really is the answer. And also, you know, this is a really common problem because most devices that have through-hole parts in them are assembled with what's called a wave soldering station or a wave soldering machine, where like a big wave of solder basically solders all of the through-hole parts that are underneath, you know, coming through the board. And that's why when you get products, consumer products, all of the holes are already filled with solder because of that wave flow, the reflow and the wave soldering and the surface tension pulls up the solder into the holes. So the first thing you usually have to do when you're hacking stuff is like suck that stuff out. So another one, I play with these things a lot. Pogo plug is this like $10 network attached storage device. I think I dropped pictures of it yesterday where I plugged PCI cards into it. But that's beside the point. I was really trying to figure out where the JTAG pins were on this guy. I knew the CPU. I knew where they were. I knew where the pins were on the CPU, but I thought there had to be test points. And so I just decided, okay, you know what, I'm just going to take the chip off. I'll figure it out. I'll look at the traces underneath. In the process of doing that, like you can see the exploded view. In the bottom right, it's kind of tiny, but there's a couple traces that in the process of taking the chip off. You know, you sit there with a hot iron, hot iron, hot air, and you blow it on the chip and the chip gets warmer and warmer and warmer, very patiently. And what's really annoying is the last thing to melt is the solder, right? Because the solder conducts the heat away. So you just sit there. You got to be patient. You got to be patient. And when you want to get something to work, you are not patient. So then I'm like, okay, well, it's almost off. So I stick something in there and try to lever it up. And in the process of doing that, the thing I stuck under there, I scratched a whole bunch of traces off. So they were disconnected. And, you know, pull it off. And after all that effort, I find out that there wasn't even test points for JTAG anywhere. I thought they were routed underneath the chip. They weren't. So, you know, in this case, though, I actually kind of said screw it. I don't care if I break this one. I mean, it's 10 bucks. And I was fine with losing the 10 bucks, but I learned something from it, I guess. Well, and this is a good example, too, of if you are hacking on stuff, if you can get multiple units to have a sacrificial lamb to do something like that. If you do need to look what's underneath the part, it's like, all right, I just want to care. Now I'll get the information I need to do an attack on another one. Like, that's okay. But if you only have one, then you're screwed. Yeah, and then you return it and you're like, hey, it's bricked. I don't know what happened. Yeah. I just opened it up. All right. So, shorting traces. This is totally, you know, something that happens a lot. And this is a Hearst scramble pad. So this is an access control device that is used at like the White House and other federal buildings and airports and stuff. And it's designed in the 80s. I think they've updated it since then. But you've probably seen them before. You push a button on the pad and the key, the number ordering changes every time. It goes like doodoo doodoo and like changes every time. So someone can't, you know, look at the wear marks or your fingerprints on the thing and try to narrow down the key space. It also has really narrow viewing angles. So I put one of these in my office, which is funny because you could just kick the door down if you wanted to. Don't get any ideas, by the way. So I got one of these things on eBay. I was messing around with it. I had some batteries. I was kind of testing out the system, taking some measurements on the linear regulator. Just a standard, you know, run-of-the-mill LM7805, a five-volt linear regulator that was taking in. I can't remember what it was, 12 volts in. So I wanted to measure the input and make sure I wasn't going to fry the rest of the circuitry because this particular board was from 1992 and I didn't have a backup of the code on the microcontroller in case I broke something. I don't know what happened, but one of my probes slipped and shorted the input, which are these very high-capacity batteries with very high current output, directly to ground, which causes a short circuit and a spark and damage to the board. You can sort of see in the exploded view like all of the solder mask and part of the board is actually missing. And I was really scared that I'd just completely ruined this device. Luckily, the regulator is pretty robust. I didn't do any damage. Still had a stable five-volt output, but I sort of sat in the corner and whimpered for a while. I had a bruised ego, but I immediately sent a picture to Joe and I was like, we can use this in our presentation. Yeah, it was great. I mentioned this idea to Joe and he's like, I can break this, I can break that, I can break that and go and broke everything he had, which is great because that's what I've been doing for the past two weeks before that. So here's another one, burning traces. This was a fun example. I was reverse engineering a vacuum sealing food, like a food thing to vacuum seal food that I was working on a project designing something very similar to that. So I was reverse engineering this board to figure out how it was designed and made a really beginner error. Using my oscilloscope I wanted to visualize some of the traces on the board, but what I didn't realize is that I was creating a ground loop and I was accidentally measuring a signal, an AC signal that I shouldn't have been measuring the way that I had things set up and probably should have maybe been using a multimeter that was isolated. And there was one trace on the board that is designed to be a fuse. So that's circled there, you can sort of see the square and then it kind of goes out in a right angle of like a really thin trace before it gets to the rest. That was a fuse designed into the circuit board, which saved not only me but it saved my oscilloscope from actually getting destroyed. And normally, this is kind of what happened. Yeah, so that was Dave Jones. If you haven't seen his EEV blog videos you should check him out. He's a very interesting engineer with lots of good technical detail and lots of opinions. Actually, you want to go back to the picture of that PCB? There's a moral to be gained from this, right? This is a food saver V850, okay? Joe's hacking on a food saver V850. This is not a smart food saver. This is not an internet enabled food saver. This is like the vacuum thing that you put like steaks in so you can freeze them, okay? For vegetables. If you're bored and you need something to hack, don't just look at computer stuff. Everything's hackable. That's right. Open your mind. That's right, yeah. And this is a good point. This was all digital logic, no microcontroller or anything. So it was a good experience actually in learning how to reverse engineer analog electronics. But then I eventually just gave up and designed a digital system to do the same thing. But you know what's great is there's no firmware to brick, right? That's right. No firmware. And so the key thing here is learn how to use your oscilloscope properly, which after this, I went and studied up on ground loops and hooking up, you know, AC things to oscilloscopes and needing an isolation transformer. And bam. Yeah, so. But if I had broken my scope, that would have been really bad. You just return it, right? I don't know if that one would work, yeah. So ways to fix, unbrick your PC boards. Be patient in the first place and don't just go straight at it with a, you know, don't turn the heat up to 11 on your soldering iron. You know, blue wires will actually work. Like, you know, the little wire wrap wires that you see on board sometimes to fix prototypes. If you get some 30 gauge wire wrap wire or some magnet wire or some angel wire, I think they call it, to fix broken traces and to fix, you know, things on the circuit board that you're not going to be able to fix a blown area, but you can just patch it with wire is a good way. Yeah. Oh, go ahead. Oh, yeah. And PCBs are actually kind of really resilient. They're made of metal and metal and they work, right? So if you, even if you have that big board from the front that's like torn, you know, you line those things up, you put some glue down there and you like solder it up well enough, that board would probably still work. Assuming it's not a multi-layer board with stuff inside. Oh, yeah, you know. But for the ground planes you'd be fine. Level of detail. Yeah. So the question is, do we ever use ChipQuick? ChipQuick is a special alloy used to help you remove surface mount parts from boards that basically reduces the overall melting point of the solder. So if you have multiple pins coming off the part, you use ChipQuick and it melts everything at once and you can slide it off the board. So the answer is yes. And the good advantage with ChipQuick is that it doesn't heat your part too much. The disadvantage is that the stuff stays molten for so long that it will dribble and get stuck on other parts. And if that happens, you're going to have solder the solder alloy everywhere. I'm completely capable of breaking hardware without ChipQuick. So you have to be really careful to use it. But yes, so sometimes you use that or you just use hot air rework, but it sort of depends. All right, anybody recognize this beast up here? Does anybody remember why it's blinking red? No? Bad connection. Yeah, you don't remember blowing in your cartridge and trying to get better connection. So yeah, breaking connectors. This is, you know, messing up more mechanical physical things of systems. I mentioned before something about the Chromebook. So see if 720s, I kind of like them because I got a lot of them. And the reason I got a lot of them is because I can get them cheap, right? You go look around, you can get them for like 100 bucks each. But I'm really cheap, so I'm always looking for a little bit cheaper. So I found a lot of 10 broken ones on eBay. And I'm like, hey, what's the worst that can happen, right? 40 bucks each. That sounds like a good deal. So I opened them all up. I got 10 of them. But then I went through the other nine, and actually the 10th one as well. They all had like loose cables in the display. So if you look in the back panel of a display, it has a little, sorry, the motherboard has a cable. It goes up through the hinge, up to the back panel display. And this one, this one model just has a thing. You keep opening and closing, and opening and closing it. It just got tugs a little bit. And so the little edge of that connector slips out the tiniest bit, a fraction of a millimeter. And that's enough for the display to not work. So all I did is kind of pop open the displays, tighten these connectors, and I suddenly went from 400 bucks worth of Chromebooks to 900 bucks worth of Chromebooks. So that was kind of fun. And there was something else I was going to say about this. This was happening with normal use too, right? Yeah, so it was normal use. You keep opening and closing, and you eventually kind of wore it out a little bit too much. Sort of bad design. Oh yeah, this one. Yeah. So this is a little mini PC that I was using. I actually ended up using this to build an AR sandbox. Have you ever seen those? Let's be a picture later. And the problem is, it was very poorly designed. It was a little micro USB connector that was used for power input. And it wasn't just a regular like USB cable. This is like Intel X86, Bay Trail, 4 core, thing you do. And what would happen is, it had a 3 amp power supply. 5 volts, 3 amps. It's quite a bit, but it used to use tiny traces inside. It was like burning out those traces and heating up and melting this little connector. So the traces really weren't well sized for the amount of current. The third more ignition wasn't well controlled. If you start using the CPU too much, it wouldn't be able to supply enough power over the connector, and it would just disconnect. It would just shut down. So I kind of got sick and tired of that. I tried replacing the cable. I thought maybe bad micro USB cables, because that happens a lot to me. Probably because I used cheap cables. I opened up the case. I soldered the power lines directly to a ground point and a power point, very messily too. Not as bad as that other one, that through hole one. But I soldered it up, it worked, and yeah, it still works. Sometimes it doesn't have to be beautiful to work. That's what my wife tells me all the time. Take that as you want. I didn't mean it that way. You guys are disgusting. You should hear what he says to her. So, another one. Again, I play with all these systems. I've got these tablets. These are the cheapest tablets you can possibly buy that run Windows. Actually, you can buy them even cheaper because everybody goes to the store and they buy them, because they're cheap, and they can take them home and they can't run anything on them, so they return them and spend more money. So I go in and buy all the open box ones, because they're even cheaper. Anyway, this TW700 tablet, it's a connector for power charging input. And I use these a lot. I charge them all, and I just charge them all and charge them all. But every time you plug in that USB cable, it wobbles this connector a little bit. The housing, the case of the connector, the case of the tablet does not have a flush case around this connector. There's a little bit wiggle room, and that wiggle room keeps wiggling every time you plug it in and take it out. Solder is never, ever, ever designed to hold any load or any strain or any physical strain. It's solely designed as an electrical connectivity. So what happened is after a while, those four little tiny, five little tiny connectors on the bottom of that USB connector all got broken. It was a pain in the butt, I had to replace a bunch of them, but hey, it works now. This is a good example of if you're designing electronics, try to use a connector that has through-hole ports on the side, or whatever they are, mechanical stability, and that's something that companies don't like to do because it's an additional step to solder them in, but it's going to prevent that from happening. So here's another thing that I've done many, many times, and this is just the most recent example. This is a low-cost consumer device. This is another cheap Chromebook, because I got to think for Chromebooks, I guess. It has USB audio running over a flexible printed circuit to the other side of the laptop, so they make another motherboard small, and then they put it in cable that floats through the case to their side. If you open the case without knowing that that cable is there, you're very likely to pull it. If you're lucky, then it disconnects, it just pulls it out of that black socket. If you're not lucky, then it pulls at an angle, and it tears a bunch of the traces. It's just a piece of plastic and metal, but the thing is that for some reason these are really expensive and relatively expensive. We're talking about Joe Fitz expensive, which means it might cost like 10 bucks for this cable, but sometimes someone else plays with much, much, much, much more expensive toys. Notice how there's no detail on this slide. Let's just say that it's a very expensive consumer device, and this mistake was very costly, and if you look on that circle, this is a flat flex cable, so it's a flexible circuit board where normally you might have one of those layers on a flexible board. This is a multi-layer flex board connecting very expensive pieces of equipment together that I accidentally tore and tried to fix it. I was like, oh, some wires, right? If you can solder the top and bottom, but there were multi-layer and it was horrendously embarrassing and never to be spoken of again now that it's on film. You just returned it, right? Actually, we just returned it. No lie. The moral of the story I think behind that is do not hack on what you cannot afford to lose. Or return. Or that you can't return. Save your receipts. Save your receipts. Okay, so solutions to unbreak your connectors. Mechanical reinforcement is actually a really common one. Just use some tape. Use some epoxy. On those Chromebooks that had that wiggly port, if I had just gone in and run a drop of epoxy on each one when I got them new, which is what I do now, they would have never broken in the first place. Vendors are just too cheap to do that normally. Yeah, too cheap. Epoxy is expensive. It costs sense. Fractions of sense. So electrical reinforcement, like Joe did, patching over weak connectors and putting in better connections there. Learning how to locate replacements if you do mess something up, see if you can source a part. Looking at common distributors for various places. Reading mechanical drawings, so you know which part to use. Did you keys your friend? You can get parts shipped the same day, delivered next day if you need to, to continue on with your project. Yeah, it takes a while to get the skill to actually find anything on Digi-Key, but usually you just keep searching, you find something close, find stuff in that category and then they ship it really quickly, which is what I like. You do a parametric search and then you just narrow down until there's like a few items on one page, you just choose one of those. Or you buy all of them and return the rest. Yeah. Okay, so now we're getting into breaking chips. So actually integrated circuits on the physical circuit boards themselves. Yeah, sure. Okay, so absolute maximums, I think we might actually have an example of this, but integrated circuits are sensitive to their voltage levels. Whether they're on signal pins or power pins and data sheets of these parts will usually tell you the maximum allowable values and things like that. And usually if you go above them, the manufacturer is not going to let you return it and then let out the magic smoke and you're done. And it's kind of an RTFM case, because if you look over here it says pretty clearly operating range, use this voltage range. If you're not going to read the data sheet, then like, oh well, whatever. Who reads data sheets anyway? Until you brick something. You go, that's why I should have read that thing. Yolo wiring. So speaking of Yolo wiring, I found another tablet because I tend to acquire a lot of these cheap tablets. This is a cheap Chinese tablet and it's got a 1.8 volt spy flash chip. And this is like, I think one of the first ones that I poked at that was actually 1.8 volts, so I didn't really expect to worry about it. So I just, you know, whatever opened it up, popped it open, grabbed whatever tool I was using, probably an FTDI chip, wired it up and tried to dump the spy flash contents. And then the system didn't boot. I also didn't get any spy flash content. So I was trying to figure all that out and it turns out it's actually, I needed to level shift, right? All these tools we've got there tend to be 5 volt and 3.3 volt tools. Some of them are 5 volt current, some of them can work at lower levels, but if you do something at a higher voltage than the device is made to withstand, you're going to do something bad. You might not totally brick it, you know, in this case what happened is I actually just erased the flash contents. So the flash chip still worked, the CPU still worked, but the process of trying to read it at 3.3 volts made that 1.8 volt flash chip die. And a lot of chips do have internal protection diodes on pins. So to protect you from accidentally doing that, but you're not supposed to rely on those. Like those are almost like, it's like getting catastrophic health insurance or something, right? Like you don't want to rely on that in case there's an accident. So yeah, another thing I was playing with, oh, is that good? Onward. So pulling up too much current. So this is what someone might call an FTDI cable. It's not an FTDI cable, it's just a USB to serial cable. It's got a chip on it that says that it's a synthetic PL2303. You plug it in on one end to a USB port. It's got TXRX power and ground on the other end. I bought a bag of like 100 of them because they're cheap that way. They're like a dollar each. And this one I was using, I forget what I was doing with it, but it kept like, stopped working. And I would go and what would happen is I would look in the de-message log of the system, the USB side system, and it kept saying like, oh, device is connected, device is connected. So I'd have to go and unplug it and plug it in. And I left it, walked away, put it in there to work on it again. And it's not working. So I go to pull it out and my finger sticks into the plastic and moshes it around. And I'm like, huh, I don't think that's how it's supposed to work. So I pulled it out and ran water over my finger and opened it up and the board is definitely a little bit singed right there. I don't know whether this was just shoddy manufacturing and there was like a ball there, or whether I was hooking it up wrong and I was actually drawing too hot and hot enough to melt the plastic and blacken the board. So did we wipe your fingerprint from that image before we submitted these to DEF CON? I don't know. It doesn't actually look like my fingerprint. Joe's thumb is there. Use biometrics. Yeah, good thing the hotel doesn't use biometrics, right? Okay, so another example of pulling too much current. This is at an actual tiny little chip level. This was for a product I was working on for a consumer device. So I had some pre-production prototypes, sent those to a manufacturer to start getting ready to ramp up for full production and they had made some changes to some parts which is not uncommon. They might say, oh, we have a supplier that can provide a similar second source part. We're going to put that in place of this chip that you, the engineer, slaved over to specify. So they sometimes just put in what they think is the right replacement and not tell you or tell you later. So these came back and we noticed that once in a while we would have failures of this particular low dropout linear regulator. So taking power in, bringing it down to a lower voltage and we just couldn't figure out what it was and this was a company I wasn't very familiar with. I think they were Chinese based I'm not sure and the only thing we could think of is let's decap the chip. Let's take the plastic covering off of the chip itself and look at the die. Look at the actual integrated circuit to see if we could locate any failure. So we sent a bunch of chips to Chris Tarnofsky who is I would say the best chip hacker in the world who's given talks at DEF CON and Black Hat and all sorts of crazy satellite TV hacking and smart card hacking and had him decap the chips and look and he went and very quickly realized that there was damage on the physical die. So like the previous one we just showed had damage on the board. An integrated circuit is really like a circuit board at a microscopic level. So there was damage on that die because the way the system was designed is basically there was current flow to that giant tab but that tab wasn't actually designed to handle current flow. It was just mostly designed for thermal heat dissipation so maybe that was a designer error or they swapped that in or it was just a misdesigned sort of under design of the part relying on the engineer to read the data sheet before they actually designed the board. Yeah but that sounds like work. A lot of work, yeah. YOLO silicon. So yeah we gotta figure out ways to unbrick these systems and it says unbricking your ICs but really we gotta talk about unbricking the whole system because most of the time if you've done like electrical damage to the chip you're just going to have to replace that chip. So replace it. But figure out the problem first because if you go and you fix your connection issues or board issues or you fix your you don't fix all those other issues first and you replace the chip you're going to end up with two dead chips and then you're like wait wait what happened and you replace it and you get three dead chips. It's like a chip killer. What do they say? You treat the cause not the symptom. Yeah yeah. So again Digikey is your friend. We're not sponsored by them. We just like them. Yeah. There's other distributors too if you like Mauser. Mauser is still your friend. So personally I like Digikey because they have USPS shipping that's generally very quick and very inexpensive. Yeah. So shipping is not as obscene compared to your ten cents worth of parts. So like two dollars of shipping for a dollar worth of resistors instead of like 18 dollar shipping for a dollar worth of resistors. It makes a big difference. Alright so here we are at our 101st section 101 first first section. 101st. Bricking scenarios that we couldn't think that fit anywhere else but sort of like WTF. What is going on? We have what? Yeah. Yeah. So anti-tamper mechanism. So this is a photo inside of an AT&T microcell. There are a couple of people who talked about these a few years ago and what happens is you open this guy up and inside there's this little gray thing that holds a bunch of jumpers. And those jumpers may either connect or not connect the thing to the case. When you pull the case open the case is designed so that it pulls those jumpers out and you don't know what arrangement they go when you put them back in. If you do this and you don't notice what you did and you power it up it sets a tamper flag and it phones home and it tells AT&T that you've been doing bad things and that you probably shouldn't have to return from this customer. So some discussion about this if you search online people have tried to open these up and pull out the jumpers and try to put it all back together. They're like oh no I saw something fly across the room and then they call up AT&T and they're like oh my microcells not working they're like well it looks like it's been tampered with and they're like oh it must have fallen on the floor or something. And they're like okay. Yeah sure. Yeah so anti-tamper mechanisms are things that are physical security to protect you from tampering with an electronic device. Here's another one. This is from a Verifone PIN pad 1000 SE. This is a point of sale terminal thing you'd enter in your PIN. They have a lot of mechanisms on this particular device and I had purchased a whole bunch of different PIN pads at a surplus store for five bucks a piece and this just happened to be one of them. That had multiple mechanisms. So you open up the device there's a button that gets depressed. But the coolest thing about this one is that there's an active circuit board there. It's like a multi-layer circuit board. Four layers the top and the bottom layers are copper planes and then the inner layers are like a mesh of wire like a maze. So if this thing is powered on and you try to like drill through it or remove the cover the system's going to know and give you some sort of tamper detected and not work and you'd have to re-key it and everything. So this was just sort of a fun one and another great reason to have sacrificial lambs if you can because if you tamper something like this on your first one and you only have one you're going to be in a lot of trouble. So another comment on the whole like having sacrificial lambs if you're ever like just looking for devices to hack on and like you decide to go like AliExpress which is a place to get like really cheap junk from China buy more than one because you buy one and you go and you take it apart and you're like hey I hacked this thing and it's really cool let me go buy ten more. You buy ten more they're going to be different. They're going to have the same color on the outside it's going to be the same picture with like photoshopped out and written over watermarks but you're going to get a different device inside so buy them all at once buy a bunch to begin and then just hack them. Yeah they just grab it from a different factory or just like iterative their design process is just too crazy. So some weird environmental conditions which are the worst things to debug if any of you guys have worked with RF systems before you know this sort of black magic around RF design and it's really really sort of a nightmare so when the environment conspires against you to mess up your circuitry like that's really it's a hard thing to deal with. This particular design is a RFID ReadWrite module. I've designed a series of RFID readers and writers for Parallax which is like a hobbyist electronics company. This was a fourth in a series that I thought would be a really easy fix to add on USB functionality to our standard ReadWrite serial version that would take in TTL level serial. So this one with USB I was like well I can just add a serial to USB device take the TTL level serial add a USB port on there and we'd be good. Not so true. About three years later after debugging this on and off for three years just I just realized this and we were able to put it in the slides is that I was receiving all this noise on the RFID receive line like I was seeing demodulated data when there was no tag over the reader. So it was like a mystery of what was being demodulated and it turns out that my reader was escalating noise from the environment because the antenna was too sensitive and I didn't know that it was one change in capacitor value to decrease the sensitivity and it really reared its head when you're powering directly from a USB device and I was sort of treating the USB power 5V power line as being clean but that's not necessarily true. So it was just generating lots of noise and I was having all these problems and thought I was a horrible engineer because I could change one capacitor and I thought I was an awesome engineer but I'm never touching RF stuff again. So this is another one and the picture doesn't do justice to what happened. This is an AR sandbox which is where you have a sandbox that's full of sand and a connect that looks down at it and looks at what you've got and when you move the sand away and make a pile it turns it into a mountain and it colors it green when you dig a hole it makes it water and it turns blue. This is a tour camp which is a great awesome outdoor hacking camp event but it is in the Pacific Northwest so it is a bit moist and the second morning I turned on my connect and my thing and it just was not working properly and this is a partial failure. What actually I had at tour camp is I had the very corners of the frame which show up as mountains they'd be red and the middle was all black like what the hell is going on and this is an environment conspiring to work against you. I had this underneath a black sheet which had worked through all my testing because the connect uses infrared to find out what's going on. I thought I had left it out and the moisture had melted or fried something inside and it was no longer working I could replace the connect. Turns out the sheet that I used had a mesh pattern on it and the infrared light from the sun would cast down and make an infrared pattern that the connect was recognizing and not throwing any color on properly so I thought I had bricked hardware I thought everything was broken turns out it was just a sunny day. Damn those sunny days were the worst. Yeah so you know test your systems in the right environments so optical glitching this is something that we just sort of threw in here because when Joe was over we were working on the slides one day and I needed to take a picture of the scramble pad to show in the earlier slides I had my camera with a big flash on it and I took a picture of the scramble pad I went locked the door we went back in for something later and my access code wasn't recognized I'm like huh that's weird and then I remembered about optical glitching that you could actually cause failures inside of chips due to photons hitting things the wrong way and since my that scramble pad had a EEPROM to store the program code which is UV erasable I actually caused the system to fail and erase all of the access control pin numbers in there so I sort of bricked it I was afraid that I actually like changed some of the code but really like you know chips do not like light but you can sometimes bend that to your will if you're intentionally trying to use optical glitching to like you know skip over something and do some glitching on the die itself that's sort of a crazy advanced attack but it sort of surprised me and it was like wow light can damage things even making a slide deck about bricking can break things yeah that's right subtext yes so the Bosch BMP 012345 0805 pressure sensor is also sensitive to that there's also the example of the raspberry pi 0 which maybe that was or 2 or raspberry pi 2 that uh oh that was the power regulator so yeah there's lots of things you assume it has a package over it right a plastic package but light and photons can still get through and mess with you and that's like it's pretty interesting so these these WTF scenarios like it's kind of tough like what the heck did you do what did you break we didn't change anything at work yesterday at work today and it just stops working um so what can you do you can get another piece of hardware be more careful um you get another piece of hardware and do like a manual diff compare every single component test step by step swap them out one by one or uh the best one I like uh you just grab a bite to eat take a nap maybe it'll work tomorrow maybe someone else will fix it maybe it'll work tonight yeah yeah these types of things like are are actually the worst that make you hate and then you solve them and it you know everything's okay again so as a little recap like we got the best ways to break and the best ways to avoid it so breaking your firmware right just wipe your flash it's wiped um yeah cut traces you know yep smash connectors smash connectors uh applying the wrong voltage work on anything in the last minute and that's when these WTF scenarios uh really win so avoiding it back up your firmware um yeah so you know have a good workspace don't rush things take your time have protective measures you don't damage components the p word patience uh double check your pinouts and voltages read the read the manual read the data sheet um have a little setup yep so and and uh unbrick who who cares about that that's no fun um restore your back up because you've got one right yep yep um enhance your soldering skills so you know you don't make mistakes with disgusting soldering right did you key is your friend order parts and did you key is you're still a friend no matter what it is and like you said don't hack what you can't afford to lose I never listen to that one yeah whatever um so yeah so benefits having a sacrificial brick maybe you brick one but then you learn from it anyway because hacking is all about learning right and maybe you learn something like okay now I know how to defeat that next time now I know how to not make that mistake next time um and share your mistakes right like it's sort of embarrassing to stand up here and say like I fucked that up I fucked that up but actually it's kind of fun yeah I guess it is but there's you know there's lots of failures and like sharing those we learn from them blog posts are great I see lots of blog posts of people breaking things that I would have bricked if I hadn't read the blog post yeah so you know that again that's the way to learn everyone's going to make mistakes and don't be afraid so step one brick hardware step three profit yeah and uh yep so thank you for coming yeah apparently you can make a whole presentation about this and thanks for sitting through it