 So who are we? So my name is Thomas Roth. This is Josh Datko. And we have a third colleague who is Dimitri Ndospasov, who unfortunately felt too cool to fly to San Diego. Before we start, we have some special thanks to Colin Oflin, just because he always had an open ear to us when we had any questions and wanted to clarify something. And also, he builds a pretty awesome device called the chip whisperer. So if you're interested in fault injection, check it out. So why are we here? So basically, we work with hardware security. And when we talk to people about the security design of the devices, they almost never consider chips as part of the threat model. So you talk to them, and they just assume that whatever the vendor says is true. Like, oh, this chip is secure. Don't worry about it. And also, a lot of people think that glitching and fault injection is some dark magic. But turns out, it's really easy. And we want to make this more accessible and make people try fault injection because it turns out you can do it very cheaply, very easily. And yeah. So before we start, what is fault injection? So fault injection is a term that professionals use. We call it glitching. And so basically, the theory behind glitching and fault injection is we want to introduce small mistakes into chips. So basically, for example, we cut the power for a very, very short amount of time. We think like 10 nanoseconds or so. We change the period of the clock signal. Or even we point a taser onto the chip and see what happens if there's a strong EMI shock. And the most common way to do glitching is called voltage glitching. And so basically, you try to cut the power at a very precise moment in time to cause undefined behavior. And so the theory behind this is that if you look at a chip data sheet, it has a safe operating area. So basically, up to a certain voltage, it will run at a certain megahertz and so on. And what's very interesting is what happens if you go outside of the safe operating area. So you drop the voltage further, or you try to even go into higher frequencies with the clock signal and so on. And the basic theory in glitching is that you wait for a trigger event, then you wait for a certain delay, you glitch for a certain duration, and then you check for a result. What does this mean? Well, on a real device, this might be, oh, I'm booting this device. I'm waiting until I'm in the bootloader. And then I'm waiting for the precise moment where the bootloader checks whether the firmware is valid. And then I drop the power for a very short amount of time and try to boot my modified firmware. And so basically, if you look at flash reads, flash reads take a lot of power. So basically, if a system boots up and it tries to read a certain variable or so from flash, and you interrupt the power, the result gets undefined. And if you boot a chip and you do a power measurement, you can actually see this pretty well. So in this region, this is just the yellow line is the reset going up. So we start the chip and you can see in the power consumption there's a lot of stuff going on. So we can see that it goes higher, which is where the actual execution of the chip starts. And then there's a small part where there's a lot of power consumption. And this already tells you that this is probably the part where the boot ROM tries to read something from flash because that requires a lot of power. And then if we introduce a very short voltage glitch, which, whoops, for example, on a scope looks something like this, where you drop it for like a couple of nanoseconds, you drop the power. What happens is that this flash read might, depending on if you are lucky or not, might get garbled and you get an undefined result. And so much for the theory, but how do you actually do this? And that's where Josh comes in who will tell you how to prepare a device for glitching. Okay, so this is our MLM. So we're gonna tell these three steps and then we want each of you to tell three people and then I think there's a profit step in there or we launch an ICO. I'm not sure, we haven't figured that out. But what we wanna do is basically prepare the device for glitching. So we have to do some things, hardware modifications. We have to make some test firmware. So it's really simple. So if you're a hardware hacker, this is really simple firmware, then we just gotta hook it up and glitch it. And those are the three steps. So preparing the device. Basically, when we start looking at the device, we have to do some analysis of it. So we have to read the data sheet and we have to figure out how the device gets power. So this could be, so a lot of the microcontrollers now are pretty complicated. They have lots of peripherals, as other people have been talking about today. So we have to understand where power is going into different parts of the chip. So main power could be 3.3 volts. The core voltage is gonna be probably a little lower. If it has a Wi-Fi peripheral, that could be at a different voltage. And so internally, what's going on is there may only be, well externally, there may only be one input to the chip. There only may be 3.3 volts. But realize that all of these peripherals are probably running at different voltages and maybe even different clock domains. So these have to get split out through regulators. So and again, depending on the peripheral less being used, there may be more than one regulator. So the 3.3 comes in and goes to the core regulator that could provide power to the CPU core. The RF may have a different one. And then GPIO may also be running directly at 3.3. So there may not be a regulator in there. And so this is again, this is the reconnaissance step. And some of this may not be inferrable from the data sheet. So some of this we just may have to make educated guesses on depending on the thing we want to glitch. But usually a good design will have some sort of bypass capacitors and this is to smooth the voltage out. And this is generally a good idea if you're making a product you want to have these bypass capacitors. And then these are external to the chip and you'll kind of see them placed around. But what's also nice is that this capacitor may actually provide a direct link into the core voltage which is the area you want to glitch. So this is perfect. This is like the side door entrance into the core, right into the CPU core. But we have that capacitor in the way and that's kind of buffering this energy. So pretty easy, we just use some hardware hacking remove the capacitor problem solved. And now we just have direct access to the CPU core and now we can provide that voltage. And so yeah, so now this is the area that we're looking at right now. We can see that VDD core core is right in the CPU core and that's the, if we provide that voltage that will be a nice area that we can provide a glitching and capacitors are gone so we've got direct access. So on this kind of like NRF example board we're trying to apply our MLM strategy. So prep one is prepare the patient basically. So we have to remove these capacitors. We identify the capacitors by looking at the data sheet doing some other reconnaissance and then we can just pluck those off. And so some practical issues if you do this, I mean there's a reason these capacitors are there but so by doing this it could be a little bit unstable. So this is just where you're in the analog domain and you have to just kind of provide a nice clean power supply. And so if you hook up the power supply there you will be basically providing that voltage that was there with the capacitor. So you have a nice clean voltage going in. You've just replaced the like 10 cent capacitor with like a dollar power supply but that's okay. It's not a problem. And yeah that's exactly where we're gonna glitch. And so on a real, I mean so that other slide just basically had those red things. Now this is what it looks like in practice and we were just soldering one wire into VD core and that's what it looks like practically. And again we take out these capacitors. So now we have to build a test firmware basically. So this is coming at it from the attitude is I wanna glitch this target. How do I go about it? Step one was this prep. Step two is building a test firmware. So we need to know that our glitch is working somehow. So we have to write some amount of code that will turn on the chip on, bring it up, exercise some function and then provide a way that we know that it worked. So kind of in the like the Arduino linguo, the way you know here's the kind of like pseudocode slash Arduino code that would do this. Basically we wanna bring these pins up. Right so the boot the device we just ignore all that stuff that happens before main and then magically we get the boot. Do like basically the system bring up, embedded system bring up. We are going to set some GPOs as high and this is gonna be a nice clean trigger to our glitching system because we'll have this nice rising edge. Here we're gonna read the flash address. So we have this flash address read and this is the value we're gonna corrupt and then basically we loop around that. So we just keep trying to loop around this and then if we get a different, so this code if we're running around it should always work, right? Should be always green. We'll see all that should never fail but by injecting the glitch this is going to, it's going to exit out of the loop and so this is how we're gonna know that this has succeeded and so if you looked at this on a scope, right? You know the how, you know what does it look like and so you know the chip is gonna boot, your pin is gonna be low because it's booting then when that rising edge comes up this is the trigger that you put in there so that now you have a nice clean trigger spot. You have to program and delay of the glitcher so this is one of the parameters that you're gonna tweak with the glitches. Okay, where in this search space do I wanna glitch? So you just delay this a little bit. Insert the glitch and then, you know then whether that glitch was effective this kind of depends on that target in that example that we showed the, right? It would see if that read failed and then possibly set another pin but if you're trying to attack a live system or a real system that's really gonna depend on what the system does but then this is just the general algorithm for going through and how to glitch something. So now we need an actual glitcher so what do we need for the glitcher? Well we need some sort of device that provides a very accurate stable glitch has to be very precise, you know kind of like nanosecond level and so we've got on board the FPGA train and so this is the kind of, this is the tool that we're using to basically build the glitchers and as other people have talked about basically FPGAs have started to come down decently in price so we're not using a very complex features of the FPGA this is just kind of like basic digital logic so these can work on low-end FPGAs like the lattice ice stick. What's nice about those is that they also have open source tool chains and so this is a very approachable way and then also there is the ice breaker which is another open source project and so it's also using the Lattice FPGA but basically you have that, you have this, the PMOD which I'll talk about in a second and then the power supply which I've previously talked about because you need to provide different voltages so you need some power supply that can do kind of custom voltages. So here is the last ice stick it's like I said, open source full chain approachable FPGA, it's more than enough to do this the max PMOD, so I'm a big fan of the digital and PMOD spec for FPGAs, there's now a, apparently a Z-MOD so if you need more IO you can use the Z-MOD spec which is totally overkill for this you don't have to do that but it's got like a 40 pin high density connector it's super nice to geek out on but basically, and this is the kit that's in there it's just an analog multiplexer it has three channel switching and it's just, it fits in the nice PMOD format so you don't have to do a lot of wiring and so what the PMOD is doing is we need to basically insert the glitch right, so what the glitch is is you have, we have that voltage going into VDD core and we need to drive it to zero and basically the multiplexer is perfect for that because if you have these two inputs and then you provide the switch it's going to switch the other one and then that's exactly what you want for kind of a glitch pulse so there's other ways to do this too you can use a MOSFET there's, people have used function generators there's lots of ways to do the glitch as we are using an analog multiplexer and so the thing to know about the kits that you got from the patio is that, oh yeah, thank you, Thomas they have, so like any tool you have to make some and in trying to make the tool there's some design decisions that have to be made so some of, all of the channels are hardwired to ground so you can only switch between one channel and ground obviously if you get the multiplexer yourself you can provide any input to other channels but on that PMOD you provide one channel the other one's hardwired to ground and then there's three resistors that are optional basically if you solder those resistors on then the result will go back to the FPGA so sometimes that's not what you want because if you're, especially if you're doing high voltages you don't want to put like five volts back into your FPGA IO but if you depopulate those resistors it'll just act as like a bridge I have been meaning to put this up on the GitHub so you'll just have to take this audible explanation of how that works the essence is you can find the full schematics for the PMOD on our GitHub and it will explain you how to use it and how to connect it and there are a lot of different configuration options to make it a bit more versatile but the basic circuit is currently on the website that will just enable you to glitch in a couple of minutes basically after soldering it yeah and then now we have the power supply this is a nice cheap power supply configurable via UART, highly recommended we hook it all up with a device on their test is whatever we're trying to test whether it's a dev board or an actual target we have to get those signals back so we need the trigger pin to come back into the FPGA the success indicator, whatever that was it's easy if you do your own firmware it may be different if you're trying to attack a real device so you need that to come back then you have the multiplexer coming back for the different voltages then you have to provide the enable signals and so yeah so it starts to add up and a lot of the practical problems here which I know at least I've had a lot of frustrations with is that you know a lot of the wiring it becomes tricky so like when it's not working you just kind of have to go through each of these wiring that's why we have that PMOD MUX helps cut down some of the wires but you can totally do this on a breadboard it's just that when there's a problem you now have to go through each wire and then you have the power supply and so yeah this is what it looks like when it's all hooked up and in a big mess of wires but the actual glitcher again we're using these FPGAs this is super great this is like I said I really am a big fan of that PMOD connector I previously was not a big fan of 3D modeling though so I was like PMOD connector check make a PMOD no problem put them together I didn't really consider what happens when you put these two together because you end up with a device that looks like that right so you kind of have this L shaped glitcher so you should do 3D modeling it's like a thing and it'll prevent you from making weird L shaped glitching devices yeah and so now we're ready to glitch yeah so basically we now just walk through the theory on how to build a glitcher and the idea is if you want to glitch the real trick is to find your glitch parameters you need to figure out after your trigger event which might be oh I turn my Xbox on and for example the Xbox mod chip is a glitcher that just tries to glitch the Xbox CPU and so that would be my trigger event and then a certain delay after I turn the device on I want to glitch for a certain amount of time and finding these parameters is the real trick so as you've seen before we have the trigger event and then we have the delay and the glitch and the delay we require might be 200 nanoseconds it might be 10 nanoseconds it might be 20 and our glitch pulse might be very short or very long and so what we have to do is we have to write a device that tries to, tries each delay there is and tries each glitch pulse there is and then checks for success and so the problem is if you do this on FPGA and you write your own software for it it becomes complicated really really quickly and so we decided to use something pretty simple for the host control which is Jupyter Notebooks so we provide a Python API to the glitcher that just lets you very easily automate the glitcher and implement your own success checks so for example if you glitch a chip to re-enable the debugging interfaces you might want to build a short script that just checks if JTAG is enabled and integrating this into a big software is kind of a pain so we just use Jupyter Notebooks that let you write your own success function in like 30 lines so for example if we take our glitcher for a chip you're about to see it's just like 20 lines so basically all we do is we tell the chip hey we want to restart you for a certain amount of time then we iterate through all delays that we want to try we iterate through all the pulses all the pulse length the glitch length we want to try and then we actually glitch and then all we have to do is check whether our glitch was successful and so for example in this case with our test firmware we just check whether GPIO was brought up whether we broke our own test firmware but if you do this on a real device your success check can really be anything so as I said it might be a JTAG check it might be a check whether oh my Xbox is now booting my own games or whatever and then we just say oh yeah we were successful and so how does this look in the wild so if you build this and you eventually have a stable glitch range you simply run it and then you get some nice progress bars and after a couple of seconds depending on how stable your glitch is you actually get a success indication so in this case the glitch takes like I don't know 20 seconds or so to execute we were successful but yeah this doesn't really give us anything because all we did was put up an IO so what's really interesting is when you start glitching real targets and so what we did was basically we looked at the processors we find in common IoT devices and so our goal was to take a look at the modern processors that people are using in new designs and the new chips that promise new security features and so the chips we tried and evaluated as part of this was the NRF 52840 which is a Bluetooth chip that you find in a lot of variables and then the ESP32 that's almost in every second Chinese IoT device basically and then the SEM-L11 which is my personal favorite because it's a secure microcontroller and the SEM-32 F2 and so our goal was to configure these all in real world conditions and test the chips in C2 and so because we just have 25 minutes we're gonna skip over the boring chips and go right to the highly secure microchip SEM-L11 and this chip is world-class secure like I tell you it's the winner of the best contribution to IoT security award and just if you search for security on this site it's 57 secure like this is a secure chip and so if you actually read the data sheet we talked about the different power domains earlier and basically we said oh we just glitch V-core right we go behind the regulator and the problem is if you read the data sheet it actually has this special section oh man we have a brownout detector on the 1.2 volt line it's been calibrated in the factory don't touch this and so I was a bit scared that we might not be able to attack this because yeah there's like this dedicated V-core brownout and so can we glitch it? Well the real problem was I started glitching this chip and then I was pretty sure I messed something up because I got success after literally five minutes so I was like oh did I misconfigure the firmware did I mess something up like I was mind blown right? So what do you do with this? Well microchip provides this secure UART boot loader for the L11 and so we can fully bypass this very very easily but that for me brought up the question how easy is this to glitch and so if you couldn't tell by the dialect but I'm German and I live in South Germany and we are very very cheap and if we look at the price of Glitcher like one Glitcher is like nine beers at Oktoberfest that's unacceptable and so the goal was what if we take the most secure chip there is from microchip and the cheapest chip there is from microchip and we use the cheapest chip to glitch the most secure chip and so that was how the five dollar Glitcher was born and so all you need is an 80 tiny which costs like I don't know one euro one dollar, two dollars and the kit that you actually get for free so it's actually the two dollar Glitcher today and then at the price for one pretzel at Oktoberfest you can glitch this chip so that's pretty awesome and it just takes like 130 lines of AVR assembler undisclosed amount of beer and then you end up with a mass of wires on your desk that is able to fully bypass the secure reference boot loader for five dollars and so this brings us to the next chip which is the STM32 F2 and Josh okay so we last year at CCC to talk wall.fl we talked about the STM32 F2 so with the timer meeting this was going to be a little quick because you can go watch the wallet, you go to wallet.fl which is also a website and you can watch that presentation but there's lots of previous work the STM32 in this case is used in a Bitcoin harder wallet don't worry if you're triggered about the word Bitcoin I'm not going to get into all those details but just know that people are storing millions of dollars on an STM32 F2 and if you search so the L11 was 57 times secure if you search security in this datasheet you don't find anything so this is obviously the chip to use for cryptocurrency now in this case basically we're bypassing the readout protection if readout protection 2 is set basically in flash it's got to be this value CCC if it's 0 it has to be this if it's neither of those two you're in RDP 1 which sounds, that sounds horrible then you have the right intuition because we just need to flip one bit and then we can get a RDP 2 to RDP 1 downgrade and that's exactly what we did and so here that value is being read out from the boot ROM and so we do, we reset the chip it goes into the boot ROM the boot ROM runs is going to read this value then it'll eventually launch a user code we are inserting, so here's where it's read so that's where we want to insert that glitch and then if we can flip that one bit we'll go into RDP 1 and so we've seen variations of this picture but this is basically that same setup FPGA, the MUX the Trezor cryptocurrency hardware wallet and this is, we're scripting this with Python so the host is controlling it so let's see how this works and so this is me, this is us running the thing and so as it's running it's searching through that space so Thomas showed that graph of it sliding around that's exactly what it is the first column there is delay in clock cycles and then bam, there we're in, right, so yeah so this is, so now it's downgraded yeah, thank you and yeah, so if a chip stores lots of money then if you do this then you get the money because that's how the seed is stored so there's still some exploitation steps that we have to do we got to read SRAM, do a lot of that stuff but you can look at wallet that fail for that we made this glitcher a little bit elaborate for this I have some of these boards if you want to try this socket there's some issues with them but again, audible instructions I'll help you work it out but we just kind of automated into one and yeah, so chip.fail has basically the GitHub source it has the schematics for the PMOD that you guys have and everything else that we've open sourced is on there so that's the thing and yeah, I think that's any, that's all we have yeah, thank you as always you can find them in the patio if you have additional questions