 Hi, I'm Laura Abbott and I'm here with my colleague Rick Alter to talk about breaking trust zones and using a privilege escalation on the LPC-55 S6-9. We are engineers at Oxide Computer, working on building a new server. Computers haven't changed much in many years and we are looking to fix that. No, I promise this isn't a sales pitch, I'm giving background to explain what we were building when we stumbled across this issue, mostly by accident. One of the reasons we are looking to build a new server is to have some level of security assurance at the hardware level. As part of the server, we are building a hardware route of trust using a security focused microcontroller. All of our software is written in Rust and the goal is to have everything open sourced by the time the product ships as we are big believers in transparency and software. This will be a theme throughout the presentation. We have some pretty specific requirements for what we are looking for in hardware. We need a strong assertion regarding the integrity and authenticity of route of trust firmware. We want to be able to use the route of trust to extend trust to other parts of the system and if we're not confident in what's running on the route of trust, we can do that. Part of building up that trust involves having a camper and personation resistant unique ID. If the advice does get compromised, we need a way to reestablish that trust. Other features you might think of such as cryptographic accelerators or key storage were considered pluses but not strict requirements. We did a pretty wide survey of hardware including programmable logic devices. One requirement we didn't explicitly state but turned out to be pretty key is that the hardware must be available for us to purchase. This ruled out more than one device. We are also biased towards devices that can be programmed with open source tools because, again, transparency. There was no one standout winner among all the candidates and based on our trade-offs we made, we decided the hardware that best met our needs was the LPC-55S69 from NXP. The LPC-55S69 is a dual core ARM V8M Cortex M33. Core Zero has trust zone N and memory protection unit support for isolation. Core Zero does not have trust zone but it can be restricted at the bus level similar to other DMA engines. Rick will talk more about this in a bit. There's cryptographic accelerators for AES, SHA, and doing filled math with ECC. The chip supports secure boot using RSA signatures as well as authenticated debug. There's a puff, physically unclinable function for use with keys. Part of the flash can be locked down after programming to prevent modification. Other variants of the chip support can, which is common in automotive use cases. There are a lot of appealing things about this chip and it seems like it can be used in a lot of places. But of course we have to ask ourselves, why should we believe this is secure? NXP presents this as a great chip to use in your IoT and other edge cases and a lot of written material sounds great but it's marketing material. How does this actually work in practice? Trust but verify and all that. What we found when we started working with the chip was that the documentation was not great. It was frequently pretty confusing and this was frequently, this was especially true when it came to secure boot. This is the diagram for the user manual for the LPC55 showing the boot flow from power on. There's a number of steps here and also ways things can go wrong. When trying to make secure boot work, we ran into some issues. So naturally when the manual doesn't provide information, you turn to the application notes and the application notes give a slightly different boot flow chart with a few more steps. This updated flow chart mentions the NMPA, which we'll talk about more in a bit. It also mentions the debug mailbox as well as the recovery loop path. These are two entirely new code paths that the manual apparently just didn't feel the need to mention. This pattern of making a passing reference to something significant or omitting something completely was a recurring experience with the NXP documentation. We continued to run surprises while reading the documentation. As part of our due diligence, we were reviewing all the application notes. I found one labeled camera interface in LPC55, which was very surprising given the manual did not mention a camera hardware block anyways. The application note contained the line, there is a hidden coprocessor in LPC55S69 can handle the signals of camera. When I mentioned this to my colleagues, they thought I was joking because a hidden coprocessor or even a camera sounds like a bad idea and a chip that's supposed to be secure. We had to ask NXP for details about this, what this thing even was because they usually don't give out this information unless you're a high volume customer. We did eventually find out that this core is named EZH. It's a custom core with a custom instructions that architecture really designed to offload wire protocol conversions that don't need to run on the main ARM processor. The documentation also contained a passing reference to DICE-CDI. DICE stands for Device Identifier Composition Engine and Compound Device Identifier. This is a feature that comes out of the trust computing group and is designed to give a cryptographally strong identity to the chip based off of the first mutable code. The idea is that if you change the first code that's running, you get a new identity. This is a really nice feature for what we're building in root of trust and being able to extend trust but unfortunately NXP decided to de-featurize this by removing detailed documentation. The feature is still actually there in hardware, you just can't know anything about it. And then there's the wrong patcher. But before we go into this part of the story, it helps to have an understanding of trust zone M and I'll pass this off to Rick to talk more about trust zone M. Thank you Laura. So what exactly is trust zone M? According to ARM, the ARM V8M architecture extends trust zone technology to, I can't do this to you, I got bored the first time I read this and I've practically fell in a sleep every time I've tried to read it since. Trust zone M is very similar to trust zone A, which you probably know as just trust zone. This is the system we've had for a long time on a profile devices used in cell phones and other embedded systems. In trust zone A, the hardware is divided into a secure world and a non-secure world and then you have multiple execution modes within each of those worlds. Now in an M profile device, there's a couple of key differences. The first is that you only have two execution modes instead of the four that you would normally have on an A profile device. On an M profile device, you're going to have handler mode, which is similar to what you might have as an OS kernel, and thread mode, which is roughly what a user space thread would be. The second key thing is that M profile devices have no MMUs and there's no concept of virtual addressing at all. In fact, everything is physically addressed and while you do have the option to include an MPU in a given chip, it depends on the chip and so you can't rely on memory protection in any form in a chip. So given all of this, how does trust zone M work? Trust zone A relies pretty heavily on the MMU, so how does M profile actually deal with distinguishing between S world and NS world? Well, if all you have is physical memory addresses, you use physical memory addresses. You divide up the memory space into secure, non-secure, and non-secure callable ranges. If you're executing from within a secure range, you can do data accesses to any address in the system. But your next instruction fetch must be from another secure range. Now if you want to leave the secure mode and go to non-secure, you have to use the special instructions BXNS or BLXNS in order to indicate I am switching to non-secure mode and therefore the next instruction fetch will come from a non-secure range. Once I'm in the non-secure range, data accesses can only go to non-secure ranges. Also, my next instruction either needs to come from a non-secure range or it needs to come from a non-secure callable range and specifically be a SG or secure gateway instruction within an NSC range. Why does this exist? Well conceptually, it's okay for a secure world to arbitrarily call into any part of the non-secure world. We trust the secure world more but a non-secure application shouldn't be able to arbitrarily jump into any code path in the secure world. So instead, you create this special NSC range that you place these SG instructions in and that creates an explicit entry point for the non-secure world to call into the secure world. And in practice, an NSC range typically gets used like a big jump table. You have an SG instruction immediately followed by a branch that goes back into the secure range. Now when you have an NSC region, it more or less acts like a secure region. The only special detail is that it can actually have these SG instructions and it's where NS code can actually call into S code. Okay, so how do we actually set up these SNS or NSC ranges? Well, it depends on which part of the chip we're talking about. If we're talking about the CPU, you start with the SAO or security attribution unit. This is similar in concept to an MPU where you're actually defining a range with a base and limit address and then specifying what security attribute applies to that range. Now this is actually defined architecturally as part of the RMV8M security extension that defines trust zone M. So if you have a part with trust zone M, you're going to have a SAO. However, the number of regions available to you is implementation defined. In practice, you would likely define four, maybe eight ranges, which is kind of similar to how MPUs are often provided. Now one key piece about this is that the default state is that any region not covered by a SAO region, any address not covered by a SAO region is implicitly considered to be secure. So in effect, the SAO allows you to mark ranges as lower secure security state than what they start out as. Now there's another piece where the chip vendor might have reasons why they want certain areas of the chip to be considered secure or non-secure. So there's the implementation defined attribution unit or IDAL. And this is a piece of fixed logic attached to the CPU core that does an address to security attribute computation. It's just that this is fixed by the chip vendor. Now in one of ARM's documents, they provide an example implementation where you just take bit 28 of the physical address and use that to indicate secure or not secure. Sadly, a lot of implementations actually just picked up on this and ran with it, including the LPC55S69. When you do that, you end up with a memory address space that looks like this. You see that things like SRAM and code are a single device, but they have double the address space and it becomes a secure and a non-secure region. So this means that you actually have two different ways of accessing the same physical device. And we call those the non-secure and secure aliases. Why is this important? Well, when you do an access from the CPU core, that address is fed to both the SAO and the IDAO. Those independently compute what the security attribute is. And then the most secure of the two is what's chosen to actually do the restrictions. So if you use the secure addresses, if you use a secure range from the IDAO's perspective, it can never be lower privilege. The IDAO will always win as having it be a secure address. And this means that you can write secure code, secure world code, using secure aliases and never have a chance of it being anything else. On the other hand, if you use a non-secure region from the IDAO's perspective, you have to go look at the SAO to decide what its security attribute is actually going to be. It could be secure, non-secure or non-secure callable. And this all depends on what the chip vendor actually did as far as what the IDAO layout is and then the application as far as what the SAO configuration is. Okay, so what happens once you have that request and you've computed what the security attribute is? Where does the policy get enforced? Well, this is mostly implementation defined, but they do have some common names for things. So first, you're typically going to have the CPU core sending out a request over the AHB interface. And in a trust-owned M world, you actually extend the typical AHB matrix to include the security attributes, and that's called secure AHB or SAHB. Some implementations actually take this a step further and include the ability to do policy-based restrictions on passing through the bus matrix based upon the security policy. It's not common, but it exists. When you reach the peripheral side of the matrix, then you encounter the MPCs and PPCs, which are really the policy going into peripherals. Now, why do we have two different ones? Well, the PPC is more of a general, here's a single security policy for a peripheral. This is the, I want to have this entire GPIO interface be available to the secure world or the non-secure world. But in the case of something like an SRAM, you might want to do more fine-grained policy of having it be something like every 2K I want to alternate through secure or non-secure. And so that gets a separate MPC that allows that fine granularity of enforcing of security attributes. Okay, so what about other AHB initiators? We have more than just the CPU core. Well, again, left mostly to the implementers to find out, but common terminology, you might have a secure AHB aware initiator. This is really what's happening with the SAO and IDAO in the Cortex M33s is you essentially have all the built-in logic to make the decision about what the security attribute should be. And so you're just asserting that as part of the request. If you have a legacy AHB device and you want to just insert it onto a secure AHB, you do need a way to actually set what the security attribute should be. So this is typically called an MSW or master security wrapper. And the common implementation is to have it be a single security attribute gets applied to all requests from that initiator. So when you put this all together, you get something like this, which happens to be the AHB bus matrix diagram from the LPC 55 S69 user manual. And so you can see that it starts to look like a network flow diagram roughly. And you can think of it in kind of firewalling rules where the IDAO and SAO and the MSWs are sort of setting your ingress policy and labeling. And then you might have some enforcement within the matrix itself. But then when you hit the egress side, you can enforce more strict policies about because it came from this source or it has this security attribute, it is allowed or not allowed to go into this peripheral. So the implication about all of this is that most peripherals are going to show up in both the secure and non-secure ranges. This is because from the IDAO's perspective, you need to have non-secure ranges in order for you to have non-secure access to a space at all. If I only had a secure alias for a given device, I could only ever access it from the secure world. And that could be useful. The rest of the system is designed to be implemented by the application software. You configure the SAO, the MSW, the MPC and the PPC to set up whatever policies you want. This is where you say, I want GPIO block A to be in the secure world. Now, given this level of configurability, you run into a lot of potential for mistakes and those mistakes can have devastating consequences. For example, you could accidentally leave the non-secure alias active for a code section that is supposed to be secure only. So with all of that in mind, we'll go back to Laura to talk about more. How about we actually found the ROM patcher? Thanks, Rick. So back to the ROM patcher. Actually really back to the secure boot flow diagrams. It turns out that just having a box labeled validate image and validate CMPA and NMPA isn't really descriptive for all the ways things can go wrong. I managed to break a stack of boards by screwing up various settings. Physically, the chips are fine, but there's no way to change any of the persistent settings or get new code on. As part of NXP's security for the boot ROM, standard SWD debugging isn't enabled at PowerOn until right before the jump to the user image. This means that if you screw up a setting, there's no good way to figure out where it's looping at the ROM because you can't attach a debugger. The lockout for these settings also happens before ISP or the debug mailbox handler, which are designed to be able to access parts of the code. So there's really no way to be able to undo this. In some respects, this is a positive sign because it means that lockout will happen if something does get compromised, but it's a pretty big problem for transparency and that we don't know all the details of what's going on. All I really wanted was complete flow chart showing all the possible ways I could rip my board, so I knew not to do that. NXP unfortunately did not provide the flow chart, but what they did provide was a ROM that was not readout protected, so it was possible to save the ROM, load it up in Ghidra, and start reverse engineering. I had never really done serious reverse engineering before, so this was a really fun project for me to figure out what exactly this chip was doing. I spent a lot of time comparing the addresses against the manual to see what was being accessed. NXP divides the flash space of the chip into two distinct parts, the regular user flash for applications and the protected flash region. The protected part of the protected flash region refers to the fact that pages in this region can only be changed under some circumstances or maybe not at all. The protected flash region holds a bunch of different settings. The region can be divided up into a couple of different areas. The CFPA and the CMPA are designed to be programmed by the user of the chip in the field and at manufacturing time respectively. That's what the FNN stands for. They contain settings for secure boot and screwing them up can result in a brick borne. The NMPA is controlled by NXP and can't be modified. The details of what each field and protected flash region does was not fully described in the manual. NXP's region in particular was really not documented. But apparently everything was more fully described in an attached Excel spreadsheet. And by attached they mean attached to the PDF. I was not aware this was something you could do in terms of attaching things to PDFs. But you download the attached Excel spreadsheet and some parts are missing. But we can show the rows and we finally get to see the full address space of the protected flash area, including the NXP area, which showed this really interesting region labeled ROM patch, which was undoubtedly interesting because nowhere in the documentation was the ability to change the ROM mentioned at all. Going back to why we were interested in this chip in the first place, part of how we build up our route of trust is assuming that the ROM always executes the thing, code. The ability to change the ROM is potentially concerning and breaks a lot of our assumptions. How concerning though? The only way to figure out is to know is to figure out exactly how this block works. Fortunately, the person that disassembly that references flash region give a full program sequence for the ROM patcher. Pretty sweet. NXP's ROM patch controller can support up to 16 patches. A single patch slot can patch one 32-bit ROM address. The block cannot be used to patch addresses outside the ROM region. Up to eight slots can replace the specified address of any 32-bit value. Other slots can be replaced with an SBC instruction corresponding to the slot number. SBC is the system call instruction on our V8M. What happens here is the ROM sets up the system call table in RAM to allow for patching longer instruction sequences. The ROM patch settings do not persist across reset, which does lessen the impact here, but changes made to a secure address are also aliased in the non-secure region as Rick talked about earlier. This is the diagram showing the register layout of the hardware block. To set patch, the entire patch controller must be turned off. The target address is set in one of the address registers. If we're just doing a value replacement, you write the value to one of the corresponding registers. The value registers are a reverse index, which is a bit of an interesting quirk. The control bit determines if we're doing a value replacement or an SBC replacement. Individual patches can also be turned off and on. Importantly, these settings are not sticky, so you can change settings as much as you like. This means it's possible to patch one set of addresses, turn off the controller, then patch another set of addresses. It's worth noting that this is probably a custom NXP design. Previous generations of ARM had a standardized flash patch and breakpoint unit to allow re-enapping of arbitrary flash regions. This was associated with at least one persistent exploit that was presented at a previous DEFCON. ARM V8M removed the patching feature of the unit probably because the designers realized that remapping could potentially break isolation with TrustZone. In fairness to NXP, there are legitimate reasons to want to have a hardware patcher. Running software is hard and you do need a way to fix bugs in your ROM. There's a reasonable motivation to want to bring this back, but they probably should have had a bit to be able to prevent modification from curious firmware engineers. The ROM is the first code that's executed at startup. And if that were the only use of the ROM, this block would be significantly less interesting. The ROM also exports a bunch of functions which are designed to be called at runtime by user applications. These are all features that are either hard to sequence or NXP wants to keep proprietary and would also be served by providing source code to see what exactly these are all doing. All of these APIs expect to be called from secure mode as all the addresses are for secure aliases. The image authentication APIs also require privilege mode. The addresses for calling the functions are accessible via tables inside the ROM itself. Okay, this is DEF CON here. So clearly it's time to do something interesting with this hardware. We can put this all together to do an exploit. For step one, we find a ROM API used by secure code. Step two, we use the ROM cacher from non-secure mode to change the code of the API since that code exists in ROM space. Step three, have the secure code invoked by API and now you've won. I'll pass it back to Rick to talk about writing an exploit and trusted firmware now. So now that we have a plan of attack for how to use the ROM patcher to create a privilege escalation, we had to go actually build a full proof of concept. And initially we wanted to start very simple but it turned out that it was much easier to illustrate the severity of the problem by taking a unmodified secure world and demonstrating it with a non-secure app only. In this case we chose trusted firmware M. So what is trusted firmware M? This is a reference implementation of ARM's platform security architecture which is really a set of APIs for common security services offered from a secure world to a non-secure app. This is designed to allow you to have different vendors for your secure world and non-secure world code. Think of it as I buy a secure TEE type environment from one vendor, maybe it comes with the chip and then I write my own code and I only have access to the non-secure world. So this provides a lot of core services like cryptography, attestation and in internal trusted storage and even some lifecycle management for your device which makes a lot of sense and is actually quite useful. Thankfully, trusted firmware M has upstream support for the LPC 55 S69. So I could just grab this and build it. Now, the piece that we're really interested in is the internal trusted storage. Why are we interested in that? Well, as you might expect from something named trusted storage it deals with persistent data. And specifically it's something that is required to be implemented in the secure world in most implementations of trusted firmware M. So this gives us an API that's callable from the non-secure space. It's required to be implemented in the secure world and it's going to interact with persistent data storage which means it will touch the flash. Does this have a good chance of using a ROM API? Well, yes it does, it uses the ROM flash API. So now we can get into building our actual POC. Now remember, we're just building the non-secure section of a full POC. We're using an unmodified trusted firmware M. So we take that trusted firmware M, I used version 1.2 which was the most recent at the time and I build that as an unmodified image. And then I build a completely independent non-secure app that does the following. We need to write a little payload that fits within the capabilities of the ROM patcher. Remember that we only have a limited number of words that we can actually write into, patch into the ROM address space. And so I needed a payload that would fit in that space. Then the ROM patch controller will be used from non-secure mode via its non-secure alias to copy that payload into empty space in the ROM. So we're essentially moving code from the non-secure world into the secure world. Once we've done that, we can patch the flash write ROM API to invoke that payload. Once we do so, non-secure world can call into the ITS API to do any sort of write and our payload should run. We just have to verify it. So we're gonna talk a lot about addressing and layout. And so it's important to understand in these devices you have a single flash region and trusted firmware M in a configuration with no bootloader simply puts the secure and non-secure images right next to each other in address space. And then you have, you know, your other persistent data storage. Now this is important because it goes into how the SAO and the MPC get configured. The secure world actually sets this up. And when we flip on debug mode for the TFM secure mode or secure world, it gives us this spew about that configuration. And I verified that it doesn't change when you turn off debug mode, but it was just handy to be able to have it spit it out to me over a serial port. But if we look carefully, you'll notice that the SAO is being configured to mark out that section of non-secure code. Now, if we look, we're to go back and look at our IDAL implementation for the LPC 55, we would find that the zero range up to 1-0-0-0-0-0 is actually marked as a non-secure range. And so this is really using the SAO to further to also set it as non-secure so that both sides agree this is a non-secure range. The remainder of that flash region will automatically become secure because there's no SAO region covering it. And then there will be a additional secure alias for the entire flash as well. Now we also have the NSC base, which if you note actually starts with a one in the address and that's using the secure alias. Remember that NSC can actually be in a secure range. So it's perfectly fine and legit to use secure alias for that. Because of this being configured, when we're executing a non-secure, we should not be able to access the secure mode flash. But there should be no non-secure window into that region from zero to 40,000 hex. Okay, so the first thing we do is look at the flash ROM API and look specifically to find a method that we can instrument and provide a call out to our payload. Now I turned out that the flash write method is actually pretty simple and provides a really nice entry point where I can replace two move instructions with a single branch and link instruction. And I have a lot of registers to play with at that moment and it's really easy to replay those two instructions inside my payload. And so constructing the payload is primarily just doing a bunch of copies of constants from a non-secure section of memory and copying it over into secure. Now this is really kind of a neat trick because remember secure mode can do data accesses from non-secure regions. So I don't actually have to put the false constant pool in the secure world. I just have to have the code that reads it be in the secure world so that it can write to the secure aliases of the other devices that I wanna interact with. So in this case, the particular example we're using constants to load in and overwrite the control registers for both the SAO and the secure AHB to basically extend the ranges and open up both the secure mode code region in as we saw in the flash layout as well as the secure memory region and extend those in the SAO regions thus turn them into a non-secure space. Now if we were to try to access them by the secure aliases this would fail but a non-secure execution via the non-secure alias with the SAO in this configuration it should be marked as non-secure even though it's accessing the secure code that's at the zero range of the flash. So how do we verify this? Well, it's pretty simple. You just need to dereference zero. Remember everything's physically addressed so zero is where the secure code actually lives and we just do that from a non-secure space. Assuming that everything worked we'll just get a dump out of the actual contents. So here we have an NXP LPC Espresso 55S69 this is just an off the shelf dev board for that part. I have two USB connections at the top the orange one is directly connected to the USB interface on the LPC 55. The black one goes to an on-board SWD debugger called the LPC link and the main reason I have that is that it has a serial console or it is connected to the main LPC 55 and it exposes it as a USB ACM device so just a convenient way to get the serial. We're actually gonna use the ROM bootloader on the LPC 55 for most of what we need to do. So the first thing is we need a copy of both the trusted firmware M. I'm using the tag for the 1.2 version and then we're also gonna get a copy of our POC. Now I've said that we use an unmodified version of trusted firmware M. There's actually a bug in their build system so I grab an upstream patch that fixes that bug for this particular version. Then the build system for trusted firmware M uses PIP so use a virtual M for that and then go ahead and have it fetch all of its requirements. Then I'm gonna go ahead and install CMake that way and then they use CMake as part of their build system so we're gonna go ahead and say build for this development board using the GNU ARM tool chain with the trusted firmware M bootloader disabled. We're not using that in this case and then also build our POC as the non-secure application. Okay, so now we're gonna go ahead and build that. Now I did grab our POC during the recording of this from a private GitHub repository. By the time this is available, we will have made that same repository public and set the default branch to be the appropriate code so you can play along at home. Okay, so with that built we can look and here's the non-secure world and the secure world binaries. Now what we wanna do is erase the flash and load these in. So the first thing we're gonna do is use an XP's tool for this for interacting with the bootloader which is called blhost and it's not the most reliable tool. So what we have to do is hold down the ISP button and press the reset button and it's supposed to jump into the ISP mode and then we can talk to it over USB. So in this case, we got it on the first try there. Now I'm going to actually reset it again and do the same thing between each step just because I get slightly better success that way. Oops. Okay, so what we're gonna do is write just the secure application of the secure world into the flash. Great. Now if I reset and let this run, you'll see that we get the debug spew that I showed earlier showing us the SAO configuration and then it actually launches the secure application and then hangs because there is no non-secure world to execute right now. So now we're gonna go ahead and put it back into ISP mode and we're gonna write in the non-secure world and then if I reset, you'll see that it actually does that same dump of the SAO configuration and then now this time it actually starts to boot our non-secure world. There's a bug that I couldn't figure out where when I first tried to use the ITS it crashes but if I just hit reset again, it's gonna scroll by pretty quickly and I'll go back here to show what actually is happening. So we started up and we got into this normal thing and the LPC ROM POC started is the non-secure application and within that we start a couple of threads. One is this touch flash API and this thread is literally just incrementing a counter using the ITS. It reads from ITS, increments by one and then pushes it back. Now we actually wait for five seconds and before we apply our ROM patch. So this is just the operation normally and then the patch begins and this is just a separate thread from non-secure space executing the payload that we looked at earlier. It's loading the payload, patching the flash write out API and doing all that work and in the meantime the counter thread then keeps on running and except that now it's actually running our POC every time this counter is invoked and so then a little bit later after we've let that just sleep for a while the POC thread then is also running from non-secure code again actually goes in and dumps out the beginning addresses of the secure space. So that happens and then the counter just continues on and we print out a tick every once in a while to show that everything is still alive. So now that we've gone through our proof of concept let's talk a little bit about how we went about disclosing this to NXP and doing a public disclosure. Like most people and most companies we started with the responsible disclosure process with the 90 day response window and we started off with coming up with our own template because frankly we're a computer manufacturer not a security company we didn't expect to have to do a disclosure and didn't have anything ready for this. So I put something together that wrote up a lot of the details and tried to provide impact and CBSS scores and then the first question was well how do I send this to the vendor? Where do I even find the contact info for their page? So this is what you have at the header of NXP's website and I wasn't really clear to me where to go. Maybe the support section, nothing particularly obvious there. How about the company section? No, no nothing there either. Okay well maybe that's at the footer. That could be no, no not the footer. Okay well back to the company there was a contact us maybe I can go there. No, no but wait there's a section that says contact support out there it is hidden at the bottom of the contact support page. Now when I did go to this page finally after finding it it does tell me how to email NXP's p-cert and it gives me their GPG key along with some details about their process. Okay sent that off we did that on December 16th, 2020 and off it went and 30 minutes later we got a response. Now this is not an automated response but it does actually mostly give the equivalent of an automated response like yes we got it but we'll get in touch at some point. So I don't really count this as part of the 90 day like timeline for a response and coordination. And in fact I waited a very long time for them to follow up. If you notice this is December 16th, 2020 here on January 11th, 2021. I had to ask like it's been almost a month have you confirmed anything? Is it look like a real issue? And they came back a day later and said oh yes in fact we did confirm it and it's an issue and thank you for reporting this yada yada we'll be following up. And then the coordination kinda went a little wonky. You'll notice from the dates and times like there was a lot of time between these interactions and it started started with hey we really looked at your POC a little bit more closely and we think we were doing some silly things. Now this was our first POC that did not use trusted firmware M. So it's a little bit understandable but this is where we came back and said look here's one the very clean slate this does not use any modifications to your example code for how to write a secure world application and we can do definitely do privilege escalation. What was really annoying about this was an insistence on trying to demote the severity of the issue in this case. A long time after that there was a hey yeah this is a real problem and can we have 45 more days? Now note this was almost 60 days after we originally reported the issue before they actually acknowledged that it was a real issue of the severity that we told them it was. So we had some internal deliberation and decided hey you know what tell us about what else is in this chip because we are trying to use it we wanna be a customer and if you can tell us that we'll gladly give you 45 days. Well they just took the 45 days and didn't really hold up their end of the bargain. We did get some material under NDA answering a few of our questions but not a whole lot. One of the more interesting ones is we asked to be able to access or audit their ROM API source code because we already reverse engineered most of it we just wanted to see what else we were missing. And this is the statement we got back and oddly enough they gave us permission to share this publicly. This is not a statement that I would want to see from a company making a security chip. After we had had some further discussions and we're working out a timeline I asked about hey when are you gonna actually have a CDE like what are you gonna do about that? We wanna have a public disclosure this is a big issue and honest to God the response was what's a CDE? And we said okay you know what we'll just file it ourselves I've done this before not a big deal but then it comes out a month later they come back and say oh hey we're gonna file a CDE on this who should we put as your reporter? And I'm like no we already have the CDE number assigned here's all the details like we kind of took the wing out of their sales but it was very delayed. So in the end we spent almost four and a half months working on this disclosure from start to finish and a lot of it was waiting for them to even acknowledge it. In the end from the Oxide side we did push out the CDE we did blog posts going into some of the technical details we did some tweets and actually given how poorly the response went with NXP that was enough to fuel a spite CFP for DEF CON so here we are from NXP's side their disclosure I call this public but in practice they wrote a security bulletin that we received under NDA and we're told that in fact they're only going to email that out to select customers I have searched and not been able to find this anywhere publicly I also did not receive it outside of our discussions with the p-cert channel I have no idea how you get these security bulletins somebody gets them but good luck as far as off the shelf customers they did update the user manuals if you know what you're looking for the two most recent revisions to the LPC-55 S69 user manual do talk about the security regions but they are very light on what the underlying cause was and are more prescriptive in terms of you must write these registers to these values without explaining why Soloro what should we take away from all of this? Thanks Rick wrapping this up this is the full scope of devices with this wrong hatcher the title of this talk mentions the LPC-55 S69 but it affects multiple ships in the LPC-55 family all those numbers in there actually need something but I'm not a chip marketing marketing expert anything in the LPC-55 S6X is affected the S in the part name stands for secure which means it has trust zone there is a variant LPC-55 2X which does not have trust zone this means it isn't vulnerable to a trust zone breakout because it isn't there this vulnerability can still be used for privilege escalation however slightly worrying the LPC-55 2X does not have the extra level setting bits without trust zone so the only prevention against it is the MPU one of the questions we asked NXP when we were discussions was if there was an ability to lock the patch configuration to prevent changes the answer there was no and it sounded like this was a deliberate product choice with the idea that maybe they would need to give a patch to customers later this seemed like a bad choice but I don't work for an XT product strategy I mean maybe they could have put the lock bit in there let these describe but again out of my hands NXP did mention that the upcoming LPC-55 S3X would include the ability to lock the wrong patch which was really good to hear we're not sure if this if this is all effective devices NXP was willing to acknowledge the trust zone vulnerability but wasn't willing to admit this was a privilege escalation there may be other chips have the same controller we don't know about which is really concerning from a security perspective finally some important takeaways code is hard configuring hardware is hard configuring hardware security is really hard trust zone M is hard to configure correctly there's a lot of knobs requiring code to drop permissions is a great way to accidentally end up with something still exposed we've seen this in Linux plenty reference implementations are frequently incomplete the trusted firmware POC we gave could have been prevented by setting a few lock bits to prevent changing the SAU but they just weren't set just taking a reference implementation is no guarantee you'll be secure or you won't hit odd edge cases you need to be doing audits and review of all the reference code transparency is so important in your hardware in order to drop privileges you need to know that you need to do so the wrong packet block was labeled as reserve the various parts of the memory map because NXP did not want to acknowledge it existed the old manual before their update we tell you not to write anything for the reserve area for various security positions protections this means that even if you were following the manual to the letter you would not be protected Oxside and NXP really seem to have a fundamental disagreement about how big of an issue this is our stance is that unless you have full knowledge about what your hardware is doing it is not possible to write a truly secure implementation turning that reference code into something correct is incredibly difficult without full documentation hardware makers should really want to make it easy to write secure code and to treat undocumented hardware in the insistences of security and really please just give the source code for the wrong the conclusion here should not be that NXP should have re-protected the ROM so that we couldn't find the bug that is not driven to be an effective solution Rick and I are not professional reverse engineers where developers didn't think this kind of problem was really neat I'd like to say there were no more bugs looking in that wrong but I really don't know people who are much more experienced in this area could probably find more bugs there's certainly been a history of bugs in the ISP mode some of which have been presented at DEF CON NXP could really increase the confidence here by letting us all review the code just saying that a third party reviewed the code doesn't really tell us anything if we don't know what was found or fixed especially if the audit didn't actually manage to find this hardware block hiding things doesn't actually make it more secure it only delays the time till bugs are found I really look forward to the future when hardware manufacturers have complete correct hardware documentation available so we can all have more secure and open software thank you