 All right, I'll hand it over to you these wonderful people. Give them a big round of applause. All right. Okay, so you guys are all here. I'm sure you know what this talk is. You should pet yourself in the back because this is the right choice for talk to attend right now. It is the best talk. You made the right decision. Your life is already done. You're good. You're good for life because we're talking about Cisco routers and secure boot and the title of this talk is 100 seconds of solitude. My name is Aung Sui. Is that on? Sisting on. Hey, my name is Jatin Gadaria. Very good. Okay. And what is primary main objective? Primary main objective is not Chimpogo Camp. It is Cisco trust anchor. And, you know, so this talk is about the work that we've done over the last three years around reverse engineering not just the Cisco router in front of you, but a fundamental piece of hardware that underpins a lot of the secure boot process of dozens of types of Cisco devices ranging from, you know, small switches all the way up to the very large enterprise level, you know, core router. So the trust anchor is something that we had no idea existed when we started this project and we'll take you through all of that story. But ultimately what this revolved around was the work that we developed, techniques that we developed to manipulate FPGA bit streams in order for it to do something useful. So, you know, even though we're going to talk about the security implications of our techniques on this router, we're really here to talk, well, okay, so the deeper primary main objective is FPGA bit stream manipulation. And the reason why this is is because, you know, yes, Cisco routers and switches, you know, all use or a lot of them use this type of, you know, Spartans, Ilings, FPGA, but so do a whole lot of other things in the world like, you know, advanced driver assist in cars, I'm sure legacy weapon systems and missiles and all sorts of things and the, you know, the techniques that we're going to talk about here are going to affect those devices too. So even though this might seem like a fairly large impact vulnerability disclosure, the real impact of using these techniques in devices we haven't even started to look at, I think is much larger. So, you know, like I said, we did this for three years and there is a long story which I'm going to take you through over the next 235 slides. And yes, I have exactly those. But before we start, you know, this is kind of an epic story, you know, there's love and loss and betrayal, more love and friendship and defeat and, you know, redemption and more friendship. So we're going to go through the cast. Jadon over here, right, that's him. Rick is actually in the front row, that's Rick. Joey is sadly not here, I think he's at a wedding. This is me, I'm wearing sandals just like the photo shows. James, right there, okay. And well, over the last three years at Rebel and Security, we've invested a ton of our resources in creating this massively complicated, highly intelligent, automated testing framework, abbreviated to Brian. You know, it's so complicated that some might even say that he is border-aligned sentient, which I think is ridiculous, but you know, we'll get to that. All right, so as with all great epic tales, we start in mid-story, okay. So, you know, we started this work in 2016. In 2015, Jadon and I worked on this little thing where we got code execution inside the on-screen display controllers of all sorts of, you know, probably over a billion monitors. And then in 2017, you know, Rick and I worked on this thing where we built an EMP, electromagnetic pulse generator in order to defeat secure boot in truss zone on a Cisco phone. So, you know, those projects are pretty simple, straightforward, you know, there's really no thinking involved, I mean, very easy to do. So in the middle of all of that, you know, I say, hey guys, like art and lives are so easy, you know, like it's going too well, let's punish yourselves for like no reason. And then Jadon says, what the f, no, it's very, very good. He says, what? All right, so ultimately, you know, fast forward to 2019, here we are talking about all of that work. And, you know, it resulted in this thing called Thrangry cat, right. And technically, you can't pronounce it like Thrangry cat because it is the first vulnerability named after three unpronounceable emojis. So you can only come close to pronouncing it, but not really, you can't really say it out loud. Thrangry fact one, the domain name for this vulnerability is emojis, okay. And for uncool kids, you can type it out as Thrangry cat, it's fine. Thrangry fact two, we, you know, have all sorts of information about the nature of the vulnerability, you know, how we did it. And actually, I will temp fate and try to tab out of this thing to show you our website. So if you go to our website, just a few hours ago, you know, we put the slides up finally and then we also have all of the libraries that we'll be talking about and the tools that we built. It's all in GitHub, it's right there on our website. So if you like getting this stuff and you're interested, check it out. The tools are actually super cool. Okay, so we wrote about, you know, we had this, like, four, a set of four reasons why we decided to name our vulnerability after emojis. And I'll just give you one. Reason number two is that emojis are indexical to the digital age, right? And that's just one of four reasons. And Thrangry fact 2.1, and this made my heart sing when I read it, right? I found it interesting that the section that explains the name of this vulnerability is way more thought out and verbose than any other sections. It's almost, they had, like, a linguist on the team and the emoji name was his prideful contribution. True fact, true fact. I, thank you. If you're out there, you're good. Um, alright, so just the immediate impact of, uh, this vulnerability, uh, it affected, like I said, probably, I think, something like 180, uh, types of. 133. 133 Cisco devices. And here's a list of some of them, you know, go like this page and more on this page and this page and this page and this page and this page and this page. I think there's one more. Nope, two more. One more, right? Alright, cool. So it affects a whole lot of Cisco routers. Alright, let's talk about how we did it. Uh, you know, so. It all started when we wanted to, um, hack ASR 1001. But it was end of life. And, uh, Cisco started, uh, you know, sending 1001 X in 2013, right? Yeah, so, right, Jen was like, let's look at the 1001 and I was like, oh, can't really buy this thing anymore. So whatever, I see this thing called the 1001 X, uh, whatever. Let's, it's probably the same, you know? Yeah, but, uh, what, um, doesn't matter, we don't care about documentation, who care about the X, you know, X is what jumps. It's probably fine, right? So we looked at it and we, you know, signed ourselves up to do this work and, you know, to be fair, look, the left is the 1001, the right is the 1001 X, looks exactly the same. Uh, you know, the 1001 X, uh, was kind of expensive. It's like $11,000 per router, but we bought it anyway, to help everyone visualize what exactly that is. It was on sale. Oh, that's right, it's on sale, so you can get it for a little bit cheaper. Uh, anyway, so what is that? Let's talk about what that means. So $1 is, uh, 6.14 inches by 2.61 inches. It's approximately one gram, so the cost of that 1001 X is approximately 3.58 feet or 22 pounds, which, uh, to put in a unit that you guys can understand, it's basically two healthy three month old babies stacked on top of each other, like that's the right height and weight, that's exactly what it is. Okay, so the 1001 X, the X, uh, has this thing called trust anchor and secure boot, which again, at the time, we had no idea what it is. Uh, so we thought, hey, maybe this is just like a new bootloader, whatever, let's, we can do it. Alright, so goal is, uh, what do we want to do? The goal is to modify the RAM on, like we just wanted to run our own firmware, our own stack on the SR 1001, and it was, that's what we wanted, so, uh, we have done, you know, previous work and we thought, you know, like it would be easy, it would just like change the root key, change the firmware, it doesn't matter, let's just do that. Alright, so, and uh, this is kind of like iron shift. The secret ingredient is, okay, we actually have one of these things in front of you. If we don't keep the lid on, uh, it'll, the thermals are horrible, so it's not great. But anyway, so, when you open this box, uh, you see this thing over here, uh, it might look a little weird, but a lot of this is basically a standard, you know, Intel based computer, right? So, the things that we care about is, um, you know, obviously there's the Intel Xeon processor, so probably a lot of that code runs there, uh, and there is a Xilinx FPGA, so who knows, maybe that's doing something interesting. And this thing has a number of different little SPI flash chips that, uh, we thought might actually be, you know, useful to look at. Uh, and the rest of it is a whole lot of complicated hardware that actually makes, you know, the routing and switching stuff work. But, you know, we opened this box and we said, okay, you know, we probably understand something about this piece of hardware, you know, this is somewhat like a normal pro, a computer, but it certainly has a whole lot of weird stuff that we haven't seen before. Okay, so let's look at the software analysis. Um, so, if you guys have worked with Cisco, they used to have, uh, this, uh, proprietary boot console called Ramon, so which allows you to boot through TFTP, you know, change images. Uh, so what they did was they changed to UEFI in 2013 and, uh, but they still wanted to work with their Ramon. So, they implemented a pre-Raman as a pre-EFI module. What it does is that it manages Ramon. It also, with those two SPI flashes as you saw, as you can see, it also handles the updates. So, in the Linux kernel, if they want to apply a patch to the boot loader, the Linux kernel will update one of the SPI flashes and then the pre-Raman will validate, uh, that whether it is right or not, and if it is not, then copy the code and copy again. Uh, pretty common stuff. Uh, then they had Ramon, uh, then they had Ramon and, uh, basically this is implemented as a DXC module. Uh, again it does the exactly same thing. It can allow you to boot, uh, uh, Cisco images, um, through TFTP. It also allows, it also validates the next operating system it's gonna boot. In the boot loader, um, the OS stage and, uh, it also has a privilege mode which allows you to, uh, introspect what is happening in the DXC. Oh, by the way, so Ram doesn't stand, uh, for what you think it does. It's not actually read only memory, it stands for romantic monitor. True fact. True fact. So, uh, then comes the Linux kernel. Uh, so they ported, uh, they started using, uh, you know, uh, uh, off-the-shelf Linux kernel on which they ported their own Cisco stack running, uh, as a privilege process. They also wrote a process manager which, uh, manages this process so anything, any kind of crash happens in the iOS, it reboots the whole, uh, system. And once you combine this, this becomes your iOS XC stack. And if you go to their website, this is the most secure stack. Uh, so we, while doing this analysis looking at everything, we saw no hashes on the UEFI, no certs on the UEFI, so it should be like a really easy mod because our goal is to just run our own stack. And, uh, we also disabled, uh, some of the checks, um, which were in the pre-ram on and we mod, uh, booted the modified firmware and everything worked. And I was, this is pretty simple. Uh, so, but, wait, then drive to reset. Then we saw, like, wow, it reset after every hundred seconds, which doesn't make sense. And, uh, we spent a lot of time. Well, so first it was, uh, you know, it, you know, it resets what he was able to get code execution on the XAM processor and it allowed us to do it. And then you would eventually kind of reboot. And then, uh, at first, you know, he was like, well, once in a while, maybe a minute, right? And then he was like, uh, it's kind of like a hundred seconds. And I say, well, how many seconds exactly a hundred seconds? Because computers don't like to count in decimal. You know, it's not like they have ten fingers and ten toes. So 100 exact seconds was kind of a hint to us that there's something else, uh, that's kind of like a, you know, a piece of code that's on engineer road that enforces that 100 second thing. So this is signified by a loud fan noise. These fans, as you can see, four fans, make a very loud noise. And this was crucial in doing all of the reverse engineering. This is probably the key, the fan noise. So we came up with the hypothesis of, like, what is this hundred second reset causing? You know, there are multiple x86, because the processor is an x86 core. So maybe there are multiple mitigations, like the one is, you know, maybe it is doing some kind of virtual machine introspection. Uh, but we analyzed the code, it was already disabled. Uh, x86 has dozens of Vosh stock timers, so we disabled all of them because we had hundred seconds to run whatever code we want. And, uh, that also did not help. Uh, then we went to SMM. And we saw, like, SMM was enabled. Uh, so we disabled that. Uh, but it was still pre-booting. So it was like, um, I don't know, man. Yeah, so we, we hypothesized, like, is there like a magical deity in this processor that somehow, despite the fact that we control every instruction that executes on the processor, still is watching us. And tempting us, right? It allows us to execute, like, our hopes are coming true, right? And it waits for 100 seconds before it takes our dream away, right? Like, what kind of thing is this? Uh. Alright, so, you know, normally, every time we ask this question, like, how do we mind read the computer we don't understand, uh, we typically like to default back to electromagnetic emanation. And if you look at some of the past research we've done, this is a really useful tool and a way to kind of figure out, you know, what the computer is kind of doing without really understanding a lot. So, you know, basic stuff, like, when you push an electron through a wire, right, it likes to induce a magnetic field, then electric field, so it kind of emanates, right, EM. So when you take a near-field probe to the thing, right, you should be able to measure out the, you know, activity around certain parts of the machine in order to kind of come out with a timeline of what is probably executing. So, you know, we have a CNC machine that we like to hook up to a near-field probe, right, that to do this type of, you know, analysis automatically, uh, I think the zip type broke, so we're just holding it by hand, but usually it's automated. Uh, we swear, right? So, uh, after gathering some data, here's what we have, okay? So, uh, on this one, on the, on the way, way on the right, we have, uh, the emanation coming out of the, the CPU power supply, okay? So, there is a tiny little dot, uh, where's my mouse? Up here, right? So this is, you know, time, right? Uh, so, you know, as we go downwards, that's later in time. So, little dots come up, right, and a whole lot of noise coming out of the power supply, you know, in like a quarter second or something like that. Now, uh, there's nothing coming out of the, the CPU, the Xeon processor until basically the bottom of, of, of this graph. So the processor is sitting idle doing nothing, but the FPGA, the Xeon Spartan 6 FPGA, has this, uh, emanation profile. So, you know, slightly after the initial power turn on, right? You see two little blips, and then it waits there for a while, thinks about its life, and then, uh, later it starts doing a bunch of stuff, right? And then the SPI chip, one of the SPI chips, which is actually connected to the FPGA, right, has similar, uh, electromagnetic emanation profile, uh, in the beginning, and then, you know, it gets red, and then basically nothing happens. So, you know, if you put, like, this is like blues clues, right? You put all these clues together, I wonder what's happening. So, power turns on, right? The FPGA reads from the SPI flash, uh, its bit stream, right? It actually reads two bit streams, and then it loads that configuration into the FPGA, and then the FPGA does some magical thing, right? And then basically at the very bottom of this, this is where the, uh, the Xeon processor starts to do stuff. So, uh, you know, we start hypothesizing that this is basically the mean little deity that's hiding in the machine that's taking our dreams away after 100 seconds. You took my joke. But, uh, there is a slide. So, uh, there were unknown bits coming out of this SPI. Uh, and, uh, we saw, you know, like it was not available in the bootloader and turned out, uh, we had 100 seconds so we can dump this code. Uh, and, uh, these were interrupt handlers, uh, for the real mode. Uh, basically the BIOS ROM, VBIOS, which is present in that range. Uh, then we also, you know, analyzed, we saw some, uh, address ranges, which was like, uh, ZXFED4, as you can see. This is usually used for external devices, uh, which are mapped, uh, uh, you know, maybe mapped as, um, I.O. devices for SQI's, you know, bus, which is basically a serial quad interface. And, uh, once you, once, this also gives us, like, kind of an idea, like some other I.O. device is mapped in and X86 is, uh, uh, breeding it's state. So, just to validate, we also hijacked the first X86 instruction, wrote the whole serial driver, literally validated and there was an external entity which was doing it. And that was a joke, but. Well, hold on. So this is where, you know, he put in the slide and I said, well, you know, let's not be culturally insensitive. So I added, we do have a cat guide and then the rest of this presentation. So cat Buddha, cat Buddha. So there is a deity, it is a vengeful one. So, uh, we wanted to, um, you know, we knew that FPG is doing the reset. Um, so this was our assumption at this point, like FPG is booting, uh, it's validating the UEFI, so any bit change in the UEFI, uh, address space, you know, it causes the whole thing to, uh, reset. And also pre-Raman checks, Raman, Raman checks the Linux OS, um, it's kind of a, you know, secured boot flow. So then we wanted to find the reset pin and, uh, uh, it was the first $10,000 device which we did because it was a BGA chip. Well, so, and this is a large part of the difficulty of doing this work because, I mean, seriously, who can afford to just blow away $10,000 by, like, touching a pin wrong? You know, that was for all of us, uh, and we've destroyed many more than just one of these. Uh, so this was, uh, for all of us, probably the most expensive soldering mistake we've made in life until this point. So in, in honor of those mistakes, we've built a fail fort, right? So the fort grows over time and there's a whole history of mythology about the rise and fall of fail fort. But anyway, fail fort version 1, $10,000, uh, humble beginnings. This is the start of the civilization. Um, then we wanted to test this theory, right? Uh, an RTL reconstruction is really hard and we'll come back to article, RTL, what is RTL? And then, uh, but we still wanted it. And, uh, uh, I want to just, like, to pull the reset pin high and I'm a systems guy, so I go to either Angad, Rick, for my hardware problems and, uh, Rick said, you know, this GI Joe guy over there, like, I'm going to take a 10K ohm, I resisted it, I'm going to just, you know, keep it high always. And then it, again, costed, it basically costed $1 per one ohm and it was total cost was $20,000. All right, so now, Router sitting there doing, like, completely unfazed except, you know, the ones that have died. Uh, and us, we're negative $20,000, uh, in a, in a hull. So version 2, you know, we started racing recklessly towards sales guy. This is, this is where we, uh, you know, took over the temple of the jungle. Um, all right, so, you know, at this time, another one of our colleagues, you know, uh, Joey was sitting and saying, well, man, why are you guys, like, breaking all this equipment? You guys suck. Like, why don't you try to Google this, you know? And he said, like, I'm a level 9 Googler. So, look, I type into the computer thing and I found this patent. So here is a patent, uh, awarded to Cisco in 2012 that talked about a lot of the things that we were kind of seeing, right? So it is a patent on secure boot through an external device and it's, it's specifically said, you know, you can implement this using an FPGA and it actually has a very similar diagram to the thing that we came up with. So anyway, this document is super useful and one key thing that the document said is you must implement the trust anchor, which is, I think what they call it in this patent, with an immutable, uh, thing, right? Because if the attacker can, like, change it then the whole premise of it goes away. Uh, and they also said that you can implement it with an FPGA. But you know, we'll get back to that. Uh, wait. So it is also, if you see, it is connected through LPC bus. So the way it is actually doing it is like it turns off the south bridge. So, uh, the code has to just reset. Alright. So at this point, you know, we can, we're kind of getting somewhere, but, you know, for, for us, and I think for a whole lot of other folks out there, you know, we're firmer people, you know, maybe software people, maybe some hardware people, but all of us found FPGAs to be scary. You know, it is a mystery thing that's really hard. A lot of smart people are working on it and, you know, they haven't done the thing that we needed. So, you know, we kind of, you know, said, ugh, this is so hard. And we said, well, you know what? Like, why don't we stop, let's give up, but say we didn't. And then, like a year goes by, right? Because, uh, you know, we have actual work and, you know, we're already defeated twice, like $20,000 in the whole. Um, and then, you know, one night, Jadon was like, yeah, I, I basically went to Ong and I said, like, give me $20,000 more. Let's do it. I can, I can hack FPGA. Yeah. And I said, okay, fine, let's go do it. So, uh, he read a ton of docs and he's going to talk about. Yeah. So, uh, again, as I said, like, I'm a systems guy, so, you know, how do you understand FPGAs? And, uh, actually we never, what is an FPGA? What does it stand for? Yeah. So what is an FPGA? Right? Like, it looks like a lot of blocks, a lot of registers, a lot of flops. No idea. Feel programmable gate array. Oh, I see. I still don't know that. Okay. So, uh, how does the FPGA design flow works? Right? Like, uh, the code basically provides the hardware circuit and an SG, hardware description language, uh, the render tool chain synthesizes it, places it, maps it, all this information about the truth tables, about what is the memory initialization, routing, and so forth, uh, gets encoded into a bit stream, which gets encoded into a binary, which we call it a configuration bit stream. So anytime FPGA gets configured, how the, uh, you know, because it is an IC, which has, uh, benefits of both hardware and software, uh, so it has hardware logic block to implement to do computations and, uh, it can be reconfigured as software. And there are different types of FPGAs, there is SRAM based FPGA, which is basically boots its configuration bit stream from an external static memory. Uh, so anytime it reboots, it has to reload itself and that's what we're talking about here, uh, for Cisco ASR. And then the other one is the flash base in which the flash is automatic, uh, is already present on the die. It's less power consumption. It, uh, boots really quickly. And then the other one is the anti fuse, which is, doesn't allow you to configure again. So it's not kind of an FPGA. So but still, what is FPGA? Now we learn about advanced FPGA because that's how smart we are. Yeah. And beginning FPGA and I just learned field program over Gator, right? Okay. And actually, you still didn't know what FPGA stands for. So FPGA can be thought of, think of as like a combination of blocks, you know, each block does something and they basically correlate with each other. But again, what is FPGA for systems people? It's y is equal to fx. So this is very important because what matters is like if we can control the y and the x with irrespective of the logic f, then we win. Doesn't matter how complex it is. So let's go inside like, what are the basic blocks of the FPGA? IOB is a group of basic elements which drives, uh, the output and the input functions of the FPGA and the value could be zero one or z. But really, what is it? It is just a thing which is connected to IO pin. And it gives zero or one. That's all you care about. The second thing is IO interface. This is how the pin gets the uh, uh, the value from the rest of the logic and provides value to the, uh, to the other part of the logic. And the third is BRAM, which is just memory. It can be of different data width. Uh, you know, it could be one bit, uh, one width or two or 32 or it could be, uh, for Spartan 6, it's like 18 kilobyte, 9 kilobyte blocks. You can configure it, you can make it single or dual port. The other, uh, which does, uh, so complex logic block is something which implements the boolean function of the FPGA. It has two slices, a switch matrix. Switch matrix allows it to con, uh, you know, like communicate with rest, rest of the FPGA or also allows it to communicate it to other parts of the IO. Uh, and uh, the slice contains flip flops and you do again, doesn't matter for system scrap. Uh, and then there is, you know, slices can be of multiple types, slice x, boolean function, slice l, with the carry logic, boolean function, and slice m also allows, gives you some memory. Uh, but this is the complexity of the FPGA. Do I- It's so complicated. I'm, I'm bored just listening to this. I know. Do we? Because there's so many, you know, like little nuances of piece of, like all of the stuff and all of the innards of this is, uh, almost always in commercial FPGAs kept as a secret. This is a proprietary thing. You know, they allow you to, they give you a tool chain that generates, you know, from the, the Harvard description language, the bit stream that configures their specific vendors, specific, you know, FPGA, and it will work, but they don't actually, they will generally never tell you that any of the internals of how that bit stream is read and what, what that bit actually does. But, you know, you have to know all of this stuff about the components and you kind of just depend on the fact that the magical tool chain will give you the bit stream that will magically configure this hardware thing to do what you kind of told it to do through, you know, VHDL or Veralog. So other resources doesn't really matter, David. So what is the developer domain? Right? The developer domain is like, you specify the source code and the attacker domain is the bit stream. So how do you reverse a bit stream? Well, let, before that, let's go back. Right? So as an attacker, you know, you don't have access to any of the information about the hardware description language. You don't actually even know how the various blocks are routed. All you have is this, you know, end of the result bit stream, like literally a stream of bits that is fed through the FPGA. And magically, like I said, you know, that bit stream makes the FPGA do what you think it should do. But as an attacker, all you have is the bit stream. So you have to reverse engineer the bit stream, you know, like you would reverse engineer a piece of binary with the machine instructions and then figure out, you know, what the logical constructs is in that massive bit stream without any documentation from the vendor. And again, you know, each vendor will have a different format, right? Each type of class of FPGA from the same form of vendor will have a different format. So the rules are literally changing on you underneath your feet. And that's what we have. Okay. So, so we wanted to reverse the FPGA, right? And there is some people have tried it. Jbits was one thing which Xilinx released in 1999. It allowed you to like change logic. But then they started to start supporting it. Then Bill, which it requires a net list, it was released in 2012. And the guy did say that it's very, it's like impossible to reverse engineer. And then 2017 Bitmare, it also allows you to like move logic, relocate logic. So by reversing, you know, we mean you take the bit stream. And then the goal for the most part is to be able to recover the hardware description language and underlying logic that went into generating that bit stream. And once you have that, and if you did that 100% correctly, you might be able to modify that logic to have it do something different than its original design. And then you will have to run the whole synthesis tool, right? Regenerate the bit stream. Well, you know, redo the optimization routing and then regenerate a bit stream. And if you did that 100% correctly, again, without knowing any of the rules about how those bits are actually used for that FPGA, then you might have a hope of, you know, taking something like the bit stream that we found in the FPGA that we think is implementing the trust anchor and then altering its behavior in a way that it might allow us to bypass, you know, the trust anchor capability. But the chances, you know, people doing this on anything but a very simple FPGA is very low. In fact, I'm not actually aware of anyone who has successfully done that entire path on something like a commercial, you know, like a real world commercial application of an FPGA on some commercial, you know, bit stream. So chances that working very low. Yeah, let's talk about the pie. All right. So like I mentioned, you know, on the FPGA you have complex logic blocks, right? So this is where, you know, your hardware description language with the logic gets configured. Then you have pins, right? Pins are literally those little metal things that stick out from the chip, right? That you wire to other metal things on other chips. So, you know, we're humans, we like to design things on, you know, few layers of PCB, right? So the number of pins we want to keep reasonable and the largest, you know, Spartan 6 FPGA has something like 560, 76 pins, right? So, you know, that's a lot, but that's a very small finite number compared to the logic blocks which go up to about 200,000. And in each block you have a whole lot of, you know, like smaller logic blocks. So, you know, we have about no more than 600 pins and we have 200,000 CLBs and, you know, standard sort of approach to FPGA reversing was to, you know, recover all of the logic from all the CLBs. And we looked at that and we said, well, you know, we're not FPGA people. Like, we're, you know, this is too complicated, right? Like this probably won't work. A lot of smart people have tried and haven't really gotten far. But why would, why don't we look at this? You know, they're not that many pins. So if we can figure out a way to deterministically reliably manipulate the pin, who cares about what the logic is doing? You know, if we can disconnect the pin, right, and hook it up to something else that we supply, then it's one of those things that like, if the FPGA falls in the forest, like, did it really happen if no one hears it? No, right? Because, you know, if we take a pin disconnected from its logic, and that pin is still connected to the rest of the circuit board, and then we bypass that logic, right? So, and this is an actual plot of the Spartan 6 family, the largest, the most dense chip has, yeah, 576 pins and 150,000 logic blocks, right? So if you put that, you know, actually in scale, the graph actually looks like something like this, right? So, and this is not even to scale. That curve grows really fast. So one of our intuitions at the beginning is, forget about reconstruction, you know, reconstructing logic. It's too hard. Let's look at, you know, a way for us to reliably manipulate the pin itself and see if we can do that. So, also, you know, BitStream does optimizations and there are no optimizations in the IOB. This is one of the logic complexity we are looking at. So this is a SHA-256 we wrote, and this is a very small part of that logic. It has, like, thousands of CLEs, but what is interesting is this thing, right? Like, this is the binary representation of the BitStream. The top part represents the complex logic block, which is implementing the Boolean functions, and the seven, yeah, six of those are basically the IOB blocks. Each IOB block is eight bytes in Spartan 6. And if you can control, if you change a little bit in this, you know, like a byte here, you can change the value of the logic of the IOB from 0 to 1 or 1 to 0. And you can, you can also change, these eight bytes represents the characteristics of this IOB. So you can convert it from, you know, like input to output. By just changing one parameter, the number of rounds in the shot of 56, look at the logic changing, but the IOB remains the same. And this is the depth. So the only depth was actually in the complex logic block. It was not there in the IOB. And this is very important. So as I said- Let's back. So the thing that you look at is an actual graphical rendering of the actual BitStream generated from this little mock example that we created. Uh, and if you can go back one more slide, right? Yeah. So the IOB doesn't change at all. It's in very predictable places. We know where the pins are because they're numbered, you know, like one after another, right? They're designed on the PC board and they don't get to change because the PC board is, you know, in hardware. And if we figured out, like which eight byte construct, um, describes which pin, then we should be able to very predictably and reliably, uh, you know, change the behavior of that pin. Uh, and we have a much more hope of doing that than trying to do RTL reconstruction on the logic block. Okay. So as we said, like, uh, we can change the output from, uh, to, um, you know, we can assign the value zero without depending on the, uh, the function F. Uh, then similarly, output two one, change output to input. Uh, and the same thing goes with the input. So, uh, what we want to say here is that, like, you know, the vendor is not, vendor should not be dependent on the obscurity of the FPGA because as we saw, RTL reconstruction is hard but not changing IO. So how do you reverse a BitStream? Right? We see BitStream as a firmware and it can go with unpack, analyze, modify, repack. And it has, you can talk about, like, it has encryption but you can probably do, uh, you know, like, uh, uh, site-channel analysis to figure out the ES keys, you know, you can do fault injection to change, to basically skip this check. And then you can also do photon-emotion analysis. Uh, so our development board, which we chose to do all this research was Spun 6, uh, LX45T, uh, uh, is the model of the Spun. And how do you do on the unpack? If you read the documentation, you can do the unpack. Uh, which is basically, like, find the sync word, figure out which, uh, kind, what kind of, um, uh, Spartan it is, uh, downloads, uh, register in description language, parse it, and uh, find the registered FDRI, which specifies the hardware circuit for Boolean functions. So, uh, then we move to analyze. There are multiple types of logic, uh, uh, type zero pa- frames. The first one is the, uh, complex logic block frame, which also contains IOI, uh, um, you know, like how to configure a BRAM and all that. Type one is BRAM, the actual memory, what it contains. Uh, and the type two is the IOB itself. And all of these, which is very important, is all of these are serially laid out. So if you figure out the range of, uh, of each of those, then you can do analysis and reversing to figure out which, uh, pin is represented by which bits. And, you know, um, the main important thing here is like, if you figure out the device layout, you can exactly pinpoint which CLB is used, which BRAM is used, and which IOB is used. And each, you know, these are details which you can find in our, uh, github. Uh, and, uh, major represents, so it is a two, uh, SRAM is basically a 2D, uh, array. And, uh, each row has multiple columns. Each column represents one resource. So, you know, like a CLB, a memory, uh, slice MCLB will be one column, slice L, CLB will be second column, BRAM will be third column. And, like that. And similarly, each, each column has its own array of rows, uh, which defines which flip flop is used, uh, uh, which MUX was actually, uh, encoded. So, uh, we, we actually analyzed the BISTream and created this really cool, uh, uh, visualization. My team has really worked hard on this, uh, tool. Uh, as you can see, it specifies the resource utilization, uh, which column is used. You can see, this is for, uh, the Mojo board, uh, which uses LX9. It's a really cool devote. And this represents the ASR BISTream. So, you, you can see, like, which, uh, uh, block, which resource represents the PCIe, which resource represents the LCLB, and so forth and so on. So, this tool, can you go back, is, is Unkid help right now? It's amazingly cool. You should totally check it out. Because, uh, you know, you can, all that stuff that Jadon said, you can read maybe 2,000 pages of Xilin's documentation and try to figure it out. Great. Or you can use this tool which has already done that. And not only that, it will show you, like, every, you can pinpoint to any piece of the binary, and it will show you what that binary, what, what that chunk of binary represents. Is it an IO pin? If so, which pin is it? Right? Is it, like, a CLB? If so, like, what is the raw data in that CLB? Uh, and for us, you know, we, after reading the, the patent, knew that, okay, the trust anchor does a whole lot of magical deity stuff, right? And then if it doesn't like what's happening after 100 seconds, it will literally assert the reset pin on the Xeon processor and physically reset the processor. So that is how the mechanism we think worked. So if we can find out exactly which pin the FPGA is using, uh, for that, we can do our thing and disable that pin and make sure the trust anchor literally cannot reset that reset pin and then we win. And, uh, the next thing is, okay, so because the pins are set in, in serial order, uh, we know exactly what part of that binary it ought to be. So the only thing that we need to know now is which pin, uh, actually is used on the FPGA, which basically involves tracing, you know, the, the PCB on this router from the CPU reset pin to the FPGA, which is very simple. Um, we have only five minutes left, so I guess we're gonna go to the demo, but, uh, this is how the encoding works, if you know the range, figure out how, which pin represent, which bit represents which pin. And, uh, you know, all this code you can easily read. Um, we have a paper coming out in Woot, we have a lot of documentation at GitHub, all the details are there. And, uh, one thing I would want to say is that modification is really easy. Uh, they have like implemented this 22 bit CRC for single event upset, you know, uh, and, uh, that is also reverse engineered. You guys can just check out the GitHub, really have to go. But, uh, you can also disable the CRC. I haven't tried that, but, uh, you can just disable using this register. Uh, and then I want to, uh, the demo basically contains, uh, we have a bit stream which enables those four pins and we're gonna, uh, just turn this pin off, uh, using the tool which is currently present again. And, uh, as on set, we basically figure out which is the reset, uh, which pin is controlling the reset pin and you disable that. And really cool thing part about this is, like, uh, there is no patch available. Once it is allowed, we disable the emulation of the FPGA so that the Linux code actually cannot update the FPGA anymore. So that's pretty awesome. Uh, and, uh, I also want to talk about, uh, Rick, uh, found a really cool way as Aang was saying, like, there are 600 pins. In this case, there were 296 and, you know, our, uh, automated extraction bit stream tool, who's not here right now. Uh, so he really, he came up with a cool idea of JTAG scanching. Uh, as soon as the fan comes up, uh, he checks the state of all of those 296 pins using JTAG and we found 10 pins. And one of them was the one. And, you wanna? Sure. Okay, so this is where Brian comes in, you know, after years of investing in this, uh, we just couldn't make it work. So we got an intern. His name is Brian. And I said, Brian, you touch all the pins, you find out where this thing connects. And he did it. He's great. Thank you, Brian. I mean, without him, this work would have not been possible. Because, you know, we need to trace through, like, the six, uh, layers of the PCB in order to find, you know, which pin goes from the reset pins to the FPGA. Uh, and then, well, you know, he touched the wrong pins. So, feel for it. Gross. Right? It was just fine. I mean, it was a very hard job. So, uh, but then, you know, we were able to make some progress, uh, and we were able to find that pin, right? Uh, and the last question is well, this is cool, but how are we gonna do this remotely? You know, if it's not remote, it's not really interesting, uh, because if somebody already has root, uh, on the router, then, okay, you can change all sorts of stuff. So, we said, well, you know, it's 2019, finding a remote, you know, code execution of all, and then privilege escalation is gotta be really hard. I mean, people have been looking at Cisco routers for quite a long time. Uh, so, man, like, do we, can we even do it? Uh, yeah. So, they have a driver, basically. That's how they use it. And, uh, we also did a lot of fuzzing on all these protocols. Uh, while this was going on, James come in. Well, you know, so we're like, man, this is gonna be really hard. Who knows if we're ever gonna get this to work? And then James, who's sitting over there, said, okay, guys, I got root. It was great. And then, you know, it's easy, you know, like, he found, uh, command injection vulnerability, CSRF, vulnerability on the Cisco routers, uh, which are passionate. So, let's just point that out. You know, he found this kick ass remotely exploitable, uh, you know, command injection in 2019 on a, on Cisco iOS. So, you know, for those who say command injections are gone, it's like, you know, it's not true. Uh, and this was a very simple, uh, vulnerability to exploit. So, man, uh, this was great. So, if you couple any kind of code injection on any part of the tax surface of any Cisco iOS, uh, you know, platform or device that's affected by this, you can immediately use that escalate privilege and then update the firmware, uh, and well update the, the bitstream and then disable the, the FPGA, right? The trust anchor. So, you can take a remote exploit, couple it with this and make permanent modification of the hardware that is impossible to undo, uh, using software again. So, with that, let's, um, we broke some more, uh, and then final cost, $50,000, uh, the Fort is very large, uh, but as a result, we figured, finally figure this out. So, I'm gonna let John show the, uh, the two demos. Uh, we're gonna exploit the Cisco router and then show how this is done. And then, um, we're gonna also show the Moja board demo. Uh, okay, so, um, okay. So, this is what I want to show right now. This is booted, it takes like ten minutes to boot, so I have to boot that. So, this is the output of this ASR1001X right here, right? It's running the latest firmware. Well, one minus the latest firmware, because Cisco did patch this. So, what I want to talk about here is that, uh, look at this message. As soon as it boots, it says initializing hardware. And, uh, that says system integrity status. This status is coming from the FPGA. And the pre-RAMON, which we talked about earlier, is reading the status and printing it out. So, now we can go ahead and exploit it. We did. Yeah, we did. We connected to the wrong port. Uh, anyhow, never mind. Uh, sorry guys. This is part of the demo. Well, okay, it's going on. So what it is doing right now is that it is doing James Hack, uh, getting the root. And then we, uh, took a driver which was Quack.co. I don't know why they wanted to quack it. But, uh, so we took, we hijacked the driver. We understood this, we reverse-engineered the CPLD driver, understood how to emulate, uh, how the FPGA emulates the SPI, uh, flash for the, uh, Linux, uh, OS and then, uh, basically applied the patch. You only require around 15 bytes of patch to disable the complete, uh, update and also disable the reset. This is going to take some time. I want to go ahead and, like, do the mojo demo. So, we're watching as a full remote exploit of Cisco AOS, uh, followed by a full modification of bitstream bypassing trust anchor and then permanently disabling any future updates to trust anchor via the FPGA with this bitstream, uh, you know, modification. So, that being said, right, look at all, think about all of the devices that's affected. This thing has been in deployment since 2013. Uh, this vulnerability is in all those devices. So, if anyone finds any code, uh, any kind of code execution in any part of that, that attack surface, uh, there is a chance that this is not only has affected your router but is persistent, uh, to the point where, in order for you to get rid of it, you probably will literally have to desolder a chip, right, from your router or your switch in order to test, even to see if you've been exploited or not. Um, okay, so, uh, as you can see, this is a mojo board, uh, and yeah, can you put the light on it? Yeah. So, this is a mojo board. You will Google it. And, uh, what matters is like those four lights, right? And, uh, those four lights are representing one on those pins. And we're gonna go ahead, run our tool, which I'm gonna run again. Well, so right now, you know, the lights indicate, you know, the pieces of logic that we've put in, very simple to understand, you know, like one, one thing says, okay, uh, you know, pull, like connect this to logic that always makes the pin zero or LED zero high. So, uh, what it's doing right now here is that we, uh, took the tool, we disabled the pin and I'm uploading it right now, the repack binary, which our tool did, uh, the GitHub link. And, uh, it's gonna go ahead and we wanna go, can you reboot the router? So, let's look at what was the integrity status. Again, this is on pre-RAM on. We have full access now, FPGA cannot do anything and here it is. We have updated the FPGA. It's still booted. You cannot apply any more patch. I have 20 SPI flashes right now on my, so I have to like literally take it out, update it. Uh, like this is where you have that. That's very good. Come on, guys. Like this is really cool though. Right? So, now we've left the output, right, of this thing saying that something is wrong, right, in Raman so we can demonstrate, prove that, okay, so the FPGA is doing its job, it just physically can't reset that processor. And now we can actually patch Raman and pre-Raman to make this little message go away, right? So, but for the purpose of the demo, you know, we're showing that this thing is trying to cry out for help, but no one is coming. And we're gonna wait 100 seconds also while they show the other demo. Uh, so as you can see, uh, there were four lights before and we have disabled the one light which was on the, uh, at the end. So again, try it out, play with it and, uh, what we wanted to talk about. Yeah, and the reason why we used the Mojo is that, you know, this is a very inexpensive device that has, you know, the Spartan FPGA inside. So, uh, if you take a look at our tool, right, you should be able to, you know, modify this yourself and play with all of the different options of taking, you know, pin-high, pin-low, input to output and all of that good stuff. So, you know, I think we're, uh, about out of time. Yeah, this is the impact. Well, right. Cisco routers are used a lot in a lot of places. Um, but so we do make a lot of recommendations, uh, and a lot more detail on the technical side of all of this and the implications of other things that's not just a Cisco router, right, in our, in our paper, which is coming out in, in a day, I think, or two days, uh, at Usenix Woot. So, you know, we don't have time to talk about all of our recommendations that we have for, uh, you know, improving FPGA security, right, making it as, as good as it can because fundamentally this is something that is stuck in hardware and I don't believe that this is something that can actually be fixed via a software patch. So, please check out our website, uh, our code on GitHub, read our paper, read the fucking paper, right? And, um, they got 20 more slides you guys can look at it later. I don't know, I think we're gonna get kicked out. Oh, in order to reward you guys for, for coming to the right talk, I'm gonna throw out some t-shirts. I mean, who else is doing that, right? There you go. You guys made the right decision in life and you get awarded for it. Yay! So, all right. Well, anyway, thank you very much. Uh, that's our talk.