 all right we're gonna get started shortly all right it is now time for the final talk of Torcon 21 please welcome to the stage Jutton who's gonna talk about a really awesome way of ponying Cisco devices hi please let me know if I'm not here you cannot hear me because when I was sitting back it was very difficult to understand the talk so I'm Jutton Katalia I'm the principal scientist and architect of our defensive technologies at Red Balloon Security and I'm gonna talk about defeating Cisco trust anchor a case study of recent advancement and I haven't put Rick who's the second author in this talk but he's here and and on couldn't join today so I'm gonna start like in this project who were the cast who were the characters who helped me or actually together we did this work myself Rick Housley who presented I was a great talk Joey on James and Brian he is the most important part in this whole talk so it all started when we started thinking about like running you know hacking up a SR 1001 because we just wanted to and we found out that it was end of life but and Cisco released in 2013 is our 1001 X they both looked the same so it didn't matter and we just bought it and the main objective of the star of this whole research was to run our software and during this research we found a trust ankle one trust ankle vulnerability because when we worked on we have been working on Cisco artists for last 10 years and it hasn't been that difficult to run code on it you basically we're not looking for vulnerability all we wanted was to run our software on it and then we found out that in 2013 Cisco implemented this proprietary bootloader verification basically their own proprietary secure boot based on FPGAs and you know and this is the CVE for that and first I want to start with the impact of this vulnerability so when we were working on it we thought that you know we'll be only impacting 1000 series right and it will be just that but we impacted 900 920 6800 9500 9800 more oh wow there is no there oh three thousand wait for it then there are ASA then there is 9700 and then or recently they updated this this is actually there's a security appliances and I don't want to say it but they did not release this before so you know given that we thought you know like it's up we have affected so many products now it's around 140 or something so Cisco you know becoming like running around 60% of the internet the world's internet obviously not somewhere but the novel the novel technique which we are proposing doesn't just affect routing infrastructure it also affects weapon systems ADAS systems and automotive medical ICS so what is this right we we started with the hardware analysis and which Rick was a great help that if you look at this diagram right B represent the A represents where the firmware of the system is stored right 2 SPI flashes basically like your motherboard sees an intellect city 6 core and D is the South Bridge B and E are more interesting so ease the Xilinx FPGA which we didn't know when we started this project like it was the trust anchor and these where the bit stream is stored and if you look at the ASR analysis and you guys were late but I was here so I know what is UEFI and those guys did a good job in the morning so as most probably some of you guys are network admins that you guys know what is Ramon Ramon is a Cisco's you know old-timey proprietary bootloader through which you can boot different iOS images what Cisco did was you started using UEFI I think around 2013 when they shifted to this model and they implemented a pre-Ramon which what it does is that it manages Ramon it also helps in upgrading the bootloader managing the bootloader and which is implemented as an a PEI phase in the UEFI if you remember the chart from the morning and Ramon is basically the it validates the operating system which will be loaded later in the stage it also has a really cool memory inspection module which is disabled by default and you can you know enable I don't I don't know if there is any flag to do to do it but you can modify the SPI and like enable it and so what it does is that it boots the Linux kernel which is the operating system and the OS but we are forgetting about Cisco iOS the real iOS not Apple iOS what they did was they took they took the old iOS and implemented they literally took it and implemented as a demon in a Linux process and that's what Cisco is XC is the latest and greatest from Cisco so we started looking at the SPI and we looked at it there were no hashes no certs that should be really because what we want to do is just run our software that's all we care about don't care to mess up with Cisco so we modified the UEFI and we disabled some pre-Ramon checks and booted the modified one firmware everything worked right easy peasy done with the project but wait it reset it so why is that it exactly after 100 seconds the router resets and it shows like it tries three times to boot itself and then you know after three times it drops into Ramon no idea what it is so then we start with some hypothesis like you know Intel has some really cool modes like SMM VMM and so we thought like maybe it's of a hype there being a hypervisor which we couldn't see and but we saw that the VMM it was disabled and all of this was able to we were able to do because we had a hundred seconds to run whatever code we want which is bizarre so then we disabled the watchdog timers we thought maybe it's watchdog timers then we saw that the SMM was actually enabled we disabled it pretty easy and it was still presetting had no idea what's going on and then you know we we started with electro magnetic ammunition like we can hear Phil probe wanted to know what was going on and we saw this is just a hunch going on you know another Friday we were trying to do something and we saw that the FPGA was there was a there was some emanation coming out from the SPI which means that the FPGA is coming up reading its configuration and we'll talk about FPGA later on and then FPGA does something probably some kind of computation and then the circuitry comes up so some hunch that FPGA is doing something and then we also sniffed the SPI bus we saw some you know how we did we saw that like a micro loader is coming up you know the interrupt handlers the bio stuff the V bio stuff and you know basically the e thousand and the f thought e thousand to hex 10,000 range you know because you have 100 second you can do all the analysis you want to we also looked at pre-ram on we saw some weird checks this is the range where usually in x86 architecture external devices are mapped and if you saw the Rick stock you know these are basically your memory mapped Ios and just for the sake of sanity we hijacked the first instruction wrote our own serial driver and we saw that the system was literally shutting off so what kind of entity is it like there is some external entity I don't know whether it's you know Jesus creation or Buddha no idea there is someone who's doing it and it was obviously Xilinx because it was sitting right next to the SPI bus so obviously it's a bus master some kind of so this was the this was the you know hunch and that device powers on FPGA reads itself configure configures itself passes the you know validates the whole SPI which is basically the pre- Ramon Ramon Linux OS IOS T and does the whole chain of integrity checks so FPGA reversing is hard so I thought you know like why not just buy it and like find the reset pin but these things even on sale they are expensive so 11 grand we went ahead and we found the reset pin we had to destroy $10,000 and this is what $10,000 looks like so we started it we found the dollars and then I went to Rick you know like I tell reconstruction is really hard I don't want to do it what should I do I want to reset this pin I want to keep this pin high I don't want it to go down and Rick said you know why not just put a 10 km and like should be okay what what bad could happen $20,000 gone resistors resistors cost $1 per room and this is how it looks in $20,000 so Joey Joey basically you know great intelligence person basically said you dum-dums there is a patent which basically talks about the exact same thing where Cisco filed it in 2012 and FPGA acts as a bus master validates this is actually what Open Titan is also doing if you guys read recently and what they are doing is you know they validate the fact the FPGA validates what performs some kind of computation validates the certificates go or validates a hash and you know sends the microloader to CPU and allow the CPU to read the bit stream the firmware from the SPI flash just not that after 100 seconds of reset the way we could tell and we did a lot of our analysis was that the fan was going you know fan used to make really high noise that FPGA has tripped the system so and now from this from this patent we were able to understand like it was controlling not just the SPI bus but also all the other hardware peripherals so and we left the project around in 2017 came back in 2018 down $20,000 I went to Ong and I said I can do with this I can basically reverse engineer FPGA and he said explain me FPG is so this is FPGA basics for humans so what is an FPGA right it's a it's an IC that has basically the advantages of both software and hardware right it can do computations with the hardware and it can be configured like software and to explain you what it is it's just you know so the developer provides you the HDL which is basically some kind of it's a language and you write code in it the vendor tool synthesizes it and then the mapping and the routing can be done by either the vendor vendor toolchain or by the developer itself and then you take you know all those two tables B ram initialization think about your BSS and you know stuff like that if you're familiar with of you basically encoded into a binary which is called a configuration bit stream with which how you can program the hardware inside FPGA and there are multiple types of FPGA as frame base which is Cisco which Cisco uses in this one you have to configure the FPGA each time the system reboots the second one is the flash base in which the configuration bit stream is living inside the diet cells so it's less power consumption and it automatically like quickly boots up as with as compared to SRAM where you have to read some volatile memory and configure it and then add diffuse I don't know why it's called FPGA because you cannot reconfigure it in here at all but what really is FPGA right it's a it's a combination of blocks different IP blocks which can do something but that's what FPGA is and you know those blocks I will come ahead but I'm a mathematical person a system person what is FPGA is Y is equal to FX this is what FPGA is if anyone talks about FPGA and talk a lot about and zap zap it's Y equal to FX and all you care about is changing Y or X you don't need to worry about F and that's what our research shows to complete the tutorial IOB block is basically what drives the signal zero and one right it's a it's just a pin IOI is what tells the pins from the rest of the logic the F of X that okay drive it I have come I've computed the boolean function now this is one or zero B RAM is just your RAM right you can configure it different ways and the CLB is the element which can which does the boolean functions right it consists of two slices you know it has flip flops look at tables and you can also act you can make it as a storage component also so how to reverse engineer FPGA this is all I need to know right and there has been some previous work done you know you can read about it but the main the main idea here is that there are hundreds of tens of thousands of CLB components in FPGA and they are you know they're way too complex like why do I want to go ahead and do this right like why do I want to like reverse engineer this because I'm a lazy hacker I want to get things done and the IO pin count usually doesn't increase right this is Spartan 6 family which is what Cisco has been using and if you see it the CLB count keeps on increasing but the IO count doesn't like it stays you know 10 15 linear constant action so this is the shot 256 algorithm like this is the complexity we are talking about this is just CLB's and look at the routing in network here you don't want to go ahead and you know reverse engineer that and this is what I understand pretty well is that's the CLB encoding and so this is the shot 256 and this is how you can represent what the bit stream contains so that's the logic all the white ones are ones and you know all the purple ones are zero this is what the configuration of the CLB's look like and you look at the IO B it's down there and it remains the same change a little like I think one round or one constant and this is how it all changes and if you diff them this is basically it you know it validates a theory that all we have to do is like go after y and x because now we can apply it uniformly to different bit streams just changing those wise so there are few modifications scenarios here we want to change we're going to take the output change it from one to zero zero to one configure it to become an input and same thing goes with input one and then reconfigure it to make it output so these are the config these are the modification scenarios and what matters is that you know like even though what vendors have been you know trying to do is they are relying on the obscurity not like actually applying a good model and security and even though the article construction is hard changing I was actually pretty easy so let's go ahead and like how we do the bit stream reversing right there is that's how we apply our firmware reversing you know you can bit streams are encrypted you can apply different kind of glitching attacks you know you can do side channel analysis to figure out what the encryption is the key is and you can basically you know use it to decrypt the whole bit stream and this is the development board we use to do our analysis and how you unpack you read this document that's all you have to do go into this go into this go to this link you know I don't have to explain all of this I'm running out of time but the this this actually we have done really good work in this and it's very modularized I would really recommend you guys to go ahead and add LTERRA and different families for Spartan and then how to and to look at the analysis there are three types of frames one is configuration logic which basically contains the you know the how what does the CLB does then there is VRAM what the memory contains in the in the bit stream and the IOV which we care about and there are certain device layout which matters you know that's so if you go into the repo we have specified Alex 9 and Alex 45t of the Spartan 6 series each frame is 130 bytes you can think of it like a matrix structure there is rogue columns and columns itself is a matrix and you all you have to do is like figure out what is the major info about that specific device you you just put it in a JSON and our tool basically passes it you specify the minor this is how our tool looks like I'm not a JavaScript person Alex from our team actually did a really good job in this it shows you the resource utilization this is one of the demos and it exactly picks out which CLB which type of CLB was being was being used where in VRAM the data is and this is how the Cisco analysis look like if you if you look at it right there are multiple CLBs which are doing some stuff there is PCI stuff there is GTP trans receiver a lot of stuff again how to do the encoding look at the link it explains you how to modify it look at the link and how to free pack they use some weird 22 bit CRC for single event upset during flashing they you know like all you have to do is just look at their software and will tell you how to do it and and then you know using the side channel says you can figure out the key and you can encrypt it back and there is also a register which allows you to bypass the CRC you can try that and there is a there is an example in the repo which explains you if you buy a mojo board which is Alex nine it and you have four pins on you can turn it one off and it has been left to the reader how to determine after this amazing tutorial how to turn it back on so now that this knowledge I went to a wrong and I wanted I got $13 so and now the you know idea was on the pin on the ASR but there are 246 296 pins on the ASR and the Alex 45 t and I went to Rick again and I said how to figure out I don't have that much time because it's a it's a literally a flash you cannot update it and you have to take off the flash each time anything bad happens because you know it resets so it takes around 15 minutes to retry the whole thing one pin so he used JTAC chain to find to find 10 pins out of 296 because remember the fan which was making the noise these were the 10 pins who change state after the fan noise was made so we focused on those 10 pins what's there I don't think these 10 pins were used not sure but then we Brian this is where Brian comes in automated bitstream extraction and testing framework so he was supposed to test 10 pins but in worst case 296 pins great help and in the testing course we lost another $30,000 at the end of the $10,000 so this is how it looks like and then we finally upon the end you know like we were able to run modified firmware because we were able to disable the FPGA to send a reset signal by reversing the FPGA bitstream but now the idea was like how can we go towards you know go to Cisco and because they're gonna say that how can you do it remotely turned out they actually update their trust anchor from Linux which is supposed to be immutable and so there is a driver which they use called Cpld.co and they're also a driver called Quack.co we hijacked that updated it, reverse engineered the Cpld and boom like you can update the bitstream from the Linux kernel now we need root to get into Linux kernel because when you log in you are in this you know like iOSD demon template shell and you have to do some kind of privilege escalation to be there and I started writing protocol fuzzers you know it's basically all their protocols which have been there for last 20 years so there must be some bug so while I was doing that James came in and James said you know they are actually writing they managed the iOS process using another process written in Lua so he went ahead he figured out two CVs which got us root which was basically command injection and CSRF bypass and then final cost was not 30k 40k because demo gods also wanted a sacrifice and this is how 40k looks like but later on actually Cisco said we're gonna replace all these routers to you but then we found out there was actually 50k worth of loss which we don't know how it happened they did give us two routers so thanks Cisco and what is the mitigation so I was looking at you know they released a patch after three months and what they did was in order to program SPI right like FPGA emulates itself as SPI and what they did was they are now not allowing the SPI select line to be selected so there is no way to update FPGA which I don't believe and in order to fix current ones they have to actually send someone to desolder the chip you know there is no if this has been exploited in the wild you have to desolder the chip and apply the patch and you know the select line is Keplo but the main problem here is that there is still a mutable root of trust and there is a way to update this and we know how to do it just need some time and more money so I want to show this demo which because I didn't want to I didn't want to bring another router humidity can cost $60,000 so if you look at this so this is when FPGA boots up shows like this you know shows the status of the system integrity and this is looking good nothing wrong in it it boots up you know validates does the whole secure boot chain loads of the IOSD and now we're gonna use the exploit to get root those are the CV numbers and then once you get the root we're gonna you know in smart or quack or KO which will send commands to you know SPI rat commands to the FPGA SPI and and we're gonna upgrade the Ramon which is basically updating those two SPI flashes which contains the bootloader which gives us persistence and this is so I you know we could have removed this because this all exists in the bootloader now because and we control the bootloader I kept it to make sure that you know to show you that FPGA did calculate the hash updated the status of the register which the bootloader is reading and it is supposed to now reset but we stopped the FPGA pin to become one from zero right and even though you know FPGA is supposed to be the master we have controlled it we can also control any kind of other pins which are going in we can you know like the way FPGA is reading the SPI bus we can control the SPI data lines and basically show it's perfectly you know like whatever detection tool they have built or you know if other people you know we others people like write something to figure out what's going on we can actually control the data lines and we can you know emulate write software and so to show you I have some picture the guy that guy is not here and we have you know it's upgrading the this is doing the bootloader upgrade because it's two SPI flashes you update one flash from the SPI and this is we have persistence and billions of dollars of research compromised so future work is you know there is FPGA I have some kind of compression you know they apply and it messes up with the layout of the system which we want to look into their hardware frugals you can you know there is no way to validate the fact that you know you can have authentication built into the FPG as these days but I think like recently F secure guys also found problems in those authentic authentication and confidentiality you know validating that so there are ways to add hardware frugals in it you can also use front end because you know the clock speed is like around 400 megahertz or something and this is our GitHub repo thank you all right well we have about 10 minutes to clear out of here but the lucky news here is that we have the beach luau going on right now there's tons of food out there go ahead on out there we'll be out there in a little bit and do some closing remarks out on the beach so please get the hell out of here