 Let's give Jay Legorio a big welcome, first-time speaker at DEF CON. Come on! Come on! Come on! Man after my own heart, we're digging into reversing here, breaking apart some encryption. So let's go. Good evening DEF CON and happy 30th. And I'm sure this has been asked like a dozen times today, who's first DEF CON? Oh, that's awesome. Okay, that's great. So welcome. Glad you're enjoying it. There's a lot more for you to see tonight. Hacker Jeopardy is here after this. I think it's in a little bit, but that's the next thing. So let's get this started. This is, this talk is called tear down this XI wall, breaking open the XISIL encrypted firmware. So we'll talk about how I got into this, into the different analysis that went into it. There's hardware, software, some firmware stuff going on, where that gets us later, and then what's left for other people to do after they take up this mantle. So, who am I? My name's Jay. I went to school at UMBC in Maryland, and then I went to school again in California. And I have a CISSP and some various SANS certs that I'm sure have expired, but they were fun. And when I think about myself, I think like I'm a Windows developer, so you're going to see a lot of Windows stuff in here, even though this is very much a Linuxy talk. I'm a security consultant, and I'm also a licensed private investigator in DC. And although I'm a first-time DEF CON speaker on the main stage, I've spoken at Hackfest in Quebec. That was super fun. I got to talk about making artificial pancreases at DEF CON 27 in the biohacking village, which by the way is not a good village. It is an excellent village. So if you can get over there, you definitely should. And I got to do sort of a re-up on that talk with some updates at KernelCon 2, which was in March of 2020, and very suddenly virtual. But boy, did they pull that off in like two weeks' notice. It was good. And yeah, I'm an unrepentant nerd. So it was December 2020, and we all just got through the first year of the plague. And things were looking up. Vaccines were just over the horizon, and things were looking a little hopeful. But then like usual, it all got ruined because I got an email. And so the text here is kind of small, but what it says is, hey, where's ISIL? We make your firewall. And we know that you diligently updated and patched for the last quarter, just like we ask you to. And we know it's only been two weeks since you installed that patch. But we've got this thing called an undocumented user account. And this is the fix for it. And so when I think undocumented user account, I think they just fucking backdoored all my firewalls. You've got to be kidding me. And so, but maybe it's not that bad, right? So you go look up the CVE, and you go look at the research, and yeah, yeah, that's exactly what they did. So that was super cool. If you've got the username ZYFWP, and the password was out there, you could log into any ISIL firewall of this series. And that kind of seems terrible. So all this is bad, right? But I'm nosy, and I want to see the bug. And even though I've not really gone into reversing the ISIL firmware before, I've always kind of thought about it. So I figure, why not? This is a great opportunity. And it's not like I'm doing much else. So even though it didn't affect me directly, because, and I'll quote from the article, the account seemed to work on both the SSH and web interface. I don't expose those to the internet. But what's to stop someone from getting into the network a different way, and then turning around and enabling more access via my backdoored firewall? So again, all this seems pretty bad. And what makes it more interesting is that at the time, there was 100,000 to 300,000 potentially affected devices. So with all this, let's take a look. And I assume it'll be easy, because everything that I go into starting with seems like it's going to be easy, and then it definitely isn't. But what do we see here? Right off the bat, PK. That's a really good sign for us, because that means it's a zip file. So great news. Let's open it in 7Zip. But 7Zip says there's a password, which is weird, because I've worked with these firewalls for a while and never had to know a password. So we've looked at this for about five minutes. What do we know so far? We know it's a zip file. We know the file's password protected. We can see the files inside. That's how zip crypto works. And some of those files are available online, we think, maybe. And the update process doesn't require a password. And the other interesting thing is that 7Zip says there's a trailer. And I don't mean a trailer like, oh, five or eight bytes got added on to the end, because it's some weird download thing. That's 124K. That's not unintentional. So what do we do first? Default passwords. Zaisal has been raked over the coals for Zaisal 1, 2, 3. It was the first thing I tried. It was the first thing not to work. Tried a few others that other people suggested, and none of that worked. So we can see the files inside the archive. Maybe we can do something with that. And what I tried next was called the zip plain text attack. And this is not novel. It's not new. It's from 1994. And what you can do is if you have one file from outside of the archive, you can derive the key stream for the rest of the files in the archive with, like, math and magic. I don't know. It's not my thing. But I tried it. And yeah, it did not work. And one reason that it didn't work, oh, no. There's nothing at the FTP site. It's just gone. My go-to for years for files from Zaisal is just gone. And what happened was between when I submitted this paper and got accepted, they took that service offline. I don't have that to show you because it's gone. And it's replaced with a thing where you need to log in and give them a serial number and register. So all of those files are unavailable. But maybe somebody else has a copy. So we go to archive.org. And it's not, I mean, granted, great, great, folks. It's not the most updated thing. This is from 2014 in the June timeframe. But let's at least give it a try, right? It's something to go on. And it's better than what I got. So we can use something called BK crack. It's available on GitHub. And it didn't work. And I'm pretty sure that what I had was just the wrong files because they have to be, you know, byte by byte identical in order for this to work. So I have one more strategy to try. You all might be familiar with it. Our very last resort, password cracking. And I tried lost password if you've used it. It's very pointy, clicky, easy to use. It's great. And John the Ripper, which is less easy to use, but really fast and really good if you can get the hang of it. And they were both touchy about the trailer. But it gave me a couple of days to brainstorm. And when I was done brainstorming, I just felt depressed. So this doesn't feel great. I don't really know where to go here until I realized what I have was a chicken and an egg problem. So I've gotten encrypted firmware and a device that knows how to decrypt it. So somewhere the firmware is decrypted and it just so happens that I'm a hoarder. And so I have the previous generation of this device on the shelf so maybe we can bench it and pull something out of it. So now we'll move on to the hardware. This is the original device in the rack. Not going to win any cable porn, you know, awards or anything. But there it is. It does its job for the most part when it's not backdoored. And so this is a typical MIPS box. You can see a serial port up at the top. You can see the ethernet ports down at the bottom. And where I've highlighted in red is a separate board where the storage goes. And this is the top view. It's your standard Samsung NAND flash chip. And yeah, we're just going to bend that back. And I cut the hell out of those pins. I hope they weren't needed later. And what we've got is a 10 pin connector. We've got this Fison PS 2251, which might look familiar to some of you. And yeah, whatever the hell this is. But it turns out that the 10 pin connector is the same thing that you'd see on your regular desktop motherboard to connect external USB pins. And the Fison chip is your typical NAND to USB storage chip that you see these on thumb drives all the time. And so I looked for a USB hub that I really didn't care about because this was going to be a bet. This was going to be fraught. And so you just shove some pins into the header and solder them onto a USB cable that I just cut in half and hope the magic smoke doesn't come out when you plug it in. But the drive mounted and I had a disk, which honestly surprised the hell out of me. So you can use like Win32 disk imagery to image the entire drive. You get all of the partitions and you throw it into bin walk and then out pops the firmware in plain text, which is great. And so we're not done yet though because when I kind of set out for this, I didn't want the firmware from this one device that's now in the trash. I wanted to decrypt these arbitrary firmware files whenever I got them off of the Zeisel website. But now we have a starting point. So let's look at the software because now we have access to it, right? The device runs the Apache web server, pretty typical. You submit the firmware to firmware upload CGI and CGI dot bin. And the first thing you do with that is run strings. And it's got a really interesting one, ZLD FS extract. Looks pretty promising. So we'll throw it into Google and we get one solitary result. And it's 10 years old, so it could mean nothing. But it says USG firmwares are zipped with a very strong password. Okay, that's, yeah, I guess so. I mean the password cracking sure didn't work. And it says the ZLD FS extract command line verifies the zip and unzips it. Yeah, okay. It also says watch the console output during the update. And I didn't do that because again, like I just cut this thing off the board. So that wasn't really an option. But it looks like we're on the right path. Now again, I took the old device that I had on a shelf, right? The devices have the same processor architecture. The firmware formats are similar, encrypted zip, trailing bytes. And so there's nothing on the internet to say how to break into this firmware. So like, I don't know, from a vendor's perspective, if it ain't broke, don't fix it. So what's to say that it ended up changing for the new model, which is what I was actually interested in. Now the first thing I wanted to run on this binary ZLD FS extract is file. Just give me something to go on. And the thing that I really hated was that it statically linked and stripped. And that means that I can't find things like mem copy, stir copy. The normal library functions that you would want to see, because it's going to help you reverse engineer the rest of the thing. And maybe that doesn't hurt us so bad, but it didn't feel great. And the next thing that I ran was strings. And I got the best one that you could ever hope for, a usage string. This is great. It tells me where to start when I want to submit a file to this thing. And the second best news is that this thing calls 7za, which is seven zip. So again, feeling good, I'm on the right path. Now let's look at our other friend here that's going to help us with our software reverse engineering. And that's Ghidra. Now if you've been living under a rock, it is a free disassembler and decompiler from the national security agency, may have heard of them. And we can see a function that is building a command line. And part of the command line is 7za. Okay, great. That's where we want to start. And the other thing that it adds to the command line is dash p. And that's a really good sign because after dash p, that's where it's going to drop the password. So this is the function that calls the function that generates the password. Now I didn't want to try and emulate the entire device just because that seemed hard and I'm lazy, I don't know. And so QEMU user mode emulation was great. It jumped right into action. So the only little trick that I had to do was I had to grab a library that you see that .so off the device and just link it to somewhere else that was in my Windows subsystem for Linux build so that it could run. And then when you run it, you add strace to it so that you can see the different sys calls that it makes and maybe try to suss out some information. So speaking of information, my first run at this, you can see I added dash i because part of the usage seemed to indicate if I do that, it will just tell me some information. It won't try to extract anything. And I thought that was a good starting point. But just when I was going to go look for where it ran 7zip, I saw this here where right off the bat, we open the file for read and seek to 124k from the end of the file. Now I'm not saying I don't believe in coincidences, but that should look familiar. So I think we're on the right path here. Now again, I tried to run dash i because I thought it was going to build this command line and have it be 7zip the whole thing. But what I actually found was that I control that parameter entirely. So if you look at this exact VE call here, it tries to run dash i, and that means I can run whatever I want in there. And then after dash i, you have dash o, dash q, and dash p, and that really long string. So we could plug a tool in here that spits out the password since we have direct control of this parameter. And earmuffs, goons, earmuffs. I didn't tell the conference that I was going to release a tool today. So I'm going to put it on the screen, and it will be our little secret. This is the most complicated tool I've ever released. Put the camera down. Please. This is hard work. All right? We're going to print out the fourth argument. And it works. So you can feed the extractor any firmware file and quickly output the password for easy extraction with 7zip. And I think we got there. So that's success. And you can see the passwords there multiple times because instead of, I don't know, being efficient, it runs 7zip for each file it wants to extract. It doesn't just dump the entire archive. And those dots there on the left side, if you see that, that's what would otherwise be just progress, you know, progress dots, indicators. And yeah. So that's really all you had to do. And there's the command line to do it. You can pass any .bin file there, including the one that was released, I think it was like a month ago. Or maybe a couple months ago. So can we see the bug yet? No. It turns out the backup of my firmware got corrupted, which was awesome because this bug exists in one, but only one version of the firmware, not the one before it and not the one after it. And it's just gone. Zaisal really effectively memory hold this backdoored firmware from the internet. I feel really good about my Google foo. And it is just gone. So I kind of sat around and said, you know, what the hell do I do now? And I thought, I'd DM the researcher. His name's Neil to sync on Twitter. And he sent over his copy for the USG 40. And even though this wasn't the model that I had, that's fine. I didn't want to load this onto a real device or anything. And so I extracted it like this. And since we already know the IOCs, we can just grep for them. And it's pretty easy. So you can see we grep for ZYFWP. And the first two files are winners. Shadow.basic and password.basic. And remember, this is a firmware for a device that maybe has never booted before. So these will become your Etsy shadow, Etsy password files. And that hash right there, in shadow.basic, that's a good hash. You know, we're not looking at MD5 over here. And the other thing that kind of struck me was a match in Capwap serve, which is an executable. And that seemed a little weird. So throw that into Geiger 2. And what you get is a function that loads up this structure called FWinfo. And it plugs the username in ZYFWP. And then it populates the password field. And this password is otherwise really strong. So nice job, guys. But, you know, maybe don't backdoor all your firmwares with it. I don't know. That's just me. So continuing on, this is for someone else who might want to pick up that mantle. Now, the first thing I would do if it were me is I would make a Python script. And here is where the deobfuscation function is. And here's the function that calls it. I am bad at MIPS. Boy, I cannot read that stuff. So I leave that as an exercise to somebody who's way better than that. But I would turn the deobfuscation algorithm into a Python script. So you can just point it at the firmware file and get the password. And I do hope somebody does this because it would be way better than this Goldberg machine nonsense that I've kind of cobbled together here. And I did also mention that I'm not releasing a tool. But if I'm giving you some offsets and saying go forth, how are you going to do that if you don't have anything to start with? Because remember, I cut this thing open. Well, it turns out it was all way easier than I made it. So the firmware releases include the firmware files, the release notes, and the .ri file, which is a recovery image. And you can blast that over a serial port if you've got your firewall in a bad state. And it's unencrypted, so you can just throw it into BinWalk. And it's got FS extract, which is great. Now, you all have FS extract, and I'm not responsible for that. And I want to thank HN Security for pointing this out. It should have been obvious, but I don't claim to be good at obvious things. And so if you want to go that route, no firewalls need to be harmed in the process of your research. So here's their screenshot of the BinWalk.ri file. You can see ZLD FS extract is right there. And so, yeah, now we thank our firewalls for their sacrifice and protecting me from just looking at what was literally right there. And the other thing that I would do is I would run the user land image through EMBA. EMBA is a really great firmware analysis tool. It's still pretty new, and it's gotten really good even in the short time it's been alive. So it'll generate this really slick report at the end that shows everything that it found, and it can find a lot. So it started out as this text output only tool, and it would generate like 15 different files of all kinds of different categories of findings, which was good, but I mean it was dense. And so now it's got this really sweet web-based output that you can browse locally, pointy clicky, look at everything. And from our process above, it can do everything after the unzip process, including the automated BinWalk, the check against known CVEs, and even emulation of the target binaries for the whole spectrum of static and dynamic analysis. So let's look at what things it found in this older firmware. Now, the only thing I love more than Star Copy is Mem Copy, and the only thing I love more than that is a binary that'll call system and has one or two of those. So I think libnetsnmpmibs.so looks pretty interesting if I'm looking at the Venn diagram of what calls those things. And the other thing that it can look at is file permissions and just bad decisions related to those. So I see a lot of files that can suede to just whatever permissions they want, and some of them make sense, but others definitely don't. So I'll give you suexec makes sense. Zysudo.suede, totally, yep, on board, that makes sense. WebAuthera.cgi, that's a weird one, DynamicsScript.cgi, that sounds terrifying. Facebookwifioth.cgi, why does that need to be root? And the two that I don't really have any questions over is ping and trace root. I don't know why ping and trace root needs suede permissions. Maybe again, I'm not a UNIX person, maybe somebody can help me with that, but that seems like you're just asking for trouble. So the other thing, this is the last screen that I'll show that I really like is it'll catalog your CVEs and give you a little more information about those. So like it shows these three, they're known exploited. It'll give you a whole bunch of 10 out of 10s if you're looking for those, and it gives you the CVE number there. But my favorite part is if there is a POC out there, it'll point you right at it. So what this helps you in other products with is if you are otherwise running the latest firmware, but it's using old libraries that are exploitable, this is the dead giveaway that you might have something there to look at. And so it turns out EMBA on WSL might not be a long way off because I was working with the developer, Michael Messner, on how this can be done. And the older version of EMBA was really, really powerful and very good, but the newer version has a dependency on Docker. And WSL on Windows 10 has some manner of incompatibility. So it's not on them, but the immediate answer was to load up an Ubuntu VM on Windows 10. And that felt maybe like a heavier lift than it should have been. But they added some improvements, not only for that, but also based on the decrypted user mode images that otherwise would have been inaccessible to them without running through all this. And the next thing, or I guess the last thing that I'd pay attention to is when Zaisal starts changing things. So I love this diagram in a prior talk, and I couldn't find it again, so if you know who made it, please tag me on Twitter or whatever, so I can cite them because I really love this kind of paradigm here. Because in theory, we're over here on the left where we can get the plain text version of the format. And then there will be a transition. And then it will be encrypted again, and we won't be able to see anything. But to get here in the middle, the firmware they've published will have to understand the format changes going forward. And it will itself be vulnerable to this process here. And there'll be dead giveaway language as to indicate which version has the new stuff. You can only install version X before you can install version Y and beyond in the future, in future updates, because firmware before version X won't know how to handle the new update mechanism because it won't conform to the old scheme. And so once you trace the upgrade flow here in this transitional firmware, you'll know how to decrypt the new protection scheme, and for whenever that happens, I leave that as an exercise for the listener too. So this went by as I was making these slides, and I think stack smashing is presenting this year somewhere here, but I thought it good to point out. And I guess I've seen some bad firmware encryption, like static XORs, like 1 byte XORs, or maybe even 4 byte XORs, because they're super clever, or rolling XORs, none of which is encryption, but you get my point. But I guess what we have here isn't the worst, but man, if you're shipping backdoor accounts, I don't know, maybe Dave's not wrong, and it's just kind of window dressing. So with that, I say happy bug hunting in this firmware format and others. And just to kind of recap what we went over, I talked the tiniest bit about myself and how I got into all this stuff. We covered, you know, a few different methods of analysis to figure all this out, and what that got us, and what continuing forward for you could get you in the future. And before I take any questions, I want to thank my friends and family, and especially the DEF CON CFP review team, because without them I just wouldn't be here. And Neil's to sync for responding to a message from an internet rando, literally out of nowhere. And Michael Messner for EMBA, the author of every other tool mentioned, because I kind of, you know, burned through a lot. And I'll call them out separately to the Geiger project, because decompilers are free now, unless you're an American, because you damn well paid for that shit. So, and with that, I'll take any questions, but I thank you for your time tonight.