 Hi everyone, I'm James, and this is Zach. Thanks for coming to our talk. We're going to be talking a little bit today about our experience at last year's Pone to Own, what we learned, the research we did, and how you can hopefully do it too if you're interested. So I'm James Horseman. I work currently at Horizon 3AAI. I'm an attack engineer where I research O-days and days, build implants, tons of fun stuff. Before that I was in the government contracting sector for about seven years, doing a lot of the cool, same stuff. Before that I studied math and compsci at University of Maryland, and in my free time when I'm not hacking on computers, I have a wife and two young kids that keep me very busy, and I enjoy jujitsu in the little bit of free time I have after that. Hey guys, I'm Zach. I also work at Horizon 3AAI and I first got my start in this industry in the military. I joined the Air Force and joined a special operations unit, and they really fostered that hacker mentality and curiosity. So if you're looking to join a cool unit and see what kind of awesome cyber things are possible in the military, look into the joint communications unit, and also join cyber operations group. After that I joined government contracting. That's where I met James, and now I'm at Horizon 3AAI. And I have a wife, two young kids as well. My wife is here, down supporting me, so say hi to her. And in my little bit of free time, I like going to shoot range and also taking my kids fishing. What will you gain? So James talked a little bit about this. The four main takeaways we want you to walk out of here with are how to break down complex problems, not just in hacking, but in everyday life. A little bit of basic reverse engineering knowledge in software and hardware. A little bit about how the PONDA own competition works, and then lastly we'll be releasing our exploit POCs that help you dump credentials and engagements. So the first one, breaking down complex problems. If you were like me and you worked in large corporations or government, there's these really silly posters plastered all over the the walls of your office and one of them is typically this one, the journey of a thousand miles begins with a single step. And it's actually true and actually pretty useful. So one of the first things we do when we're breaking down hard problems is we want to set milestones. It seems simple, but breaking down really complex things into distinct individual steps helps you plan. And so to make this a little more concrete for the PONDA own competition, what we did was there's a ton of targets out there. You got to choose from routers, printers, cell phones, and so you want to be able to first say, hey, this target set like routers, we have experience in this. Let's go after routers and let's not worry about these more complicated things like cell phones. So picking targets, that's a milestone. Then once you pick a target, let's look into how we actually acquire one. What's the prior research in these targets? And you just go down a line, you set these milestones and help get a little plan going. And the second weapon we use is what's called the OODA loop. So all you military people out there probably already know what this is. It's called the OODA loop. It's the observe, orient, decide, and act loop. And typically you're trying to have the fastest OODA loop against your adversary because that'll make you win. But in this case, our adversary is a printer, so it's not that important. But really the most important part of this concept is getting feedback as quick as possible. And so feedback can be good or bad. And you want to experiment, but you want to move fast. And so you use the OODA loop to get feedback and progress from one milestone to the next. So how we apply this in reverse engineering is say the Lexmark printer has a web service. We want to build a client of some type to interact with that web service. And once you have a client, you want to now start to fuzz that web service. So you use the OODA loop to begin to get quick feedback and you use it to progress from milestone to milestone. It's awesome to have a teammate like Zach who is well versed in these techniques of setting milestones and using the OODA loop. Myself, I've definitely been prone to fall down rabbit holes. I'm sure many people in the audience as well, you can go to a certain spot of a target you're looking at and you can spend hours or even days on something that ultimately ends up being unfruitful. So using those techniques and having a teammate like Zach to balance things off of is immensely useful. So target research. So the known competition for those that don't know is a competition that's held a few times a year by Trend Micro that they give you a couple categories of devices to choose from. And if you can demonstrate successful remote code execution on those devices, you win a certain amount of money and you win a certain amount of points. So the one that we participated in last year was a small office home office themed. So some of the categories of devices you got to choose from were printers. The main topic of this talk we'll be talking about the research we did on this Lexmark printer. Routers, you could either attack the land side or the land side and the way they come up with these cash amounts are by perceived difficulty. So you can see in general the land side of a router is generally more difficult to attack than the land side. So the cash prize that you would win is a little bit higher. There is also some smart speakers and smart devices, again, getting up there in the cash amount because it's perceived as a tougher target. And then finally phones. We didn't even bother touching phones. We knew from previous research you'd really need years of expertise and experience normally to go after these hard targets evidenced by a $200,000 prize for RCE. So like I was talking about, one of the first milestones we said is actually acquiring one of the devices to test with. So first contact with the Lexmark printer. Acquiring a device is actually much harder than you think. The point of own competition actually listed this Lexmark printer, which was end of life and we could not find it new anywhere. So the only place that we found that we could acquire them is on Mercari, which is kind of like a Facebook marketplace style app where you can just buy from random people around the world. So we ended up finding one that was listed as fully functional. And before the printer even arrived, I got this message from the seller that asked me to please play as soon as possible because they have lawyer bills. So I was not inspired with much confidence that this printer was going to work out. And we actually found one more printer. There's only two in the world we could find. And this one was four parts, as you can see. It doesn't look very functional. But the game plan on this one was that it had an intact motherboard. And we were hopefully going to extract the firmware off one of the chips. So obtaining the Lexmark firmware, like Zach said, we had two printers to work with, one we were hoping we'd be able to test, and then another one for parts. When we started our reverse engineering experience with this printer, we first tried to download the firmware straight off of Lexmark's website for this printer. They come in these FLS files. And you can see it's listed here as an HP printer job language data. That's a pretty standard printer protocol. So it's a little bit encouraging. Maybe we should just get the firmware for free. But if you examine a little bit further into that file, you'll find it's a PJL command at the top there and then an encrypted blob. And that encrypted blob contains the rest of the firmware that we're interested in. So in order to get the firmware, or actually to decrypt the firmware, our plan was to find an existing printer, which we did, extract the unencrypted firmware from the flash, hopefully find the keys within that flash to decrypt the latest firmware. One of the rules for Pone to own is that your exploit has to work on the latest firmware, so we couldn't just operate on the printer and decrypt the firmware that came on those broken printers. So this is our first experience with hardware hacking for either of us. We both have a lot of experience in software reverse engineering and software vulnerability research, but almost none in hardware hacking or hardware research. And so I'm sure there's a ton of people out in the audience that have a ton of awesome experience in hardware hacking and some of the stuff that we did might make you cringe. But first we needed to buy a couple tools. So on the left there, one of the tools we needed was a basic iFixit kit. It has a whole bunch of different screws and screw heads and just things that make it easy to take apart different electronic devices. In the middle there's a universal chip reader and writer. And what that will allow you to do is to take different flash chips and then read or write to them. And that came with a bunch of different adapters, including the one that we needed to read the printer firmware. This one we were lucky to find on eBay for a few hundred bucks, knew I think these are over a thousand dollars. And when it came to us it was covered in like grease and grime and everything and I was kind of worried it might not work. I think it came from maybe a mechanic shop. I think you can use these to read and write ECUs on cars. And then on the right side there we have a heat gun that we plan to use to desolder the chip from the motherboard. So I took the broken four parts printer and I wasn't very gentle. I just ripped the plastic off because I know I didn't have any hope of putting it back together again. And eventually got the motherboard out. And in the lower left-hand corner there you can see the actual flash chip that we're interested in extracting and reading from. We only had two of these printers, one we needed to keep functional, one I had already destroyed to get the motherboard out. So I wanted to make sure that I was confident in my ability to remove the chip without destroying it. So I looked up the chip format and I tried to find some similar things that were maybe a little bit cheaper or easier to find to practice on. And the first thing I found was basic USB sticks used this chip flash format. I believe it's called the T-SOP48. It has a 48 pins. So I went around my house, gathered up a whole bunch of old USB sticks and some old routers. And I was practicing and I was able, after a while, to get them off pretty easily at pretty low heat. Unfortunately, after I was pretty confident, I was like, oh, I got this. I went to go do it on the printer and it did not come off. It did not work. I ended up cranking up the heat quite a bit. And you can see I burnt off some of the engravings on the flash chip and getting kind of nervous. So I called Zach on Zoom and I'm like, the chip isn't coming off. I'm not sure what to do. We talked for a little bit and ultimately, I said, I'm going to go give it one more shot, crank the heat way up. And then this time what I did was actually heat up the entire motherboard first along with that ethernet jack there that you see on the left. And in hindsight, that's obvious. You can look up stuff on YouTube that says if you're trying to desolder stuff, heat up the surrounding area. But at the time, that was not obvious to me. So ultimately, we were able to get the chip off the board, but I wasn't convinced that it was functional. I thought I probably fried it. So we put the chip into the chip reader. And like I said, I had all these adapters and I had the right adapter. I inserted it in the chip reader. But I didn't have any drivers. The chip reader came from eBay, and it just came in box with no CDs or drivers or anything like that. So I had to go online and try to download these drivers. And of course, all the websites that I found them from, they were listed by Chrome as serving up malware. And I couldn't find anything official for this chip reader. So I downloaded some of the ones that were marked as malware and said, I'll give them a try. I threw them in a VM. And then this is a picture I sent to Zach. I said, this seems a little bit sketchy, but hopefully it'll work. It was all broken English, very old Windows, Win32 style GUIs. And I was able to detect the chip reader, and I started trying to read from the chip. Again, it wasn't working. So I was worried. I was like either I burnt the chip, the chip reader's bad. I'm running it in a VM because I don't want to run this thing that's marked as malware on my host. All sorts of stuff could have been wrong. I was a little bit encouraged because as I was trying to read from the chip, it would give me error codes and say, these pins, like pin 32 through 38 are not detected. So maybe it's just loose in there. And I tried for probably about 45 minutes to get the chip to read and couldn't get it. So I gave it a break for a while, went back to talk with Zach, again, went back. And I had to hold my finger on the chip at just the right angle and hit read a few times and ultimately got a good flash read. So now we have the flash dump. And we need to take that flash dump and ultimately get the user mode file system for this printer. If we didn't mention it earlier, this is a Linux based printer, so it has a normal Linux kernel, Linux user mode file system. So the first thing that we did was take that flash dump and remove all the out of bound data. On these flash chips, they don't just store all of the bytes of your file system or your kernel. They also store this management data so the flash can do wear leveling and detect bad blocks. So our first script removed all that out of bound data. And we saw that it was a UBI file format after that. UBI is another kind of block management technology that allows the computer to interact with this flash chip and just read and write files to it without having to worry about all the sectors and the wear leveling and things like that. We then found this cool tool on GitHub called UBI Reader. We were able to use that to extract the volumes from this UBI image. And then finally, one of those images was the root file system for this Linux based printer. And it was just the basic SquashFS file system. So the normal Linux tool on SquashFS gave us the user mode file system. So once we had that file system, we did some reverse engineering, and we were able to find the program that was responsible for handling the firmware update. And after a good amount of reverse engineering, we found the RSA key is that this firmware update binary used to decrypt the firmware. And this is just a picture in Ghidra of those RSA keys. So from here, we were able to decrypt the latest firmware and move on from there. We'll be releasing a blog after this that has a little bit more details on the vulnerability, or I'm sorry, on this reverse engineering portion after this talk. So now we have the firmware, and we can actually begin the vulnerability research to actually compete in this competition. So in vulnerability research, there's really two types of analysis. There's dynamic and static analysis. So I'll talk briefly about the general approach you would want to take, or at least I take. So what I like to do is I like to explore the functionality of the device you're looking at, like a user would, and get really familiar with it. So in this case, it's a printer. It can print. It can fax. It actually can get synced up with Lexmark Cloud, and you can control it remotely. So you begin to get a little bit more complexity there. There's also other types of services. But ultimately, what you want to do while you're doing this is open up something like Burp. And as you're exploring it, look at the type of requests that are being sent back and forth. How are they exchanging data? What type of messaging protocols are they? Is it JSON? Is it XML? And as you're doing that, if anything stands out, just write a little note to yourself, because when you actually get to open up a service in Gidra and start looking at the source code, you can then take those notes and look at where the functionality is actually being implemented in the source code. Also what you want to do is you just want to see what the external attack service of the device is. So in this case, you would just run simple Nmap. You see that the Lexmark device has quite a few web services that ultimately all are on the same back end. But they also have other services like PGL. Unfortunately, that one printer that we thought was not fully functional, there was a toner issue. And so in Lexmark printers, when there's a toner issue, it disables quite a bit of functionality, and PGL was not accessible. So we decided not to investigate it for vulnerability research. So like I said, and what goes with dynamic analysis is you pair it with static analysis. So you saw those services with Nmap, and now you want to find what are the actual binaries backing those services on the file system. So you want to see what is actually being started with the init, startup scripts, system D. Typically, those things are things that start services on boot. And so you'll find the binaries that back services that way. And often in IoT world and embedded world, these services are written in compiled languages, and most often in memory unsafe compiled languages like C and C++, where they introduce a lot of memory corruption issues. From use of experience, I'd say that the most common pitfalls in these type of devices are first authentication message, authentication mechanisms in both HTTP and the messaging protocols. So a lot of times there might be simple authentication checks that are failed because they didn't anticipate the type of input that somebody would send it. Next, there's a ton of parsers that these vendors write for well-known messaging protocols like JSON, XML, SOAP. And these protocols are complex. And if you write your own custom parser, there's a lot of room for error there. And then the classic unsafe call the system with user-controlled input that will just get you remote code execution like you read in textbooks from the 90s. And then like I was saying, just basic memory safety issues, they could incorrectly check the length of a buffer that they're about to copy into, and just checking those basic calls and looking for just really, really easy things. So to give some concrete examples with those I just talked about, the Flashback team who's competed often in the Pondone competition a couple years back posted a write up about one of their vulnerabilities they named Tokyo Drift. And in this one, the vulnerability was an authentication bypass because they used a stir-i-stir method, which essentially just checks for the presence of a string and another string. And by stacking two specific actions that they wanted to get executed, the binary incorrectly thought that one action was being passed and they actually able to execute the second action and bypass authentication. Another concrete example would be this last year I was looking at a Netgear router, not one for the Pondone competition, but it was similar and it was just simple untrusted data being fed into a system and it's usually as easy as you think it would be once you actually get to the source code. So Zach just gave a good overview of a general approach for vulnerability research using static and dynamic analysis together. When it comes to the Lexmark printer that we were targeting for this competition, static analysis, we wanted to look at the web server first. Like Zach said, some of those other services were disabled in our kind of broken printer and we were looking through Burrp and when we saw a bunch of different types of requests you can make through the regular web UI when you connect to the printer. So when we started reverse engineering it we saw it was a basic Apache web server with a whole bunch of different end points that you could hit and one of those end points included a CGI directory and that CGI directory was full of bash scripts. So they had a large portion of their web server functionality written in bash of all languages and we just grep through that CGI bin directory for eval and if you're not familiar with bash it's very difficult to handle untrusted user input with the eval command. You're basically giving them a shell. Anything you pass in the eval will be evaluated like you typed it into the terminal and the command variable there did contain untrusted user input. And so this facts change, facts trace setting is the end point that we decided to go after. A little bit of a sign note for this. This end point can either be authenticated or unauthenticated meaning that the user might have to enter some credentials to access the end point or it might not and what controls that is when you set the printer up. If you factor reset the printer you just hook it right out of the box and boot it up this is the screen that you're greeted with and it's gonna ask you whether or not you wanna set up an admin account now or later. If you choose set up later that end point is unauthenticated if you choose set up now that end point is authenticated. So we were a little bit concerned because for Pone to own you need to have unauthenticated RCE and we were going back and forth of like maybe we need to find authentication bypass or something like that. But ultimately what we found is that within our own customer base and on Shodan every printer that we found on the internet did not have this end point authenticated. So everybody chose set up later when they asked you if you wanna set up an admin account. And it's a little bit worse, even worse than that because if you choose set up later and then actually go through the web UI and actually set up an admin user later that end point's still unauthenticated. There's a very specific little box you have to press to reauthenticate that end point. So we moved on like cool this is unauthenticated command injection let's keep going. If you open up that webpage on a web browser this is what you're greeted with. It lets you change some settings about the fax tracing they have a tracing library that you can use to debug your printer I guess. I'm not sure exactly what it is used for but you can toggle all of these switches to either on I'm sorry off low, medium or high. And if you run that request through Burp this is what it looks like. And at the very bottom there you can see that they just take all of those variable names and then sent them to an integer value of zero, one, two, three. And so what we're able to do is just replace one of those integer values with this basic command injection. So we told it to create a file in bar temp hello and later on when we actually got our shell you can see that that file is created. So it's about as simple as it gets in terms of a vulnerability. One thing I wanted to point out here is Zach and I were doing this research kind of part-time as part of our normal day job and I would say about 70 to 80% of the time was spent figuring out how to get the printer, how to get the flash off the printer, how to decrypt the firmware. That was a large portion of the time and it was only maybe 20% of the time once we actually had the firmware that it took us to find this vulnerability. So I just think it's interesting that if you have the opportunity to maybe go after a target where it's a little bit harder to actually get to the code or get to the binaries, you could find potentially easier bugs. This command injection in Bash was on their latest firmware and just recently patched. A few lessons learned from the competition is that your eyes are bigger than your stomach. Like I said, Zach and I were doing this part-time as part of our normal job and we bought a couple other targets. We looked at one of the TP-Link routers and the PR4100 NAS device and we just didn't have time to dig into them. Also don't give up on older targets. This Lexmark printer was in this competition for years and there's a similar one in it this year that uses the same type of firmware. And so just because people way smarter than myself and Zach have looked at these targets for years, doesn't mean that a firmware update won't introduce a new vulnerability or there's just an old vulnerability that's been in there that nobody has caught. And then finally, go back to basics. When I first started learning vulnerability research, I learned about the basic stack and heap overflows and so when I would go to look for vulnerabilities, those are the kinds of things I would look for. But after I gained a little bit more experience, I kind of found out for myself that those are actually normally the harder vulnerabilities to find and even if you do find them, they're normally much harder to weaponize. So when I'm doing research now, I like to start with the easy bugs first. I'll look for basic command injections, basic authentication bypass, SSRS. And then only after I've exhausted all of those options, then I'll move to the memory corruptions, the stack overflows, heap overflows. So I'm gonna talk a little bit about state of security in 2023. After the competition at Pone D'One, they released stats about the different bug classes and the percentage of them. And authentication bypass and command injection and these real simple ones are overwhelmingly the largest percentage of bugs found, which is kind of concerning, like all these devices that we use in our homes and it isn't just, it's not just in our homes, like we've been doing this for enterprise applications and appliance as well and it's the same story. So two talks that I really liked from this last year that I recommend you go watch are Deep Attack Services and Shallow Bugs by Chompy. In this talk, she talks about how she wanted to look for a bug in an authentication library within Windows that hadn't had any vulnerabilities found for a while and it was used across a lot of different authentication mechanisms. So she popped it open and within the first 10 minutes, she finds a bug just briefly looking at it. And then the second one is information security is an ecology of horrors by David Till. And my favorite quote from the talk is he said that his friends ask him, Dave, why is it so hard to find bugs? And he responded, stop looking for hard bugs. And I think it's laughable and it applies to all of this because like you saw or like I'm telling you at the Pondo and Competition, all these appliances, enterprise apps is the simple bugs that are still out there and it just takes a set of eyes to look over them and they fall out. So definitely give these a watch. I think another thing that comes into play here is businesses view security as a cost center. Recently SZA announced or not rather announced but they commented that they were thinking about making companies liable for software insecurity. And then that same talk by David Till, he said adding software liabilities to the equation doesn't mean these companies hire more programmers and QA people or threat modelers or whatever, they just hire more lawyers and insurance. And I think back to my wife as a nurse and when she first entered the workforce and was getting hired at a hospital, the hospital asked her if she wanted to carry additional liability insurance. And that's kind of scary for a person just entering the workforce. And I think about how our industry could change where companies begin to pass liability down to the employee and we could begin to look like the healthcare industry where now software engineers may be needing to question if they need to carry additional liability insurance if their employers start passing that liability down to them. It's not all terrible news though. Just because they had a web server written in bash doesn't mean that they're completely out to lunch. What I'm trying to demonstrate here is that Lib Uranium which is a central library in their firmware has been written in Rust. We're looking through the strings and we see Rust EH personality which is a common symbol that you'll find in any Rust binary. And so they are definitely trying to make strides in the right direction by moving to more memory safe languages that make it harder for the developers to make mistakes. Like I'd love to see somebody write a bash eval statement that is totally safe that takes user input. I think that'd be very difficult to do. Languages like Rust go, some of the more modern stuff make things like that a lot harder to shoot yourself in the foot. So after all this we wanted to see, hey there's all these application security tools out there and both developers and security researchers like use these tools to find vulnerabilities. We wanna see, will any of the tools out there be able to detect the bug we found in the Lex Park printer? So we went after two of the common open source tools that are out there. The first is CodeQL. We looked at the type of language support they have. Unfortunately bash is not supported. But luckily we found that SemGrip does have experimental support for bash for whatever reason. To give a quick little intro to SemGrip, it has a couple modes of how it detects different insecure coding practices. The first is a regex style pattern matching where it just looks for common rules that lead to unsafe things. But the second is Taint. And what Taint mode does is it breaks down a language into its abstract syntax tree and tokenizes the language. And it's able to programmatically tell you if a certain variable may lead into or be used by a certain function. So what you can do is say, are any of these variables eventually fed into the eval statement, like in this case? So I ran SemGrip out of the box on our vulnerable page and it did not find it, unfortunately. And I wanted to dig in a little more. So I looked and there's four rules that are in SemGrip and none of them are for a command injection style vulnerability being fed into eval. So I wanted to give them another chance so I made a rule that would hopefully track this type of input. And unfortunately, it still did not find it. I believe it's because the language doesn't properly tokenize all of the bash language and that's why it's still in the experimental phase. That's not to say SemGrip is a terrible tool. I mean, we're talking about bash and I don't think many enterprise level web services are commonly written in bash. So I ran it, I was looking into a very large security vendors appliance and I found a vulnerability that allowed you to do authenticated arbitrary follow right as the root user, allowing you to take over the device. And so I just ran it out of the box SemGrip and luckily SemGrip did find this. So hats off to SemGrip, it does find things. So finally we wanna share a few of our proofs of concepts for this Lexmark printer vulnerability. The first one here is just a simple Python script that abuses the command injection to give you a shell in the context of the HTTP user. We first send a couple wake up commands to make sure the printer is awake. We found that if you would let the printer sit for a while sometimes it wouldn't respond appropriately. It'll start a listener on your local machine on a certain port and then the remote host would be the printer itself and then it'll allow you to get a shell to do further exploration. The second POC we'll be releasing is to dump credentials and on the Lexmark devices there is a database called auth.db. Inside that database all of the local users, their passwords are stored with the passwords hash with SHA 256 script. And if you integrate your printer into Active Directory all of those credentials are stored in clear text which is awesome for you and me. So we'll be releasing this POC that would dumps all those credentials. And surprisingly a lot of these devices are integrated to Active Directory so I recommend if you find a client that has a bunch of Lexmark printers that you run this and hopefully get some sweet domain credits. So we wanted to have a little bit of flair for PoN to own. Often you'll see people who compete they'll change the screen to like their logo on whatever device they're hacking or they'll make the device play a song. And so we decided we wanted the printer to play Super Mario Brothers. There's a command on the printer called Rob that does a whole bunch of different stuff but one of the things you can make it do is beep. So on the left there there's Slack conversation between Zach and I where he attempted his first composition of Super Mario Brothers that we will play for you now. You're not supposed to clap, that was bad. So that one was, they had a couple wrong notes in it if anybody heard and the joke down there is I told Zach, I said there's a lot of wrong notes in their LOL, he said what are you some of musical aficionado. So we got together and we figured out that whatever thing he was reading you had to convert the notes to a frequency. You can see here it's a little bit complicated so Zach did his best on that first try. And finally we were able to get a good version of Super Mario Brothers. What James didn't tell you is that he actually plays guitar and I've never read music in my life so when I sent him that little video he was like really looking down on me there. I'd never read music before. Zach did add a few Easter eggs in that bit conductor script that generates it so if you want to, I think he added some T-Swift into there so if you go and download that you can make your Lexmark printer replace some Taylor Swift. That's all we have, the proof of concepts will be up on GitHub, you can find us on Twitter, that's where we'll post the links to the proof of concept, these slides and then also our blog that'll have a little bit more information. Thank you everybody.