 Hi, I'm Patrik. I'm the co-founder of Village Idiot Labs, which conducts security research in IoT and helps deliver content, including CTF, labs, so on and so forth, for IoT Village, which I am also a co-organizer of. I'm also a DEFCON26 black badge holder. Deal with it. My name is Zoltán Balázs and I'm the head of vulnerability research at Kujo AI. I developed the zombie browser toolkit, the hardware firewall bypass tool, malware analysis sandbox tester tool, played with some crappy IoT devices and my exploit code ended up running on 600,000 IP cameras. I invented the idea of encrypted exploit delivery. I'm the co-organizer of the hack-assuring meetup program committee member of the activity conference and volunteer at IoT Village. Are you into emulating IoT devices? Are you interested in the suffering of embedded device development? Have you ever debugged MIPS assembly? Do you want to build a lab for IoT devices? If yes, I recommend that you enjoy the show or you just stay for the MIPS, but still, please stay and you will have a learning opportunity to find zero days in your IoT devices. So, why are we doing this talk today? Well, IoT is cool and all and we already know about it, so what? Well, let's have a look at all these photos. As you can see, there's a lot of spaghetti wiring, there's book-ended devices everywhere, they take up a lot of space, and quite frankly, it's tough to just focus on the virtual aspect of attacking or conducting research of these devices. The physical parts are important, don't get me wrong, but that's going to be beyond the scope of this talk. So, we looked around and tried to find some tools which can fit our needs, but turned out there's none. We checked, for example, the Furman 9 and it's cool and awesome for vulnerability research purposes, but it's a bit bloated for our lab purposes. You can also find a lot of outdated and incorrect blog posts on how you can do this manually, but I promise you that it's really a pain. And we also checked the Armex, which is really, really cool. The only thing we missed was the MIPS support, so we figured let's add MIPS support to Armex and let the fun begin. And now, without further ado, let's review what we'll be discussing today. We'll dive into a precursor on architectures, we'll talk about the differences between virtualization and emulation, and then we'll dive right into the pain that is our new project, MIPSX. So, we'll be talking a little bit about the kernel, on the file system, and VRAM, key new virtual machines, Docker, and last, but certainly not least, a demo. As you know, there are many different CPU architectures out there in the world. The most common one is, I believe, the Intel one, which is a complex instruction set based CPU. It's mostly used in desktop servers. The second most common one is the ARM CPU architecture, which is based on risk, which means it's reduced instruction set. You can mostly find it in embedded devices, for example, smartphones, and now also in MacBooks. Our talk will deal with MIPSX. It's also a reduced one. You can find it in embedded devices, mostly, for example, in cheap Chinese IP cameras. Another famous architecture is the PowerPC one. Again, this is a reduced one. You can find it in embedded devices or in some very old Apple hardware. And last but not least, there is also Spark CPU architecture. You can find it in mostly in high-end servers or in embedded things. There are also a lot of different others, but that's not important for us right now. On this slide, you can see a sample when you run, for example, Hello World compiled for Intel system and how this looks like in GDB. And what's interesting to see here is the lot of registers, but still not that many. And if you look at these assembly instructions, these might be very familiar for you. Here, you can see a screenshot from debugging an ARM executable. As you can see now, a lot, lot more registers we have here, especially because it's a 64-bit architecture. And let's check the assembly instructions. For example, Alder, it's a very common one used in ARM and also the stack pointer. It's important or the PC, which is the program counter. Here's MIPS32 in assembly. Again, a lot of registers, as it is the case with the reduced instruction sets. And here you can see some not very known MISP assembly instructions, for example, UI at UI, things like that. I have no idea what these are doing, but soon we will figure out. And just for fun, I also created a screenshot of debugging a PowerPC binary. And here you can see, again, a lot of registers. And here are the PowerPC assembly instructions. I don't even try to spell these out. Virtualization versus emulation. So you're likely already very familiar and comfortable working in a hypervisor-based environment. Think virtual box, VMware, and even parallels for Mac. At a very high level, what's happening here is that your host computer is providing a hardware interface layer that is passed through to your guest VM, whether that's Windows or even some other Nix flavor. The expected behavior here is that your VM just boots into the kernel and provides you with a workable operating system. So the guest VM is effectively leveraging the host's CPU and other hardware resources through abstraction and translation layer. That's great for x86, x64-based architectures, which is pretty much the standard out there. But what about other architectures like ARM or MIPS? Well, we could use target CPU architecture development boards, something like a Raspberry Pi, but that's ARM-based. So what about MIPS boards? Well, these are extremely rare and difficult to find, and generally they're a discontinued product. So this is where emulation comes in. Emulation works by providing a Rosetta Stone-like instruction set of the target architecture to that of the host architecture. Now, I'm greatly oversimplifying things here, as this is quite a complex implementation. But fear not, we already have a solution available to us, and that is KeyMew. KeyMew supports many different CPU architectures, like ARM, MIPS, Spark, PowerPercy, even x86, x64, and various other boutiquey other architectures. And so what happens is it's working to provide a reliable, emulated BIOS, basic input-output system, in order to boot to your target architectures kernel. One gotcha you need to be aware of, though, is you can't just necessarily take a firmware kernel that's compiled for an IoT device and just right inside of KeyMew. Again, firmware relies on hardware and pinouts and buses and all those other channels, unless you're extremely customizing KeyMew in terms of its drivers, in terms of the way how it handles and emulates certain functions and capability, chances are it will not work. So which kernel should we be using? Well, if you check some blogs on firmware emulation, generally speaking, almost everyone's going to recommend to use BuildsRoot to build your kernel and host file system, which on the surface sounds great, most of the time. If you don't want to deal with this process, which can be very tedious and arduous, just check out the repository we have listed here on the slide. The individual has taken the time to actually compile some very old but pre-built Debian kernels and host file systems for Intel, ARM, and MIPS. What makes these Debian images great is that in contrast to Buildroom, once you actually run the kernel inside of KeyMew, you're able to change the app repository listing, for example, to archive.Debian.org, and then you can just install apps and binaries as you normally would. With Buildroom, though, if you forget something at compile time, it's generally going to be a pain in the butt to get you working. So what makes things challenging in the kernel identification space here is to build an old kernel like you would see in an IoT device, which is generally kernel versions like 2.6 or perhaps even 3.x. You need to make sure you select all the right packages, any libraries, dependencies, crypto components, and legacy-shared objects from wherever the sources may be. Buildroot is going to try to get them as part of the build process, but if the repositories are no longer online, you need to hunt them down, and it's a very iterative and frustrating build process. So on the surface, Buildroot is great, but be forewarned. Here be dragons. If you want to compile your own kernel, well, you will see some errors during your journey into this. For example, let's use 12 years old Ubuntu to build an 11 years old Buildroot, and first thing you will notice that it cannot download some of the necessary packages, because the SSI connection will not work. That's kind of expected, and you can easily get around these things. Or you will see that some of the downloads and our websites are pretty slow when you download. You will see some errors, some of them are fatal, some of them are, I don't know, mortal. If you try to debug things, some stuff will not work out of the box, especially if you try to debug with some older versions of GDB, you will see some compile errors, missing tools, missing tools, another compiler errors, more errors, missing tools, you cannot debug errors, internal compiler error segmentation fault. That's kind of interesting. You will end up, as I mentioned, using very old, like 12 years old Ubuntu to build stuff will not work if you try to build an old kernel. You will see some errors, and some errors, and more, and more, and more. And yes, this was one of my favorites, because I never had any idea that to emulate something in MIPS, I would need ninjas. And clearly, there were no ninjas around. But at the end, yes, this will be a lot of frustration that you cannot compile your kernel, or it will just not run the way you expect it. Continuing down the kernel radical, we already know about buildroot and the pre-buildw images. But what's next? Well, in order to compile a MIPS kernel, you need the appropriate tool chain. What is a tool chain, you ask? Well, tool chains are the tools you need to compile things, e.g., CC, or even GCC, and there's a variety of others. In this case, we would be cross-compiling, so we need a cross-compile tool chain. Cross-compiling means that you are taking a source project and compiling it for the target architecture. So in this case, we would be compiling for MIPS, but on an Intel-based CPU. Cross-compiling comes with its own unique set of challenges, mainly it can be pretty buggy and sometimes painfully slow, particularly if you're doing it inside of Keynew. Linux kernel 2.6 is pretty common with quick-to-market IoT devices, think Amazon specials. This kernel was a very long-term and stable release, and it's been around for quite a while. Thus, if you want to emulate this device, you would have to compile your own old kernel, mainly kernel 2.6. If a newer MIPS kernel is okay in the 4.x category, then that's great, but chances are the binaries that were compiled for your target IoT device will not be compatible with that kernel version. So here we go again, there be dragons. So if you do go the build route, instead of the already provided Debian image, you better be prepared to build your own binaries as well. So things like GDB, GDB server, Netcat, just to name a few. And when you compile them, you also want them to be compiled statically, so they become a standalone and more universal binary that you can transplant between your devices that you're looking to research on. Toolchain, hell. On the building from scratch route, there are many options for building the kernel, the host file system, which houses the target architecture libraries, some binaries, the compilers, all that other fun stuff, and even your standalone static files. The short list of toolchains out there include build route naturally, open WRT, which is a common and very popular platform for open WRT OS compilation. There's cross tool NG, there's muscle, there's Docker cross and Debian cross tools. During our journey, and after much trial and error, we landed on build route being the best option for the custom kernel as well as the host file system. And the muscle C library was best for compiling our static binaries. But after some weekends spent on this, you will be created with these nice logins. And finally, you can see that, hey, now I'm able to emulate some stuff on their mix. You just need some development environments for this, like five different versions of you going to end the colleagues to do that. Easy peasy. Single binary versus full by system emulation. So even though so far our talk has been focused on full system emulation, you can actually just run a single binary with the help of key new user emulation. The catch 22 here, however, is that if you're running a dynamically compiled binary, you'll need to preload them so that key new knows where to point to. Otherwise, you'll just encounter a bunch of lovely errors. On the screen here, you'll see the following commands that you can both run and even use to start compiling binaries on your Debian x86 64 based operating system. Also, our friend Maria from Azaria Labs has an excellent write up on how to get started using ARM inside of key new. This serves as well since the process will be nearly identical except for MIPS. And if you follow all these instructions, then as you can see, you can run your MIPS binary from an Intel based operating system just by using the QMU user land tools. Unfortunately, if it's a dynamically linked one and not a static one by default, it will not run. But we are used to this. Luckily, there are some easy fixes how you can correct these. And after that, you can run even dynamically linked binaries from a different architecture by using the QMU emulation. File systems. So not all binaries are created equally MIPS isn't necessarily MIPS across the board. So tying back to one of our previous talking points about CPU architectures, there's a few different flavors of MIPS. So we have MIPS one, which is the first generation registered to register architecture generally contains 32 bits of data types. We've got MIPS two, which is focused on increasing shared memory, multiprocessing still uses 32 bit data types. We then move forward to MIPS three, which is a backwards compatible extension of MIPS two, but also features 64 bit data types. Then we've got MIPS four and five, which are just super sets of MIPS three, which are obviously backwards compatible with previous versions as well. Then we've got MIPS 32 and MIPS 64 self explanatory. Then we've also got micro MIPS, which is a super set of MIPS 32 and 64, and is the only form of architecture that supports code compression and works with 1632 and 64 bit code data sets. Cross compiling, which is what we spoke about earlier, is going to be necessary and very important in this case. Sadly, most IoT devices don't follow a particular standard and it's up to each manufacturer to figure out what architecture they want to run. From our perspective then, that means you're going to want to compile a whole crap ton of the same binary except for the different variations of MIPS architectures. Another consideration is that we will need to chroot the target IoT firmware file system. What's happening here is chroot is an operation that changes the apparent root directory for the currently running process and any subsequent children that they may spawn. Why this is important is that when it comes to full system emulation, dynamically compiled binaries will search for kernel library specific shared code, which is always, always, always provided on the actual device. So what do we do in the virtual emulated perspective? Well, we'll have to copy all of these files across to our team UVM, and then chroot to the guest file system, and then voila, everything in theory should work. Persistence of a file system pretty much just affects how you want to do your workflow when it comes to IoT research. If you plan on modifying any contents of the virtual file system, you probably don't want it to be persistent. So non persistent. If you plan to simply throw everything you can against it and just reset back to a known good config, this kind of acts like a snapshot feature, if you will, if you're familiar with traditional virtual machine software like VMware, virtual box, KDM, etc. Non-voluntile RAM embedded devices have to have the ability to retain critical configuration data beyond the former file itself. This will help the device survive an upgrade, a reboot, a power failure, or even the occasional drop off a table. NV-RAM is typically used to store configuration data, credentials, manufacture information, default configs for certain services. A simple example would be if you were to log into your home Wi-Fi router and tell it, hey, here is my SSID, here's the username I want to use, here's the password, MAC address filtering, port filtering, any of that fun stuff. Well, when you hit save, what actually is happening on the back end is that information is being saved to NV-RAM versus a database. Where this causes issues for security research is if the binary you're trying to assess does any form of interaction with NV-RAM the values probably won't be found and that binary is going to crash. So inside of KEMU you're going to get a bunch of errors. This can be a very, very frustrating process. So it becomes very much so a trial and error activity to figure out what is your binary trying to look for and then basically use NV-RAM Faker to simulate that value set. Just get the bare minimums in place so that it will run. KEMU, KEMU, KEMU. We've used this term quite a bit now and we did cut on what it is. So just to reiterate KEMU is a nice little architecture emulation program. It has been around for a number of years now. You can do both hardware emulation as well as user level process emulation. For the purposes of MIPSX we are more interested in the hardware emulation. So, again, as I spoke about earlier, we're all used to virtualization tools like the M-Ware, VirtualBox, KVM, Parallels, etc. and how they're set up and allow us to interact with the guest operating system residing in them. KEMU doesn't behave nearly as friendly or as nicely, particularly when it comes to networking. On that, KEMU has two networking components. The first being the virtual network device, so a PCI network card for the guest or VM, and second being the network backend that interacts with this virtual NIC, aka interacts with your host network and conducts or provides pass-through to your host network. Why this is important is because PING, for example, does not work out of the box. TCP and UDP will, provided you actually expose those ports from the guest OS to your host. A typical workaround we've ended up using is a TAP interface between our host NIC and the KEMU NIC. Now, this sounds easy on the surface, but definitely causes a myriad of issues and makes troubleshooting super fun and complicated. Note the eye roll here. When it comes to stability, think about what happens if your binary crash is inside of the KEMU VM. What's happening on your host? What's happening inside the guest file system? Where are the logs, the dumps, any of that fun stuff being generated? And how do you get them out of the virtual file system? What if the binary crashing causes a reboot? But that reboot ends up spinning the guest OS, but your CH rooted inside of there. Okay. How do you diagnose where that went wrong? Did you have a script to automate your CH root inside of that guest file system? Hopefully. NVRAN. So we've talked about this. We're going to use NV on Faker, and we're going to simulate some of the value pairs that it's looking for. That's great. But what about consideration for like the shared objects in the shared libraries that also have values that come or are from being set in the NVRAN? Accessibility of the guest, let alone the services inside of that guest. We need ports to be exposed. We can SSH through the KEMU guest OS, and that's great. But what about finding other services that are running on that host? You probably want those to be exposed so that you can run your favorite tools like Nmap to do port discovery. Debugging, using GDB server is probably your best bet, and simplest option for reviewing the binary crashes inside of the KEMU VM. But what do you do for more dynamic interaction for export development? In such cases for buffer overslows, for example. Another consideration is something really neat you can do with KEMU is tap into the kernel itself. How practical is this? Well, you tell me, what level of experience do you have with kernel exploitation from a binary? For our lab purposes, we also created a ducker-based version of MIPSX, but luckily before this presentation got recorded, Chomil also created his own version of RMIX in ducker. So right now, it works really, really nice. And finally, we don't need some old or dumb VMs to run our RMIX or MIPSX framework. First of all, you have your host machine, and in that you have your docker container, which will run QMU in our emulation environment. And this will run our host kernel and FI system, which will match the CPU architecture of the firmware you want to emulate, whether it's, for example, MIPS Big Endian or Little Endian or whatever you want. And inside that, you will run your Siege-Rooted IoT firmware with the rootfsh copied from the IoT device. And in that, you will have your interesting binaries, which you want to attack or, for example, debug like a web server. And the host kernel and FI system runs your SSH server and GDB server and everything you need for research purposes. And basically, that's all you need. What makes this really, really fun if you want to debug things, because it's sometimes very hard to find where the issue is, or sometimes it's even hard to figure out in which terminal am I at the moment. Is this the host machine, the docker container QMU Siege-Rooted environment? It's a lot of fun. I can promise you that. And now, let me show you how MIPS X works in the latest docker environment. First, let's start the docker container. Then just log in to our docker shell. With the launcher, we can start this eminent IP camera, which is a MIPS-based device. So this will start the MIPS little and the host FI system with the kernel and the basic most FI system, which will have SSH and all the debugging stuff we need. Then we SSH into this QMU and start the CH-Rooted firmware IoT environment. You will see a lot of different errors. Whenever you try to emulate something, usually it's some missing hardware or some other issue. As you can see now, we are in the MIPS CPU architecture. And what's fun that you never know what is basically a real error or what is not a real error. But here, you can see the different services running like telnetd. So let's open another docker shell and from that, let's GDP into this demon process. And as you can see, GDP nicely starts. And now, we have full debug access to the MIPS environment and to the binaries, which can be our target. So from now on, we can start fuzzing it or whatever we want to do. So what's next for MIPSX? Well, we plan to continue on refining the NB RAM abilities in terms of identifying values that it's looking for and helping you generate a list. We want to develop a CI CD model in terms of letting you identify custom firmware that you want to build and auto deploy that into the MIPSX framework so that it just stands up your guest VM as quickly as possible. So that's our next steps. But there's also profit in this, right? And by profit, we mean paying it forward to the security community. We continue to drive positive momentum in bug bounty. And effectively, at the end of the day, it's about putting pressure on the vendors to help improve and make secure code and secure systems development. This project is open source, which means anyone can contribute, anyone with an idea can throw it in there and we can talk about, you know, taking this to the next level. That's what would mean by profit, helping everyone do good in the future. And if you want, you can check out our repository where you can find prebuilt VM and also our Docker version of MIPSX. And also, if you want to build your own host file system, then we have some scripts which can help you in creating our MIPSX compatible host file system. So that's all folks. Thank you for your attention.