 Hello Hakers. This is a talk about a bug in microchip. And I'm Nini Chen, a security researcher at Devcore. We are from Taiwan. And I'm also a member of Belsen city of team and a member of ondefine. Okay. Why? Why the router OS? Because we want to participate in the Ponzu on Torrento in 2022. As one of the targets, the rotable of microchip hasn't been exploited in the previous event. More correctly, no one tried to demo exploit against this target in the context. You may already know the rules of the event. Participants must find a pre-O's RCE with no user interaction to acquire the points of the target. And the sole host mesh up, a special category was introduced in 2022. Participants have to change the router when site exploits and another device exploits to earn the points and bounty. The bounty and point in this category are relatively high. So it's a good target for us to earn more to win the contest. Let's recap some better things quickly. The router board of microchip is quite popular and the router OS is the operating system of router board hardware. By scanning the microchip, we can observe about 3 million devices online. Router OS is a standalone operating system based on the Linux kernel. And microchip also offers ISO images so we can run it as a VM. It's closed source and also closed ecosystem. According to the download terms on microchip's website, you pay 45 bucks to get a GPOS code. What a bargain. And currently, there are two versions of router OS, the V7 and the V6. They should be considered two branches because their implementation are slightly different. And the cloud hosted router is the router OS image for VM. But only the AMD64 version is available. Most of the services on router OS are implemented by microchip themselves. And they share a similar architecture. So we named them NOVA binary afterward. And they use NOVA message to talk to each other. Though the router OS is based on Linux kernel, users are trapped inside a customized COI. And there's no official method to access the underlying Linux system. Only developers can install a backdoor package to access the Linux system for debugging purpose. And the router can manage their routers via the web panel or Winbox, which is a Windows GUI application. It's also a popular attack service because the Winbox interface on the router can be used to talk to other NOVA binary. Some of them can be accessed without authentication. To understand how people research the router OS, let's have a look at past CVEs of the router OS. There are a total of 80 CVEs. And 28 of them are pre-auth exploitable. And the victim is the router itself. 16 out of the 28 are those in various components. 8 out of 28 are DNS poisoning, allowing attacker to MITM, or firewall evasion. Only 4 pre-auths vulnerability in history can lead to a shell, or locking us at a mean directly without user interaction. These are what we care about. 3 out of 4 were found between 2017 and 2019. And 3 out of 4 are in the wild. Let's have a close look. CV 2017 20149, aka Gmail rate, is wonderfully exploited from the CIA in the 2017. By sending an HDTV request with crafty content lens, we can corrupt stack to execute arbitrary code by stack clash. CV 2018, 7445, is a buffer overflow in SMB, found by black box fuzzing. CV 2018 14847 is also one of the exploits leaked from the CIA in 2017. This bug allows unauthenticated attackers to read arbitrary file on router OS. It sounds not a big deal in the first place, but since the password stored on router OS was from simply repeating XOR password in plain text with the hash of the username and the fixed string at late time. So the password can be recovered easily, and the bug with arbitrary file reading is sufficient to lock in us at a mean. CV 2021 41987 is a hip hop overflow in the code of the ACP service. This was discovered on an APT C2 server in 2021. As we can see, these vulnerabilities are mostly in the wild. We can learn too much from them directly. So let's take a look at some talks and articles about the router OS. In 2017, there are three talks, and they are about a package format adopted by the router OS, the developer's back door, how the router OS store the password, et cetera. Talks are interesting and helpful to have a big picture of the router OS. In 2018, the talk bug hunting in router OS analyzed the normal message in binary and JSON format. Introduced the how Winbox client talked to the Winbox interface by normal message and found the post-OS stack overflow. In 2019, the first one is the same as the previous one. So the next one is my critique firewall and Netbypass. It's a talk about the vulnerability in the Winbox that can be used to scan the intranet if the Winbox is exposed to the intranet. The next, finding an exploiting CVE found a stack buffer overflow in SMB by black box passing and explained how they got the shell with their exploit on the router OS VM. And the deep dive microchip exploit is a security analyze about the CIA exploits and the massive exploit event based on the liquid payload. The next one, help me vulnerability, you are my only hope, release a tool to help users to jailbreak their device to check if liquid exploit was already compromising them. And the last one, router OS trying to root is about chaining DNS poisoning vulnerability and a downgrade attack to get a shell. Finally, in 2022, polling microchip into the limelight talk more about NOVA message and disclose post-OS vulnerability in triple W, they can be used to jailbreak. They also release a tool to help researchers quickly get a shell on the MD64 version of router OS with the vulnerability. It seems like the whole world is talking about the NOVA message in the IPC of the router OS. So we also try to understand what it is. Let's talk about IPC of router OS with an example. While I use the login through the telnet to access the console of router OS, the telnet will immediately call the EX ECL to execute the login process to ask the user for a username and password. Then the login process will send a NOVA message with your username and password. So the user process to check if they are correct. The user process also responds in NOVA message. The NOVA message consists of key value pairs and the key can only be numbers. The key, the sys2 and sysform are arrays containing the ID of NOVA binary and the ID of the handler. The communication between two process is something like this. Every message is sent to the loader via socket and the loader dispatch the message to the corresponding binary. The login process send a request with a username and password to the user process. The value of sys2 means the target is the handler 4 and the binary ID is 13. And the process itself doesn't have to fill the sysform with its process ID because the loader we do that based on which socket is received the message from. Next, the loader remove the target binary ID in sys2 as the source ID to sysform and then forward the message to the target process. When the user process receive the message from the socket, it also remove the handler from the sys2 and pass it to the corresponding handler. Then the handler does the rest of the thing. We mentioned that Winbox interface on the router OS is a popular attack service since it can be used to access all the process. That's because the protocol Winbox client use to talk to the interface on the router is basically a normal message in the TCP packet. So if the target handler fail to check the sender's privilege, that can cause problems. No, normal message is type value mapping and can contain various types and also normal message itself. Because the JSON format of the normal message is deprecated, we only talk about the binary form today. Okay, the M2 at the beginning is the magic bytes for the normal message. Then the useful bytes to describe a key. The first three bytes of key are ID and the last byte is type. And bytes following are the value for that key. So this message is ID types and values look like this. Types like bool can be expressed with one bit directly using the lower bit of the type bytes. To understand which binary corresponding to the ID in the sys from and sys2, we need to pass X3 file under the loader directory. It's kind of XML but in binary form. If the binary was introduced by installing the package, its ID is under the loader directory of that package. Some IDs can be found in the X3 file because the process is not persistent. The ID is just a serial number like the locking process we just mentioned. Our binaries also have X3 files for different purpose. For example, the X3 of triple W describe which URI should be handled by which servlet. It seems like it's a good target for fuzzing, right? But we can't just fuzz it and expect we get something after two months. Now, it's a spoiler. While we were writing our exploit with the vulnerability that I will introduce later, talk at POC 2022 micro tick router OS security, the forgotten IPC message. Though I think it's far from forgotten but anyway, the talk is about fuzzing and the NOVA message. And it found lots of post-os CVEs. By the number of CVEs, it looks like the father has been running for a long time. A bit of an effort thought. I don't think we can get a better result if we do the same thing at the time. Not to mention that we only have two months. So, is this still possible to point the router OS? Review these CVEs again. Most of the vulnerability we think interesting will discover many years ago. Researchers also found lots of vulnerability. But as we can see, the micro tick becomes safer and safer. But in 2021, a vulnerability in the SCEP server we just mentioned was discovered by analyzing and in a wild exploit on the APT C2 server. Was it discovered until land because it was not a default feature? Or is it just because people have overlooked something? Most talks and articles focus on jailbreaking, analyzing the in the wild exploits and NOVA message in IPC. I have a summary after reading them and starting to reverse by ourselves. That is, no one with sanity will like to dive into the details of NOVA banners. I mean, expect for the exploits from the CIA and APT. Most of the research about bug hunting is fuzzing the protocol playing with NOVA message or fuzzing NOVA message. By the outcome, it seems like attackers are familiar with the router OS laners. We need more details about the NOVA message. And don't get me wrong, I'm not against fuzzing, but we have to ensure we check everything to take advantage of the context. I don't think the system is perfect now. There is a gap between researchers and attackers. So what was missed if we want to find some pre-oath RCE bugs in router OS? The first question is where are the entry points to the customized IPC? Most of them can only access by authentic user. We can foresee the playing with NOVA message in IPC will result in lots of post-oath bugs. IPC is just a part of implementation of the most functionality. We want to review the crucial part directly and carefully. For example, how does the DHCP process pass a DHCP packet to extract the info in it? Then we have to mention the architecture of the NOVA message, NOVA binary. Every NOVA binary has a looper. The looper use port to check the status of socket and process event in each loop, just like the regular event loop you've seen everywhere. Looper is responsible for the communication between the current process and the loader. The looper register a special callback for a unique socket, which is used to communicate with loader. The special callback on MSG sock will dispatch the message to the corresponding handler once it received NOVA message from the socket. So how does the handler work? While the looper receive a NOVA message, you dispatch it to the corresponding handler. Look at the right hand side. And let's assume that handler one is responsible for the NOVA message. And then your code function in V-table with the CMD prefix, like corresponding to the command specified by the CMD. There are many functions here, but because the handler is the best class, commands relative to objects are unimplemented. The CMD unknown is often overridden and used to extend the customized command. Sometimes the developers override the handle CMD instead of the CMD unknown. You may have already read some of this in the past research. But most of the time the NOVA binary use the derived classes rather than the best handler class. The derived classes can store a collection of objects of a specific type. For example, while the user is setting up a DHCP server, they will send a message to the handler zero of DHCP process to create a DHCP server. The handler zero, which is an instance of AMAP, calls the function on its V-table with offset hex 7C to create a DHCP server object and insert it into a tree. The 7C is the offset to the constructor of the inner object. And you can find the offset of such function in other derived classes by reversing the CMD 8 object. In the previous example, the web panel sends a NOVA message to DHCP to create a DHCP server. But why if I told you that some functionality don't even use the NOVA message? Take the CDP and the LLDP for example. They are implemented in Discover. The looper registers two codec functions. The looper keeps looping and check the status of sockets. If there are incoming packets, the looper call the codec function. The codec of CDP will check if the interface they receive the packet is not filtered by firewall rule. Then extract the information from the packet and store them directly into another ASIC map without sending NOVA message. So there's no NOVA message here. So we started to review the router OS by enumerating functions and reversing them. Then we found something strange. One day, while plugging and unplugging some cable, connect to the router, suddenly the log file showed that RADVD crashed several times. So we tried to reproduce the crash by manually plugging and unplugging cables about a thousand times. But couldn't figure out the crash condition. The condition is unknown and the crash appears randomly. Instead of mindlessly unplugging and plugging the cables, we reversed the RADVD to see if we can find something interesting. We didn't find the root cause of the crash, but we got something better. So why is RADVD? RADVD is a service about SLAAC of IPv6. Let's say a computer or a node want to get an IPv6 address. It will first mud case the router solicitation to all router prefixes. Then if a router receives the OS, it will mud case the router advertisement with network prefix to all nodes. After that, a node can calculate its IPv6 address by combining the network prefix and EUI64, which is calculated from the MAC address. Another scenario is that if an upstream router, router A, maybe it's an ISP, they give you a prefix with slash 40A, which assign in advance by email or something. Then you can use them with whatever you want in your LAN. Then here comes the link prefix and router prefix. First, the router B act as a node on the interface with the router A. The router B ask router A for a prefix. The router A send a slash 64 prefix for router B to calculate its IPv6 address. Router B calculate an IPv6 address. Let's say it's 201, colon DBA, colon colon 1. And use it on the interface that links with router A. So now the router A can route all traffic to the address on the slash 40A to router B. Now when nodes want to get IPv6 and send RS to the router B, the router B just mudicates the slash 40A prefix to nodes. Then the nodes act as usual. So the functionality of RADVD in router OS should have two functionality. First, it can act as a node to receive RA. And it can also mudicates RA to the LAN. Let's follow the execution flow of RADVD to find the bug. The RADVD have a similar design pattern to the discover process we mentioned earlier. We register a callback, the looper pull a socket in every loop. And if there is an incoming ICMPv6 socket packet, the looper call a register callback. What callback do is roughly like this. It check if the packet is a valley RA or a valley RS. If it is a RA, then extract information and store them in the handle one, which is a map. And if it is an RA, it will mudicates the RA immediately. Sorry, if it is RS, it will mudicates RA immediately. There are three cases the router OS will mudicates the prefix to nodes. The first one is when you receive the RS from nodes. The second is when you receive an RA from upstream router. The last one is when it, sorry, the last one is when you receive the RS from the router, it will mudicates prefix in the default period, which is in 200 to 600 seconds. And yes, the router will mudicates the RA to nodes once it receives the RA. But it's hard to trace this behavior directly because this is about a subscription I will talk about later. We didn't know that then. But the other two cases, the RADVD code send RA right after it receives an RS. The handle one registered a timer to send RA periodically. So we can find the send RA function by tracing these two methods. Following the send RA function, we found something interesting while the RADVD deal with DNS advisory. It will advise DNS to nodes based on the DNS list it received from upstream. The ADD DNS function is used to flatten the DNS tree and add it to the ICMPV6. The RA row is the ICMPV6 payload, which is the local variable with the size of 496 bytes. Step into the add DNS and we can immediately find out there is probably a stack bubble overflow. The function add DNS to packets by memory copy without any bundling check that causes stack bubble overflow if the DNS is big enough. The DNS addresses are from the RDNSS of the RA. But the max length of the RDNSS of an RA is 8 by 8 bit, which is insufficient to override the return address. But if this is not the first time the router always receive an RA, the RADVD have to inspire all RDNSS. They make the RADVD flatten another tree to override more bytes for us. So, what we have to do is we have to act as a Bay neighbor, let's send a router to the RADVD and then send another RADVD with the big RDNSS list. Now the return address is overwritten with an IPv6 address in the RDNSS of the first RA. It is time to construct the exploits and it looks easy because there is detection enabled. So, let's find a way to jump on the stack and run our shell call. That should be easy. Thank you. The overflow IPv6 addresses are sorted since they were stored in the tree. To make the order of the IP addresses stable to ensure the shell call is good, we can use part of the IPv6 prefix as a serial number and put a jump at the end of each address to jump over the prefix because there are not very instructions. But because of the delay slot of MIPS, which means the instruction after the branch instruction will always be executed before the branch instruction itself. We have to move the jump instruction forward. But the other problem is that you cannot do a syscall in the delay slot. So, basically, this is a bad idea. It's actually a CDF 101 problem. Just make a prefix a valid instruction. They won't affect our gadget or shell call and everything is fine. And because we didn't leak any stack address, plus we didn't find any gadget that can be used to move stack pointer to register T9 for jump instruction in the first place. So, what we did here is first write our gadget on memory, then jump to that gadget. Finally, our shell call will be executed. Sounds good. But there is iCache. The MIPS CPU has two cache, the instruction cache and the data cache. While writing our gadget on memory, we are actually writing on dCache. Now, when we try to control the execution flow of execute, sorry, to execute the gadget for running our shell call, the processor first check if the instruction are already in the iCache. Because we wrote our gadget on the data section, the cache always miss it. So, the memory content will be loaded into iCache. They cause the processor to execute a bunch of nope until the program crash. To flush the dCache, we need to make the processor content switch or exhaust the dCache whose size is 32 kilobytes. There is no slip in RADVD. Other functions we also trap into the kernel, but they don't guarantee the content switch will always happen. To make our exploit consistently successful in maximizing the chance to win the game, we have to flush the 32 kilobyte dCache. Luckily, the value in randomized VA space of router OS is 1, which means the heap is not randomized. The only things we need to do is do some memory allocation to extend the heap and write some gadget on it. We may run out of bytes for shell call because of that. But the DNS addresses were stored in a tree that caused lots of memory space. The heap is already larger than 32 kilobytes during the processing of a request. So, we can call the memory copy by GOT hijacked to write 32 kilobytes garbage on the heap to flush the dCache for our shell call directly. And profit. The vulnerability was assigned CVE 2020, 2023, 32154 and it's already been fixed. After that, we found the vulnerable call had existed at least since the router OS v6. This means the vulnerability has existed since 2013, nine years until we found it and ten years until they're fixed. So, no one with sanity would like to dive into the details of a normal binary, QED, you're welcome. Combine the exploit with our exploit for the Canon printer for the point to own. The attacking scenario is that an attacker is a bad neighbor of the microtik router, sending crafty ICMP v6 to attacking the router. And if we compromise the router successfully, we can forward our payload to attack the printer behind the router. The whole setup can be simplified as such in the contest. Just let our laptop act as the upstream router of the target router. We can almost see our 100k bolting. But, another but, our exploit only working on Mac OS and fail on Ubuntu, whether it's a VM or not. We try to debug the exploit by running the exploit on VM and recording the traffic, download traffic and replay it, it just fail as we expected. We also done the traffic from Ubuntu and replay it on Mac. However, it somehow succeed. So, we guess there is an OS that will reorder the packet. We put a signature on router to monitor the traffic from Mac and Ubuntu. And because the AF packet is not affected by any firewall chance, the result should be reliable. But they look the same. The last result could be that I have to go to torrento in person, but we can't live it like that because who knows if the bug will suddenly happen again during the contest, even on the Mac OS. Eventually, we find the problem. It's all about speed. The gaps look tiny and you may never notice in Washak. But the difference between them is actually 390 times. The problem is not on Ubuntu, it's just because the Mac sends to packet too quickly, causing rest condition. Plus, we were lazy and did not actually calculate how many bytes we need to override the return address. We just send a bunch of garbage and do the pattern matching. So, the solution is put a slip before sending the second RA and fixing the offset of our payload. Then, it's very stable. The probability of success is almost 100%. But how? The RADVD is only one thread. How could there be a rest condition? I didn't mention that while the RADVD parsing the incoming packet, the DNS address are stored in a vector. But somehow, the handler passed a DNS tree to the addDNS function. What's the relationship between these two members? It's actually about why we can't find where the Colbert function, multi-casular RA, when you receive an RA packet. Let's look closely at the Colbert function. There is an array responsible for storing something called a remote object. Then, it looks straightforward. Just eat the new DNS vector, then do RODNS on each of them and append the result to the DNS remote object vector. So, what is the remote object? The remote object is a mechanism to store common resources in the process, and other processes can access them via ID. For example, a DNS remote object is actually stored in the handle 2 of the resolver, and the RADVD only preserves the ID of the object in handle 1. Now, let's talk about the subscription in Nova binary. The binary IP pool 6 is responsible for managing the IPv6 address pool. The DHCP subscribes to it, so the DHCP can react to the change. It's done by sending a subscribed Nova message, and the sys-notified CMD can specify which command you want to subscribe to. The FE-000B means notification. So, if there is another binary, try to add an object to the handle 1 of IP pool 6. By sending a Nova message, the add object of handle 1 will be executed. In most cases, the add object will send notifies to notifies subscribers who subscribe to FE-000B to inform them that the IP pool 6 has changed. So, when the RADVD receives an RA from the upstream router, it calls RODNS to create DNS remote object for each IPv6 address in the RADNS of RA. The command add object of handle 4 in the resolver is responsible for such request, and will add the IPv6 object into the handle 2. Also, because the handle 1 of RADVD subscribes to the resolver's handle 2, the handle 2 will notify it with every DNS address it has now. So, handle 1 constructs the RA for LAN with least DNS addresses. The problem is in the implementation of RODNS. Let's send Nova message by post message which is non-blocking. It means the remote object will get the object ID immediately. So, if we send request too fast, the second RA arrives before the RADVD gets the object ID of the old remote object. The RADVD can only stop the delete those old objects rather than really delete them. They cause the RADS condition. This is what happens if we follow them step by step. Because both processes are single thread, let's assume they are in the first loop of the looper while the RODN RADVD receives the first RA from when. With one DNS address, the RADVD creates a remote object in the resolver by sending the normal message. And the resolver will set up a timer after receiving the first request because it cannot tell how many operations should be grouped. It should just set a timer and send only one notification when the time is up. Apart from notification, the resolver should send the response back to tell RADVD the object ID of the remote object it just asks for. But if we send the second RA quickly before the resolver sends the object ID back, the RADVD can only mark the old DNS remote object as destroyed. Then keep going to create the DNS remote object for the RADNSS in the second RA. They will send a request to the resolver, but as I say, we send the second RA too quickly. The resolver is still in the first loop which tries to send the object ID it created in the first loop, so our request will stay in the socket. Now, the resolver sends the object ID back. The response handler is a callback registered by the first request will handle the response. The handler finds out the object has already been marked as solved delete. So it won't update the object ID. Instead, it will post another request to the resolver to delete that object. The request will be processed in the third loop of the resolver after the request of creating the DNS remote object for the second RA. In the second loop of the resolver, it processes the creation request with the DNS address from the second RA and the response with object ID. Now, time's up. The timer of the resolver thinks a group of 18 operations should have been done, and it's time to notify everyone what DNS addresses it has. And because the handler 1 of RADVD subscribes to the handler 2 of the resolvers, it receives the DNS addresses in the normal message, then store them as a tree and use them to construct the RA for LAN in the RA routine. At that moment, the result of two requests is missing. That's why our first version of the exploit can only work on Mac OS because we trigger the REST condition. It seems hard to achieve a REST condition because there is a timer, but the mechanism is actually more complex than I just explained. So the REST would always happen as long as the two RAs arrive at the router in the short period. So the pattern of the REST condition in the remote object is as follows. Use non-blocking method to create, delete, remote object. And the target process subscribes to the remote object. Then we can mix the result of two requests into one. They may make us bypass some checks in the binary. We try to find other vulnerability with this pattern so that we can, so that we could register the LAN category of the router in PON2ON. But we ran out of time and did not manage it to find other vulnerability to exploit the LAN side of the router OS. But at least we are the first team to exploit the so-host mesh up category successfully and also won the title of Mesh of PON. Okay. In summary, MicroTik re-implemented everything with its own design IPC. The business logic is scattered all over. We have to have an abstract concept of normal binary to find the most critical code to reversing. And there is a pre-auth RCE on when that has been leaked for nine years. Also, we introduce a pattern of risk condition in remote object due to non-blocking method to you. And the last thing, we will drop tools that we use in the PON2ON event to ease the reversing of router OS at GitHub. And that's it. Any questions?