 Hi, I'm Carl Kosher, and this talk is a look inside the 737 Communications Management Unit, and some of the interesting things we've learned while reverse engineering systems. But first, the obligatory Who Am I slide. So I'm a research scientist at the University of Washington, but previously I was a postdoc at UC San Diego, where I did a lot of this work. In general, I'm a low level system security researcher. So I've worked on things such as RFID systems, looked at enhanced driver's licenses, transit cards, and built some anti-reland defenses. I also led one of the original car hacking teams. So we were the first to demonstrate a full remote compromise of a vehicle over the internet. I've reverse engineered wireless SCADA systems involved in power distribution. I've done some proof of concept malware against DNA sequencers and analysis pipelines, and I'm also working on wet lab automation systems these days. So of course, I'm not doing this work alone. There are a number of people involved in the AeroSEP project. And this project really came out of the previous car hacking research that we did. So if the people listed here, the ones in the top row and listed in orange were part of that original car hacking team, and that project had some pretty big impact. She thinks she's going to be able to stop right with those cones? Let's make sure that she can't, and she's going to drive right through them while complete control of that brake. All right, there we go. Oh no, no, no, no, no. No, no. No, no, no. Brakes didn't work. Oh my god. I can't operate the brakes at all. Oh my word. That is frightening. So obviously these results were pretty impactful and inspired DARPA's approximately 70 million dollar high assurance cyber military systems or HACM's program to make systems provably secure against attack. But avionics systems are already engineered to a high assurance level, albeit for safety not security. So we wondered if these safety assurance requirements translated into improved security. So the 777, 787 and A380 all have new and interesting architectures, but you can't really buy spare parts for those off of eBay. As it turns out, as 737 airframes are retired, parts do become available from auction sites. And so that's why we decided to focus on 737s. And there's actually a great website with a ton of information about the 737 that we used in our research, and they've even released a book. After investigating how the systems were integrated, we decided to focus on the communications management unit as it connects the rest of the avionics systems to the outside world. So for example, the CMU can take messages through the VHF, HF or satellite radios and relay them to other avionics systems. It's also used for sending messages between the pilots and ground. So since the CMU is sort of this central hub for all this messaging, it has a pretty large attack surface and it's a pretty juicy target for looking for vulnerabilities. So aside from the display unit, all these systems live in the electronics and equipment bay. This is a view from the E&E bay from the outside, and this is what it looks like from the inside. So all of these boxes are called LRUs or line replaceable units because they slide in and out and connect with an interconnect back plane, making them very easy to swap out for maintenance or repair. So in the cockpit, pilots interact with these systems through these control display units. Currently there were separate display units for messaging and for the flight management system, but they've since been combined into these multi-control display units or NCDUs where you can access different systems through one common interface. So for example, on the left, the MCDU is interacting with the flight management system. Well on the right, the MCDU is interacting with the communications management unit, which allows the pilots to send and receive messages, request weather reports, do a whole bunch of things. So to put the communications management unit in context, we first need to talk about ACARS, which started out as a simple messaging system designed to solve one very specific problem for airlines, and that was ensuring that they paid their employees only as much as they were entitled to. The way that it did this is through reporting these OOOI statuses, which basically indicates whether the plane is out of the gate, off the ground, on the ground, or back in the gate. And this is how airlines determine how pilots and flight attendants are paid. But since then it's been generalized to transfer avionics data. So for example, you can push a flight plan over ACARS, or you can request a clearance from air traffic control. Airlines typically have their own applications that they add on top of this. So for example, if you've ever received connecting gate information while you're in flight, that information was likely relayed through ACARS, and it's increasingly being used to replace traditional voice-based air traffic control communications, especially in Europe. So ACARS works over multiple RF link types, including the legacy VHF ACARS radios, the new VHF VDL2 data radios, over-the-horizon HF data radios, and even satellite links. And since the standard came out in 1978, and has largely been unchanged since then, these messages typically aren't encrypted. And when they are, they're often encrypted using questionable methods. So the communications management unit has some interesting requirements. So obviously air traffic control is flight critical. And so that software needs to be certified to a certain assurance level. But airlines want to add their own custom apps to these communication management units. And it's cost-prohibitive and extremely time-intensive to certify these. So how do CMUs allow uncertified apps to not take certification? As it turns out, there are two major vendors of CMUs for the 737, and they each take a different approach to solving this problem. So the two major versions of the CMU for the 737 are the Collins CMU-900 and the Honeywell Mark III. So the Collins CMU-900 has 46 for running applications, 386 to handle IO, and does inter-process communication through dual-ported RAM between the two processors. It runs the Vertex real-time operating system, which I'll talk a little bit about later, makes extensive use of x86 segments, which makes reverse engineering a giant pain. And they invented their own virtual machine instruction set that apps are compiled for, and this allows apps to be isolated without tainting certification. The other major player is the Honeywell Mark III, and it has a slightly beefier Pentium for running its applications and uses PowerPC to handle IO. And similar to the CMU-900, it uses dual-ported RAM for inter-process or communication. However, the Pentium runs Honeywell's custom DOS real-time operating system, while the PowerPCs use Vertex like the CMU-900. On the Honeywell Mark III, apps are database-driven, and the database processing code is written in a verifiable language that I'll describe later. So let's take a look inside some of these units. So this is the CMU-900 main processor board, and on there you have the 486, RAM, a CPLD for glue logic, a timer and interrupt controller, a real-time clock, and a serial interface chip. The IO board is responsible for interfacing with all the 429 buses. On the board, we have the 386, the dual-ported RAM, and some custom ASICs that we're pretty sure are 429 transceivers, along with some analogie bits to interface with the 429 bus. So this is the Honeywell Mark III processor board, and underneath that heatsink is a Pentium. There's also RAM, some glue logic, and a very annoying non-volatile storage system, which relies on a battery, which of course has gone dead by now, basically breaking the system. So the IO board on the Honeywell Mark III is a rather interesting beast. The first thing that pops out to me is that there are two PowerPCs, and through our reverse engineering, it seems like only one of them is used for IO, while the second supports re-flashing the system. So in the middle there is a CPLD, which we suspect interfaces the 429 transceivers with the PowerPC chips. There are a number of interesting connectivity options on here. So for example, there's an Ethernet controller on there with magnetics, probably for supporting a gate link upgrade, whereby you can push messages to the CMU over Wi-Fi. There's also a DSP chip next to an ADC, and we suspect that's used for sending A-cars messages over legacy VHF radios. But also interestingly enough, there's a V.90 modem chipset, and I'm not really sure why this exists, especially with that other DSP chip on there. Perhaps they were leaving the option open to plug a CMU into a phone line. I'm not really sure that would make sense on an aircraft, but it might make sense for maintenance. And of course, there's lots of JTAG connectors everywhere. It warms my heart. So let's take a deep dive into how the Honeywell Mark III is actually constructed. So to begin our reverse engineering journey, we sacrificed one board, removed the conformal coding, pulled off the flash chips, dumped the flash chips, and ran strings on the binaries. And what we saw next was surprising. This program must be run under Win32. Whack. By looking for PE headers, we were able to carve up the flash ROM into a number of interesting files. It also turns out the flash had multiple partitions. One seemed to be for flight mode, another seemed to be for update mode, and another for recovery. So as it turns out, Honeywell wrote their own real-time OS called DOS. And one of the interesting things about DOS is that its scheduler was ported to Java and formally verified by NASA to ensure that it always met its real-time requirements. As you might have guessed, it seems heavily inspired by Windows. It loads statically-linked portable executables. The APIs look a lot like Win32 or the Windows NT kernel API. And in fact, while reverse engineering the kernel, if I didn't know what a parameter to a function was, I would just go to MSDN and it would probably have the right answer. And it even has a concept of a registry, which is basically just a small binary blob of configuration parameters. So why is DOS so similar to Windows? Is it because they wanted to develop the CMU code against a Windows target? Well, this is pure speculation, but here's a map of Microsoft campus in Redmond, Washington. And right across the street is Honeywell Aerospace. So again, just speculation, but I'm guessing it was likely developed by ex-Microsoft employees. So outside of the kernel, the bulk of the CMU code is in a single executable called CMU Core. And this is written in a language called SDL, the Specification and Description Language. And this was created by the ITU. And it is a language that only telecoms can love. Metastructures are defined by ASN-1, and basically it's intended for building telephone switches. If you're familiar with Erlang, it's spiritually similar. SDL lets you define loosely coupled state machines that pass messages to each other. So it's ideal for protocol handling. SDL has a couple of forms that it can take. There is both a graphical and a textual form. The graphical form is shown here. So basically you write state machines that send and receive messages or triggered by timers. And you can also execute arbitrary C code. Now there's a companion language to SDL called MSC or Message Sequence Charts. And this lets you define protocol flows and test cases to validate your SDL against. And by writing test cases for handling 429 and ACARS messages, they can verify that the SDL code is correct. So like I said, SDL programs are composed of loosely coupled state machines called tasks. And in CMU core there are a lot of tasks. And many of them send messages to each other. Now I'm not going to read through this entire list here, but I encourage you to pause the video and take a look at the list. So at this point we have SDL code that is verified. And that SDL code is then compiled down to C code by a product called Object Geode, which was made by a company named confusingly enough Verlog, no relation to the hardware description language. But how do airlines make their own apps? Well, it turns out that the SDL code actually reads app logic from databases stored in Flash. So I'm sure you've all heard of SQL, but there's also another database language called Quell. And the database on Honeywell Mark III is an embedded version of Ingress Quell. Now the crazy thing is that queries are directly embedded in C files, which are then pre-processed into other C files, which are then compiled and linked against. So if you want to write a query that iterates through employee names, you can write it sort of as a pseudo for loop as shown here. And that'll be turned into a database query and that printf will be executed for every row returned. So unfortunately, these queries are directly converted into C code, so there are no query strings and executable. And furthermore, we don't even have any schema information since it's known at compile time, so there is no reason to store it in Flash. So how are these apps created? Well, Honeywell has this tool called the Ground-Based Software Tool, which is used both to create HGI, the Honeywell-Generated Information, as well as the AMI or the Airline Modifiable Information. And basically this tool lets you populate the database tables that define your app. And this is used both for certified apps made by Honeywell, as well as custom operational apps made by Airlines. So how do these state machines actually interact with real hardware? Well, these SDL messages are serialized and sent through DOS's Interprocess Communication Mechanism mailboxes to a process called DP-RAM-IO. And DP-RAM-IO takes these messages and shoves it into the dual ported RAM, which is shared by the PowerPCs. And the PowerPCs handle the low level interface to the hardware. So as I said before, one seems to handle the PCMCI update cards and those little else. The other one seems to handle almost everything. Unlike the Pentium chip, the PowerPCs run the Vertex Real-Time OS. And the code on the PowerPCs is also written in SDL. So this makes it fairly easy to pass messages back and forth between the x86 and PowerPCs. So Vertex is a fairly old but well proven real-time operating system. And to emphasize this point, here's an ad from 1983 where they will buy you a VW bug if you find a bug in their OS. So there's one tricky bit about passing messages between the x86 and PowerPC. And that is x86 is little endian, well PowerPC is big endian. And so the way that they actually solve this is with one giant switch statement based on the message type to do the endian as well. And that code is written in C, which of course is far more promising for bugs. And in fact, we found some. So we found some sign versus unsigned confusion and some integer overflows as well. Fortunately, they do not seem to be exploitable. So now that we have a general idea of how the CMU works, let's actually try going and see if we can find some real bugs. So static analysis of this auto-generated code is hard. So we'd like to do some dynamic analysis. Unfortunately, getting the CMU to boot was challenging. It does a lot of self-tests on startup. If something is wrong, it just won't boot. And we spent several months with it just blinking the fail light at us. Fortunately, we found some serial ports on the front maintenance port, which gave us some clues as to what was going wrong. So when we powered up the CMU, it would state that it was starting its built-in functional test. Unfortunately, after a minute, it would start spinning out error codes. And as it turns out, this was because the APM was missing. Now, the APM is the airline personality module, which has the tail number, the aircraft configuration, the type of aircraft it is, and a host of other settings. So if you need to replace the CMU, you don't have to reprogram it. You just pull it out, slide on a new one, connects to the existing airplane personality module, and the new CMU knows exactly what it needs to do. Unfortunately, the only APM we could find was for the Cullen CMU-900, which, despite being somewhat specified by an ARINC standard, was totally incompatible, both logically and electrically. Fortunately, the CMU saves a copy of the last known good APM image in Flash. And so we were actually able to create an APM emulator with an Arduino. It basically just emulated an iSports EEPROM. And we used the APM image that we pulled out of Flash. Now, the second issue that we ran into was NVGram battery was bad. And we didn't actually know that this was preventing the CMU from booting. We basically had to take one of the cryptic error codes out of the serial console, find where that was in the binary, figure out what triggered that string to be printed, and then discover that it was doing a test on the battery voltage. As it turns out, we just removed the battery and replaced it with a power supply. And that was good enough to make it boot. So there was actually a second serial port exposed on the front maintenance port. And once the system booted, if you pressed Enter, you got an interesting maintenance menu. Another weird behavior is that if you provided input to the first serial port, you would get another menu on the second serial port. So now that we can get the CMU to boot, let's try doing something with it. And to do that, we need an MCDU and a VHF data radio. Unfortunately, we didn't have those. So we ended up writing emulators for them and connected them to the real CMU over the 429 buses. And this really kicked off the development of a test bed we called Triton to support this kind of testing and security research. So the Triton test bed is designed to support a mixture of both real and emulated devices. So for example, we have a real communications management unit, but we don't have a real printer or airborne data loader unit. So we've created emulators for those. And these devices are connected through a 429 message routing daemon, which we call our 429D. And our 429D both supports real 429 interfaces, as well as accepting connections from emulated avionics units. Similarly for ACARS, we have a separate daemon that emulates different network segments. For example, we have one daemon that emulates the air segment. So all the VHF data radios are connected to it. We also have a separate instance for the ground segment and we have a virtual ground station that bridges the two segments. So a little bit about the 429 devices that the Triton test bed supports. This box here is an Ethernet to 429 converter, and this was one of the first things that we targeted. Unfortunately, it can only be configured with a static IP address and it kept getting virtually bricked by having its IP address randomly changed to something that we didn't know. And we actually suspect that UCSD was doing network scans against this device and it just so happened that it sent a packet to reconfigure the IP address to some garbage address. So after a couple of times of opening this up, pulling off the eProm, putting in the correct IP address, soldering it back on, getting it back on the network, only to have it bricked again, we decided to find a better solution. One of the primary interfaces that we use now is this Ballard USB to 429 interface. Unfortunately, it is rather expensive. We did also develop our own custom USB to 429 interface, which we called Scorpion. And if you ever saw the pilot episode of that TV show, you might understand why we called it that. It's basically an ATMEGA U2 with a whole 429 transceiver. It supports four transmit channels and eight receive channels. And this basically interfaces with the R429D daemon to expose a TCP IP interface to the 429 buses. So now that we have all of this wired up, we can start doing some really interesting things. So on the left here, we have our virtual MCDU, which lets us control our real CMU. Now connected to the real CMU is our virtual VHF data radio, which we can send real ACARS messages that we pick off over the air, alter the tail number so that our CMU processes them, which then forwards it to our virtual printer program, which sends it to a real credit card receipt printer. So interacting with the CMU is all well and good, but we'd really like to have some low level debugging capabilities. And as it turns out, the main processor board has a Pentium debug port. Now, unfortunately, the Pentium probe mode is undocumented. Details are in the infamous appendix H of the P5 external interface specification, which was available only under NDA from Intel. And back in the mid 90s, there were a lot of rants and Dr. Dobbs journal about this. There was one beta version of the P5 external interface specification that had these details, but all these copies were supposedly recalled by Intel and destroyed. So of course, there's no copies of this on the web. But while we were searching for it, we did find that there was one physical copy of an unknown version at an undisclosed university lab halfway across the world. So we sent them an email and asked if they had it. And if so, what version it was and that we might be interested in a scan of it. And so they wrote back with a scan of the title page, which was version 1.3. And unfortunately, we needed version three. So before we could write them back and say thanks, but no thanks, they wrote us back and said they found something very interesting inside. And sandwiched between the pages of version 1.3 was version three. And they happened to send us a scan of the entire thing, including details of the Pentium micro op instruction format, which is exactly what you need to use the Pentium probe mode. So armed with this information, I added P5 debugging support to open OCD. And so now we can use GDB against the CMU. Unfortunately, if you halt the hardware, the watchdog timers on the other processors will notice something to miss and reset the entire system. And the solution for this is to basically run the CMU's x86 firmware in QMU, but relaying messages to the power PCs over JTAG. And this is based on some of my earlier dissertation work on a system called surrogates, which lets you run firmware and QMU, but proxies peripheral access over JTAG in near real time. So now we can instrument the CMU firmware, collect IO traces, do snapshots and do all sorts of interesting dynamic analysis. So in summary, the communications management unit supports flight critical operations, which requires the firmware to be certified, but airlines desire customization. And through our reverse engineering, we've identified some somewhat unusual ways that vendors have isolated these uncertified apps from the rest of the certified system. And finally, we built a flexible test bed for further security research on avionics systems. So if you would like more information about our Triton test bed or surrogate system, check out our site at aerosec.org. Thanks.