 is titled, dissecting modern 3G, 4G, cellular modems. This is by Harold Welter and Holger Freyter. And that was totally mispronounced, sorry about that. Freyter. We saw in the previous presentations on Smart Cities that there's a lot of IoT that doesn't need to communicate. And while there's ZigBee and LoraWan and other protocols, most likely they're gonna fall back on what is really there, tried and proven and those are the 3G, 4G modems. I don't really need to introduce our speakers today. They're well known for years and years here at the Congress. So I'm just gonna pass it right over to them and have a great talk. So today we're going to talk about cellular modems just to differentiate it. It's not about baseband or baseband exploitation. It's really about a GSM module or 3G or 4G module. Our talk is going to be structured in a couple of phases. First of all, our motivation. Why are we looking into it? Where are we coming from? The second part will lead to the history. What have we done before in terms of modems? Looking at how we picked the modems we actually looked at, then something that we actually didn't expect when looking at it. Also, then looking at the firmware upgrade mechanism if there's one how it works, what has been done, and we will finish with our recommendations and wishes. First of all, we're implementing GSM specifications for more than a decade now. It started with humble beginnings of sending AT commands to modems on mobile Linux devices to actually working on a free software smartphone at OpenMoco and then working on OpenBSC and OsmoCom to implement radio area network software and core network software. And has been eight years since LaForge presented about how our Zen modern smartphone hardware looks like. It's seven years since we worked on OsmoCom BB to run our own baseband software on our commercial grade hardware. And professionally, we have worked with M2M devices and have built M2M devices ourselves using 2G modems. And from this point, we started to explore how which kind of device would we use in modern embedded devices or M2M or IoT devices these days for our implementation of 3G and 4G network software. So if you send messages over the air and you don't get a response, it's always difficult. Like, did you encode it correctly? Was it sent? So we looked into having a device that allows us to get logging to see if the message arrived or throws an hour, even able to extract traces from it. And what's also important for us and got us into OsmoCom and OpenBSC is to build tools to allow others to understand how cellular technology works. So while TCP IP might be well known to many of us, how the IP is actually transmitted back to the CoreNet and routed to the Internet is not set clear and we want to make it more visible and having technology and tools helps for it. For a brief moment, picture yourself trying to build our classic M2M device. You might pick a modem because it's already certified and easy to use, but you need to run some application code on it. And the traditional approach would be to get a microcontroller or a bigger processor and connect these two devices using USB or serial. But it means that you need to have a bigger PCB more power consumption, so it would be nice if you can run application software on the modem itself already. And one of the driving factors for it is to reduce the PCB space to have a lower power consumption to save on the bill of material to have fewer components. And back then we found something that's called OpenAT. It's done by Sierra Wireless, and mostly it allows you to write C software, which is then compiled with GCC and uploaded to the modem and you can start running it. It will be loaded into the real-time operating system. It runs as a normal process. There's no MMU, no privilege separation, so if your application crashes, the entire modem will crash. To ease debugging, Sierra Wireless has like a nice tools to get output and send AT commands and logging debugging. But the problem with this approach is that if you build an application and make it stable, you know so much about this OpenAT stuff that you're mostly locked into this API. So your architecture and software is following what an OpenAT application will look like. And then you spend years developing an application and suddenly there's no pass for 4G. So you're locked in and even your 2G modems will be discontinued. So it's a nice platform to get started, but it's kind of a dead end. And this brings us to our modern requirements. So what does a good modem look like? And one is we still want to be able to run our own code in it and not like some Python script or some limited Java, but like our real C application with access to the device and no artificial control. We don't want to be locked in by a single modem render. So it might be okay to use a specific ship set, but we don't want to be forced to follow whatever the modem supplier wants us to do. For debugging purposes, we want to be able to get log messages to see what C modem is doing, see if it's throwing away stuff, debug output, be able to control it. And for 3G and 4G development, we want to be able to see the radio messages. And you might know a tool called X-Goldmon has been written by 2B as Engel. It allows you to get tracing information from Infineon basement, but it has some limitations. So we want something like X-Goldmon, but for GPRS, Edge, UMTS and LTE. The modem market or general, the cellular market is kind of dominated by Qualcomm, and it's kind of set by itself, but it also means that if you pick a modem most likely, it runs or it's based on a Qualcomm ship set. And Qualcomm is very close to what we want from a modem because they expose something called the DIAC protocol. And it's also used in many different Qualcomm products from DVBH to GSM to FAM2 cells. The first time I personally heard about it was at the 28C3, but I talked from Guillaume that looked into the baseband Qualcomm stack. And this DIAC protocol, the framing is very simple. It's like classic HDLC with a start and an end marker with a comment bit, some payload and a checksum. And it's used for events, like if your modem is switching a network, you get an event, you can enable logging and then you get a lot of textual output, but also for comments and response. So you can send a comment to read a memory address and to get like the value from this memory address back. And literally there are thousands of different messages that can be sent or received. And in terms of free software implementations like the modem manager or GSM parser only use a fraction of these available messages. But it means like this DIAC protocol is something you want to have direct access to because it allows you to see what the modem is doing. And it brings us to the point of selecting a device that is exposing DIAC. And in the past, the option icon stick might have been a very good device to use because it's exposing DIAC on USB out of the box. But it's kind of old. It's using old Qualcomm software. It's limited to 2G and 3G. So we look into something more modern. And one of the devices we found is from a Chinese modem manufacturer called CrackTel. It's a UC20. It exposes DIAC out of the box. It's even documented in their hardware interface that it's a diagnostic interface. But sadly it doesn't support LTE. So if you look at a modern device, it makes sense to not go for 2G and 3G only, but also have an option to go for 4G, which brought us to the EC20. It looks like the UC20, but it has LTE. So that sounded quite nice for building a product that can be soldered to your BCP. But for development purposes, they also offer it in a mini PCI Express form factor. So you can just plug it into one of your devices that might already have mini PCI Express. So we picked the EC20 as a module we want to look at it. And once we started to look, it has the Qualcomm MDM9615 chipsets, which surprisingly is also used in the iPhone 5. But beyond that, there's not a lot of documentation of what it has, which brings us to the unexpected surprise. So after I got the modem, I used the AT-comment interface just to play with it to see if my application could do what it wants and was hanging after a day. And I got involved with the supply of the modem and they gave me a firmware update, which was a TIP file. And I unpacked it and all the file names looked awfully like a Linux system, which is why would my modem have a Linux system in it? But maybe I'm just mistaking it and then they have a flash tool and it looks like the flash tool of Android was fast, but maybe it's coincidences or it was... It was just convenient to use it. But actually other people have already seen that the MDM9615 runs Linux on it, like at the DevCon 26 by Mickey Sharkotov. So apparently it runs Linux on your modem. And the question is why? Why would it even run Linux? Because Qualcomm is known to have put their IP stack into the modem, IP stack, SIP. Why would they stop doing it? It didn't really make sense. So we started to look at it and also see there was no information that Linux actually runs on this device. Which means no written offer that I use the R-Mines GPL tools to untax easy flash file system. It really looked like Linux. And I started to look at some of the binaries that are not standard, but either QuakeTel or Qualcomm specific. And you see funny strings that look like AT commands, but like AT plus Linux or Q Linux command. What would it do? And at this point, we started to explore both technically. So what is this platform really looking like? But also from a legal point of view, like can we please get these source codes? Can you please put a written offer in it? And at this point I hand over to LeForge. Yeah, so yeah. So as a lucky coincidence, I have been doing some GPL enforcement in the past. So, well. Okay, but first we started to look at better the hardware. And well, if you're doing hardware anyway professionally and you have all the tools and the process and your partners for assembly and so on and so on, why not just do a couple of boards to help you with the task at hand? So many of the mini PCI Express modules that you can find for cellular modems, they have additional signals on undocumented pins of the mini PCI Express connector, like PCM audio or even URs and so on. And well, of course, if you just put the modem in a regular PC mainboard or an embedded device, those signals, they don't terminate anywhere. They're just not used on the slot side. And soldering wires to the pitch of a mini PCI Express board is not very convenient. So I created what we call a breakout board. And you can see a picture of that. It's an open hardware project, schematics and everything. Design files have been published. So you have this connector on the right-hand side which exposes all these extra signals so you can actually easily access the various undocumented signals. So the EC20 solder module documents, there are a debug UART pins, which are not the normal UART pins on which you speak AT commands, but it's additional UART. But it seems like not all modules have it enabled. So we bought modules from three different suppliers. They all have different firmware versions and different configurations, and some of them have it enabled, some not. But those that had it enabled have it at 1.8 volts. But how can I say, in their wisdom, the designers decided not to expose the 1.8-volt voltage rail on any of the other pins of the modem. So you have external voltages of 3.6 volts, 3.3 volts and so on, but no 1.8 volts. And since in previous projects, whenever I needed to attach to a UART of an embedded device, I built another level shifter for a specific voltage. I already had built for 2.8, 2.5, 2.3 volts before. I decided, okay, let's do a board that is a multi-voltage UART, so you can actually select the voltage of your UART with a small rotary switch and connect to serial ports of various devices. And that's what's here. It's also another open hardware project at Osmo-com. Yeah, and then you attach to the serial port and you get a Linux login prompt. And interestingly, if you look at the firmware update file that Holger just mentioned, it contains, of course, an ETC password file and there is only a desk hash. And if you do a little bit of password cracking, you see it's OE Linux one, two, three. It's the root password of the device. So by now we were 250% sure that there is no, there is Linux in the device. And yeah, this is by the way, how the full setup looks like. You can see I sold the small additional three pin header to get access to the UART on this module and this way we could start to explore it further. So, okay, brings us to the topic of well, where can we find the source code? The module didn't ship with a written offer. It didn't ship with a license text. It didn't include source code. It didn't even mention that Linux or other GPR license software was used. So while digging around the official supplier of the manufacturer didn't release anything or didn't provide any such information, we were looking a bit around because I mean, it's likely the other devices do the same and we found that actually Qualcomm is publishing a complete open embedded build system for building Linux for those modem chip sets. So this includes the open embedded meta layers, the kernel, the bootloader and various other bits and pieces. And it's almost completely undocumented. They're basically some Git refos and you can peek around them, but there's not really a documentation how to use it. And well, if you look at the entry level website of this open source project from Qualcomm, which is at code Aurora, well, there's like literally hundreds of branches and tags and you don't really know what to use for what and they say, well, this is one example version that you can build. And then somebody I think years ago posted, well, I tried your instructions, but well, it doesn't compile. I'm facing some issues. And of course, nobody ever responds because well, you know, why would they ever respond? But anyway, it's public. And on that website, you can find it and you can start building. So we started building that. Of course, we also ran into the issues that didn't build, but anyway, in the end, we managed to build some of the code. And meanwhile, we were talking to the manufacturer of the modem and asking them for the complete and corresponding source code. And then what they sent is the complete and corresponding source code to the firmware update tool that you run on your host PC, which we didn't ask for. And it was not GPL license. We still received the source code. Okay, nice. It's good. At that point, we could understand how the firmware update protocol works towards the modem. And then while you ask again for the complete and corresponding source code, I said, oh, we never been. By the way, all the typos and grammar issues are not introduced by us. So we never been in legal dispute and we always make sure to understand intellectual property rights ahead of using technology belonging to a third party. Well, clearly they did not in the Linux case and then they sent us this nice little letter. This is an excerpt from it. It's like, oh, we always respect the importance of intellectual property rights and laws and we actively engage with known essential intellectual property right owners, apparently the copyright owners of Linux and other free software are not essential. And so on in order to be compliant with the rights. So then you ask again and you ask again. So you see basically we always ask the same question. It's like, oh, we appreciate the efforts. This was to the lawyer of the Netfilter IP table project. They missed the fact that it's multiple tables with singular, but anyway. Well, then your client, me, doesn't have the right to empower copyright. It's like, ooh, okay, that's new to me. And they claim that I had transferred my copyright to the Free Software Foundation, which I never did and which nobody I think or at least it's highly unusual if you work on Linux kernel code to do so. So I did not have copyright. Okay, anyway. And call, sorry, my mistake, Quacktail always respects intellectual property rights, of course. Okay, so still no source code. Then we ask again and ask again. Well, then thank you for your detailed explanations. Da, da, da, da. We will provide a tar ball. And then we're always willing to achieve GPR compliance and so on and so on. And then another month or so passes and then I finally, well, we do some legal enforcement, we do some warning notices and so on. And then, yeah, we are not perfect and we cannot construct a perfect website which I never requested. We just wanted a source code. And then, well, you get some source code and it doesn't build. And then say, well, there is a header file missing. And by coincidence, it's a header file that I wrote. I don't know if they did this on intention. I mean, how many header files did I write? And if you know IP tables, there is a match for the differentiated services code point for the DCP field in the IP header. And that has a header file with just like eight lines of boilerplate definitions. And this was missing. And I said, no, we don't have this file. And Qualcomm also doesn't have this file. And Qualcomm never provided this file to us. I mean, it's in the public repositories that Qualcomm hosts on code Aurora and the kernel doesn't build without those files. And then, by the way, we will not discuss compiling issues by email anymore. Okay, then some more time. I get more and more. Then you get individual header files and individual emails and you put those header files in your kernel source tree. And then you see that the scripts in the kernel have missing executable bits, but certain C files suddenly have executable bits. And then another header file is missing and so on and so on. But by now we have received various source code tar balls. They interestingly contain not only the GPL-LGPL code, but also other code with like VSDs type or Apache type licenses where they wouldn't have to release it, which is good. And they did this intentionally. And I think it's a very nice sign of them that they don't release only what they have to, but they release more. For the EC20, it's still not license compliant. There is no busybox source code included, for example. Not that busybooks would ever have done any GPL enforcement in the past. And well, but that's not my primary concern. And I think it's getting there and it's workable and you can use the source code that they release. Interestingly, there's other motor manufacturers like Sierra Wireless, which are also building modems on such Qualcomm systems. And they release not only the source code, but extensive documentation. So this is a small excerpt from a screenshot where you can actually, oh, they describe how to build it with open embedded. They describe how to use fastboot to install the firmware on the module and so on and so on. However, as good as they are in the open source side, they try to lure customers into a proprietary framework like they did with OpenAT in the past and that, well, again would result in vendor lock-in. So it's not really recommended or I think a smart move to go ahead that way. And with that, I'm going to hand back to Holger before returning later. We're going to very briefly look at the hardware. It's a Qualcomm MDM ship that I've already mentioned, also in iPhones and it turns out maybe in your future car. At least right now it's in modems from QuackTel and Sierra Wireless, but sadly from a free software point of view, like it runs Linux, it talks to the hardware, but there is absolutely no documentation about the hardware in the internet. Like even on other websites, not nothing because it brings us to the hardware overview. So we know there's a ARM processor inside, probably hexagon, they're connected somehow, nothing. So that's very frustrating to have spent many years on free software to see Linux winning, even getting into the modem devices, but no hardware documentation being available. Not even a block diagram. You have to repeat it. Not even a block diagram here. Yeah, so nothing. Let's look at the software part and explore the system from a software point of view. LaForge has shown how to get a serial console on it, but not every modem has it enabled and we didn't look at what it takes to enable it and also soldering is not that nice. So after exploring the root file or the unpacked root file system, so we saw it runs an Android debug bridge. So if you have used Android, so ADB shell should give you a shell. And nicely, we have already seen the AT plus Q Linux command to execute something on the device and we found a script that is reconfiguring the Android USB gadget to actually put ADB in it as well. So let's try it, you execute the AT command and then suddenly the serials on your host Linux don't work anymore because the way the QC serial kernel module is written, it's matching a device based on the number of interfaces and if you added ADB to it, you suddenly don't have four interfaces, but five interfaces and your driver cannot identify the device. So we had to first hack it a bit and LaForge has made a more clean patch to actually get it. But after the small experience, we have ADB shell on it. It works on any module, but that's... And the ADB shell is root, of course. I mean, so you don't even need to root the device. It's like you get the root shell immediately. So there's no lockdown as well. It's root, there's no SE Linux and just a very nice and open Linux system. To build it, it's a bit odd that it has a bootloader that seems to be proprietary, then it has the Android bootloader and Android Linux kernel, the Android debug bridge we've mentioned, but surprisingly, it's not using the rest of the Android system, but it has a GNU Lipsy with Busybox, UserLand, System5 in it. So it's a very classic, open-embedded build and it's actively developed and maintained by Qualcomm, but they make so many releases, you actually don't know which one to pick. It's a bit of a zoom. Yeah, and then you start to look a bit at the Linux kernel that is released and luckily and interestingly and to my pleasant surprise, there's no binary-only kernel modules, so everything in the kernel is released in source code. Nevertheless, of course, it's a lot of source code, so if you look a little bit at the number of lines, a diff shows up between the closest mainline version and the kernel that's used in those modules, you end up with 1.5 to 1.9 million lines of diff. I mean, this is not actual code lines, this is counting all the lines of a diff output, including the context, but still it gives you an idea about the size of the differences compared to mainline. And of course, you expect on those kernels that there's all the CPU-specific stuff and lots of driver code and so on and so on, but then if you look at it in more detail and as a disclaimer, I haven't looked at Qualcomm, Android, Linux kernels during the past 10 years or so, well, let's say eight years, no, six years, whatever, a long time, and I know there's a lot of code in there, but I didn't expect all the different things that I found in there. I mean, they have their own shared memory-based logging infrastructure and shared memory is not shared memory with the modem processor, it's only shared in the Linux system. You have an inter-processor communication logging process. IPC is not inter-process communication, like you would know it's inter-processor communication. And you have something which completely flabbergasted me, it's called remote spin locks. I mean, you know, spin lock, you basically, you have a mutually exclusive, exclusion mechanisms are only one thread one CPU can enter a critical part of the code on your multi-processor Linux system, but then here you can actually also block the modem processor, the hexagon on the other side from entering a particular section. What could possibly go wrong if you hold, if you keep the real-time operating system in busy waiting, but okay. Yeah, then you look at the source code and I've actually, since I haven't looked at Qualcomm kernel source code for quite some time, I was expecting, well, this has been all these Linux Android phones that have Qualcomm chipsets and lots of, since it's open source, plenty of people must have analyzed it and there's certainly some documentation, some high-level overview about all these individual subsystems, how they glue together, how this works, and I could just look this documentation or look at that documentation. But interestingly, it doesn't exist, so I had to start to write that documentation. There's now a lot of information in our wiki and some of the interesting parts that you find in there is the shared memory device, which is the core of all the interaction between different CPU cores, you have inter-processor communications, you have remote network, you have a BAM, that's also a BAM to BAM, is the bus access manager, and we have IPA, this is not your favorite beverage, but the internet packet accelerator and there's some diagnostics forwarding and so on. And if we look at these SIP subsystems, I draw a graph, you know, if you normally know how rare that is, I draw a picture. And it, to symbolize you what's happening in this modem. So if you look at this picture, you will see basically the large square at the top is the application processor, the bottom small part is the modem processor, about which we don't know much, of course, but then on the left side, due to the availability of source code, we can see what's happening. So you have the shared memory device and then you have channels implemented by the shared memory device and individual different subsystem binding to those channels as two for AT commands, which attach to the serial function gadget of the USB gadget code in Linux. So basically the important part is to see that the USB you speak, you don't speak to the modem actually, to the modem processor, to the basement processor, but you speak USB to the Linux gadget inside the Linux ARM core on those devices. And then that Linux ARM core forwards or handles different interfaces on your USB configuration in different ways. And you have this small box symbolizing the user space and you can see how the different paths go. It's quite interesting if you look at the serial ports, you have a serial port for GPS and you have a serial port for AT commands and you think, well, okay, these are both serial port devices, they must be handled quite similarly, but no, the GPS part is actually handled here, over here it goes into user space, it's bridged, it goes into a virtual serial port here again and it goes into the serial gadgets, whereas the AT commands go straight here inside the kernel and never end up in user space. So I don't know why, but it's quite sophisticated to say. If we look at the diagnostic subsystem, which is particularly of interest to us, and now you can see I didn't draw graphs anymore, I just wrote a little bit of dotty. We have the modem DSP on the left-hand side, we have the, then we have the Linux kernel where we have the shared memory device. On this, a DIAC forwarding module in the kernel binds, on that, we have a connection to the diagnostic function gadget of the USB gadget driver and that just goes to your host. So if you talk the DIAC protocol to the modem, it actually goes this way through Linux, through a shared memory device in the modem DSP. But what's even more interesting is that there is a diagnostics character device on Linux called DevDIAC, which, well, for example, to QMocs to your other processes, they basically attach to this diagnostics device and all the logging that you find in those Linux user space processes, they don't use this log, they don't use an Android logging framework, but they log through the Qualcomm diagnostic subsystem and you get the log messages of those processes through this kernel device over to the function device, over to USB into the host. So if you manage to figure out which logging flags and so on to enable, you get the log output of those Linux user land processes over DIAC. Well, very Qualcomm-like, not so surprising, but still sort of unusual in the Linux world. If you look at the networking, the QMI, which controls your, basically, the modem, which network you attach to, whether you activate PDP context, your QOS parameters and so on, you might have used QMI, CLI, or other tools on your Linux laptop to talk to such modems. And in this specific case of those Linux-based Qualcomm modems, well, you again have the modem DSP, goes through the shared memory device, talks to the RMNet device, the USB gadget, USB to host, and you have your host PC somewhere on the right-hand side over there. So this is basically the path your QMI takes. But then you have also QMI in the user land on the modem itself, which is what's presented here. And they have what's called the QMuxD, the QMI multiplexer demon, which then offers Unix domain sockets to various different client programs. So basically, all these user space programs, by using this Unix domain socket, they can talk QMI to the modem as well. And all of them can basically configure, get status reports, and so on, all the different parts and all the different services on the modem, which is interesting and it's across something that we want to have coming from our initial motivation. We want to run our own applications in there and we want to talk to this QMuxD and talk QMI in there. So we created a couple of tools to help the analysis. On the one hand side, we used the open embedded that Qualcomm released to build a matching OPKG and the packages for tools that you need, like Socat, S-Trace, LSOF, and so on, for some exploration. We also have written a couple of C programs for testing, basically for accessing the QMI from code inside the modem that's successful. And then we have a couple of, those still link to the proprietary libraries that are provided in the modem. And then we have started with some entirely free open source programs, like the QMuxD wrapper, which is an LD preload library, so you can trace this QMuxD communication. There is ongoing work for LibQMIGLib transport for this QMuxD, which then would enable you to run a program that's linked against the free software LibQMIGLib inside the modem. So you can develop it like you run it on a laptop but you can run it transparently inside the modem after cross compilation. There's also a tool which we call OsmoQCDIAC, which is basically a host tool for obtaining these DIAC-based logs from the modem. So you can run this on your laptop, attach it to the modem, and then you get all kinds of traces, not only the interface traces, but also QMI protocol traces, which we then again decode using LibQMI, so you get a textual representation. There's ongoing work here to basically move all of that into Wireshark, so you get the full decode of that in Wireshark, but that's not yet there. So what kind of user-based programs do we find? Okay, ADVD, we know what that does. We have an ATFWD daemon, well, AT Forwarding daemon. Well, what does it do? It implements those things like an AT plus QLinuxCMD and other AT commands. So basically, a user-based program on Linux can register a callback within the modem to forward certain AT commands into the Linux user space where you can then implement them. So you can basically implement custom AT commands in user-based programs. There is all kinds of other software, which we haven't really figured out yet, a lot what they do. There's one monster called the QC Map Connection Manager, which basically allows you to run a Linux-based Wi-Fi access point with LTE backhaul, excuse me, for the typo. So basically, you have to attach a Wi-Fi chip to the SDIO interface of your modem module, and then you have a full personal access point device that has an LTE backhaul. And then the Wi-Fi, like the parameters, for example, the key and the SSID and the channel and so on, you configure all of that through what? AT commands, of course. If you ever wanted to look at software that receives AT commands and then generates textual config files for a WPA supplicant and for host AP demon, then you can look at this code. I prefer not to. So we have the Quacktail Bridge, which, well, is a very simple device. It reads from one device and it writes it to another device. And apparently, this is such a complex task that you need to schedule a different process, and I think it has three threads or something obscure like that. So, okay, well. Which brings us to the funny bits and pieces that you find in those modems. Well, the first thing is AT plus Q Linux command. We spoke about that. You can run arbitrary shell commands as root in a read write rule file system in there. So, basically, you can do anything. I mean, you can send an RM dash RF slash and then, well, it's gone. It's dead gym. So we also have commands to switch into fast boot mode so you can update the firmware. You have a special AT command to print a D-mask and you have also all kinds of other AT commands. And when you do a strings, just a strings call on those executables, it looks like shell scripts in many cases. And one of the most funny things I found in this modem is how many processes and threads does it take to reboot a system? It's apparently a very complex question. How do you reboot your system? Apparently, this was the easiest method they could find. So, there's one process, the reboot diag app, which registers a diag command with command code 0x29 with the diag infrastructure. And it spawns a thread which executes another executable called QMI simple real test with an input 5.0. And then it calls system echo modem reset to write modem reset into the 5.0 of the input of the process it just spawned, which then causes this to send a QMI message to the modem, which will reboot the baseband processor. And then, of course, you clean up, you remove that temporary file, because slash TMP is not a tempfist, but it's read write file system. And then it goes on to write the string reboot into slash def slash rebooter def using fwrite, not using echo this time. This is a C program, not a shell script. So apparently, they discovered that and used fwrite. And then we have a reboot daemon, a second process again with two or three threads, which reads this rebooter device. And then actually they published the source code, so this is the actual source code, right? So you read from this def rebooter device, then there's a nice comment documenting what it does. You do a string compare. You have the first printf going for reboot, you have the second printf initiating reboot, and then you call system on the reboot executable. So this is apparently the most simple method. So if you ever were wondering how we would reboot your Linux system, this is the new reference. Yeah, then we have C programs that look like shell scripts. So this is an actual, of course, condensed output of strings on the QuackTill daemon, right? You see like echo in SysFS files, you see copying files, even with semicolon in there. You see echo into the duty cycle of some pulse width modulation, no idea what that does. And then they even grab for the process and they kill processes, you know, with the most obscure things. And even they don't use reader, but they do like LS and then parse the output of that rather than use opendir reader and the usual calls you would do to get a list of files. It's quite amusing. Yeah, which brings us to the topic of firmware updates. But you can, I actually missed this example. Sorry, I have to make a quick interruption here. All these machine to machine modems, they typically have an LED and the blinking rhythm of the LED indicates to you whether it's registered to the network, whether the data connection is open, whether it's searching for networks, that's all different blinking patterns of the LED. And how do you implement this on this modem? Well, you run a user space daemon that calls system echo one to the dev GPIO which controls the LED all the time, you know, to blink the LED. It's not like the kernel wouldn't have infrastructure for LED blinking patterns and so on and so on, but okay, so you have a daemon that does nothing else but basically toggling your GPIO by spawning shell processes using the system syscall. Okay, with that I hand over to Olga. Now the question is, do you expect anything after the topic of firmware upgrade or is it going to be an empty slide? And the answer is, they know that they have to offer firmware upgrades through over-the-air updates making it as small as possible. And actually it's something that Qualcomm is preparing for modem vendors. So it's based on the Android from around 4.0 recovery kits not using Android myself. I haven't looked at how it worked before. Many of you might be more familiar with it, but it's mostly a SIP file and it contains Delta updates and surprisingly they are somehow hashed to a SHA-1 and then the SHA-1 is being signed with a private key and the result is being put into a comment of the SIP file. That's probably pretty standard but looked a bit odd and it's nice that they prepare probably secure firmware upgrades with like minimal Delta functionally. So what has QuackTel done to this code? They still use it, that's quite nice but they have removed or actually not really removed but they're not using the RSA code to verify a signature and instead of using the standard Android way to patch these systems they use a proprietary component from a company that used to be called Red Band but now it's Harman and probably soon Samsung. And this Red Band component is nothing that QuackTel has written but it's a commercial product and it's used in the QuackTel UC20 module as well and also in other automotive projects we have seen Red Band updates being used. So at this point I started to look into how does this update format look like and instead of presenting a very complex form of the format I have this slice. So other people like Matthew Solnick I think Black Hat presented something called attacks on all my device management. These can be remotely triggered but the update mechanism used here cannot be remotely triggered. The modem needs to be asked to start an update so that's already a bit more secure but I started to look at the hackstamps of a specific Delta update and with a lot of help from Detaspa we actually managed to understand how the update binary looks like and we have created a small tool to take an existing update and put it into smaller parts and also be able to create our own defiles. The format itself has many different pointers and offsets so in the example you might already see the offsets here and compression size so it starts with a common header and then after the header you have a LZMA compressed table of contents or however you want to call it and outside in the header you have an offset into the decompressed version of this table of contents where actually file update starts and when you start creating your own file you might get the offset wrong and the update binary just crashes with your malformed update file so it's like not very robust code, a very complicated file format and nothing is cryptographically signed so when you use strings on the binary you see the word signature but it only refers to a CSE32 checksum. The next part like now we understand how the update format works. We can create our own update files. The question is how do they end up on the device and it's something that is implemented in the AT Forward Demand that Harold or LaFoche has mentioned and if you grab for some specific strings like WGet, this QC map connection manager or photo or update.zip you already kind of guess how this application works. So you issue an AT command with a URL for updating it. In turn it will disable your normal IP connection that you have my established on your host and opens our PDP context on the device itself using the QC map connection manager then it will spawn WGet to download the file then it will use system to move it to the right directory. It will remember what it wants to do with this file and it will reboot into the recovery partition and system and at this point the update zip file will just be applied and the system will reboot once again. Again, without any cryptographically signature or checks. So if you manage to hijack the update process you can install any binary on the device or on the modem or anywhere else on the system as you want. But instead of just seeing how bad it is we want to say like what do we expect them to do and I'm handing over to Harold again. Yeah, so rather than saying oh this is all bad it's all unlocked and it's insecure and so on well it's fun for us of course because that's what we wanted, right? We wanted a modem device where we could do basically whatever we want to and where we don't have to break sophisticated security mechanisms that are designed to keep the user or the customer or the owner of the product out. So yes there are security issues and those security issues must be fixed but we need security mechanisms that work without locking out the user or the owner of the device of course. So this is our public call to the manufacturers if you fix those issues keep in mind that the openness of the platform is interesting for all kinds of legitimate use cases and while you want to protect against malicious attackers you of course still want to enable the actual owners of the device and the users of those devices to use the flexibility they provide because there's a lot of, yeah, thanks. Yeah, so what's the status today in Outlook? Well, we have just opened the wiki on the, it's on the last slide, there are all the links so on the Osmo-com project we now have a wiki for QuakeTail Qualcomm LE modems where all the information that I gathered from reading the thousands and thousands of lines of source code is in there. We have released the debug tools presented in this talk in a Git repository and source code. The hardware boards are released as open hardware and available and what's unfortunately still ongoing is the libQMI GLIP integration which is well to some extent to the fact that I've never written anything against GLIP before or anything inside a program that uses GLIP. It's like all this infrastructure. I'm usually more low level than that. And while we hope to grow this documentation and we kindly invite all of you with an interest in understanding those platforms better to help us out, you don't actually need to necessarily reverse engineer and disassemble things, it's just read the source code, understand what it does and play a bit with the device. We are planning an open embedded package feed so we can actually easily install additional and everyone can easily install additional packages on those modems. There's plenty of flash, I think it's like 20, 30 megabytes of free flash so you can install quite a number of additional packages in there. And our aim is to have free software only user land on this Cortex A5 CPU so to do away with all those proprietary processes that run in user space and the libraries and run the open source kernel and have basically the libQMI GLIP integration and all other bits needed to run our own standard Linux user land code in there and have custom images that we can run on modems for all kinds of use cases. Okay, now before we go for Q&A in a minute, there's an unrelated announcement that we would like to present here. The OsmoCom project has gained support for running your own 3G 3.5G network during the last year. Not sure who has noticed that, unfortunately it suffers a bit of a lack of contributions. So we want to motivate people to contribute more and we have just started an Accelerate 3.5G program which provides 53 3.5G femtosells to people who can convince us that they would contribute something reasonable to our project. So these femtosells are already supported by the OsmoCom code so using the femtosell and the OsmoCom code you can run your own 3.5G network and we're giving those away for free so if you're interested in any of that, please submit a proposal until the end of January and then you will hopefully receive your free femtosell until the end of February. Which brings us to questions. Yes, it's Q&A time. We have a total of eight microphones here. One, two, three, four, five, six, seven, eight, I can do math. Please step up to the microphones to ask your question. Meanwhile, we have from the Signals Angel we have a question. Yes, the internet wants to know if there will be, in quotes, next open moco. Not from us, no, we're not working on a mobile phone. We're looking at modems that not at mobile phones at the moment. So microphone two. So just to clarify, does that mean that Linux runs on the iPhone 5? We don't know. We don't know, we're happy to hear from you if you can find it. But I think you can run the trip with something else on this Cortex processor. But we don't know. Signals Angel? Yeah. The internet wants to know why there are no mini PCI Express slash M2 to USB 3 adapters because they think LTE is capable of at least 300 megabits and USB 2 could be a bottleneck. Well, I'm not really in the business of manufacturing or selling adapters. So, I mean, yes, we did this open hardware device out of a specific need, but you would have to ask the hardware manufacturers that I'm sorry. Could I ask the people who are leaving to leave quietly, please? Microphone two. Can you buy the entire EC20 with the breakout board and the UART as a single kit somewhere? Well, you can contact us, but it's not really something that we have prepared for. But yes, it's certainly an option. But I mean, we're not here to sell you anything. We're here to invite you to help us learn about modems. Thanks. And with the Android debug shell, you actually don't really need the serial. So you can refresh the device using USB. Even if you break it, it can be refreshed. And with ADB shell, you have your root shell. So that works. And with ATQ Linux command, you can actually start a lock-in on the NMEA console as well. So even if ADB doesn't work, you can get a lock-in on it. So it's very easy to get started without serial. Signals, Angel? Hello, okay. Do you have, have you tried to get a source code from other manufacturers? Well, we know at this point of, I think three different manufacturers that use this MDM9615 plus Linux combination. From QuackTail, we have just described how it went. Sierra has published all this already by themselves in a very good way. So there's no need to ask them. It's out there. You can just download it and it's documented. And the third one is an old end-of-life Huawei module where we have also asked, but this is still ongoing. They are now, it's interestingly that from a Chinese supplier, you get the excuse, oh, there is Christmas coming up. And it was like, okay, is there so many Christians in China? But okay. Well, I think I'm quite sure if I ask in January, you guys, oh, there is Chinese New Year coming up. But we'll see about that. Microphone 4. You mentioned the Qualcomm chip used in the iPhone 5. Does this mean, well, is it likely that this, the iPhone application processor talks AT commands to this Qualcomm chip? Is this still a state of the art? No, it's not state of the art. And I don't think it's happening. I mean, also on these modem modules, the AT command is there for legacy purposes. You have the QMI exported over USB and normally your modem manager or whatever you use, a phone or whatever infrastructure talks QMI to those devices. I'm gonna go back to the signals agent again. Yeah. The question is, what is the total size of the flash or the root FS on there and how much RAM has this thing got? I don't know. I don't remember the flash size. Was RAM a 32 megabytes? Yeah, this 32 megabytes of RAM on this, the successor, the EC25 has 128 megs of RAM, but a flash size I also don't remember right now. But check out the Wiki. We have boot logs and all kinds of like D-mesk outputs and so on all on the Wiki. So I'm quite sure it's somewhere in there. Microphone four. You said that you tried to check Legato Linux from Sierra. Wouldn't it be possible to completely build your own kernel there and your applications? Yes, it's certainly possible, but then of course, if you do this, you buy a lot of things with your money that you don't use in the end. So, and you support a vendor that tries to lock people into this proprietary framework. So it's, yes, it's possible. But do you have to use a framework? I don't think so. No, you don't have to, but I mean, they developing this and part of what you pay for is this framework and that's why their products are more expensive. So. Microphone two. Yes, do you have some ballpark figures about the power consumption in the lowest power quiz and current mode? No, I'm quite sure it's going to be high. I think Holger has some comment on that. So when you look at this device, you use S-trace against the QMux D and it's like mostly waking up every couple of milliseconds. So it's not power efficient, the C code. It's really, really, really annoying to see processes that run all the time. And from the previous slides, it's echo mem to CISFS. That's actually the advanced power management. So you can trigger an AT command to have the device sleep and it writes echo mem into it to go to sleep. So it can be tuned and with a free software user space, it can probably be better than what it is right now. Signals Angel again? Yeah. How accessible is the baseband? Like memory and like EMI and stuff. We haven't really investigated this in detail, but we also haven't seen any signature verification and so on there. So I think it's completely open. So yeah, the DSP firmware is in three separate partitions and there's no RSA signature and it's the end of these partitions. So it seems that you can modify it if they have locked down like the poke command. That's something we haven't tried or looked at it. Microphone one. Are the modules readily available in the LGA package rather than the mini PCI dev board? Like if people want to start trying to use these in open hardware? They should be available again, yes. I mean, they were temporary, not available due to the TPA enforcement, but I think now they should be available again. And what are the costs of those modules like? I think it's like 47 something euros around that region or maybe, no, it was more than 50, but anyway, somewhere in that region, yeah. Okay, thanks. So one more question from the internet. Yeah, and the internet wants to know if you can capture layer two network stuff with this. Yes, yes, you can. So I think that's the end of our Q&A session. Please join me in thanking LeForge and Holger on this fantastic talk about things that I really didn't want to know about 3G modems.