 Let's welcome Leonard Wolter from Belgium talking about Starlink Compromise. Okay, good morning everyone. Thank you for being here. This talk is titled Glitched on Earth by Humans, a black box security evaluation of the SpaceX Starlink user terminal. A quick introduction on Starlink. I guess everyone knows this, but for every satellite internet system, you need a space segment and an earth segment. In the space segment, we have satellites. These satellites can, in some cases, communicate with each other over laser links. And the idea is that you as a user buy a user terminal that sends you data up to the satellite. The satellite relays it back down to earth through a gateway. And in this way, you can access the internet. Now, for some reason, SpaceX wouldn't give me a satellite, which meant that I had to buy a user terminal and try to get in it that way. So, we bought the user terminal, we put it up on the roof of our university building and connected it to the network. Everything was working well. But at that point, we just had a second internet connection and there wasn't that much to do with it. So, I started looking at what other people had done or what other people were working on at the time. So, the top three or the first three videos on the top were already out when I started working on that user terminal. So, some people had bought the user terminal, tore it open to see what was on the inside. The video by the signal pad went in more depth about the RF side of things. And then, when I started working, some other people were also working on it, like Colin O'Flynn did a tear down with a blowtorch and he started poking at the board a bit more than the others. There's a nice blog post by Olek on the Wi-Fi router that you get with the system and then Dan Murray did a tear down of the square user terminal and even made a 3D printed case for it. Now, one thing that was missing for me at least in all of these videos and tear downs was that I was mostly interested in the security of the system and that's what this talk is about. Now, I think, I've already mentioned there's a few hardware revisions of the user terminal, but I think very few people realize how many hardware revisions there have been over the time. So, the circular user terminal, the one that we have here on the stage is about 60 centimeters in diameter, was meant for residential use and as you can see, there have been multiple hardware revisions of it. You can also see that there was an SOC cut three and cut four, so meaning that there have been at least two silicon revisions of the main system on chip. Then we had the square user terminal. This is often referred to as the third revision of the hardware and you can see, even of that third revision, there have been multiple revisions. Then there's also a high performance user terminal now that's more meant for business use. And finally, we have transceiver units. There's some information about these in the code on the user terminal, but there's not a lot of public information. So, the idea here is that you have a transceiver box to which an external antenna is connected. Now, the research I did used the user terminals we could buy at the time and that meant that we worked with a ref two proto two and a ref two proto four. At some point, I broke one of the two and then it was replaced and that's how I had basically two dishes. So, what you can see is that I worked with an SOC cut three and SOC cut four. I did the attack on both and this also means that the attack will work on all of the other hardware that's listed on the screen because they all use an SOC cut four, at least for now. So, this is what you get if you open up the user terminal. You're greeted with this big metal shield on the inside. There's a small cutout that reveals part of the printed circuit board with three connectors. The first connector is for power over Ethernet. The second one is for motors. So, normally this dish can orient itself towards the satellites. And then the third unpopulated connector in this case is a UART connector. And UART is always interesting if you're hacking hardware because often times it leads to a root shell quite easily. Now, that wasn't the case here. So, you can see that while the user terminal is booting, it prints out some information. You can see it's using the U-boot-boot loader, but you can also see that SpaceX disables all of the serial input. So, whatever input you would provide over the serial console, it would be ignored and you wouldn't be able to do anything. If you allow the dish to boot fully, then you get this login prompt. You can try to enter as many passwords as you want. It won't work because there is not really a password set. As long as the dish doesn't believe it is development hardware. So, that meant we had to go deeper. So, we removed the metal shield from the PCV. And then you see this big PCV on the inside. The one that's also here on stage. And there's a few interesting areas on this board that we can mark. So, you have the area with the GPS receiver. There's an area with a clock generation circuit that's used to generate a clock for the SoC and for some other parts. And then there's a power over Ethernet circuit, of course. And then the part you're most interested in is the SoC. So, this part contains the SoC, some EMMC memory and DRM. And then there's, of course, one obvious part on this entire printed circuit board that I haven't circled. And that's all of the other parts, basically. And the remainder of the hardware looks like this. So, there's a digital beamformer made by SD Microelectronics. It's codenamed Shiraz. And each of these beamformers is paired with 16 frontend modules. So, these are the smaller chips on the screen. And their codename is Pulsar. Now, when I was working on one of these dishes, I had broken it. And SpaceX was nice enough to replace it. But I didn't want to just let it go to waste. So, I desoldered a few of these chips from the board. And I sent them off to John McMaster. And John was nice enough to decapsulate these chips for us and image them. And the pictures are online now. So, you can view these pictures in higher resolution in your browser. And there's a few interesting things you can see here in the picture. So, the chip on the left is the beamformer. And you can see that the same pattern is basically repeated 16 times. And these are the 16 channels for all of the frontend modules. On the right, you see a frontend module, which is divided into two parts. And it has basically a receive and a transmit pad. Now, this is the area on the PCB that we are most interested in for this talk. And it contains a system on chip. A system on chip is a custom quad-core ARM Cortex A53 processor made by SD. And as I said, they're having multiple silicon revisions of this chip. The code name for this one is Katzen. You can also see that there's a separate secure element. And at the end of the talk, I'll give a bit more details about what this is used for. Then we have some EMMC memory. So, this stores all of the firmware of the dish. And then, finally, we have 8 gigabits of DDR3. In red on the slide, you can see some pull-up and pull-down resistors. And these are basically used to indicate to the SOC which hardware revision of the board this is. Now, what you can see on this picture as well for the system on chip is that it has this metal shield on top. And this means that it's a flip chip package with an integrated heat spreader. If you remove the heat spreader, you see the backside of the die. A backside of the die means that you can also make images of the chip through the backside. So, this is a picture we made in our lab with a laser fault injection setup. And a picture like this is useful if you're trying to do physical attacks. Because for some physical attacks, you want to target a specific area of the die. So, one thing you can see here in the bottom right corner are the four CPU cores that this chip has. I donated this picture to the Silicon Prom project, so you can also view it online. So, the first step now was to extract the firmware from the dish in the hopes of finding an easy software vulnerability to get into it. To read an EMMC chip, you basically need three test points, command, clock, and data zero. The way I identify them is by soldering wires to all of these test points, then connecting those to a logic analyzer, restarting the user terminal, and then by just looking at these signals, you can figure out which signal is clock, which one is command, and which one is data zero. I have marked them on the slide, so if you want to redo this at home, it should be easier. Now, an EMMC chip is quite similar to an SD card in many ways. And that also means that you can use an SD card reader to read out the chip. So, on the left, you can see an SD card reader, paired with a level shifter in this case because the EMMC is running at 1 volt 8. And then you can simply plug in that SD card reader into your PC and just make an image of the chip. Now, if you want to redo this at home, I would recommend that you buy a low-voltage EMMC adapter from the exploiteers. These are only $12, and they make your life a lot easier. The only reason I did what is shown on the left is that I was too impatient to get one. Now, once you have a dump of the EMMC, you can start pulling it apart. And for this, the Uboot GPL sources that are available are quite helpful. So, what we can see is that there are multiple trusted firmware boot stages. These are firmware image packages that you can unpack with the FIP tool. There's a flatted UI image tree that you can unpack with the Uboot tools. And there's a SpaceX runtime, a SpaceX calibration partition that's specific for every dish. There's an EDR partition and a dish configuration partition. Now, one thing you can already see here is that the dish is using trusted firmware A, so secure boot implementation. And all of the other partitions are verity, protected, or looks encrypted. So, this means that if you would just rewrite the firmware on the EMMC, the dish wouldn't boot and you wouldn't be able to do anything. If you want more details about how to extract the firmware, I wrote a blog post on that and it's available online. So, there's quite a few interesting things that you can find in the firmware. So, one thing people have asked me in the past is, how does the dish handle its terminals? How hot can it get before it shuts down? If you extract the firmware, there's actually a file that nicely explains it with this in a lot of detail. Another thing you can find are all of the RF channels that are being used, so the uplink and downlink channels, and what frequencies they operate at. And you can even find the frequencies of the lasers that are being used by the satellites. And this is because some of the code base is shared between the satellite and the user terminal. Another interesting thing we found are development geofences. So, if you have a development user terminal and you would use it outside of one of these geofences, then SpaceX would get an alert, basically. Most of these development geofences make a lot of sense. So, some of them are clearly at SpaceX facilities. Others make less sense. So, for example, the one on the right is the Connections Museum in Seattle. And for some reason, they have development hardware. Now, if you zoom in far enough on Google Maps, you can actually see that there are user terminals on top of this SpaceX parking lot. So, if someone really wants development hardware, this might be the easiest way to get it. Another thing we can see in the code is that SpaceX is collecting all of telemetry on how you use the user terminal. I know some people have mounted user terminals on top of their car. I even know someone is here that mounted it in their plane. And you can be sure that if you do that sort of thing, SpaceX will know. Now that we have the firmware of the dish, we can also start looking at how this login prompt is implemented. And we can see that it's a simple shell script, basically, that prints development login enabled. Then it checks if this is a production unit or a development unit. And then, accordingly, it prints yes and sets a password. Or it prints no and you're not allowed to log in. Now, if you look at this from a more physical perspective, this is what the logic analyzer trace looks like for this UART output. So, the dish prints development login enabled. Then there's a two millisecond gap. And then it says no. Now, if we want to get into this user terminal, it would be very convenient if the dish would think it's development hardware and then prints yes so that we can log in. Now, how can we make it believe that it is development hardware? Well, we did this using fault injection. And fault injection, there's a lot of different ways of doing it. In this case, as I said, it's a flip chip package and the back side of the die is exposed. And then oftentimes, we start thinking about very fancy types of attacks like laser fault injection, body bias injection or EMFI. But we had a problem. This PCB is so big that it doesn't fit on any of the lab equipment we have to do these more sophisticated attacks. And, of course, if you would want to do a laser fault injection attack while the dish is mounted on a roof, that wouldn't be very easy to do. So the first picture on the left is a micro stepping table you could say that is used to accurately position EMFI equipment on top of the die. And you can see the dish is way bigger than the setup itself. On the right, we have a metal box that contains a laser fault injection setup on the inside. And the dish wouldn't even fit in there. Another problem we faced is that there are no development kits for this chip available. So this means that the attack was completely black box. We didn't have any data sheets. We didn't have open development samples. So all of the work in figuring out glitch parameters and offsets, yeah, it takes more effort this way, let's say. And there's a few more attack factors we can look at. We can look at the clock signal to the SOC. But most of the time, there's PLLs in these chips that sort of eat your clock glitches. There's a reset line that we could play with. But ultimately, I chose to attempt voltage fault injection. So initially, we started with a very simple setup. So we have a chip as per light connected to the core voltage supply of the main SOC. And this chip as per basically momentarily short the core voltage to ground. And this can cause a glitch. Now, as you can see, I didn't remove any of the decoupling capacitors from the board because we're trying to glitch it while it's already running Linux. And at this point, you could say that the SOC is already unstable because it's being pushed to higher clock frequencies. Something that you can't see here on the slide is an external oscilloscope that we used to trigger on the UART signal. And that then triggers the chip as per. Now, this is an example output of the sort of results you can get with this. So the DishPrint development login enabled and even prints yes. But then there's also a no pointer dereference in the kernel and the entire system comes crashing down. So you weren't able to log in. Now, if you try this often enough, then the DishPrint development login enabled yes. You can log in with the username root and the password falcon. And you have a root shell on the user terminal. So this was nice for us because it's meant that the proof of concept worked. And we knew that the SOC is at least somewhat susceptible to voltage glitches. And the attack was also reproduced by the SpaceX team. So it's easy to produce undesirable faults. But the issue with this attack is that it's very slow. So we have to reboot to the user terminal, then glitch it and hope we get lucky. And you can only do this once every 12 seconds. So this meant that it's a very unreliable attack. So I decided I would move up in the boot chain and go to the early boot stages to try and do the attack. So because this Dish implements ARM trusted firmware, it has a ROM bootloader of course in the SOC. And this ROM bootloader reads the root of trust public key from ETH uses in the SOC. It will use this root of trust public key to cryptographically verify the signature of all of the consecutive firmware stages. So in a bit more detail, the ROM bootloader or BL1 will load a certificate related to the second stage from the EMMC. The certificate was in a custom format from ST Microelectronics. And the certificate contains a hash digest over the firmware and a signature over that digest. So the next step of course for the ROM bootloader is to verify the certificate. And it does that by verifying the signature. Afterwards, if the signature check succeeds, it loads the firmware itself into memory and then verifies the hash digest over the firmware. Now if you've done fault injection before, there's two obvious points here where you could try to attack. So one is attacking the certificate signature check. And the second one is the hash comparison. Now because this is a completely black box scenario, we don't really know which point to attack. We just have to try a lot of different combinations and be very patient until we get lucky ones. This is the setup that we made for this initial exploration, you could say, of the attack. A few tricks that you can apply in a completely black box scenario are to try to boot the dish with invalid signatures, invalid hashes, invalid firmware images and so on, and then compare how the boot flow changes. And this gives you an idea of what is happening at what point in time. You can also try to glitch a valid certificate into a signature verification failure and that way you can at least optimize the glitch with. So at the top right of the picture you can see an EM probe on the backside of the die, right in the same corner where the four CPU cores are. And this EM antenna gives us a nice side channel and gives us some information about what the SOC is doing. In the bottom right you can see that I removed all of the decoupling capacitors. This is required to get the glitch to work and the reason for that is that initially when the SOC boots there's only one core that's running and it's running at a much lower clock frequency and that makes it less susceptible to voltage glitches. This is a plot that shows the UART output and an EM side channel that was captured using that EM antenna. And basically what you see at this point in time is that the dish prints I've loaded the certificate and I'm going to start verifying the signature. And you can see from the EM side channel that the signature verification is already starting when the last byte of this message is being sent out. And this information for us as an attacker is very useful. So this side channel is also very useful and in this way we can figure out when the signature verification starts. So sometimes when you glitch something it's easier to attack the start of the operation than the end because in this way you might skip the entire operation altogether. And that's what ended up working for us. So if we glitched at the very start of the signature verification then it was skipped entirely. So you can see that the dish printed that it loaded the certificate it's going to start verifying the signature and immediately after the signature check succeeded. So we skip the entire process. And this you can clearly see in the EM side channel if you compare them. So at this point we can execute our own second stage bootloader meaning that we can already start exploring the system in a bit more detail. And in this way I started reading memory regions on the SOC chip and I figured out that the ROM bootloader is mapped at the address shown on the slide. And it's also readable from the second stage bootloader meaning that we can dump the ROM bootloader and start analyzing it. So I emulated the ROM bootloader in Unicorn engine and tried fuzzing it with AFL++ in Unicorn mode but this didn't result in any software vulnerabilities. And that's of course unfortunate because that means that we cannot easily scale our attack and we will have to glitch it every single time we boot the SOC. Another thing I did in Unicorn engine is I tried to simulate the effect of the faults I was injecting and I did this using a very simple fault model in which we skip a single instruction. And it turns out that skipping a single instruction is not sufficient to compromise secure boot in the system because there are fault injection counter measures implemented in the ROM bootloader. It's difficult to say what really happens when we do our attack but I think it's likely that we are skipping or modifying at least multiple instructions and my slides no longer work. Okay. So here's an example of a glitch that was detected by a counter measure in the ROM bootloader. So the first part of the video output is basically saying that it has loaded the certificate and it's going to start signature verification. In this case we loaded the certificate with an invalid signature and then of course we glitched the SOC and it accepts our invalid signature. Now at this point it's loading the second stage bootloader verifying the hash. The hash is correct but still at the end of the boot it says authentication error and this is because one of the counter measures basically detected our glitching attempt. Here in the slide you can see how this or one of these counter measures is implemented. So there's some control flow checks and there are some redundant operations in the code. I don't have the time to go into detail here what's happening but you can look at the code in the slide. Now an issue we had when glitching this chip is that if we want to bypass signature verification in the ROM bootloader we have to remove all of the decoupling capacitors to get this to work but if we remove all of the decoupling capacitors the dish is no longer able to boot completely because the system becomes too unstable and this meant that we had to come up with a way to enable and disable decoupling capacitors at will. This took quite a lot of manual experimentation. I tried different types of MOSFETs, high side and low side switching, the different gate voltages, different types of MOSFET drivers, different capacitor sizes and of course timing how I was turning off and on the decoupling capacitors. The picture on the left shows a hand soldered example that didn't work and on the right is the first example that did work so we made a small interposer PCB that you solder to the main board and this then allows you to switch on two capacitor banks. So this meant that we had the full attack working in the lab. We could glitch the ROM bootloader, modify all of the following boot stages and end up in a root shell on the dish. But the setup was still using of course an oscilloscope, power supplies and signal generators which would be very impractical if we wanted to do this on the roof of the university building. At this point I contacted SpaceX and I told them I have the full attack working in the lab and they offered to send me a Yubiki that would allow me to SSH into the user terminal to continue my research. But at this point I decided I was already too far down the rabbit hole and I didn't accept their offer. So I decided I was going to make a more mobile setup and this is basically the first prototype of this mobile setup. It uses a Raspberry Pi Pico and that Pico drives basically MOSFETs to turn on and off the coupling capacitors and to inject the glitch. So this is the first working example of this but as you can see it's still quite messy. So I decided to make a motship that we could solder onto the user terminal. The way I did this is I made a scan of part of the dish because it's too big and then I loaded it into Inkscape and in Inkscape you can then draw the board layout basically at real size. You can then export this and load it into KiCat as an edge cut layer. This is the finished motship. You can see that we have a Raspberry Pi microcontroller overclocked to 250 megahertz and that's driving the MOSFET driver. The MOSFET driver is switching the two decoupling MOSFETs that allows it to switch on these two separate decoupling banks you could say and then finally we have a glitch slash crowbar MOSFET that inserts the actual glitch. So the board itself is about six centimeters in size. You can see it also soldered onto the user terminal here on stage. And one of the goals for this talk is that other people can recreate this research and can use that to start looking at network infrastructure that's behind all of this so that people can also look how secure that's implemented and because of that I made the motship available on GitHub. The repository is still private at the moment so once I get a minute after the talk I will make it public. This is a picture of the motship installed onto the user terminal. There's a few extra wires that you have to solder as well so one wire is used to leach 12 volts from the user terminal. One wire is connected to the enable pin of the voltage regulator and basically allows you to power cycle the user terminal and then we have a connection to 1 volt 8 for a level shifter. This is a setup that someone at the university made for us. It allows us to hang the dish outside of the lap window and this is more convenient than always going up to the roof when something isn't working and so on. Now everything was working. I could connect to the Starlink network. I was planning to start looking at all of the network infrastructure was there and then I made a big mistake. I was reading on Reddit that the recent firmware updates made the connection a lot better and because the user terminal is mounted right up to the sky the connection wasn't that good so I decided I will do a firmware update and that will fix my connection issues. Now it turns out that SpaceX still had a hidden effuse that they could blow to disable all of the UART output of this dish and my motship was of course designed to trigger on this UART signal. So this meant that we had to improvise, adapt and overcome. I was quite lucky that I started working on this early so I had logic analyzer captures of both EMMC data and UART output and I could of course make a new capture of the dish with the effuse blown and I had this very nice side channel trace and this meant that even on the dish with the blown effuse I could take a new side channel capture and then basically visually find the new offset in time where I had to insert my glitch. I slightly modified the motship to trigger on EMMC data zero instead of UART. It's a bit ugly but it works so if you want to make a nicer solution you have to basically spin a new PCB revision. So at that point you can start exploring the network and as it turns out things don't really get easier here because all of the interesting communication with backend servers is using mutually authenticated TLS and for that the STSafe secure element is being used. So this means that most standard libraries don't work. I started implementing or I have implemented STSafe support in the TLS light and G Python TLS implementation and with that I made a Python script that allows me to fetch the latest firmware updates. So whenever I see someone post that they have a new firmware update I just take the version number I put it into my script and I can download it and this is convenient because usually firmwares are rolled out in stages and you don't always get the newest firmware immediately. I also started making my own firmware update archive and this is convenient to compare what has been changing over time in the firmware. Most of the network communication is over IPv6 which also means that you can't easily scan all of the hosts that are up there. One host in particular seems interesting because it has a lot of open ports. A lot of the communication is using GRPC with the backend servers and again there is no nice support for the secure element in these libraries. So this means that everything takes more time to implement and this is also where I hope that other people will start making a mud chip and will start playing with the network infrastructure. So as I said my hope is people will make these mud chips, will implement them in their own dish and then explore the network. Another thing you can start playing with at that point is you can play with the digital beamformers. It seems that they run their own firmware that you can likely update from the main SOC. There's also some nice binaries that allow you to do some diagnostics of the beamformers and so on. So maybe someone could even try to implement a point to point link using two of these user terminals. So to conclude we were able to bypass secure boot using voltage fault injection in the ROM bootloader. This is a quad core Cortex A53 processor and we did the entire attack in a completely black box scenario. Meaning we didn't have any documentation, we didn't have an open development sample and so on. We also had to figure out a way to enable and disable the coupling capacitors. And it's important to realize that fault injection countermeasures implemented in software are only as good as the fault model that is used to implement them. And in this case the used fault model doesn't reflect reality. I think this is a very well designed product at least from a security standpoint. I didn't find any very obvious low hanging food in the firmware which forced me basically to go the hardware route. In contrast to many other devices even if you get a root shell there's no obvious way to come up with a scalable attack. Of course I reported all of these vulnerabilities to the SpaceX product security incident response team. They were very responsive and helpful. Even when I broke a dish during my research they were happy to replace it. So with that I'm almost ready to end my talk. There's a link to the repository on the slide. You can always email me if you have questions or send me a message on Twitter. But before we really end this talk I want to try and do a demo. Now this is tricky especially if it crashes. I can already try to answer a question maybe in the meantime. Do you have any questions? Difficult to say the entire process from getting the user terminal to having the full attack working was about one year. But it's not the only project I worked on in the meantime. So I don't know exactly how many hours I spent on it. Did they blow everybody's eFuse or just yours because of something you were doing? And I'm wondering if security is going to be increasing and increasing now that this is deployed in a war in Ukraine? So for the first part of the question, yes, all eFuses are blown so if you connect to the yours, connect on your dish now you won't see any output whatsoever. I didn't completely get the second part of the question, sorry. He's talking about is there going to be any outcome because of the war in Ukraine where these have been deployed? I still don't really get it. The war in Ukraine, do you think there's going to be any outcome like increased security because they're being deployed in a war environment? It's difficult to fix this issue without making a new revision of the SOC and that's probably going to take a while. So I have some issues with my demo. It worked fine at Black Hat, but apparently... So first and foremost, a lot of respect. A lot of work. I appreciate it. So one of the things you mentioned that you made your own custom bootloader, like a second bootloader. So it was a black box, how you did the second bootloader and what was the output of the memory that you investigated. So what were the memory outputs? How you actually get it through the serial or through the... It's very difficult to understand the questions up here for some reason. Okay, you made your own second bootloader, right? Yeah. So was it like a completely custom thing or you used something? I patched the original one. Oh, you patched the original one? Yeah. I loaded that one into Hydra, started analyzing it and then modified it to do what I wanted basically. Okay, so it was like a binary patch? Yeah, binary patch. Okay. And you were getting the output like a memory prints through serial? Or... I was getting what, sorry? So you mentioned that you dumped the memory through the second bootloader. Yeah, I was outputting the ROM bootloader over serial. Okay, so it was through the serial. Okay. Yeah, it's pretty cool. Thank you. There's one more question up here. I didn't because I don't have a new terminal, but it's the same SOC, so the same attack should work there. So you basically have to design a new PCV that fits that board and that whole work. I know from the code it's the same chip. I'm sure that the demo worked like two hours ago in my hotel room, but yeah, I guess we'll call it and then I'll try to set up the demo somewhere else and then you can see it there. Sorry.