 Welcome to Fuzzing NASA Core Flight System Software. My name is Ronald Broberg. I work for Lockheed Martin. I've been there for 25 years. I work in LMSpace, started my career there back again. I've been a software engineer. I work test and integration, network security engineer, cybersecurity engineer. My, one of my first tasks was working on programming for the Space Defense Operations Center, Spadok at NORAD. And in the recent years, I've been involved in more site, more active cyber security type of work, including a red team engagement at Cape Canaveral. And that's what brought me into testing of satellite software. So what is the NASA Core Flight System Software? Brief intro. This is the Solar Parker probe, the magnetospheric multi-scale satellite, and the lunar reconnaissance orbit. These satellites plus maybe a dozen more recent or current satellites all run core flight system software. It's real flight worthy software as being flown in missions today around the sun, the earth and the moon. And so it is certainly of interest from a cyber testing perspective. The core flight system software from NASA's own description is reusable software framework composed of a set of reusable software applications. They stress this reusability for good reason, having to do with time, schedule and money. There are three key aspects related to the architecture. It has a dynamic runtime, a layered software design and a component-based design. All of these are meant to decrease the time to develop and test flight software for individual NASA missions by emphasizing this reusability core. And here's an architectural view from NASA documentation. If you focus on the blue ring, that's the software bus over which these modules communicate with each other. If you focus on the lower right-hand corner, you can see the software bus time services, executive services, event services and table service modules. Those are the core of the core flight system software. But the open source release also includes many of these others components and includes two what they call lab applications designed for test and development but not for a mission deployment. And that's the command ingest and the telemetry output by which commands can be injected from the outside into the software bus and to data flow from the software bus to outside clients. There's also, for instance, in order to make a complete mission set of software, you'll need guidance, navigation and control and that's what's being represented here but in the purple in the center. And there are some file system management components as well, some of which are in the release that's available open sourced. If you look in the upper left, excuse me, upper right-hand corner, the space wire 1553 bus and instrument manager, these things would be specific to it but particular satellite and are not included in the open source release. Another view of the core flight system is this application stack view again, adapted from NASA. The items in gray are those that would be required to be developed specifically for a new mission and wouldn't be shared necessarily between missions depending on the payloads that you have perhaps. The items in blue are the reusability components. So you can see that there's quite a bit that's already provided to help us start a new mission off with their software development. In the lower layer here, the OSI abstraction API, this provides the portability to the core flight system software. You can run this on Linux x86 on Raspberry Pi arms, power PCs, there's real-time operating system versions. So a multitude of different platforms this either is running or can be easily ported to. The NASA core flight system software and what attracted my attention originally is open source software. It's been open source since at least 2011. Don't know it's been on GitHub that long but you can find it at github.com slash NASA slash CFS. There are two major releases, the Akela release and the Buddhist release. These aren't just left in stasis, they've been updated over time to try to keep them current. And in addition, there's of course the main development branch. You can note in the upper left-hand corner that they have a very active GitHub repo with updates occurring almost weekly, if not even more frequently. Now that we have an introduction to the core flight system software, the question arises, how do we go about testing it? A common testing approach of course is fuzzing, which fuzzes the user input into any particular application. In this case, the user input is via telecommand on the command ingest module or application, which is listening on UDP port one, two, three, four. Note that the command ingest module itself is not multi-threaded, although the CFS application overall is. When fuzzer is fine that there's a network interface, they often go to replacing that with a file system interface. Prini provides the capability of doing that pretty easily for TCP, but doesn't have UDP interfaces as part of its core package. So that would be additional software we'd have to write in order to build out that direction. Another option is to build the file system interface directly into the application in USA use AFL. This is a great approach because it'll give you coverage, but the CFS executable is multi-threaded with the high dependency on shared library applications and table spaces that I found difficult to configure with AFL. So I chose to use a pure network fuzzing tool called Fuzzatron, which natively supports both UDP and TCP. It uses Redomsa for blind fuzzing. That is, you provided a set of inputs and it will just start randomizing based on those. Or it can use BLAB for protocol aware fuzzing. In that case, you have to build a protocol definition file first and it will start fuzzing the individual components of that protocol. I did occasionally see it crash early on start. The Fuzzatron issue, there's an issue for this in the Fuzzatron GitHub repository. Deterministic mutations completed, but that wasn't really much of a problem because I was running this manually. It was simple enough to just restart and go. You might need to pay attention to this if you build an automated framework. An issue I did have though is that Fuzzatron sends tens of thousands of packets per second. That was way faster than the core flight system software processed. To resolve that, I had to slow Fuzzatron. I took the code a little bit to add a timer and I had to speed up the core flight system. There is a time out there where it was, I believe it was 500 milliseconds to process a particular packet before it would process the next one. So I reduced that to like 50 milliseconds. In the end, I was able to get a sustained 800 packets per second for test. The CCSDS packets that we're processing have a primary header composed of packet identification, packet sequence control and packet data length. It also has an optional secondary header that can be included as part of the data field. Some of the NASA software, core flight system software indicated the secondary header was actually a mandatory field, but I did not find that to be true in the open source release. As you can see here in the middle, there are a couple of packets that have been broke down to their binary format. And we can see the application ID. We can see the sequential counter. We can see the packet length, but we don't see a secondary header here. So read the code or in this case, read the packet. In order to begin the blind fuzzing using Redamso, we have to feed it a core group of inputs instead of inputs. And in order to collect those, you can use the ground system application that's included as part of the open source release, which has both the telecommand component and the telemetry component. The telemetry receives from the core flight system on one, two, three, five. It sends to the flight system the telecommands on one, two, three, four. So we set up a TCP dump. We send one packet using this application. We save that packet off, reset TCP dump, send another one. Using this, I collected 24 or so different packets from different modules that I could use to begin the fuzzing. Walking through the methodology here, yes, you can start the core flight system application named core CPU one, name of the executable. You start the packet capture. You run fuzzetron until it detects the seg fault of the executable under test. You stop your TCP dump. You search through the kill packet. It's worth noting that it's not the last in the file. There's a slight delay between when fuzzetron sends its packets and when it can detect the seg fault of the executable under test. That delay means that you're gonna find that packet. Several dozen, maybe even a hundred or two packets up from the bottom, probably not quite that much, but several dozen. So once you have carved out your kill packet, of course you need to save it. So a brief demo of this process. We're going to start the executable that we wanna fuzz the core flight system here. That stop flywheel is a good indication that you've reached the complete bus initialization. Then you run the demo. You can see here that the TCP dump is being launched. It's gonna save its PCAP file and that out directory. There's a little bit of logic here to find the process identification, the PID for the core CPU. And then we run our executable. We've got our set of packets in that command directory. We're running on local host port one, two, three, four. And there's the PID that we're watching. We're not gonna run to the point of seg fault here. It can take a hundred thousand or a million or so cases before you get that far. Once you've carved out the packet that you need, you can then examine that and verify that it is indeed a kill packet. So we're gonna restart the core flight system software. You can see it's ready now. We're gonna go up and take a look at the packet we're about to send. You can see here that I've collected a couple dozen kill packets. And when we look in it, if you look at those last eight words there, you'll see in little indian 800, that's the application ID and 57AA, again little indian reversed. And I believe that's about 31K bytes. That's a declared length of this. Even though you can see the data, here's those trailing zeros as much smaller. When we launch the kill packet, we're going to see our seg fault. So verify that indeed we've sent the kill. Note that it's transmitted eight bytes. So this is an eight byte packet we sent, even though, I'm sorry, the length of that was probably 22K. So we found a collection of UDV packets that can seg fault the core flight system when sent to the command ingest application. But what's actually causing that? We need to do a root cause analysis in order to see that. A couple of tools that we can do as part of that analysis is to examine the core dump, to examine the UDV packets themselves, to look at the code and do maybe some dynamic analysis in the debugger. We'll take a quick tour through those. This is a snapshot of a part of the core dump. And we can see here that frame four, the CI lab command ingest, main function is entered and then moves to the read up link. That's where the UDV packet is actually being read into the application. Once it gets that data, it calls a CFE function, the software bus send message function. And that in turn calls the send message full after which we see the core dump. Using a more robust version of the core flight system software, if we send the packet, we can see here we have the 800 application ID, but most importantly, the declared length in the packet, 22K bytes, is not the same as the bytes received or in this case labeled expected bytes. This mismatch is the core problem, the root of our problem. We can look at the read up link and here we can see that where we're going to receive data from a socket and the amount of data received is actually going to record it in the status. That's the number of bytes received. In this case that we were just showing that would be eight. We checked to make sure the status is larger than the minimum number of bytes needed for the command header and it is. And we checked that it is below and less than the CI lab max ingest. It is. No, that is not the declared length. That is the length as received. So then we call the send message function in the CFP. When we get there, part of the processing is to determine the length or the size of the buffer. And to do that, it hands off the message pointer to this subroutine and get total message length, which in turn reads the CCSDS packet length. So that's the declared length. So our actual buffer was built on received bytes, but we're about to prepare a buffer based on the declared length. So we go back into the send message and we can see this little bit of logic here. If it's a zero copy, we are not going to pull a buffer from the pool, but if it's not zero copy, we do so. Asking for a buffer the size of the declared length. Then once again, if this is not zero copy, we're going to do a mem copy, where we're going to copy the declared length from the message pointer into this buffer. This message pointer though is a buffer that was only built for eight bytes in our example. So we can also try looking at a debug version of this. We're going to run GDB here in a moment, but before that, we're going to set up the core CPU software. And this is the more robust version. And then we're going to send a, this one's a slightly different packet to that. And we can see once again, there's our application ID, there's our declared length and our received length. 31K for declared, 14 for received, which matches the 14 bytes that UDP replay tells us we sent. We can take a quick look into this packet to see what we've actually delivered. And from here, you can see that little end in 800. We can see the 4079, which matches the 31K of, and again, that'd be 7940. And that matches the 31K of length. The, we're now we're going to run the vulnerable version and we're going to run that in the debugger. Once we get that started, we can note, as I stated earlier, this is a multi-threaded app. So we can see that different threads have been launched. And we send that same kill packet, it kills it. We do a back trace. And you can see this looks very similar to the core dump we looked at earlier. We note that our function of interest is going to be in frame one. We pull back to frame one and there's the mem copy that kills us, complete with the different variables declared there. So we look at the total message size, see it 31K bytes. We can see the message pointer. This is where it's going to copy data from. And if we take a quick peek at that, we'll see that if we look at those first 14 bytes, I believe it was, we'll see here the 800 app ID, again, a little in the order there. And then we'll see the 4079, which I skipped over, that's the length. And there's the payload data that we recognize from the packet itself. So there indeed is our packet in memory and CFS. But we didn't copy this 14 bytes, we copied 31,000 bytes, which is going to be about 7,700 of these words here. So first thing we notice that we quickly overrun the memory dedicated to the CI lab application. We're into a bunch of other stuff. And now we're into other people's stuff. There's C libraries here as well as lots of other application libraries and tablespace app main processing. And so lots of stuff for us to stomp all over and pull into other parts of the application. But it's going to get even slightly worse as we try to copy the whole buffer over. We're going to completely overrun the memory that's been provided for this application, for these modules. So no surprise in that we have a sec fault. Just very briefly, you'll recall that the length of the copy comes from the CCSDS packet header. The amount that should be copied is determined by the UDP packet length. This misalignment of buffers is what leads to the sec fault. And though we only briefly looked at it, there is a zero copy approach that partially mitigates this or at least bypasses it. That's completely implemented by April, 2021. But it doesn't really take care of the input validation. Well, the memory copy has been made safer. The input validation isn't being properly performed. And interestingly in the CI lab, there is code that does exactly that. It compares the received bytes to the declared bytes to determine if they're the same. And it's using a different part of the CI lab processing right now as of late 2020. But hasn't been used in that read up link function to verify that what's been received is actually valid. So work to be done there. So now that we understand what the root cause of these kill packets is and what the problems are in the core flight system, where do we take it from there? Well, we can certainly do much better job of fuzzing. There are protocol aware types of fuzzers. At the very least, we can break the header from the data, the payload data and we can exhaustively fuzz each component of the header. In turn, we can exhaustively fuzz a certain length of payload depending on your test rig. And we can build out test cases to make sure we reach the edge of what can be ingested by a particular module such as command ingest. Also, the command ingest is not the only thing available for fuzzing. Each of these modules has an interface of the software bus where it sends data and receives data. So we can fuzz each of these modules individually. In addition, there's these file system management components. The core flight system stores applications as shared library objects and table space there as well. So there's a place that we have the ability to do manipulation of data input. So there's a lot more fuzzing left to do in the core flight system. I had originally approached the CFS developers in October, 2020 with the UDP kill packets just with their existence. And at that time, they were comfortable with that. That was not a security problem. After I've done, completed my root cause analysis due to other priorities that wasn't until this summer, I reach back to them and they verify that their stance hasn't changed on this. First and most importantly is that the command ingest app is just not flight software. And that's kind of obvious. It's an ethernet UDP interface. And that's not what's gonna be deployed on a satellite. In addition, the core flight software as seen on open source is not exactly the same software that is flown. While there's a lot of reuse, it's also customized per mission. So just because we find a particular vulnerability in the open source software, doesn't mean that it's reflected in the mission software. And on top of that, this was only against the Buddhist release. So the current development releases of core flight system don't even have this vulnerability because they use that zero copy approach for memory management of the messages. So the object of this wasn't just to find or particularly to find holes in the core flight system. It was to use the core flight system software to develop test approaches that we could then bring into other environments for testing a flight software. The space domain is quickly changing and a lot of the assumptions around security and satellites are changing and the test approaches are gonna have to become more comprehensive and more rigorous. And the core flight system offers us an open source application to develop some of those approaches, prove them out before we try to find time, space in a resource constrained environments for testing. Even this simple example that I walked through here, we needed to understand the system under test well enough to do the tuning that was required and to flow through the root cause analysis. We also need to understand the limits of the test rigs we do that we use. In this case, you saw that I had to go in and modify the timing that we were using for the packets. So these aren't necessarily plug and play. Again, emphasizing that we can develop these methodologies using open source software such as the core flight system in order to test the flight software that's not necessarily so. I appreciate your attention through this and we'll be happy to take questions. Thank you very much.