 OK, so what are we going to cover today? We're going to explain what a vehicle diagnostics adapter of VDA is to you. Then we're going to talk about some research we did attacking the VDA. And then we're going to talk about ways and approaches we took to defend the VDA. And then finally, we're going to talk about some educational things we did with the VDA. So specifically, we had four challenges in last year's carcane billet CTF that were based around the VDA. This is a picture of it here. We had it in Vegas mode, so yeah, look at that. Binary modification for Vegas mode, awesome. And we also did some educational things at the Cyber Truck Challenge we'll talk about. So Alex and I are on stage today. But Edward and Sam did a lot of this work as well as a team effort, so we wanted to acknowledge them. We also wanted to acknowledge Dr. Jeremy Daley, who put us on the path to looking at VDAs. And Friendly Vendor Person, who will not be named today, but helped give us some of the VDAs for some of the educational things and was a really good partner working with us to shine light on this part of trucking security. Cool. All right, so finally, what is a vehicle diagnostics adapter? So your truck has an OBD port in it. It's a thing that you technicians can plug into to pull diagnostic information off the truck. I think the VIN number comes through there as well, so you can pull the VIN number through there and also some information from some of the ECUs or the controllers on the truck. It also sometimes is used to flash the firmware of the ECUs within the truck. So the idea is you pull into a service station, you can plug in there. And I guess the way they do it is there's a little computer, the VDA, that actually plugs into the truck. And then there's, for some reason, it's always a Windows computer that plugs into the VDA. So you have a Windows laptop with these vendor DLLs that there's kind of like a joint protocol. I think it's called RP-1210 that everyone used to talk to the VDA. Then the VDA talks to the truck and it pulls data back and forth from it. And you can kind of see here on the right, this is kind of a diagram of the VDA we looked at. So on the right, there is the computer interface. So the USB, Bluetooth, and Wi-Fi also, is how you connect directly to the VDA. And then the micropostor connects to the vehicle network. Usually over J1939, unless it's really old. Okay, so congratulations. We all now know what a VDA is. Okay, so now we're gonna talk about attacking the VDA. So we do a lot of embedded device security or a balloon. So attacking embedded devices, it's a pretty simple formula. First you need to get the thing. The internet is really good with this. Pro tip, get more than one of the things because you'll probably break one of them in the process. I think you broke one of these, yeah. Then you wanna do the hardware reversing. You know, figure out, I guess enumerate all the components on the board. Figure out if it has JTAG, UR, stuff like that. Then you wanna get the firmware either from the board or from some sort of firmware update thing. Reverse the firmware, figure out how to update the firmware. And then if you can modify the firmware and put the modified firmware back on, you can change how it works. And then you win. So this is pretty complicated. So when we started working on this, we decided that it was so hard that only our interns could do it. And so that's where Alex came in. He was an intern at the time, he's a full-time employee now. So he's gonna walk you through the work he did as an intern, trying to figure out how this thing works. Awesome, awesome. So the first step that we always go through when reversing a new device is grabbing your favorite screwdriver set and taking it apart. So that's what we did. This is a shot of the PCB of one of the VDAs. And as you can see, the vendor did a great job making connecting to JTAG and UART super hard by giving us really awesome adapters to use and just plug into. We've identified the flash, the microcontroller, and then a Wi-Fi chip and a Bluetooth chip on this board. Now, the more astute of you might be noticing, the Wi-Fi chip has a Wi-Fi and Bluetooth stack, but there's also a Bluetooth chip on the board which begs the question, why? We don't have a great answer. I think these memes probably illustrate some of our confusion as to why there were two independent Bluetooth stacks on this device, but it had both. So once we took a look at the hardware and read a couple of data sheets, we were able to make a super legit power adapter to plug in the VDA in a way that it's not normally powered. Yeah, this is what happens when you tell an intern to power a VDA, but it gets the job done, so that's cool. And then just grab your favorite female adapter cables to connect to the UART and not have to do any work with that, really, because the vendor provided a handy-dandy interface. So once we connect it to the UART, unlike most devices that try and obfuscate and make it really difficult to find out what the debug options are on a shell, the vendor just gave us a really easy way of dumping memory. So we typed in dump invalid option and then got every possible permutation of dumping we could ever want, so it was pretty simple. We're just like, hey, can we get the first copy of firmware on the device and then it spits it out handy-dandy over ASCII over UART. So I wrote a really complicated extraction algorithm which is dump the whole firmware to serial and then write a Python program to convert that into binary. So once we actually had the bytes of the firmware sitting in memory on the device, what we wanted to do is reverse engineer the file format that actually puts those bytes on the device so then we can see whether we can actually change the bytes that are put on the device. So as Wyatt mentioned, this piece of hardware uses a Windows laptop in order to do the firmware update. And so I was a little worried because I hadn't done Windows reversing that this would be kind of complicated, but it turns out the firmware update format file was just put in the files for the application on Windows like called firmware.bin. So that took about two seconds to look at and then we had the full firmware update file for the manufacturer. This is a colorized gif of me reversing the firmware update. It did not take that long. We're not going to go into the details of how this firmware format actually works, but if you spend enough time looking at it, you will be able to figure it out. And once we did that, we knew how the vendor firmware update file translates to the bytes actually sitting in memory on the device. So once we've done all that really complicated reversing work, we wanted a way to be able to replicate that so no one would have to go through the same process. So internally, this is our tool called OFRAC. This is what Red Balloon Security uses internally. It's also open source, you guys can try it at home. And it allows us to very easily go through all of the steps that we normally do to reverse engineer and modify firmware. So we can identify arbitrary binary formats, unpack them, analyze them, make the changes that we want, and then repack them up in the format that the device expects. It's in Python and we have a really snazzy gooey on the right. If you guys want to take a look at it in more depth, we'll be at our booth over there and demoing it. So after I've reversed the format, we can write a script in Python that specifies the firmware format and allows us to make the modifications programmatically and have it spit out a binary that we can just drop back on the device. So saving a lot of time for anyone else to have to do any of the reversing. So now that we know how to make the firmware file for the device to take, we wanted a more fun way of putting it on the device than just putting that file back in the Windows file system and clicking through the vendor tool to actually put it on. So the firmware update happened over USB. So, you know, computer connected to a thing over USB, what do you do? You sniff the USB traffic and store it as a PCAP. And then we spent a really long time reversing it. We're gonna go through the protocol later and then wrote our own little Python script to just do the firmware update and then got rid of our Windows laptop because we don't really like Windows. So to go over the super complicated update procedure of this device, you first have to send hex three and then the size of the firmware update in hex as a USB packet and then wait for it. You just send the entire bytes of the firmware update. There's no encryption. There's no signature. It has a CRC check, but you basically just slap it in USB packet and it just takes it up and rewrites the firmware. For those playing at home, there's bonus points. This same unauthenticated firmware update works over HTTP. So if you have Wi-Fi on the device, you can very easily just connect to the access point and then do this exact same update format over HTTP. So now we have a full way of having bytes that we can modify our computer and putting it on the target device. So what do we do with that? Well, the first thing that we always do when performing this type of analysis is testing that are unpacking and repacking and this whole process works correctly. So the first thing that we normally do is just by modifying a string and seeing whether the device still boots. So that worked. And then we wanted to do something more fun, which we showed you at the beginning, which was the LED toggle that we hooked into ping. So you can see the code on the right. Pretty complicated, but again, we used Ofrac, which is our custom unpacker and packer tool in order to put this code into the binary. And specifically, we used Ofrac with a tool we wrote called PatchMaker, which allows us to compile C into bytes and then link it into the full firmware image. So it's really easy to iterate and write code and just plonk it back in the binary without having to deal with any linking. So this case was a little bit more difficult than normal. If you can see on the top left, the compiler directive underscore underscore regarchs. The calling convention for this toggle LED function was kind of abnormal. So the architecture of this was Motorola Cold Fire and for this specific function, you needed to pass in the first argument as a register, which kind of messed with our compiler tool chain. So we had to use a different tool chain written by the wonderful Dr. Volkl Barthaman. Shout out to him, who made the VBCC tool chain and then we just had to integrate it into our tool and compile that code in and then boom, just add in our modification. Yeah, Dr. Volkl Barthaman. His website's pretty cool, I would definitely check it out. VBCC supports a lot of architectures. It looks like it's from the 90s, but we built it and it worked right away. So just to highlight this, this is kind of how we do binary modification with OFRAC. We can use, so run ping cmd as a function we found in the binary and we wanted to over place it so whenever we pinged, we would toggle the LED lights. And so in this case, we were able to call toggle LED directly in the C code that we're writing and then using PatchMaker, we can compile this code and then link it in so that all the addresses just work with the target binary and then just use OFRAC to copy it back into the target binary. So you get pretty readable like C, it's really useful for developing binary modifications. Awesome, so far we showed you a cool demo with LEDs, like why do we care about this ability to modify the former of the VDA? So most VDAs are used to update the engine control unit in these big trucks. So the workflow is you have a new engine control unit firmware for the vendor that needs to be updated. It gets passed to the VDA, which then directly uses it and sends it over to the ECU to flash it. And so the engine control unit obviously controls the engine, very important part of the car. That's why it's worrying in particular that the VDA is normally completely unsecured, right? Like this is a device that technicians use at the dealership, not really locked up, not a part of anyone's threat model, anyone could sort of walk in and do this kind of firmware change. And then in addition, since we found so many ways to update this firmware, really not a good thing that this is a critical path in terms of getting firmware on the engine controller. Excuse me, so as a result, you could for example, put an implant on the VDA, wait for an engine controller update, add your own modifications to it, and then unbeknownst to anyone else, you now have code running on the ECU of that big truck, which is obviously really not great. Yeah, so this is kind of where the attack work ended. We're able to, as you see, modify the firmware to put in Vegas mode, flash the LED lights, which is fun, but those LED lights are what tell the technician that the device is on or off with Wi-Fi's connected if data's being transmitted. So you can use your imagination to think about what things like that can do. Alex mentioned, you know, sometimes firmware updates these use pass through this device, some diagnostic data come out, so it's a really nice place for an attacker to sit. And this work kind of extends work that Dr. Jeremy Daley and Shreeka Kumar did. They presented an SCAR this year about attacking the Windows application that communicates with the VDA. So I'd recommend checking out that work. They looked at the Windows DLLs and found that it was pretty easy to insert Shim DLLs into the process and peak and poke and manipulate packets on the Windows laptop. So kind of all the way down to the truck, there's all these places where you can insert and manipulate the traffic. Something interesting just to think about. Okay, so now we're gonna defend it. We're gonna fix all the problems and go home. Yay. So we're not actually gonna fix all the problems today. We're just gonna focus on some of the low hanging fruit. So the immediate problem here is you might have noticed there's Wi-Fi and Bluetooth on this VDA. And I think you can imagine that some people might not want that Wi-Fi and Bluetooth enabled. I wouldn't want enabled. I think people maintaining military vehicles might not want to have Wi-Fi access point on their VDAs they're plugging in or Bluetooth as well. But this is kind of a classic situation in not just automotive but all embedded devices where the vendors either they don't want to or can't remove the hardware components and software components that these features are implemented on. And so the way you disable the features is essentially flipping a bit in a config file saying don't turn on Wi-Fi, don't turn on Bluetooth. But as we all know, the code that underlines it and the hardware is still there. So an attacker can still modify the firmware or use an exploit to then turn on these features. So it's a very ripe attack surface. So we're gonna be talking about VDAs but I encourage you to think about this is a broader problem. We have component X as features ABC. We don't want those features. We can't remove them because the vendor won't remove them. So how can we remove them in the binary? Okay, so what we're gonna do today to talk you through on what we did is like how can we apply a binary patch to the VDA firmware that removes all the code that underpins these features without having any side effects on the device. And of course, this is an example of a general problem. How do we remove features XYZ from component A so there's no side effects and those features can't work and the attack surface isn't there for people to use. So we use a technique called Autotomic Binary Reduction. So the name is derived from nature actually. It's a nice place to look for inspiration. So Autotomy is when animals will shed a part of their body when they're being attacked to make it harder for predators to catch them. So ABR was proposed by our CEO Ong when he was a graduate student at Columbia. That's where the work comes from. So if you wanna really get into the math behind it, you can look at his dissertation. But to simplify, there's really just four steps. First you identify the high level feature that you don't want and then you identify the entry point in the code for it. Then you can use analysis tools to identify all the code and data that are exclusively control for dependent on those entry points. You wanna do this very carefully. So if you're not sure, you'll leave something in and then you remove that code and data and then you recompense the high level entry point with a return patch that makes sense if the program ever calls it. So if you think about like a Bluetooth subsystem, there's usually some sort of process that initializes everything. You can remove the code for that and insert like a return negative one or it depends on what is actually in the system but return code that will tell the system this is not up and it's not gonna run. So this graph I think simplifies, obviously it's more complex but at a high level this is the idea. If you have feature A and feature B, we wanna remove feature A. We look at the control flow graph, find all the functions that are exclusively control flow dependent on feature A. So function five, we cannot remove but functions one and two, three we can. And then we insert that return patch there. So the workflow for this is fairly standard and actually Alex did most of the work already that you need human input for now in. So we've understood how the firmware works and how we can update the firmware. So next step is just to identify the feature points and then all of the actual autotomy work about identifying the code that needs to be removed and removing it and packing the firmware. We can completely automate right now with the work we have. And what we're doing as a company is we're trying to move up the stack to automate this entire process. So there's minimal human interaction needed into it. Okay, so identifying the feature entry points. How do you do that? Well this is where you have to roll up your sleeves and do a little reversing. So on embedded systems like the VDA this can be a little complicated. But sometimes you get lucky. So in this case you can use symbols to find leak source code online that you can use to then label the functions in your project. This was using cold fire. So in this case we use Gija but you can use any other disassembly tool as well to do this. And then you can kind of start labeling the features inside the binary. So a new binary comes in. This whole process can be automated. You can start with the debug strings within the binary and then work up to find the functions and then automate this process removing that code. So for this like proof of concept we generalized five entry points. The OS tasks that initialize the subsystems. We also wanted to remove the UR debug commands because we found them super useful. And yeah, it should be harder to figure out how to dump the memory from the device I think, yeah. Then we also targeted some driver initialization code for these things as well. And these were the results. So we used the ABR algorithm that we have, ABR balloon, we removed in total 574 functions and end up being 144 kilobytes of code. So that was about 7% of the binary. And you can kind of visualize it here. Everything that's dark was unmodified and the stripes of green here are the areas where ABR removed the binary code. So it's kind of cool. You can see like throughout the binary there's huge swaths of functions that are just not needed and attackers can't use it anymore. Then we worked with friendly vendor person to test the image using the functional test suite that they used and they confirmed that performance was consistent with the original firmware image. So that was great. We basically showed we were able to remove this feature and the regular performance of the device wasn't modified at all. Okay, so we didn't fix all the problems. So as we're reflecting on this, it's kind of Zen. We were able to use some of the reversing that an attacker would use. It's the same workflow in the beginning to identify the features and remove it. So removing the Bluetooth and Wi-Fi code is good. We would challenge vendors to reconsider whether they even need it. Yeah, I think that's something we can all think about. Do we need to put this in? Can we support firmware builds that don't have it in? Another thing to think about is ideally you want to at least not leave the headers on to JTAG and UART. Ideally disable them as well. That just makes it a little harder for people to mess with the device. For production devices, of course. So then we get to some kind of the lower level challenges. How can we make it so anyone can't modify the firmware? So some sort of secure firmware update mechanism, maybe a secure boot mechanism would be useful here. And then how do we know that the firmware hasn't been modified? How do we know that the VDA that's on the shelf in my technician's office hasn't been tampered with? There's some sort of runtime protection monitoring could help there, because then you could see that the device has been corrupted. Okay, so we've attacked, we've defended, and now we'll just talk about some education we did. So we repurposed this research to build two educational things to kind of take these real world problems and make it challenges that people can use to kind of learn about this specific issue. And this is another area where the friendly vendor person helped us out, because we only had two of these devices, but we were able to, at the Cybertruck Challenge, have, for the last few years, a two hour workshop. So the Cybertruck Challenge is a pretty cool event that's held every year where a bunch of students are paired up with industry experts in automotive-specific truck security, and then a bunch of vendors come with fleets, and there's a bunch of education that happens, and then a bunch of assessments that students do on the devices. So I encourage you to check it out. There's also some really good educational resources on the website as well. So if you're not super familiar with J1939, that's a good place to start. But anyway, in our workshop, we walked them through this process. They physically took apart the devices, dumped the firmware, getting you out was a really fun moment when you get the shell there, also because one thing that we've kind of observed is that familiarity with command line tools is something that probably we could build on more. So connecting the UR, getting screen working and stuff like that, it was a really good moment for the students. And then we walked them through the boot string modification and a contained environment where they could toggle the LED lights there. A little more fun though, last year as part of the Caracan Village CTF, we had a bunch of these VDA boards here and four challenges. So challenge one was a real world challenge. We gave them a Peacup and we gave them a modified firmware image and said flash this image on the device. And so there the flag printed in the boot shell once you're able to flash the device. Challenge two was a little too difficult. It was ambitious. I think we would remove it if we did it again as CTF, but there the challenge was to essentially put the device in Vegas mode. And then challenge three, there's a really good write-up online that kind of walks through the process, but challenge three involved finding a super military grade encryption scheme on the device. I think it was just a bunch of like X soaring data against the XOR like repeating key. And so the write-ups really cool because they showed you how they use the ability to map the firmware to load a Geiger project, find the function. And I think they actually emulated the function in this to get the decrypted key there. And then challenge four was OTA to save the day. So we had the Wi-Fi access points turned on and you had to do the firmware update procedure but over the air. Okay, so just to recap, yeah, so that's the binary modified image in Vegas mode. We talked about what VDAs are. We showed how it's relatively straightforward to try to modify the firmware on them and attack it. We can think about what that implies to the industry. We talked through some ways which binary modifications could be made to harden the image. So we don't need to go to the end manufacturers who want to. And we talked about how we can use, I think this is a really good takeaway point. You've used real world research and convert it into challenges to educate both students and also push people out of their comfort zone. I think this was the first VDA that was at the Carkinville CTF. And yeah, I think we just want to acknowledge Edward and Sam who couldn't be here today, Jeremy and the friendly vendor person for helping out with this. So, oh, and lastly, we're Red Balloon. We're hiring now, so if you want to learn more about OFRAC over this work, we can take some questions now but we also booth over at the Carkin Village and we have two different CTF challenges that we contributed to the Carkin Village CTF this year as well. So yeah, thanks for your time. Thank you.