 Welcome. Thank you for spending your time and skipping the last testimony to come see me. It makes me feel great. So I'm here to talk about automotive bootloaders. So I guess, first of all, a little bit about who I am. I'm Phil Lepchinsky, so it's lap with all the consonants and then ski. I'm a software engineer, senior staff engineer at Renaissance. And we make chips and all kinds of things. So what am I here to talk about? Embedded bootloaders in automotive. So what does bootloader do? Essentially, it owns the reset vector in the microcontroller. And on reset, the bootloader is going to determine if your application is valid or not. If it's valid, it allows you to load the application. And on a big system out of chip, it might copy that over into RAM and execute it. For embedded devices, though, it typically will just jump to the start vector of your application. Also, in like deeply embedded devices, the bootloader usually allows you to reprogram the memory, the flash and the micro. So you can see the mouse. So here you have the roulette vector. Jump into the bootloader. It's going to do some stuff to determine if you have a valid application or not, and then jump to the application vector. So we've got these in the car. We know that they're in there. How do we talk to them? Well, we make it easy in automotive. We have a proprietary connector for OBD and you have to get special dongles like the car hacking village dongle. So everybody go buy one of those. They're 80 bucks. But that allows us to connect to the CAN bus. So that's defined by ISO protocol, released in 86. But it's 2008, become mandatory. So we know that any vehicle after 2008, it's in there. We can talk to it. And so that's the physical layer connected through the OBD2 dongle. To start talking to these things, so CAN, only 8-byte payloads. That doesn't give us a lot of information with some small commands, but we can't send large amounts of data. So I know. So we got the transport layer, 15.765. That's the underlying transport layer that sits on top of CAN and between CAN and the diagnostic layer. So this basically allows us to send payloads up to 4K by segmenting the messages. So on top of that, we're just going to make layers and layers and layers of protocols to make it confusing. We've got ISO 14229. That's Unified Diagnostic Services. So this is a ISO spec that basically defines a whole bunch of common services, essentially creating the API. It's a client server. So you send it a request with your client. And the server running on the ECU will send the response. Essentially, this spec defines all of the different standard services. There can be also OEM or supplier-specific services as well. And there's also security levels. So there's different sessions to get elevated privileges. For bootloaders, we're typically talking about the programming session. ISO Define Session 2 has the programming session. So if you send a session control change request to go into the programming session, now you go into the bootloader. So this is where things get interesting. And I guess before I jump to that, let me explain. On an embedded device in automotive, the bootloader is typically a whole separate executable program that's separately compiled from the application. So there's some kind of inter-process communication that happens between your normal application and the bootloader when you're passing the information through. So that's usually done through a reset. So you might get a programming session control request. And it'll set some flags and then cause a reset. And then the bootloader says, oh, hey, I'm in programming session. Let's go do something interesting. So what does this look like, the typical programming sequence? It's broken up into three distinct groups. There's this pre-programming sequence, which happens before your program. There's the programming sequence, which is when you're doing the programming. These are really easily named. And then the post-programming sequence, which I bet you guys can guess, it's after your program. So pre-programming, what happens here? This is usually OEM specific, really what you're doing. But in the ISO spec, there's common things that typically happen. Oftentimes, you'll do a session control and jump into this, the extended session. So that's session three. And the extended diagnostic session gives you elevated services than your normal diagnostic default session that would be like you're driving around in the car. So what you do is you go into this extended session, and you start doing things that are going to prepare the vehicle for programming. So one of the common things is to turn off DTCs. So I'm going to change my application software on the car. And all these other ECUs are still talking to each other, or they're looking for messages. So if I have an ECU that's down not sending its messages, everybody else that relies on my messages is going to say what the hell's going on. It'll start setting DTCs. It doesn't know what's going on. So typically, when the diagnostic tester is going to initiate a programming session, it turns off all the DTCs. It might turn off, disable the non-diagnostic communication, and so say, OK, everybody be quiet on the bus because I want to start reprogramming ECUs. This may or may not require security access to get into this session. Again, this is all OEM-specific, and the OEM specs are all secretly guarded and under NDA. And there's a bunch of OEM people here that they weren't. They don't let me tell the secrets. So then there's the programming step. And this is when you basically activate the bootloader and unlock it. UDS defines a security seed key exchange, so service 27. And you basically, the tester will say, hey, go give me a seed, sends the seed back, and then the tester will send the response. It's a challenge response protocol. But again, the way this is programmed, the actual algorithm is not defined by ISO. That's defined specifically by the OEM. One thing to note, again, the industry is continually trying to strive for better security. But historically, OEMs didn't always do a great job with their security algorithms. It might be a secret algorithm that's, you know, some engineer made up one day. And the secrecy of it is because it's hidden in a desk somewhere. But lots of smart reverse engineers out there that can figure things like that out. So historically, that's been a problem. The industry is working towards better types of security there so other things like better entropy on the seeds. That's another problem. Anyway, so you unlock this thing, service 27. There may be some write data by identifier service where you write like a fingerprint or some specific information. So this is typically like the tester might do like write the programming date into the ECU like the last time it did it or write it's like serial number. So like the tester serial number and say, hey, this was flashed by Ben's serial number. One, two, three, four. Very professional. So then what it'll do is, again, this is, a lot of the stuff is defined by ISO, but exactly how it's done is OEM specific. So some OEMs will have a secondary bootloader that you download and typically what's in that is like the actual flash routines. So residing in the ECU normally, they're not in there. So you can't like just, you know, gain, do like some buffer overflow and get into there and rewrite the ECU because the code for that's not even in there. The flash drivers are actually downloaded by the tester and they're set up specifically for that ECU. So you download that and then you activate that. And so that enables now the programming routines. So the next thing that happens then is we're going to get into this. So once we've activated the bootloader, we're going to get into this download stuff. And the way that looks is you have a request download, so service 34, which says you send basically your address and the length. And there's different types of addressing like some OEMs will have like actual address based addressing. Some have what's called like logical blocks where you get the logical block identifier. Again, it's all of the stuff is proprietary. Then you have the service 36, which is the transfer data. So you're sending this and you can send basically, you know, 4k at a time until you have all of your data there. Once that's happened, you do a transfer data exit. And so this is typically where you do something like a checksum or do an integrity check of all of this data before activating it. So there may be like a routine control after that. It says, hey, do this check routine and check the checksum on it. That's the basic least secure way is just doing the checksum. You know, it varies wildly. It's all OEM specific and proprietary. So a lot of times this could be also where you do like the code signing check verification. But really that could be done in the transfer data exit or it could be done with a special routine. It's all up to an OEM to decide how to do that and their specifications. So, okay, all the data's got there. We sent it to the ECU. We've verified that the data's good. It's good to go. It looks authentic-ish. Now what happens? The tester will say, okay, go back into the diagnostics or the extended session. We start getting it ready for the post-programming. So you turn the DTCs back on because now your application is all working great and you can turn the normal diagnostic communications, non-diagnostic communications on and then you usually do like a session control to the default session. And so this usually happens through like a reset. So the bootloader says, okay, we're done. We reset and you jump back into the application. So, okay, now that you have like the most basic understanding of how ECU bootloaders, or automotive bootloaders work, how are these used potentially as an attack factor? Well, there's lots of things that it can do. It's a very powerful. It's its own application. And so just by what it does, it's normal operation, it necessity, like by necessity, has to do things like, well, it prevents the application from running because it's its own application. So if you can say, hey, go into the bootloader. The application is not working. And why would you want to turn the application off? Well, what if you're trying to like spoof messages? If you're spoofing messages on the bus and the ECU is also sending out its valid messages, well, now you're, you have this contention or confliction. If you like to make up words. The, so if you have this, you know, now you can turn this ECU off and you can send its messages and spoof the messages. Most automotives right now are not using, like can is not by the protocol itself, is not sending authenticated messages. It's a bus protocol. So if I send messages with this ID, with this signal, I can now, you know, the ECUs that are listening to this can now have to listen to it if it's, if you're on the same bus and all those other things. So turn the ECU off and send your own messages and spoof those. It's also, it is out of reset the decision maker, right? It's deciding, is this application valid or not? And meaning you could also say if you can make it so that it thinks the application is not valid, well, now it's not going to run that application and you're permanently disabled or if you can somehow subvert the validity checks, now you can execute non-signed code or you can execute whatever you want, right? It also, by nature, has ways to input data or ex, or data, you know, exfiltrate data. So some OEMs implement request upload. Part of the ISO spec. It's available. The general practice is not to do that because then you could just send requests to pull all the code out of there, your firmware. But it's, some OEMs do it or the supplier implements it for whatever reason. By the nature of the ruler you can write data to it, request download. That's, you know, if you can write something into RAM then maybe you can execute it like a couple slides back. We've got this programming step where you're downloading a flash driver. So that's basically downloaded and run out of RAM. You can read interesting data, right? There's the service 22. Read data by ID. So if you have IDs we can read certain things out of there. That might be interesting in getting things out of it. Again, it might have, because it's so it's a separate application, it's a different session. There may be different dids available than what are in the application. You can read information sometimes out of the boot loader. You may be able to read a specific spot in memory. There's a service 23. Read memory by address. And again, these are all like, these are just the ISO services that are defined, right? You can look them up online. Write specific parameter data. So write data by ID. That's available. Write memory by address. That's a service. Usually there's range checking and stuff in there if it's done, implemented correctly. But again, there's a lot of powerful things available if you can get around security there. Another attack factor is the programming sequence. So if you have a valid tester even without knowledge of the security or anything like that, you may be able to record the programming sequence. You may be able to replace certain parts of it. You could potentially intercept firmware over the wire because lots of OEMs don't, like, encrypt the data once it gets to the CAN bus. It might be encrypted at rest. It might be encrypted in the cloud when they pull it down. But the the ECU doesn't have, like, the symmetric keys in it to, like, decrypt the firmware. So it's usually just transmitted over, you know, unencrypted. Again, this it's all OEM specific and people are moving towards encrypted firmware, of course. But you know, that could be with a valid tester that could be easier than you know, trying to get JTAG on a device and pulling the firmware, right, if you can just snoop it off the bus. Again, I mentioned this before, replay attacks on certain sequences. So if it's, if the tester's validly doing something interesting, you can record that in CAN and then replay it. And how would you do something like that? Well, I'll get to that in a second. Also, by nature of what the tools have to do, right, the programming tool is going to have the images, right? You have to flash these images to the ECU. So if the tool itself is not well protected, that may be like an easier attack vector than trying to like you know, get a buffer overflow in a boot loader or trying to get JTAG or somehow like taking this, you know, pulling the firmware off the ECU because the programming tool itself needs to have the firmware because it's sending it to the ECU. You can also get information about the proprietary programming sequences and things like that because you know, unless you have specs from an OEM or you know, have access to this stuff, it's all highly secretly guarded, right? Also, anytime, you know, it's using any kind of like symmetric keys, like if it's the CT algorithm is not authenticated all the way to like a back end, the tool itself, you know, may have the symmetric keys for CT. So instead of trying to reverse engineer this firmware or, you know, hack into an HSM on an ECU, the programming tool might just have all the algorithms in it. Another thing that's awesome about UDS is that so that the way the seed key is designed is basically it's like a, you know, it's like a blind butler, like you do the secret knock and he opens the door and then anybody can go in. So once you've, you know, once you've unlocked the ECU or gotten into that session then you can just kind of keep that session alive from a valid tester and then send whatever messages you want. So kind of snooping a valid session, you could hijack that and send kind of other things. So having access to these testers or if you're on the bus at the same time, it's kind of you have lots of access there. Other tag vectors. A boot loader trojan. So, you know, there's ways to even gain persistence on an ECU by downloading a boot loader that overwrites the boot loader, right? There's products that do this for updating boot loaders and essentially you program an application into the, as the new application has to look like a valid application and then it just overwrites your boot loader with its own boot loader. You might lose your keys, you might, you know, now you gain permanent persistence on the ECU. So, what makes this even more scary? So before, for all of this, you had to have physical access to the vehicle, right? So, it doesn't scale well. I can't physically walk to a thousand or a million different vehicles and start changing the firmware on them. But, don't worry, everyone solved this problem because now everyone wants to add over-the-air programming. So, many OIMs are incorporating this. Implementing is just like a diagnostic tester that's connected to the internet, right? So, depending on how secure that is, that can put in a lot of challenges, right? OT is awesome. It's great for fixing bugs. We can add features. We can do security updates. It's, it's helps facilitate, you know, rolling out recalls, like fixing recalls and stuff, because you can push these things out quickly. OTA, basically, is remote code execution. And generally, that's bad in terms of a security model. So, it's like, it's the savior, but it's also the thing that can really hurt us. It's kind of a two-edge sword. And it opens up some interesting challenges that we might not have had before. So, attacks on, like, repositories itself there's, you know, Defconn's huge, right? There's all kinds of fun talks going on. I'm sure somebody maybe this year or certainly in previous years have talked about, like, repository attacks on updaters, like, for Ubuntu and, you know, Windows Update and all that. Same thing with, with Automotive, right? If we're doing OTA updates, it's opening up attack vectors for, you know, exfiltrating code, signing keys, you know, repository poisoning, things like that. Also, there's some challenges in keeping OTA going, because the car has intermittent connectivity. You know, if it's parked in a garage, it doesn't, might not have access for who knows, right? Or somebody might park this car in a garage for 20 years and turn it on and now you have, you know, 6 million software updates. You can't drive your car for three weeks because it's downloading. And then, like, man in the middle type attacks to the vehicle. You know, it's, it's literally a moving network, computer network, so it's, it's a really challenging IT problem to maintain this kind of infrastructure and securing all of that. So there's a lot of new challenges that could prevent updates or pass bad information to the cloud or to the vehicle. And so it kind of, it's pushing, you know, OEMs to have to really think about the full end-to-end lifecycle for updates. Oh yeah, and cars move, so if you brick your phone, usually that's, you know, that's a bummer, but if you brick somebody's car and they're in the middle of, I don't know, the desert, they're not going to be particularly happy. There's also all kinds of, like, OTA attacks. There's a, so there's a project called Uptane that's doing, like, a security model framework for OTA attacks, and they've kind of enumerated a whole bunch of different things, like malicious software attacks, you know, updating, trying to poison the cache with bad updates, freeze attacks where you're stopping the updates, basically stopping the updates from having by slowing it down, or maybe you have, like, bad, vulnerable software trying to get the ECU to roll it back to a bad version. There's a lot of challenges, and lots of ways to, like, DOS the system, so it's quite a big challenge. So, now that it bummed everybody out, or maybe they've excited because maybe they're trying to hack cars. This is the car hacking village, right? So what's the industry trying to do to make it more secure and for people that don't care about that, that also means what might not be implemented right now? Things like code signing, right, where you take a message digest, you're signing it with private key, penning that to the data, and then verifying it on the ECU side. So this is one of the, like, things in that obtained project. This is one of the things that that project is really trying to, like, set up delegations for how you make a signature scheme for automotive really fault tolerant against different types of attacks. So a lot of work being done there. Secure boot, push and secure boot. You know, so this is, like, building a chain of trust to ensure the authenticity and integrity of code at startup. Like a, like, a she module, which is, like, a small HSM. You know, these might have, like, a really simple secure boot, but if you get into, like, the systems on a chip, that there's the main, mainline talk, I think, yesterday about the Qualcomm firmware, and he showed, like, the Qualcomm firmware secure boot chain, and it was, like, 45 different things. So each one of these might have, like, could have a vulnerability, right? Protect that whole thing. So these can get really complicated and really, really tricky. And also who owns the keys for a secure boot chain as it becomes really complicated, because it could be in automotive, we have this big, huge supply chain, and different stakeholders might need to own different parts of it. So, like, an OEM might want to own some things, but the original equipment, like this tier one supplier might also need a stake in it. So it's, like, these things can get really complicated and they're hard to, they're tricky to really secure. But we're working on it. Another big thing, like, this is, you know, what I work on all the time, hardware security modules. So this is, like, hardware acceleration to help facilitate some of these things. So, like, I mentioned before, you know, you've got the seed key. So, one of the big problems with that is, like, really, really poor entropy on your seeds. So somebody might just implement, because we don't have, like, there's no, like, real-time clock, there's no, you know, good timers, you don't have a good source of entropy on these ECUs. So, you know, people would implement this, like, super awesome proprietary random jumber generator, and you know, it throws out four different random numbers in a row. So it's really easy to replay these seeds. So hardware facilitated true random number generation. That really helps with certain things like that. Protected key storage. AES hardware acceleration for symmetric crypto or other symmetric crypto accelerators. Having public key crypto accelerators in there. Again, you know, the idea is to protect all the keys and also try to help accelerate some of these things so that we can do, enable things like real-time authentication and encryption and message authentication and things like that. There's also things to, like, enable secure boot in hardware. One of the challenging things with this, though, is that there's not, like, a one-spec to rule them all in automotive. So a lot of stuff is using, like, TPMs and, like, IoT or PCs and, you know, traditional IT. In automotive, there's this, like, baseline specification from this is, like, the hilarious German pun here. They call it the HISS SHE. So HISS is, like, the Hirsteller initiative software. It's, like, a consortia of, like, a bunch of German suppliers and OEMs. It's the SHE spec, which is the secure hardware extension. And it just defined, like, certain things in hardware that would, they specified for security. And it's kind of, like, a stand-alone peripheral. And so a lot of silicon vendors have implemented that and have support for that. But OEMs are, you know, moving even beyond that. A lot of people need more keys or they need, like, more specific stuff, like a secure execution environment. There's another there's another few years later group in Europe called Aveda. It was, like, a government EU-sponsored consortia. And they came up with, like, another spec that kind of divided it up into, like, three different levels. So this low, medium, full Aveda spec. And so this, like, the highest one, you know, has a secure code execution, a coprocessor, and everything like that. So again, silicon vendors are, like, implementing stuff like that. Like, we have one called ICWEM for that. But other silicon vendors, like NXP has the CSE, which is the cryptographic services engine. And Phineons got their own stuff. ARM has the trust zone. And, you know, there's been a bunch of talks on trust zone hacking stuff. So if you're interested in that, there's lots of information on trust zone stuff. So it's, it's, it's kind of a big challenge because there are so many different varieties. And there's no, like, just common, you know, API for, okay, here's the automotive API and spec for building this secure hardware as a chip vendor. So every OEM kind of just has their own, their own flavor of how it works. So again, this is a challenge. But there's also, you know, committees that are working on kind of creating more common specs. There's a CSE group for that. And there's also a ISO group that's working on kind of, like, common security practices. Some more stuff. So ECU hardening. This is just, like, general security, good hygiene stuff. But, you know, you'd be surprised how often this doesn't happen. And most of the OEMs have specs now that say, like, yeah, you have to lock the JTAG. Or, you know, tamper protection, things like, if the tamper protection is detected, it erases your keys out of the HSM. So even if somebody gains access, they can't get the keys out. Or can't use the, you know, can't use the thing. Again, good crypto hygiene. Don't use the same key across all of your cars, like some OEMs have done. But, you know, this hardware stuff basically helps enable some of the higher level, protocol level kind of security, like doing authenticated CAN. So there's a thing called AutoSAR, which is, like, a software and it's open standard software initiative or something I've had and I'm forgetting what it stands for. I should know that. Yeah, but anyway, it defines, like, a protocol for doing secure authenticated communication on CAN. And also authenticated diagnostics. So there's not any kind of specification for this right now, but OEMs, this is all OEM proprietary stuff, but, you know, what the industry is working on basically is enabling, you know, much better authentication for diagnostics. So instead of, like, having a tool that has all the symmetric keys in it and, you know, being able to have access there, you have to go out to, like, an actual server and it sends, you know, some signed piece of information that you can verify with a public key. So, I'm gonna do one on time. So, on summary here, bootloaders give you powerful access to automotive ECUs. We started adding OTA to these things and it's making everyone's lives really difficult. But it's keeping us all employed, so that's good. But yeah, it really opens up the attack service for remote exploitation. It's making, you know, it's the enabler to be able to do updates and to push security patches and things like that, but it makes stuff extremely difficult. I think automotive vehicles is one of the most difficult things to do OTA on, because it's this big thing that's moving that could smash into something. And then, of course, you know, we're working on this, we're working on securing the ECUs, but are we doing it fast enough? Are we ever there yet? Of course the answer is no, we're still working on, still working on that. It's improving all incrementally, of course. So, thank you. I really appreciate you guys taking your time to, you know, lose your happy hour, drink prices and sit here and watch and listen to me, so. Bye.