 There we go. All right, welcome back everybody. We are here with another of our great DEF CON speakers. We have Patrick Hiley here, and you can check out his presentation on YouTube where he talked about reverse engineering, the Tesla battery management system for more power. Welcome Patrick, how are you doing today? I'm great, how are you guys? Doing all right. Pretty good. So can you just kind of give us a little bit of an overview of who you are? Let people hear a little bit about your presentation just in case they haven't seen it or kind of give a little bit of an overview recap. Sure, so my name is Patrick Hiley. I'm a security consultant for Rapid7, part of the penetration testing team. And I got interested in this project back when I just started looking at how the Tesla kind of worked inside the Model S specifically. I'd seen how some people had hacked them. And once I got a chance to peek under the hood, I just wanted to learn more and more. I was already into car hacking and learning more about this. And once I realized that there was an upgrade that this car had that nobody had really researched and actually published information about, it kind of gave me a path forward. So I went to see like, okay, how did Tesla actually make this car capable of ludicrous? And it sent me down a path that was probably the most complicated project I've ever worked on. That's awesome. And I believe this is your first time being a main track speaker for DEF CON, is that right? That's correct, yes. All right. Well, for anyone that is familiar with DEF CON, we have a tradition that you should be very familiar with by now. It's called Shot the New. New speakers are welcomed into the DEF CON speaker collective with a shot of a drink of their choice to bring them into the community, celebrate their joining, the people that are giving back to the community knowledge. So thank you. Here's to you. Bottoms up. Cheers, congrats. Thank you. That's enough. Okay, yep. All right, good. Loosen you up before the questions start getting serious. Okay, so first off, like I was blown away by the amount of different techniques and skill sets that you cruise through in your talk, you had hardware reversing, hardware debugging, assembly reversing, a decompiling, compiled Python, diagnosing obscure third-party issues, binary extraction, just like a crazy amount of things. How did you go about piecing all these things together? How did you find your next steps to go through everything? Yeah, so piece by piece. I already had like the mechanical skills. So, you know, in the past, I've had like a Mustang that I've either replaced the heads on or put a supercharger on or, you know, things along those lines replaced exhaust systems on. So I had the mechanical knowledge already. I've been doing that for longer than I care to admit. I think even when I was a teenager, I was tinkering around with automobiles and taking them apart, taking carburetors apart. So complex, small items like that. And then just trying to take things apart and figure out how they work has been something I've been doing since I was a child. My parents told me about how the time I locked them out of the bathroom and then just to disassemble the toilet. And I don't remember that, but they remind me of it on pretty much every occasion when I talked about something I've been working on. So there's that. So really, once I started getting into it, I already had the reverse engineering skills from just doing some other stuff and had the CAN bus skills. If you look at some of the previous presentations I've done, I've been poking around CAN bus for some time. So having the DBC files, it was really trivial to just diagnose the stuff going on within the CAN bus. From there, it was just, okay, how do I do these other things? I know that CAN has this protocol that sits on top of the thing itself for writing diagnostics called ODX. So I found those, or called UDS. I found those ODX files and worked with a car fucker who helped show me how to use them. He's one of the main guys for the car hacking village. He showed me how to actually, import those into vehicle spy and play around with them. Helped me figure out some of the security access stuff. I had already seen some of that from Craig Smith's work. Nori, he figured out how to pop airbags on cars. So I knew about security access so I can understand the algorithm. But Tesla's algorithm, fortunately, for this vehicle was incredibly simple. It's a static seeding key. There was no transform required. It's like, you request the seed and you reply with a fixed reply and boom, you're in. So that made that part easy and then the rest of it was just piece by piece digging through the Python code and then building a test bench, which for any reverse engineering project when you're doing the very expensive piece of hardware, try and replicate it on a bench because that was invaluable. I never would have attempted half the stuff I did if I hadn't proofed it out using a much cheaper variant of it that I just wired together and figured out how to work. I was, there was one jump in particular that it was right after you had your Tesla towed home not figuring out what those particular, like what was going on with there. Like, do you remember like what was the final thing that got you past that hump? Like, I think that- Oh, I remember it really well. I imagine that there was some stress involved there. So, yeah. So I mean, I was into day two of trying to. Oh, having a little video loss might be some Wi-Fi interference or something. I said, no, I'm not here for the weekend. I'm over on Irvine. You're here in Rancho Cucamonga. I'm not coming here. You can come like Monday. I'm like, no, I'm gonna, I'm gonna like bail out and figure the rest of this out at home. So I only needed your garage to drop the battery and do that stuff. If I have to come back for that, I'm really in a SOL type situation. But so stressing out that for day and a half, I just started capturing a bunch of logs. So just from past experience, I was like, maybe the logs are telling me something. And the logs weren't great, but they did mention this one file and it was a firmware.rc. It mentioned that a couple of times. It kind of just had that around the point where it said error and I could see that it failed. So basically I can stamped. Okay, when the car said, you know, fuck you, I'm not actually gonna upgrade or let you redeploy the software. I kind of noted where that time was and looked right there in the logs and saw an error about that. And then I just did some searching. And it turns out the 10 cent guys when they did their Tesla gateway reversing, they mentioned that file. They mentioned how that file was loaded into memory. I'm like, I've never seen that file before. I've seen it mentioned, you know, I searched through all my stuff, found a couple references to it. And then I just like, well, let's just go to the gateway and see if they'll give it to me and just give you a little background. This is not why I have the car. I had already spent the two days. I was already flustered. Had to buy a last minute ticket home, flew home and I was messing around on my bench. So I was like, I went to my bench version that had a gateway and said, give me firmware at RCA. It gave it to me. I'm like, cool. Okay, you've got this BMS error. I've already replicated that. Looked at the values in it. I actually have a copy of the map here, but there's no real way to share it. But there's basically a tab separated value file. And within that file, it tells you the firmware that that particular battery pack ID needs. And it has a little CRC code. That CRC code needs to be in that firmware.rc file. And once that was updated with the correct CRC code and the BMS I knew was first the version of firmware. I could also see that CRC code being broadcast by the BMS again. I did that. And I was like, okay, well, there's this final CRC file. How do I figure that out? I asked some people that I knew it was like, hey, I don't know a ton about CRC 32s, but here's where I have. And here's the file itself with the CRC value. And then it said, oh, it's a jam CRC. Here's a file, here's a website you can go to to recreate it. So I recreated the last CRC line, uploaded it back to the gateway, rebooted the gateway and the error clears. So I'm like, good, I've got to pass forward to the car to actually try all this. And the car was basically being an asshole as well because the battery wasn't engaging. It was just dealing with 12 volts. So a lot of the components, it would just basically shut down after a little while. So I basically had to make sure it had a good battery maintain honor on it. So it sat there for a few minutes to make sure that the 12 volts had enough juice. And then I tried to get the gateway to go through its process. And as soon as that car woke up and I heard the clicks of the contactors in the rear, it was like, hallelujah. There was one other thing that you said you missed that you would only say over drinks. We did just have a drink. Can you tell us? No, okay. It was basically me messing around with the car and putting on a feature that it didn't really need to have. It was causing the error and it's just, you know, different times I was screwing around with something I shouldn't have been. My own fault is really the crux of what I want to say there. Fair. So to catch you in person over drinks, I tried to slip that in. Didn't expect it to really work. That was pretty sly. Let's try pastes. Yep. So go ahead. Got one question coming in for you that you would have had to pay Tesla about $5,000 to make this transition. What was the approximate cost for you to do it yourself? Counting the towing or not. Let's say not, because maybe somebody will learn from your lesson. So the fuse itself, where I found on the user market for about 350 bucks, I found the contactors for about 200. The rest of that was just basically my time and effort and labor. So just basically me being my labor. The rest of the stuff didn't really cost me anything other than just time. And then of course there were all the components to my bench. You can buy a Tesla MCU for about $700 to $1,000 on eBay or some of the other markets. And the BMS is running anywhere between 150 and 300. I wanted to make sure I got a BMS with the shunt. That was a little bit harder. I just happened to get lucky and saw one. But the bench parts I guess you can consider were major portions of that all. And then there is a second kit that I bought just to be safe. That was actually the exact part number that Tesla used to do the upgrade. So basically people would pay for this kit and maybe be part of their whole upgrade package. I bought one of those as well to make sure there wasn't something I was missing. And that was about another $1,000. Can people just go to their local junkyard and pull pieces themselves to create a test bench? And if so, do you think the junkyards really understand the value of this equipment that you're using? Well, so many people are doing weird things with Teslas. So I'm sure they can, but I'm not sure how many junkyards you may be able to find that actually have Teslas out there. I found the secondary market to be really all, but if you can get one, if you can get to a junkyard and you have one that has an MCU, great. Getting the BMS would be a pain in the... I don't even want to think about it. You basically have to drop the battery pack out of the vehicle to get at that BMS. There's no way to get at it from any other means. So doing that in a junkyard would be problematic. But if you can get that center display and it's not already been snagged, that's the part that you would start with with any test bench, because that's where you learn how to route the car. That's where you learn to do all the changes. That's where the gateway is stored. It really is kind of a central hub of the vehicle. Yes, go ahead. I was gonna say, I keep hearing you mentioned the gateway. Like what exactly is the gateway in the car? Can you talk about that? Is it like actually like a network router or is this something else? So it's called the security gateway. It's a function you'll see on newer Canbus automobiles. It basically is a device that kind of exists like a firewall, but not sort of like a firewall that exists between multiple Canbuses. And in this place, the infotainment unit and the part that actually connects to the internet. So it acts as a ethernet to Canbus gateway. So the ethernet side is what connects to the central display, connects to the instrument cluster. And then from the central display, there's like the cellular connection, the wifi connection, the Bluetooth connection, the USB ports, and then has all the logic for how it communicates out on the internet. And the rest of it communicates with the rest of the Tesla mothership. But then the gateway also bridges the various Canbuses. So it can take a message from say the powertrain Canbus and copy it onto either the ethernet Canbus or the chassis Canbus. Consequently, it can take messages from those other Canbuses and push them onto the BMS as well. Because sometimes devices that are not on the same bus need to communicate with one another. So when you're requesting the firmware, is it coming directly from the gateway or is the gateway like asking these sub-devices to say, hey, send me your firmware? So just to clarify your question, are you talking about the firmware.nrc file or are you talking about the firmware itself? Well, kind of both. I think you did both in your, throughout your talk. There was one particular place where I noticed that you were issuing a command line to retrieve a file and then you made some changes. I think it was, you even used Vim, which represent. And then you pushed it back. Yeah, so that was all from the CID. So the CID on the vehicle that we're talking about is an NVIDIA Tegra running Ubuntu. That, and it's, so it's basically, it's an ARM version of Ubuntu. All the firmware for all the modules of the vehicle sit within that firmware image. And then when people talk about how their Tesla gets updates, it has all new firmware and it pushes out firmware updates, all the various modules that need it. Some module way over here may not need an update, so it may not get updated. But that's part of what the gateway and the main system do when they do an update. The gateway itself does store a few files, but it doesn't store a copy of the entire vehicle's firmware. The entire vehicle's firmware is stored on the central display, you know, on that Tegra-based Ubuntu system on that little EMMC chip. That's the same EMC that people complain about wearing out because of all the logging that's going on, but the, the, the image itself has that entire copy of the firmware. The gateway only has things like the firmware.rc file, the internal.dat file, and a few others, like hwids.acq, which is all the hardware IDs of the vehicle, and it gets that from the CAN bus. So the gateway does that, just queries during an upgrade, but there are also some like, like crash files that are stored on the gateway because it's easier for it to store on the gateway. I don't know why it does that, but some of the crash files are stored there as well because I saw on my bench system some images right before the vehicle was crashed. Okay, so, so when you say crash, it's not like software crashing, it's like a black box from the actual component. Yeah, there's an actual, I think there's another like actual black box device, but the gateway stores quite a bit of that. I'm not going to pretend to understand how the whole, you know, emergency data recorder function of the vehicle works, but it's there. Some of the files are on the gateway. Yeah, that might be an interesting future talk for someone just perversing the black box of Tesla. Yeah. So you also do a lot of stuff with the car hacking village. Is there anything interesting going on that you want to sort of announce while you're here to get people to come and show up in the car hacking village, anything like that? Your question broke up right at the point where you asked the critical part. So do you mind repeating it? Oh yeah, sorry, sorry. So you're very active in the car hacking village. Is there anything like interesting going on, like maybe Tesla related or related to your talk that you're going to continue working on something that you want to like pimp out? So tomorrow at 10 o'clock, I've got a deep dive into many of the techniques that I covered in the main doc. And that's basically because, you know, due to the whole change of the online format, some of the stuff got cut off to the talk. So I have some, some of the stuff that was cut from my, you know, talk itself, but then I also expanded on some additional topics as well. So I go into some of the binary analysis. I load up the former insight IDA. I don't get very far, but I show that I was able to get an IDA. I do some of the like live UDS techniques, you know, the security access as well as the shunt calibration. And then it's just another brief overview. So it's like three different things that I'm kind of talking about. And then, you know, I can't do the C stuff as well. Other than like the setting up a bench itself, is there any particular like software that, or tools that people are required to do this kind of like investigation and playgrounds work with car hacking? So I was using a not free tool called vehicle spy only because it was very powerful and made it a lot easier. Only had it available. You can use tools and some of the other stuff out there pretty much make it possible to do whatever you want to out there. There are a ton of CAN bus interfaces out there. One of my favorites is the Panda. So for this, I'm going to give a shout out to the comma.ai folks for actually making such an awesome CAN bus interface. Comma.ai is a open source self-driving tool that I'll just let people go out and check it out on their own. But they also, they make a tool called the Panda works really well with the Tesla. It can connect to three CAN buses at once. A lot of people that have done hacking on the Tesla use that and I'm very impressed with it. It's a very inexpensive tool that'll give you physical interface. And from that, you can connect over WiFi or USB and from there directly, I'll start messing around with Linux tools that are all free. And those are all I can can use tools. That's cool. What's been Tesla's reaction to the kind of work that you've been doing to get you kind of ludicrous speed? Yeah, so when this talk was accepted I actually reached out it several times and Tesla was very supportive. They actually just asked to review the slides to make sure they're wearing any surprises. I kind of told them, hey, I'm gonna talk about this, this and this. They said, yeah, just do us a favor, send us a copy of your slides, white paper, et cetera. So we'll make sure there are any gotchas in there. And they said, yep, everything looks good. You're good to go. So very supportive. They actually have a program called the Security Vehicle Research Program in addition to their bug mounting. So a vehicle that you're interested in doing security research on can be registered with them and you won't violate any warranties. So basically, and supposedly I haven't tried this, they'll even help you with some breaking situations. Didn't need their help on this one, fortunately, but they do some of the other stuff. Sorry, excuse me. I've got a really annoying thing going on. There we go. Sorry for anyone that was on the stream. I was just constantly getting calendar alerts for the next things that I'm responsible for. Are there any other things that you're aware of in a Tesla that you could possibly try to unlock or bypass or anything else that either you want to look into or you think that other people might wanna look into to build upon your research? I'd have to think about that one for a bit. Yeah, there are some things. I would like to see if where the actual limits are in the BMS. I would like to understand those, but I don't really have a desire to push past them because you're starting to get into dangerous territory at that point. I just wanna find where they are. So it's like, all right, here's the variable that actually controls max power. Here's the power curve that's basically, defined state of charge, battery temperature and how much power is available for the various drive inverters in the battery itself. I'd like to find those. So that's, hence the additional item stuff. And then just more or less actually get more underneath the hood of how the battery management itself and the drive inverters work. The drive inverters are not something that I actually have hardware on a bench for, but because they're very expensive, the whole drive inverter is inside the drive unit and drive units a multi-thousand dollar piece of equipment, but yeah. I thought that's where I would like to get from there. I wanna see where the steps are for the P90D and P100D because those cars are faster. So we did get a really good question that was, for those not familiar with the whole gateway canvas stuff, the car be able to phone home after you've made the ludicrous speed modification. Yeah, so the vehicle within its firmware itself has a function where actually it uploads the config to Tesla on a regular basis. I think it's like vitals.json puts it in json format. So basically they take the vehicles config and whenever, I think it's when the car goes to sleep, it actually uploads that information to Tesla. So anytime someone makes a change that wasn't done by someone else, or I'm sorry, even when the service center makes a change, that is gonna be seen on Tesla's end unless of course you figure out a way around that. And I imagine that you're probably no longer under warranty as soon as you root the front panel or something like that. You know, Magnuson Moss Act affects a lot of stuff. But I'm not a lawyer, so I'm not even gonna go there and try and figure it out. But it's like, if you root the center display and you have a problem with your brakes, how are the two related, you know? Yeah, fair. But if you root your center display and you break your center display, it's a totally different situation. That's all a great legal area that I'm not gonna dare go. And to follow up on that question, if Tesla does get made aware of this bypass that you put into place, could they then reverse it and remove it from you? I'm sure they have. I mean, there are stories out there about how people had a ludicrous speed vehicle and then after they purchased it, it was removed because Tesla said they audited it and found out that it shouldn't have had it and removed it. And then of course, the person who bought the vehicle was pretty pissed off. So don't know what the resolution on that was, but that's also, there's a huge amount of controversy around the whole super charging thing. So if you buy a salvage Tesla, it has super charging disabled. Tesla does that, it's, you know, they're prerogative, it's their supercharger network but it makes it very difficult to go on trips. And people figured out using the same techniques that I'm talking about here on how to re-enable that again. And then there was a thing that came out recently that basically says, you know, you're opening yourselves up to getting sued by us because you're basically getting something that you shouldn't be. My only suggestion for Tesla is like, why not verify the safety of it and then only enable it to where they have to pay for it? Because now you've got a source of revenue, but. That was gonna be my next question because I know that the, after a certain point they stopped allowing superchargers to be free for Teslas, but it sounds like it's completely disabled on these. On the salvage vehicles, it's disabled. The way I understand it is if the vehicle has free supercharging and has free supercharging, but as soon as the vehicle is sold back to Tesla, it's lost. This particular vehicle has free supercharging because it was purchased with free supercharging. So it should always have that, but you know, as I understand it, they haven't taken it away from anymore. They said, you know, this vehicle will have free supercharging for life unless it has been in an accident and been, you know, totaled out by an insurance company. All right, so it wasn't just like Tesla purchasing it back and then reselling it with a disabled or something like that. It's like, you know, people trade in their cars. People trade in their cars. I think when they sell them out again, they don't have free supercharging. I don't know. That makes sense. I don't keep track of them. Yeah, I didn't know like this. It seems like a really kind of sketchy area. So any kind of details that I can pull out of you. Yeah, their frog is. Yeah. So it seems as though with the, where electric cars are going now, that this is going to be a really great area of research. And what kind of advice would you give to people that want to get started with car hacking research? Like if somebody wants to start from the ground up, what sorts of things could they do? Where should they start looking? Start learning CAN bus. That seems to be a real out of it. Gets from there, learn UDS. And from there, learn about like binary reversing and just reversing engineering in general. Go to a junkyard, find a car that you find interesting, verify, you know, it has CAN bus and start ripping modules out. We did something with our company called a junkyard hackathon where a bunch of us went out and just ripped apart various vera vehicles. Craig Smith helped us. He's the writer, you know, the car hacker's handbook. And we got like several vehicles worth of modules. Even if you just get one module, take it apart, figure out what CPU it has, you know, learn how to hardware hack it. And from there, have fun. Yeah, I thought I've seen in some of the videos with people hacking on electric cars. Maybe I'm remembering this wrong, but is there any risk to a car sitting in a junkyard still holding any kind of electrical charge that people should be aware of? Yeah, because high voltage is high voltage. It'll hurt you either way. I personally have never been to a junkyard where I've seen electric cars. I've only seen the salvage yards where they're like the warranty auction places. And you see those online. But the actual junkyards themselves, I don't think I've ever seen an electric vehicle in the junkyard. Even a hybrid would be dangerous though because the hybrid still has high voltage because it has an electric motor. So we got one question. Oh, it's kind of more like Hawkeye is wondering if Teslas have a problem with shutting down if their error message has reached a critical amount of storage space. I'm guessing that has to do with the EMMC logging thing. I've never seen one run out of storage space. It's just the volume of logging. And I believe at a certain point it overwrites itself, but just it's the volume of constantly writing to the EMMC that wears it down. Not for the device storage actually filling up. And it's unfortunate because it's like a $20 part. The EMMC module itself on that DAGRA is a very inexpensive part, but it's a $2,000 repair if you take it to a service center. Excellent. So as we start wrapping this up, what sorts of takeaways do you want people to have from your presentation and what would be your call to action or what would you like to see come about based on your research and your presentation that you've put out there? Well, I'd love for some day for Tesla owners to be able to work on their own vehicles for there to be like a consumer version of the tools that are used to work on and diagnose the vehicles. Cause right now you just have to take it to the garage and pay the service center to do that. I can understand I have a high voltage components why you want to do that, but as these cars age and continue to be out there in a fleet there needs to be another method. In addition to that, just this is kind of like the next version of hot riding. People are going to figure this out. Now that kind of electric cars are becoming more and more mainstream, there are going to be people that want to buy them and make them faster and you can make them faster by either lightening them or you can make them faster by tweaking what's under the hood just like you would on a standard internal combustion engine. So this is kind of like the next phase where that's going to go. Awesome. Thank you so much for doing this. Thanks so much for your presentation that people can go see on the YouTube with the Defconn channel. So thanks for doing this Patrick. Really enjoyed you discussing this with us. All right, thank you.