 All right, I'm Zaz and it's a great pleasure to be back here at Defcon China 1.0. My talk is going to be on hacking driverless vehicles and I'm also going to be including connected vehicles in that because basically unmanned vehicles are just another Internet of Things device except they're an Internet of Things device that's covered in sensors and that's able to move around in the real world and cause all kinds of mayhem. So I'll be looking at all of that stuff and one important distinction that I want to make right from the beginning is that I'm not going to be looking at attacks that include a network foothold on the platform. If you have a network foothold on the platform you can do anything you want that's well known. I'm going to be looking at ways that you can attack these vehicles from outside the system. A little bit of background about myself. I have an autonomous robot's background. I have a PhD in robotics and I am best known perhaps for a TV show that I did with another speaker that spoke at 10 a.m. this morning, Joe Grand. The show was called Prototype This and in the course of that show we actually prototyped a number of autonomous systems. For example, we did an unmanned aerial vehicle that delivered life preservers out to a swimmer in the ocean and we learned a lot about the various failures of aerial platforms and how to iterate on those platforms as part of that. We also turned our hands to ground vehicles looking at one of the principal problems in the autonomous ground vehicle space, the efficiency of pizza delivery. So we did two vehicles there, a local vehicle for delivering pizzas on the streets and a long range vehicle for long range pizza delivery which included the first ever autonomous crossing of a highway bridge in the United States. So this was pretty serious work. Although autonomous vehicles seem like a new and shiny thing, I want to take you back to the sort of origins of autonomous vehicles which weren't in fact in the United States. The first stuff was done in Germany. In the 1980s, 1986, a vehicle called Vamors that was done by a pioneer in autonomous vehicles, Ernst Dickmans. And in 1995, they drove an autonomous vehicle from Munich to Copenhagen at up to 175 kilometers per hour on the Autobahn using nothing but vision, no GPS, no LiDAR, just using cameras. So this stuff is pretty old, right? So this is kind of like, what's this talk about? It's about hacking Windows 3.1. No, a lot of new stuff being done in the space. So I'm just going to do a quick summary of what's happening in the autonomous vehicle area in Asia because America is getting all the press, but a lot of interesting stuff is happening here in Asia. So Singapore, they're looking to be a regional leader. They have autonomous bus services that they're planning to schedule by 2022. But China right here is the place that's positioning itself to be a technology leader in unmanned systems. So they're doing autonomous bus testing in Shanghai right now, in Guangzhou, public road testing since last year here in Beijing. Baidu is testing on 105 kilometers of suburban roads, and they have a partnership with Volvo. So they're looking to deploy level four autonomous taxis, fully driverless taxis by 2021. Pretty exciting. So there's a revolution that's going to happen in driverless systems because of so many advantages, right? The energy efficiency of not having to haul a human driver everywhere you go, the time efficiency of not having to deal with the needs of that driver, and the new applications that it opens up. So lots of areas in the civil space, obviously transportation, also oceanography, right? The ocean is a dangerous and boring place for the most part, it's a great place to send robots, mapping of course, filmmaking already, autonomous drones have taken off there, engineering tasks like power line and structural inspections, and logistics delivery, right? That's something that has to happen everywhere. So in particular in terms of logistics, don't just think about the air and the ground. The sea is a very important part of logistics chain, and unmanned cargo shipping is the future, because the accidents are mostly caused by the humans. 75% of maritime accidents caused by human error, and a major technical challenge that has to be overcome though before we can deploy our unmanned systems is dealing with these long voyages, months at a time, and having hardware failure. A lot of reason you have a human crew is to fix things that break. But here's an example that's just been announced, the Kongsburg company has announced they're going to deploy this boat called the Yara Birkeland, which is going to be an electric zero emissions vehicle with autonomous capability. And so this will just give you an idea about how these systems are likely to be rolled out. It's going to replace 40,000 truck trips a year, so they're going to start in 2020 as a manned vehicle. They're going to drive this boat back and forth delivering its things. 2021, they're going to start down crewing and letting the robotic systems take over, preparing for fully autonomous operation in 2022. So that's the kind of thing that you'll see out there in the real world. So the priorities of the industry in the civil space are agriculture and self-driving cars. And the things that are in the way why we don't have our roads filled with driverless cars right now when Dickman's was doing this stuff in the 80s and 90s is the fact that we have shared infrastructure. We have to share the roads and the airspace and the oceans with human piloted vehicles and the public acceptance, the perception of safety and robustness. So on the topic of safety and robustness, let's talk about how these systems fail. So I'm going to look at two case studies of classic failures. Number one, there was an event held in 2004 called the DARPA Grand Challenge. That was an autonomous robot road race from Los Angeles to Las Vegas, the home of the original DEF CON. And this was a big deal. The prize was one million dollars and a lot of teams put a lot of work and a lot of money into trying to be the first to win this road race. And the favorite for this race was a vehicle called Sandstorm from Carnegie Mellon University. They developed all kinds of new sensors. They loaded it up with everything that it had to offer. The vehicle took off and everyone was super excited. This was the one that everyone thought was going to win. Well what happened? This vehicle got a few miles along the course and then it veered off the road, crashed, rolled over and caught fire. So there it is on fire off the side of the road. What went wrong? Everyone was very disappointed. Well what went wrong was the fact that they knew the course a couple of days in advance and they manually walked the whole course. They had a perfect GPS map. But when they actually went to run the race, they had to set the waiting to trust the sensors versus the GPS and they trusted the sensors too much. The sensors got a wrong reading, drove the car off the road. So it s a constant battle to know what the robot actually knows. And the correct estimation of that state is the key to decision making. And so a successful exploit of an autonomous vehicle will most likely subvert the state estimation process. Here s another more recent big failure. This was a Tesla in autopilot mode. This is from last year, March 2018, a huge accident that unfortunately was fatal for the driver. What went wrong? So the Tesla was in autopilot mode and so it had two types of autonomous behaviors that it was doing. Number one, dynamic cruise control. It was following the vehicle in front. Number two, automatic steering with lane following. It was following the lanes and staying in its lane. It got to a fork in the highway and it could have just kept following the vehicle in front and it would have been fine. Instead it followed bad lane markings and it crashed right into the barrier between the two roads. At 120 km per hour, the crash attenuator that was there on the highway was supposed to stop that from being a terrible crash but it was previously damaged. It wasn t fully functional and so it was a bad accident and as I said, the driver unfortunately died. And this all came down to one decision. The vehicle selected those poor lane markings instead of following the lead vehicle. So autonomous vehicles is an area that s full of fragile decision making and edge cases that have to be decided in a split second. So let s take a look at the logic structures inside an autonomous vehicle. There s a hierarchy of behaviors that operate at different speeds and at different priority levels. So at the bottom you have your control loops, the maintenance of stability and operation of the vehicle. Above that, collision avoidance. So making sure once you have control of the vehicle that you can control it not to run into things. Then you ve got your navigation and localization, getting where you want to go. And at the top level, your task planners and your reasoners. If you re a taxi, getting to the right place and then billing the customer, delivering pizza, so on, which I ll get to in a sec. So if you re attacking one of these systems, then you might want to attack lower level in the stack because they defeat everything above that. However, because those lower level things are more mission critical, then more engineering effort is spent on guaranteed robustness. So they may be a harder target, right? You ve got your juicier but harder targets. Just like attacking many types of infrastructure. So two quick examples from the very beginning of this talk. The lifesaving drone first and its control loop level, it has its PID loops on its autopilot tuned to be out of cope with wind gusts and warm rising air, cold descending air and so on. Then the collision avoidance level, absolutely nothing. This is a big problem for aerial vehicles is that there s very little sense and avoid in place. Navigation and localization, that was simple waypoint circuit going from waypoint to waypoint. And then the mission reasoner was to set up this dynamic bombing run to drop the life preserver to the person in trouble. So this is a system that s extremely vulnerable to collision because there s no collision avoidance. And its high level logic depends on one single sensor, GPS. So if there s any problems with that sensor, the system is going to have problems. Pizza delivery. I ll look at the local pizza delivery because it s kind of more interesting. The control loop level, this is a two wheeled vehicle so it has to do balancing and dynamic weight shifting. It s got to deal with the fact that the weight on top changes when a pizza is removed. At the collision avoidance level, that s what this system is all about, dynamic obstacle discrimination and avoidance using the LiDAR as a primary sensor. Navigation and localization, route planning using a map that the system builds itself with a process called SLAM, simultaneous localization and mapping. And then the final high level reasoner is to dispense the pizza to the correct person using the credit card as an authentication mechanism. So this is a system because it s doing its dynamic obstacle avoidance and it s building its own map. It s vulnerable to all types of redirection, trapping and map confusion attacks. So let s take a look at the suite of sensors that you might encounter in autonomous system. There s a basic difference in sensor types. You have your active sensors that send information out and read a return and the passive sensors that just read in information from the environment. So commonly, you ll find GPS and they usually mounted on the roof so they have an uninterrupted view of the sky. LiDAR, also often mounted on the roof, that s a laser rangefinder which we ll get to later. You ve got cameras for doing vision, often a millimeter wave radar, usually on the front and back of the vehicle. Ultrasonic transducer is a common for parking. Usually, you have a digital compass. It s always nice to have an absolute direction reference. An inertial measurement unit so that you can measure acceleration and rotation. And then wheel encoders so that you can get some kind of absolute odometry. And then for other types of vehicles, for example, under the water, you might have sort of more esoteric sensors that are designed to operate in that domain. So a Doppler velocity logger, for example, under the water, scanning sonar and pressure transducers often used both under the water and in the air to give you, for example, a barometric altimeter. So sensors have a number of sources of uncertainty. They constantly have noise that they have to deal with. They drift. So that s where they over time get a reading that s further and further away from the true reading. And then you have latency problems. So you have some time delay between when you read the sensor and what the actual value is. So the real world may have changed during that latency period. So you have to model that noise. You have to model that uncertainty under a series of assumptions. When you have more than one sensor, that s great because then you can do sensor fusion. So fused or registered data can be more useful than separate individual data from one sensor alone. However, what do you do when those sensors disagree? This is very topical right now because of an automation failure that s been in the news, the Boeing 737 MAX, which had two angle of attack sensors, but they were only using one of them. They chose not to do sensor fusion for whatever reason. Boeing has never explained publicly why they chose not to do sensor fusion. But it may have been what do you do when the sensors disagree? And what they re going to do now with the new 737 MAX fix is to display a cockpit light that tells you tells the pilot when the sensors are disagreeing. So again, your robustness of your system may come down to how good is your system at discounting one malfunctioning or one intentionally spoofed or hacked sensor? That was Boeing s problem. So let s take a look at our general classes of sensor attacks. Two basic kinds of attacks. We ve got denial, just like denial of service, that s preventing the sensor from getting any good data or spoofing. That s when you are deliberately causing the sensor to retrieve specifically bad or incorrect data. So you ve got a basic attack mode choice as well from the attacker s perspective. Do you attack sensors instantaneously, give them immediately bad data, or do you do a more subtle attack where you give it aggregated bad data and you build up a poor model over time to cause it to make bad long-term inference? So I m going to take a look at all these various sensors and how we do both kinds of attacks on them. So GPS. Denial is very easy with GPS. We can jam it. You can buy online a GPS jammer. Check your local area for legality. But China is a place that manufactures many of these GPS jammers. You can also build your own. Schematics are available online. And they just throw a bucket of radio noise out and they just prevent the receiver from receiving valid satellite signals because the actual signals that come from the satellite are very weak. Spoofing is also possible. So basically you re sending out valid GPS signals with a radio at a higher power than the genuine signals from the satellites. So here s an example of that from the UT Austin Radio Navigation Laboratory, which is one of the first places to really popularize the possibility of active spoofing of GPS. So they re demonstrating here that to take over the GPS of a vehicle, you send out a valid signal first. You take over the tracking points of that system and then you move your fake signal away. And by doing that, you can affect the controller on board the vehicle. So for example, an aerial platform, if you send your fake GPS signal, send out a signal that makes it look like the vehicle is going up, then the controller on board will drive it down to compensate. So here s an example on a real aerial vehicle. They re about to send a fake signal and tell the vehicle it s going up. And so the controller s going to drive it down. And if the safety pilot didn t take over, this vehicle would drive itself right into the ground. This is something that the Iranians said that they did when they captured this American military drone. They captured this RQ-170 drone and they claimed that they used GPS spoofing to bring it down. That s almost certainly not the case because everybody knows that GPS is very spoofable. The military knows that. They have their own encrypted GPS signal, but they don t use GPS as a primary sensor because they know that it s jamable and spoofable. What happens though in a real system when a GPS system causes a problem? So here s an example from the second DARPA Grand Challenge. This is a vehicle called Sandstorm, and we can see a straight road, but it s pulling it off the road. That s what will normally happen when you have a vehicle that s got multiple sensors and it s getting bad GPS data, is that it won t immediately go off where you want it to go. It ll sort of be trying to pull it back onto the track. Here s another example from the DARPA Grand Challenge of the perils of GPS-only navigation. This vehicle completely fails to do collision avoidance because it s getting bad GPS data and that s pulling it off the road. It s not just cars that use GPS. In particular, for ships, GPS is often the primary navigation sensor because out in the middle of the ocean you have no other navigation references, especially during the day when you can t use the stars for navigation reference. So that same lab, the UT Austin Radio Navigation Lab, decided, well, let s see if we can take over a super yacht. So they found a rich person and they said, hey, can we borrow your yacht to see if we could potentially steer this thing off course? And sure enough, they had their attack system on board the yacht and instead of sending a drift signal up or down like you would send an aerial vehicle, they changed the angle of the direction of GPS travel by just three degrees because three degrees over the thousands of kilometers that a boat might travel across an ocean can put that boat in a very, very different location. So they were able to use their spoofed GPS to send a different navigation track and cause that ship to go off course. There s the radar image from that ship to prove that that was possible. So originally, this might have been an expensive attack, but now SDRs have made this attack very, very inexpensive. The first one to be presented at a hacking conference was this one using the Blade RF. So for about $1,000, this was the QUIHU 360 Unicorn team at Defcon 23. They showed that for about $1,000 using a Blade RF, they could spoof GPS. And that has become even easier since then as SDRs have come down in price. Also, they went to car showrooms and they showed that the in-car navigation system could believe that the car was in the middle of a lake somewhere. Now, we can do this with a Hacker F1 for about $400. You do have to modify it. So you need to get that the built-in oscillator in a Hacker F1 is not load drift enough to do this attack reliably. You need a 10 megahertz TCXO load drift oscillator and you make a little daughter board for that. So here s the Osh Park images for that daughter board and that shows it installed on the Hacker F. Then you go to NASA s site. They provide the GPS ephemera data, all of the details of where the satellites are for the previous day. So you can spoof for any day in the past by doing that and then you can also fake the time to spoof in the future. Then you use the open source project GPS SDR sim to convert that ephemera data into active GPS signals that are then transmitted out through the Hacker F. So I m actually doing that in this auditorium right now. Unfortunately, it just stopped. I m just started again. It may take a little while to get the signal but I ve got the maps here. We can see here my GPS device, if we can switch to the camera. Can we grab the camera view over here? Hopefully you can see that I m getting full satellite signals on this device. It hasn t quite picked it up yet because I just restarted the system but it s getting them. You can see there it s got quite a few satellite views already. Then this iPhone has already got the signal and I don t know if you can see it there, if there s any way to zoom in on that. But right now, and you can check this on your own phones, I recommend turning Wi-Fi off before doing this so it s only relying on GPS but hopefully we can see here that I m broadcasting GPS right now that s putting everyone in this room in the middle of the White House in the United States. Here s my rig here with a Hacker F1 connected to an Raspberry Pi connected to my computer. You can take down UAVs with this technique very easily if they are a little commercial drone that has no fly zone capability. Here s a video that I did of forcing a Mavic, DJI Mavic drone to land. You can see here I m simulating the GPS of the middle of an airport and that s going to force it to land. It ll take a few seconds but you ll see that the no fly zone capability has been activated. Well, I don t know why it keeps transitioning to the next one. Anyway, the drone lands because it thinks it s in an airport which is a no fly zone. Moving on to LiDAR which is a laser range finder. These were originally industrial monitoring sensors. They were used for factories and so on to keep people out of places they weren t supposed to be. They re a mechanically scanned laser and they re primarily used in robotics for collision avoidance and map building. So in order to deny LiDAR access, you can actively overpower them with a strong laser which or you can prevent the return signal because it needs that signal to bounce off something and come back. You can make it so that signal doesn t come back. For spoofing them, you can manipulate the absorptions of the reflectivity of items in the environment so they look like something else or you can actively spoof them and I ll discuss all of those methods. Here you can see the kind of point cloud map that the LiDAR builds. So as a 2D sensor, there s this laser that s being scanned backwards and forwards or left one way to the other, they re highly orientation dependent. So if you have a platform that tilts like this two-wheeled vehicle, the kind of thing that the LiDAR sees depends a lot on the angle that the vehicle is at. So that means inclines can look like obstacles and it means it may miss low obstacles and discontinuities. It s an active emission sensor. It s sending out information so it can only see something that bounces a signal back. So if it doesn t see something, it thinks nothing is there. It has no information about it. So think about it. Everything over the horizon returns nothing. It s just empty space. So most of the world returns no data to the LiDAR. You actually, even though you get a dense point cloud of things that are there, looking 360 degrees in every direction, most of it s got nothing. So also absorbent things look like nothing and transparent things. The laser goes through them. So that means we can do some physical attacks. So we can paint absorbent material on a wall just like the old Wiley Coyote gag and the vehicle might try and run right into it. We can also use obstacles that let light through and the LiDAR won t see them. Reflective things can cause problems for the LiDAR. For example, puddles of water on the ground. This is something that causes all kinds of difficulties for real world vehicles. So that means that things that are far away, the laser can bounce off, reflect back and they look like they re close. So a tree in the distance can look like a tree coming out of this puddle. And it means if you don t get a return, it s indistinguishable from a hole in the ground. You see a big gap in the road. Is that just a puddle or is it a huge hole? You can t really know. These techniques, by the way, are known by insurgent groups that are trying to deal with military applications of this technology. So here s a document that was captured from an al-Qaeda in the Arabian Peninsula group. And it mentions GPS jamming with a Russian-made Raqqal system. And it mentions using reflective materials to thwart laser designators. So these are valid adversarial techniques. Reflectance, though, in the case of unmanned vehicles is also a feature because one thing that reflects is road line markings. It s actually a very robust method to take those holes in your LiDAR point cloud and do shape matching on them and use them to find road markings. So that means that s a way that we can do an attack that fakes road markings in a way that the robot sees them, but a human safety pilot doesn t see them. So we can paint black on black reflective material that a human can t see, but we can cause unpredictable behavior from the vehicle. Or we could just send some jokes in for the people back at the mapping zone. I m sorry, I went too fast on that slide. But to send things back to home base. Solid looking objects are indistinguishable from actual solid objects because the laser just knows when it bounces off something so we can have lightweight obstacles. And in the case of denial, we can use a strong laser source to overpower the LiDAR in a certain area. So here s some research from KAIST in Korea showing that if we point a strong laser at a multi-beam velodyne so we can see here all those horizontal stripes are the individual beams of this multi-beam LiDAR and by pointing a strong source at it we can blank out part of the area of its scan. So we could hide an obstacle by overpowering the LiDAR selectively in a spot. But it gets more interesting than this. We can actually spoof returns by using a property of the way that most of the LiDAR is built. So here s an example of two very popular LiDARs, the velodyne pucks. And both of those things have curved glass on them because that way they can scan 360 degrees. Because we don t want to have sharp edges to the glass which are going to be difficult to transmit the laser through. But we can use that refraction to move our false points by modulating the power of our spoofing laser source. So that means if we were attacking a vehicle from the side we could make it think there was something in front of it. So we don t have to put our fake returns in line between us and the target. We can make them show up somewhere else. So this is using exploiting the fact that the curved glass refracts the light inwards to a different location. And then we can modulate that with source strength. So here s an example of using a weak source and we can see the false returns between the spoofer and the actual LiDAR. But then increasing the strength of our source from the same location places our fake returns off to the side. Now we can also do active spoofing. We can do kind of a relay attack. But this is a much more complicated attack. This involves characterizing the behavior of the remote laser. So we have to watch for a while, see what types of pulses it puts out, what the frequency of pulse is, and then we can start sending predictive returns from the spoofing source. This also gives us great control over where our fake returns are located. But this means the timing is critical. So you have to very, very accurately characterize the pulse rate of the source and its location. But we can see here the spoofer is located to the left of the target LiDAR, but we're inducing fake dots further to the left. Or changing the timing, now we can put the dots between us and it. So that gives us control, not of the full area, of the full xy plane, but it does give us control of the distance from the target of the fake returns. Now a system that's out there in a large number of vehicles is the Tesla autopilot because they've put that into a lot of their vehicles and they're able to update that as an over-the-air update. And Tesla's philosophy here is to only use cameras. They do use a little bit of millimeter wave radar for long-distance collision avoidance, but mostly their theory is we do everything with our eyes when we drive, so our cars should be able to do everything with cameras. We already saw one example of where that's caused a fatal accident. But so let's talk a little bit about cameras. So cameras are used for specialized object detection, including signs with traffic control features and lane markings. Sometimes we'll use stereo cameras to get a depth map, so to get a noisy version of what the LiDAR gives us. And sometimes we'll use it to fuse with a LiDAR in a method called colorizing the LiDAR. That's getting color information from the camera and melding that with the distance information from the LiDAR. This is useful for this reason. This is a video from the second DARPA Grand Challenge. This is the vehicle that won that one. The second one actually did have a winner. It was a vehicle called Stanley from Stanford, and they won because they were able to drive extremely fast. And the way they did that was to use the LiDAR to give a sense of drivable versus non-drivable, and they matched color to drivability. So that was a way they were able to look further out than the LiDAR could look by saying, okay, this is what color the drivable road is. The rest is non-drivable and being able to drive at speed. Cameras are very easily denial of service. You can dazzle them with a bright light. And then spoofing is all the ways that we know about from the ways that our own vision is fooled. Camouflage techniques, assumptions about what color things are, and then in the case of stereo, repeating patterns really confuse the stereo system. I want to say very briefly about something about all the research that's come out lately about spoofing deep learning camera recognition models. This stuff right now is kind of overblown, but I just mentioned it for completeness. So there's a very famous paper on adversarial examples for stop signs. So they trained up a recognition model on stop signs, and then they were able to add some very small amounts of black and white tape that prevented their model from recognizing the stop signs. Of course, feature-based sign detection is pretty robust, and you probably wouldn't train a deep learning model to find stop signs in the first place. There have been other examples where by adding adversarial noise to the incoming camera image, you're able to get it to make false recognition. So in this case, the pedestrians are missed when you've added some very fine, invisible to the human adversarial noise. Of course, if you have the ability to add adversarial noise to the camera signal post acquisition, that means you have a foothold on the platform, there's lots of other things you could do. Generally, these techniques are white box techniques. So these are researchers that have trained their own model. They know how the model works, and so then they attack it based on things that they develop to fool their own model, and they don't tend to work reliably in the face of parametric distortions. So when you have different angles, different amounts of glare, and things like that, different lighting conditions, they don't work so well. Here's an example of something that was pretty robust. It was robust enough that they were able to 3D print objects, and then they could show it to the camera in any orientation, and it would still work. These are the adversarial turtles from MIT. So this, for example, is a turtle that's obviously a turtle, but it has patterns printed on it so that the vision system thinks it's a rifle. But again, these techniques are not robust, and they often involve the injection of noise that requires a foothold on the platform. I mentioned it for completeness, as I said. Cameras in general involve fragile discriminators. So here's an example from just this year, from the Tencent Keen Lab. They were looking at Tesla Autopilot. They found some results that are not entirely surprising, but it's proof of what I'm saying. So for example, they added white tape to the road here to blur a lane marking, and they showed that the system successfully didn't pick up a lane marking. So this is real world blurring to prevent the Autopilot from seeing a lane, and then the converse is also true. So they were able to add just a few dots to the road. You can see them there in the upper right, and just those three fake markers were able to cause the vehicle to make an unplanned lane change. So just like that fatal accident, this is something that requires a small amount of real world preparation, but can cause a very bad accident. Moving on to the millimeter wave radar. This is the system that you find at the airport often to scan you and see under your clothes. Primarily use the collision avoidance. They're a lower resolution than the light are, and most things are very, very reflective to the radar. So it returns a lot of noisy data. So you can deny them with a radar jammer or with a chaff, which are little pieces of metal that are spat out. That's what the military uses to fend off radar guided missiles. Very flat, reflective in the RF domain, things like overhead signs cause problems for the radar. So this is an example of a Bosch millimeter wave radar sensor that's used in the Tesla, and this was shown to be jammed well at DEF CON 24. So if you had an oscilloscope and a signal analyzer, a signal generator, and a harmonic mixer and frequency multiplier, it's about $100,000 worth of equipment, then you could jam the Tesla autopilot and cause it not to see a vehicle that was in front of it. So a very expensive attack, but it's possible. They theorized spoofing and relay attacks but did not perform them. Moving on to the inertial measurement unit and the compass, here's an example of three such systems at various price points. You think there's a wide, wide range of IMUs available from very cheap, hobbyist level ones for a few dollars all the way up to many, many thousands of dollars. These are very, very robust, or they can be very, very robust. So they are used as the primary navigation sensor for some systems, such as certain underwater vehicles or such as some military systems because of the high fidelity. If you're going to shell out for, say, an IMU from a Boeing 777, you can expect 0.1% of the distance traveled as cumulative error. So that means if you go 300 kilometers, let's say under the ice, when you pop up to the surface, you're 300 meters away from where you think you would be or within a 300 meter radius. So pretty accurate. Not much you can do in the way of attacks. Very, very difficult to interfere with. You can do physical attacks with magnetic fields and exploiting thermal drift if you have physical access to the platform. One particular interesting physical attack on a particular type of readily available IMU is the acoustic attack. This is pretty exciting. This is because those cheap MEMS IMUs actually have physical components in there that move. There's a vibrating element in the MEMS gyroscope, and it has a resonant frequency. So this is what a MEMS microelectromechanical system gyro looks like, and you can disturb that with an external acoustic source. So this is similar to attacks that have been publicized recently on hard disks, what they call the screaming in the data center attack. So you blast some noise at a hard disk and it vibrates the read head and you can cause the hard disk to not be able to read or even cause physical failure of the drive. Similarly, you can do this to a MEMS gyro. So here's an example from KICE in 2015 where they took a real multi-rotor platform and they blasted acoustic noise at it, and you can see here this is the control outputs showing that when the noise was blasted, the control went nuts because it was trying to deal with the fact that these MEMS gyros were vibrating at their resonant frequency from the noise. And so they were able to successfully crash at whatever distance you can blast that level of sound pressure, a multi-rotor craft. Next sensor, wheel odometry. These are physical encoders that measure the rotation of the wheel. So here are some that you might see on a commercial driverless vehicle at present. These are good to know what your real speed is and when the vehicle has actually stopped. And here's an example from our show, from prototype this, when we were doing our bridge crossing on one of our attempts, the vehicle took this turn too closely and it actually scraped into the barrier and it knocked its wheel encoder off. You can see it there knocked off. At that point, the car got very confused. It wasn't able to get ground truth odometry and it wasn't able to proceed. So a physical attack that knocks that off can cause stopping of the vehicle or other unpredictable behavior. So ways you could attack the wheel odometry is to change the diameter of the wheel, to make the surface slippery and to scrape them off, to force the car to go close to something and physically remove them. Another sensor that I'm including really just for completeness here is the ultrasonic sensors. These are most commonly used for automated parking. So this is stuff that happens at a very slow speed. You can attack them in all the same ways as the other remote sensors. You can jam them. So here from Defcon 24 sending out ultrasonic noise to prevent them from getting a reading. You can send fake pulses back to make it look like something is close when it's not really there. And you can also do an interesting thing because it's in the acoustic domain. You can do cancellation. So if you characterize the sensor, you can send an inverse pulse and cancel out the real returns that are coming from the sensor. Again, it's a low impact attack quite literally because this is only happening at parking speed. So you know if you were going to be James Bond and produce a vehicle that was going to be able to have various attacks involved against autonomous vehicles, you would have a GPS jammer of course. You'd have smoke and dust and vapor to interfere with the LiDAR beams, lightweight decoy obstacles, radar chaff, transparent things to drop down and puncture the tires and of course no bond vehicle is complete without an oil slick and it works a little differently in this case. You're following the wheel odometry. And then based on more recent research you have your active LiDAR jammer and spoofer, active LiDAR jammer, your acoustic blaster and of course your fake lane markings and your adversarial total dispenser. So I'm just going to do a quick time check here and see who I am on time. All right, so I'm going to move pretty quickly through these next things just to try and get through to the last stuff. But the map is very, very important. There's a great emphasis these days on pre-acquired map data and that's primarily because some of the big developers of autonomous vehicles like Google, they already had a lot of mapping data. And so here you can see a LiDAR-acquired point cloud. You can see how detailed your map can be. So detailed in fact that it's often considered by designers to be the truth. And that's because it's very, very useful. If you know where everything is in the world you don't have to do as much work on recognition. Things that are hard to spot like traffic lights like vegetation. You can just know where they are, speed bumps for example. You don't need to see them if you know exactly where they are and if you know exactly when you are. Here's a quick example on traffic lights. You can have a camera that's programmed to just look where you know the traffic lights are and then you can check the status of the lights easily. You don't need to detect lights, you just need to look in the right place and detect state. But that means fake traffic lights won't be detected by your system but they will be obeyed by human drivers. These are shared roads. So they create a difference between stuff that human drivers on the road are going to pay attention to and that robots won't. So if you put a fake green light, a person is going to think it's okay to go or a fake stop light, the robot is not going to see it, you can cause an accident. Vegetation is an interesting one because vegetation changes. So you might often use say a colorized light art and a transmission classified to determine if a piece of vegetation is crossable or not. But things that overhang and change and grow like vegetation, if they're not on your map and it hasn't been updated, they can cause problems. So dependence on the map may exacerbate the brittleness of your sensor discrimination algorithms. Here's an interesting little video that shows what you can do if you have a perfect map. If you have a perfect map you don't have to have any kinds of traffic lights. You can just drive straight through an intersection. If you have perfect knowledge of the world then you can make these like split second decisions that are always right but you don't always have perfect knowledge. The map has to be constantly updated. So a local map is vulnerable to unexpected real world features and a remote map is vulnerable to denial of the update process. So jamming, spoofing through man in the middle of tax and so on. So the map is an interesting vulnerability and an attacker is going to try and exploit the logic structure that your vehicle has. So the goal is to maximize the uncertainty of the system. For example, a blind right hand turn is a very brittle traffic maneuver and this is an area where you can force the vehicle to require manual assistance. Confuse and annoy the occupants. Inconvenience the other road users. If you can prevent the robot from successfully navigating a fragile maneuver. And of course the best thing to do, if you are an attacker, you have access to the map as well. You can find exactly where these fragile decision points are and cause bad behavior and maybe even cause a bad accident like this. Trapping and redirecting are a general class of attacks that attack at the collision avoidance and navigation layers, forcing the robot to postpone its high level tasks so you can use moving obstacles. You could perhaps even use robot swarms, obstacle swarms. You can use artificial traffic control features to force the robot to go somewhere that's under the attacker's control. And you can make things that are subtle. There are ways that a human driver would just feel free to ignore them. But the robot has very, very fixed rules and it can't ignore the attacking features that you're putting in its way. The other big attacks area is clobbering. So forcing the robot to crash into something, subverting the collision avoidance layer. So the goal here is to incapacitate the vehicle to strip off sensors or to damage them physically. You can do this with subtle map deviations if the robot is very dependent on the map. You can imitate vegetation that it thinks is crossable but is not. You can simulate high impact obstacles while the vehicle is at speed that will cause it to make defensive maneuvers that will cause an accident. You could disguise entrances like overhangs or gateways that are within the GPS noise level so the vehicle might scrape its side removing those encoders for wheel odometry or so on and then placing dynamic obstacles underneath large signs to fool the radar. So all of that said, there's some debate about whether we'll see level four and five autonomous vehicles in widespread use on the road anytime soon. This is a funny quote from 2014 talking about how 99% of the country is areas where the autonomous vehicles cannot drive. So another area that I just wanted to cover very briefly at the end here is the recent announcement of the rollout of standards for vehicle to vehicle and vehicle to infrastructure communications. This is from the US Department of Transportation. These are little radio messages that cause transmit between each other and between infrastructure that warn them of certain safety hazard things. So the early V2V components are going to be rear end collision warning, lane change warning and intersection warning scenarios. Right now, just warnings. No active control of the vehicle is proposed at this stage. So it uses these radio messages both on board and to roadside communicators. The protocol is called DSRC, Dedicated Short Range Communications. These are omnidirectional 200 to 500 byte packets and the protocol is called the BSM, Basic Safety Message. These packets are not encrypted but they are authenticated via certificates. So standard PKI issues apply. These are what the packets look like. Part one is the core that's transmitted with every packet. Part two is appended whenever it's changed and there's a lot of vehicle specific information in part two. GPS is transmitted unencrypted. So if you can receive these messages from a vehicle, you can tell where it thinks that it is. So if you're sending out your GPS spoofing, you can actually get feedback to make sure that the vehicle actually is being fooled. Here's a breakdown of the security model. I want you to pay special attention to the stuff that's initial deployment versus full deployment. So those dashed lines are the initial deployment. Most of this is slated for later on. The initial deployment is very small. A lot of functions are split for security and privacy reasons. Privacy has been a big component of the development here. The certificates come pre-installed so your car will come with thousands of certificates and they're valid for five minutes each. There's a location obscura. So privacy again, very important here. The bottom line on V2V, the rollout is very careful. It was 11 years in development. They're estimating 37 years to the full fleet. Okay, so that's a long time. Compare that with all the hype about autonomous vehicles and like, oh, we're going to have robot taxis by 2022, maybe in some limited cases. But just for this short-range communications protocol, 37 years to full fleet rollout and tracking and privacy are more immediate concerns than other malicious attacks. The standard PKI concerns apply and remember, certificates are going to be very, very easy to get. You're going to have thousands of them in every vehicle, but five minutes is a long time on the road and there's no direct control even proposed. So actually the driverless vehicles may be with us before we have V2V direct control if it's going to take us 37 years just for warnings. But hopefully they won't make the same mistakes that were made in other traffic and infrastructure sensors. So here's some stuff from DEF CON 22 from IOActive. They showed that these traffic sensors that were sending data into traffic control systems had no encryption or authentication. So the messages were clear text wireless transmissions that were fully spoofable and firmware updates were neither encrypted nor signed. So hopefully they won't make these particular specific mistakes, but no doubt they will make others and we as hackers have a duty to find out what they are and notify so that things can be changed and made safe. So here's just a quick example, very recent research. This was released by Georgia Tech in March, I think. They just looked at what are the results if we had widespread driverless vehicles that were hackable. And so this is looking at Manhattan Island, New York and just looking at what happened in terms of access to the grid network. If we had a lot of vehicles because we need emergency vehicles to get through. We need ambulances and fire and so on. We want our roads to have full access. But hacking just 10% of the vehicles at rush hour, you can see they used a percolation theory to develop the simulation. And you can see a large chunk of the island is now completely inaccessible to any vehicle just by being able to do denial of service attacks on about 10% of the rush hour vehicles. 10 hacked vehicles per kilometer per lane. And then doubling that to 20% and we can see that the island is basically unnavigatable. So that's really, really serious. So this is why we need to take this problem seriously and why we need to do research here and make sure that we're putting these IoT systems out into the world in a way that's going to keep people safe. So that's it. In summary, I will just say I want you to remember that driverless vehicles are cool. We're in favor of driverless vehicles. We want all the advantages that they offer. We're not trying to stop this in any way. Don't do any of these things in the real world. Only do it in a research context. Don't hassle the half or rather don't hacksaw the bots. And hopefully we can bring the world together into the place where we're living the driverless dream where we can be doing our romance in the backseat of the car while the robot does the driving. Thank you very much.