 Hello, and welcome to my talk, Remote Adversarial Tentum Attacks Against Tesla and Mobileye. Now I will start by introducing myself. My name is Ben. I'm a computer scientist and a former Google employee. I'm a PhD student at Bangalore University of Venegab and a researcher at Cyber at BGU. My research focus our privacy, which includes side channel attacks, cover channel attacks and tempest attacks. And I also focus on security of IoT devices, mainly on drones and advanced driver assistance systems. Now you can read more about my research at my website. This talk is based on three papers that were published during the last two years. One of which was accepted to ACM CCS 2020. The other two were accepted to AutoSec 21 and you can find them online. Now this is the agenda for today. We will start by discussing adversarial AI attacks. We will then discuss about object detectors and we will move on to discuss about split second phantom attacks. Finally, we see demonstrations of data and we will end with some interesting conclusions about data. Okay, so let's start to discuss about adversarial AI attacks. And in our context, adversarial AI attacks are physical objects that are perceived differently by AI than by humans. It was already demonstrated by previous studies that object detector misclassified a detected object if attackers add a physical artifact to the objects. Now, for example, the physical object can be stickers, it can be graphically, it can be even printed blood colors and all of which cause object detector to misclassify the road signs. For example, the stop sign as, for example, a speed limit or to misclassify this specific speed limit as another type of speed limit. And this was all done again by adding some physical artifact, either stickers, graphically or even blood colors. Now the area of adversarial AI is actually exists for, I would say, about six, seven years, something like this. And many works have been published in this area. And I would say that on a monthly basis, we continuously hear about new type of adversarial AI attacks that demonstrated by guys from academics. And an immediate question that arises is, why haven't adversarial attacks been observed in the wild, given that we are continuously hear about them in the last few years. Now, adversarial AI attacks have many disadvantages and there are the reasons why these attacks weren't demonstrated in the wild. For example, one major disadvantage is the fact that these types of attacks require the attacker to approach the attack scene. Now, from an attacker's perspective, this is actually something that he would like, he or she would like to avoid, because they would like to stay anonymous and not approach the attack scene. Another disadvantage is the fact that some of the attacks can only be applied by skilled attackers. Some of the attacks require to train an adversarial network, the adversarial neural network, in order to create or to produce the adversarial instance. And bear in mind that the set of skills that require to create or to train the adversarial network to produce the adversarial instance is somehow very different from the set of skills that the average or the standard hacker basically holds. Also, some of the attacks demand full knowledge of the attack model. In our language, they call the white box, white box attacks, they require the attackers to extract the algorithm that is used by the AI in order to attack it. And as a result, they have somehow very tough preliminary stage in order to start at the beginning to start the attack. Also, some of the attacks leave forensic evidence at the attack scene, bear in mind that maybe the gravity, the way that the gravity was painted or the stickers or the way that the sticker was printed may even help investigators find the attacker. So they have some disadvantage in terms of the forensic evidence that is left in the scene. And also some of the attacks require complicated preparation, bear in mind that training a neural network is what in my definition is a complicated preparation because it takes a lot of time. It may takes a lot of resources in order to do so. And this is something that may be a limitation for some of the attacks. Okay, so we understand what disadvantages adversarial AI attacks have. Let's try to understand how object detectors, which are the target of the adversarial AI attacks work. So first of all, object detectors are algorithms, which are used to identify the location of instances of objects of a certain class. Now, for example, this can be pedestrians, this can be road signs, this can be even cars and object detectors receive frames as input and output a set of bounding boxes with the classifications of associated objects. Now here you can see, for example, in the left side, a pedestrian detector, you can see a bounding box across the pedestrian. In the middle you can see a car detector, you can see how each and every car is actually have bounding box around it. And also in the right side you can see road sign detector, which you can see how each and every road sign have a bounding box across it. And this is what object detectors intended to do. Now, object detection starts with a video stream, basically frames, and each and every frame is being received as an input to the object detector. The object detector, now the object detector actually find the boundaries of objects, basically boundaries of objects are detected and suppressed. And when I mean suppressed, it means that duplications of the same objects that are being identified multiple times are suppressed to a single detection. And at the end of this stage, we will have this nice picture, basically, you can see how both pedestrians are being detected by the object detector. However, there is also an additional component, which I call a persistency function. A persistency function is actually an interesting component that is being added to real time object detectors. And a persistency function, mainly used to decrease false positive detection, by verifying the existence of a detected object in a few consecutive frames, which corresponds to an appearance pressure. Now, a persistency function works as follows. It receives an object and accepts the objects, if the appearance duration of the object is greater than a preliminary redefined appearance pressure. Otherwise, it rejects the objects and consider it as a false positive. Now, a persistency function have one constraint, because you can understand that by using a higher appearance threshold, it will result in a lower false positive rate to the object detectors. It means that an object detector with a higher appearance threshold will yield a lower false positive rate, which is good for the overall system. So in terms of real time driving, we have an additional constraint, which requires that objects, for example, the objects can be pedestrian, need to be detected as quickly as possible. So the car can immediately react to the detected object. So on the one hand, you have a constraint which requires, which you want to basically set a higher appearance threshold. And on the other hand, you have a constraint which requires where you want to set a lower appearance threshold. And as a result, you have a trader point where the appearance threshold is set by the manufacturer by taking both constraints into account. Now the appearance threshold must decrease the false positive rate. Basically, it needs to be the sweet spot that decreases the false positive rate, but also should be short enough to detect objects. And in our case, it should be milliseconds for driving cars in order to detect the object and in order for the car to react to them as quickly as possible. Now, as a result, the appearance threshold actually creates very interesting, I would say, reality where objects, only objects that appear for a period of time, greater than the appearance threshold are considered by the existence function is real objects. Otherwise, they are completely undetected to the, to the persistency function and to the object detector as well. So here is a question which will introduce us the split second, the motivation for the split second phantom attack. Can we apply a split second phantom attack that appears in the detected area, but still imperceptible to the human eye. So that actually takes us to the next section, which is the split second phantom attacks. Now, let's try first of all to understand what split second phantom attacks are and let's discuss about the threat method. Now split second phantom attacks are basically attacks, where the attacker presents a phantom object to a car for split second, and this object actually triggers an undesired reaction from the car. This can be done by attackers, by for example hacking a digital connected billboard and embedding a phantom object into an existing advertisement. This is DEF CON and basically I think that some of your familiar with the attacks that were already demonstrated in DEF CON. How easy this type of billboards can be hacked. And you can find them there's an interesting lecture talk a few years ago that was demonstrated by that was demonstrated by a hacker that investigated what exactly need to be done in order to, for example, attacking internet connected the digital billboard. However, there is another way to apply the attack. This is by projecting a phantom object from a drone. Okay, in both cases, the phantom is appears only for split second, which should be somewhere in the undetected area, or the human but detected for the car. Now let's try to understand the significance of split second phantom attacks with respect to previous adversarial attacks. So, as opposed to the adversarial attacks that require the attacker to approach the attacks in split second phantom attacks can be applied remotely either by hacking the digital billboard or using the drone. Also, as opposed to the previous adversarial attacks that rely on skilled attackers, which may be the data science data science capabilities split second phantom attacks do not require any special expertise, as I will show you a few slides. Also, as I mentioned before, some of the adversarial attacks are white books attacks, which means that they require the algorithm in order to apply the attack the original algorithm is being attacked. Split second phantom attacks on the other hand, do not rely on white box approach they completely with that box. As opposed to the previous adversarial attacks split second phantom attacks do not leave any forensic evidence at the taxing at all. They do not use any stickers or graphic your things like this that can help investigators to find or to locate the attacker. So, as opposed to the previous adversarial attacks that require complicated preparation split second phantom attacks do not require any complex preparation at all. Again, I will discuss about it in a few slides. And also, as opposed to the previous adversarial attack that exploit the space domain split second phantom attacks exploit the time domain. This is somehow very interesting property to the expert. Now, some of you probably wonder, why does a commercial advanced driving assistant system detect the phantom as a real object. And here is an experiment that we did. We took a picture of a road sign and projected the picture on a tree. This is an interesting fact. Mobile I 600 and Freddie considers the projected road sign a real speed limit. And this is despite the fact this is completely. As you can see, project a projection that basically was projected on a tree. And probably I'm not that surprised by this result because mobilizing standard and Freddie consists only of a video camera. So doesn't have any other type of sensor which can be used to maybe identify an object. However, here is another interesting experiment that we did. We took the picture of Mr Elon Musk and projected it on a road and replaced Tesla in front of this, as you can see, projection of a pedestrian. And here is another interesting fact. This model X considers the projected pedestrian, a real pedestrian, despite the fact that this car is equipped with radar and ultrasonic sensors. Now, bear in mind radar and ultrasonic sensors are sensors that intended to detect depth. Projection completely depth less, which is being identified by a system that have the needed or the required the sensors in order to know this type of projection. Now some of you maybe again are not that surprised because I would say that pedestrians have, I would say lower cross radius section or radio signature. Then, for example, cars and as a result, a system which basically consists of video camera set of ultrasonic sensors and radar may follow the video camera instead of relying on the radar or ultrasonic sensor for the case of pedestrian detection. However, when it comes to, for example, cars, the case is somehow very different because cars have, I would say very strong radio signature that can be used to identify them. And here is another experiment that we did. We took a picture of a car, as you can see here, projected it again on a road. And as you can see, Tesla model X, consider the projected car, a real car. And despite the fact is completely deathless. And the system, which is that Tesla model X consists of the needed sensors to detect this type of object, which have much stronger radio. So again, despite the fact this is completely deathless. And the test that consists of the sensor that can, for example, be used in order to cause correlate the data that is obtained by the video camera with data that is obtained from the sense from from the radar or from the ultrasonic And despite this fact, it is still being considered this projected car, a real car. And this is somehow very interesting fact. That you will agree with me that there is a perceptual challenge in this case. There is a difference between what an advanced driving assistance system thinks it sees and what is actually there and this is actually demonstrated in these three pictures that you can see in this slide. Now, I do not consider phantoms as bugs. They are not the result of full security implementation that can be attached with writing the most secure code. For example, an SQL injection is something that can be prevented or mitigated by using, I would say by programming the code with security in mind. This is not the case for phantoms. Phantoms exploit the fundamental inability of object detectors to distinguish between a real and fake object. And as a result, we consider phantoms a scientific gap. So why are phantoms a scientific gap? So, object detectors are basically the output of a long train process. Previous studies have already discussed the implications of using the training of the training process to create artificial intelligence. They discussed about the high level of computing capabilities that are needed in order to train or to create an artificial intelligence. They also discussed about the energy consumption, the high energy consumption that the training process costs. They also discussed about the high cost that are required to train a model and artificial intelligence to create it. Finally, they also discussed about the long train time that it takes to train a model. However, it seems that the training process, despite all of these disadvantages, are able to create robust AI models that even function better than humans in real life. Now you can see this, by the way, taken from about two years ago, where Tesla claims that its autopilot is almost nine times safer than the average driver. So it seems that the train process, despite disadvantages, creates robust AI models. However, we tried to ask a different type of question. What object detectors don't live? And as you can see from the following pictures, context isn't taken into account by object detectors. You can see how unrealistic objects, despite the fact that they are completely unrealistic, are being identified as real objects. You can also see how color, in some cases, isn't taken into account. This entire road sign is made from gray use and is being identified by one of the state-of-the-art object detectors, which is then found online. Also, as you can see, in our case, texture, basically, either whether it's a transparent or skewed object isn't taken into account. And as a result, transparent and skewed objects are being identified as real objects, despite the fact that they are completely fake. And we would like to mention that object detectors are essentially feature measures. They lack the ability of humans to ignore fake objects. So let's try to understand how this fact can actually be exploited by attackers. And the first thing that needs to be determined by attackers is the appearance pressure. Now, in order to basically determine what is the appearance pressure that is needed to project or to basically present the phantom, we performed an experiment, a black box type of experiment, where we placed Tesla with Hubble-3 and Mobileye 630 in front of the wall, and we used a projector to project on the wall. And for Tesla, we projected the stop sign that you can see on the left side, and for Mobileye, we projected this nice speed limit. And we actually projected this type of phantoms for those durations. And we tested how the system actually identified them, the success rate in terms of identification or detection of this speed limit or this road signs. Now, we found that Mobileye 630 detects a phantom that appears for, I would say above at least 125 milliseconds for 100% of the time. It means that if you project a phantom for 125 milliseconds, it will be identified by Mobileye 630. Now, the case for Tesla is a bit different. Tesla, for example, detects a phantom for 100% of the time if it appears for at least 416 milliseconds. And you can see the results in the graph on the right side. Yet in mind, you can actually use a drone by mounting the projector to a drone and fly a drone, as you can see in this video demonstration, and to project, for example, in this case, a road sign on a structure that is located nearby a road and cause this specific car which equipped with Mobileye 630 to consider this road sign, a real road sign, despite the fact that it's completely a phantom. This is not what this is not the speed limit that allowed in this specific road, because by the way was taken in our university, which in this case, you can drive only up to 30 kilometers per hour. You can see how this is identified by the specific car as 90 kilometers per hour. Okay, now, here is another interesting experiment that we did. We took a picture of pedestrian and projected it on a road, and we engaged Tesla in cruise control. And cruise control was set to 18 miles per hour. And you can see how Tesla approaches the projected pedestrian detects it and immediately reduces the speed of the car from 18 miles per hour to 14 miles per hour as a result of the projected pedestrian. And again, attackers can, as a result, trigger an undesired reaction from a driving car. Now, however, in terms of this specific threat model where we need to use a drone in order to project a phantom on a road, this is what I call a semi remote type of attack, because attackers still need to be within the flying range of the drone. But bear in mind that most commercial drones, the limit is somewhere between seven and eight kilometers. So it means that ID, attackers need to be within this range in order to apply the attack against a location that should be located in this specific range. So you can understand this type of attack cannot be applied from a rival country, for example, because, again, it requires somehow the attacker to be, I would say, in the approximate approximation of up to seven or eight kilometers from the attack scene. So in order, for example, for attackers to apply the attack from a rival country, they can use the second type of the second threat model, which is exploiting the digital Internet connected billboard. And let's discuss about this type of model now. So as I mentioned before, this specific threat model is more interesting. And this is because digital billboards are located near roads. Also, they are somehow easier to attack than you might think of. And also, we will show you that attackers can actually exploit them in order to apply this attack over the Internet from an even animal company. And in order to do so, they can even do it with, instead of projecting the specific object, the specific phantom for on their advertisement, they can actually manipulate an existing advertisement by embedding the existing advertisement phantom inside of it. And in order to do so, they can find what I call a dead area, which won't be identified even by the person that maybe sits inside the driving car, for example, in the case of a semi-autonomous car. There is an algorithm for embedding a phantom in advertisement. Basically, an advertisement consists of a set of frames. And in each and every frame, we compute a local score for each and every block in a frame. And the local score is computed as follows. Key points are extracted from the frame. You can see the key points on the right side. They are marked with blue. And the score for a block B in this frame is computed based on how much a dead area the block is with respect to the key points extracted. It means that for blocks that are located that are more distant from the, this set of key points, they will receive a higher score because they are considered more a dead area inside this specific. Now, however, I discussed before about an appearance threshold that, for example, the case of Mobileye, it need to be 125 millisecond. Now, this is actually may correspond to a few consecutive frames, maybe even four frames. So we need to take the time domain into account. And in order to do so, we compute a global score for each block by computing its score with respect to the scores. To the scores in the subsequent consecutive frames. Now it means that if you have, for example, you know that Mobileye thresholds was set to 125 millisecond, which corresponds to four subsequent frames in this specific advertisement. And what you need to do is to compute the global score of each and every block with respect to its four consecutive frames. And by doing so, you can locate the best block that can be used to embed a phantom inside of it. Here's an example for an advertisement that we took, and we embedded a stop sign in order to attack Tesla on it. It's a McDonald's advertisement that we took from the internet and just to demonstrate it, we embedded the stop sign for 500 millisecond in order to attack Tesla, and Tesla have a threshold of 416 millisecond. Okay, and here is the demonstration. In this demonstration, you can see how Tesla's autopilot automatically triggers the car to stop when the car is approaching to the presented advertisement that I showed you slide before. And this advertisement and presented it on digital billboard. And when the car approaches this specific digital billboard, it immediately identifies the stop sign, which was embedded for 500 millisecond. And as a result, you can see right now on the dashboard in the left side, it triggers an immediate stop of the car. So a phantom that was added to an existing advertisement caused Tesla's autopilot to trigger an automatic stop in the middle of the road. Okay, so we have a few conclusions that I want us to discuss. And the first conclusion is that sensor attacks are a very dangerous type of attack. They are not easy to counter and they exploit the gaps between a human and machine perception. Now, the commonly used approach to counter sensor attacks, the case in where you have additional sensors is to apply a multi sensor fusion force or approach. However, in the context of semi or fully autonomous cows, this is a bad solution. And I want us to understand why this is a bad solution. Now, first of all, in some cases, multi sensor fusion is not effective at all. Now, for example, some objects can only be detected by single sensor. For example, road signs and lanes can only be detected by video camera. As a result, the existence of this specific phantom cannot be dismissed by applying multi sensor fusion. In other cases, some areas are covered only by single sensor. For example, you can see how this specific area of the Tesla, which is only covered by single sensor, won't be able to detect this type of phantom, which may appear in this specific area. Because it is only covered by one sensor, so it cannot be dismissed by applying multi sensor fusion. Now, in some cases, multi sensor fusion can be used to detect the existence of an object. Now, this is, for example, the two cases where multi sensor fusion can be used to detect the pedestrian and can be used to detect this projected cow. However, if you were a car manufacturer, would you program the car to ignore a detected object if the object was identified only by single sensor, given that you know that sensor detection accuracy may vary at adverse or ambient or weather conditions, for example, heavy rain. Now, in some cases, as a result of this adverse ambient or weather conditions, some objects might be undetected to one sensor, while still be detected to other. Now, with that in mind, how would you program the car to resolve a disagreement between the radar and the video camera regarding a detected object. Now, here is another interesting scenario. Sensor detection accuracy varies for different object is a known fact. Now, for example, it is known that video cameras are more accurate at detecting pedestrians than radars. Now, in some cases, the detection of one sensor is more reliable than other sensor. With that in mind, how would you program the car to resolve a disagreement between the radar and the video camera regarding the detected object. So, these are basically questions that I asked you, however, the industry takes into account another type of fact, which is the damage that may be posed by two, that may be posed to pedestrian drivers and nearby passengers. If a real object is ignored due to a mistake that was done by, you know, configuring the car to ignore a specific object, if it was identified only by a single sensor, this type of mistake is, I would say very risky because it's also have great impact. It may cause for an accident may cause to a loss of life. And as a result, it is better to react to a detection is a real object. If the detection confidence confidence process a predefined threshold. And this is probably the reason why the Tesla, in this case, detects the pedestrian and the car is real. This is a real object, despite the fact that they're completely deafness and despite the fact that they do not have any validation from the data that was obtained by the radar or from the data that was obtained by the Now, in practice, phantoms are not easy to counter with multi sense of fusion and this is one of the most interesting conclusions that I wanted to take from this talk. In addition, phantoms, as you can see in this pictures have already appeared in the wild. You can see them how physical basically objects cause Tesla's to consider them as real object. For example, this is what's taken from this week where a moon was detected as a traffic light and will an advertisement was detected as pedestrian. And in the right of you can see how a booking their sign was detected as a stop sign. And again, phantoms are not a new concept. They've already appeared in the wild. However, phantom attacks are more dangerous than this type of phantoms, because they can be applied remotely. And the other nut is easy to count them. So I would like to thank you for attending this talk. Thank you very much. And this is it.