 Good afternoon. I think it's time for me to get started. My talk, actually I'm going to, me and my colleague are going to give two talks on automotive security. One is on the canvas, canvas anomaly detection and another one is on the immobilized security which we are including demo and then we will open, we will try to hack the immobilizer on the site. So let me introduce my team first. My team is from the largest security company of China called Chiku 360. We have like over 7000 employees. So it's a huge company and we are also, our team specialized in the wireless security including the GPS, V2X security and anything that use video we are all interested in. So we have made some wireless hacking gadgets you can use. If you are interested in you can drop by our vendor area to say that what we have brought here. So let's get started. The outline of my talk would be like a quick recap of the status quo of connected vehicle security research. I would like 20 pages of power point to explain the current status quo. And then I will talk a little bit about automobile working principle because I had an automotive background so I want to share with you how a typical automobile work and finally I will get to the course, the meat of this talk, canvas anomaly detection which I will explain later. So this is a timeline of the car hacking development, car security development. It all started from like performance tuning. The hackers or the car tried to modify the firmware in order to like increase the performance of their cars. Maybe for example, if the car manufacturers have like a speed limit program inside the controller of the automobile, they might change that. And then immobilizer hacking which is with the purpose like they want to steal the car. And then it started from started to like remotely hacking the cars physically so you can remotely control the car too. That's kind of like a qualitative change because it started, car hacking had started to threat the driver's life. So these are the famous car hacking instance examples that have happened during the last few years. For example, the Tesla hack which is done by my colleagues and the BMW, the Jeep hack last year released by Chris Walasek and Charlie Miner. So that's why we should pay more attention to the automobile security. So this is a talk that Charlie Miner just gave just a few days ago. He talked about how to inject canned packets to control the physical features of modern automotive. My talk is targeting this kind of attack. I'm trying to detect anomalies by building a mathematical model that can detect this kind of attack like the parameters proofing and the packet injection. So this is how the Jeep hack, how they can remote control of the vehicle. You can see it affects a lot of cars because it's internet wide. A modern car, especially the ongoing development of the autonomous vehicles, they are facing various threats. For example, the sensor security that my colleague is going to talk about tomorrow, the LiDAR ultrasonic sensor and the camera that are all susceptible to the hackers. So this is last year's research by those researchers that they have done some experience on the autonomous vehicle sensors. And the GPS proofing also done by my team mate that we cannot hijack the car's location capability. So a little bit about the automotive working principle. How does a typical car work? I will explain only the aspects that only related to my research. A typical car has a transmission, an engine and the steering system breaking. So the engine output is powered via the transmission and the transmission controls the gear ratio so the car can vary its speed. So what exactly, why we only talk about the car hacking recently because the car has changed a lot. For example, this is a drive-by-wire system and the car has changed from like using the physical connection between the acceleration pedal to the throttle body. It was a mechanic connection but nowadays they have introduced like a drive-by-wire system which they use several motors to control the throttle body and they use the computer to control that and the throttle acceleration pedal now is just a sensor. So under the ECU, ECM, the electronic throttle control module is connected via canvas and on the canvas there is typically another module called infotainment system. The infotainment system is connected to the internet so that's what's introduced the danger, the hackable. So you can see the drive-by-wire systems, the acceleration pedal, the signal is gathered by the electronic throttle control module and the throttle control module outputs the signal to several that controls the throttle. And the engine control unit is connected to the throttle control module so it's dangerous if a hacker can gain control of the canvas that is used to network those ECUs and the steering by-wire system is also the same. Used to be a mechanic connection so you cannot hack the steering but nowadays you can see that there is a clutch and they're also controlled by ECUs so it's very dangerous. So automotive control system actually usually a typical automotive takes as its input the environment, environmental variables, driving conditions, vehicle variables, user variables, user variables, for example the driver's input, the gear, the acceleration pedal so that's the user input and environmental variables are like the sensory input and the location etc. And it's based on this input the vehicle controls the engine and transmissions to change those input. It's like a closed loop control system. So this is a typical vehicle communication system that consists of multiple protocols like lean and drive. Drive can, there is multiple canvas, diagnostic can and there is infotainment system which often use like most protocol. So actually I will, in my work I have to acquire the can traffic so I got my can traffic from the engine can and the drive can. So let me take the speed as an example to see how the system is susceptible to hacking. The speed is produced by the electronic, for example the electronic stability program they will use speed as its input. The engine management also needs speed to control the throttle so many ECUs on the canvas are using the speed as its input and the transmission need to use the speed to make a decision on how to change the gears and the adaptive cruise control of course it also needs the speed because you have to keep maintaining the speed. And then the initial navigation system they have to use the speed as well to calculate the location. Okay, this is canvas signaling. You should have two wires and it's high canal and it's differential signal is using differential signal. So this is not closely related to my research so I just skip those but you can see that the can signal is not like this. It's a differential signal so it's not susceptible to the common mode interference. So this is a can typical can frame structure. Can packets usually only have a can ID. It doesn't have a source or destination address so you cannot use like this. You cannot block the packets or filter the packets based on the typical computer firewall does. So you can only make your decision based on the contents that it carries. So this is a canvas access arbitration process. A canvas, all the nodes on the canvas can send packets with any ID value. So how and there are all connected to the canvas like this. So if two nodes on the network try to send a packet at the same time so which can, which node gets to transmit the data to send the data. It's determined by the arbitration process here. So packets with can ID value is lesser than the value of another can packets gets to send. So you can see that the lower the can ID value the higher the privilege of that packet. So the difficulties of canvas defense as you can see from the network architecture here if you detect a malicious, for example you detect a load A is sending is transmitting malicious packets. How can you, how could you block that, how could you defend that kind of dark. You cut off the communication of the load A. For example if the load A controls the engine if you block the communication of that module the car will be more function. So it's hard to like simply block the communication. And another problem is that the can, it's very hard to trace back to the senders. So as I just mentioned the car, the can bus only have a can ID. So if you detect the malicious packet you cannot even decide which ECU transmitted that packet. So that's a, that's a problem too. And the third problem is that high cost of positive I just mentioned that if you just simply block communication between certain ECU and the other ECU so you will cause more than likely to cause problem. And all the packets on the can bus they are very often they are, they are real time parameters for example the current speed, the current RAPM. So the, so those parameters are required has time constraints. So you cannot like so block the, like, like do the, do a filter for example you, you, you, the decision you, you collect this, you, you receive a packet. You do some calculation on it to see if it is malicious. And then you, you send that packet on the network, onto the network. That work would introduce delay. And it's not, it's not very good. So can bus attack usually includes this two kind, two kind of attack packets injection. For example you, an attacker, an attacker can, can observe on the, the traffic on the can bus to see what packets contains what features of the car. So and then he replays that packets or in, or modified packets to do some injection. So this kind of attack is very often. And the parameters proofing for example in the G-PAC the transmitter conducted that in order to control the, like the steering, steering at high speed you might, you have to like spoof the speed to make this car thinking that it's traveling at low speed because the car won't allow you to control the steering when the car is traveling at above five, five miles per hour. So it, you have to spoof the parameters. So, so that's, okay. So related research of car security, the SAE has also published the separate circuit guidebook for, for car security. And, and there's a national highway and transmission, transmission department they also published the, the, the white paper that stated that it's the, the need for intrusion detection solutions. So that's what my research is about. Now let's get into the meat of this talk. This talk is talking about like to detect the parameter injection and spoofing attack. So I have mentioned that the kind of network architecture is like they have two wires. One is kind of high and another one is high, high low. And these two wires is connected next to this. So how can you defend, defend against the, the attacks? The first, but not practical method is to introduce a gateway between every ECU and the canvas. But this requires very many changes to the, very, very, you can change the basic kind of network architecture and it's, so it's hard to implement. So I, I propose this architecture to defend, to defend against car, car hacking. So this is the original canvas, not modified. You just, you just add an IDS intrusion detection system to, on the canvas to detect the attacks. And then you use the, when an attack is detected, you, you, you control the wireless gateway that, for example the infotainment module that has internet, which has an internet connection. When you detect an attack, you can, you like control the wireless, control the serial connection or other kind of connection like Wi-Fi or Bluetooth. You just control those connections to prevent the hacker from further getting, getting control of your system. So you, you can block the, and fire up an app. So how to detect the, the, this kind of a, those attacks, those attacks I, I, I for mentioned. So the, the, the can manufacturers come up with like the intro fee based anomaly detection, which is, which is used the, the, or the intro fee on the network to detect an attack. They, they assume that if a, can network is, is, experience, or experience some kind of attack, the intro fee of that, of those traffic will, will change. But the, and the, you can see there, there is out there, like they injected four, four packets and they, they detected three of those. So what's the, so why those method is not very good, are not very good because, because they are not considering temporal features. For example, the speed is like a continuous variable. So if, if those next, the intro fee based anomaly detection, they cannot detect, they are not considering these kind of features. For example, the speed at time t is like 60 kilometers per hour. And t, t plus one, the speed suddenly changed to like 100 kilometers per hour. So that's, that kind of anomaly they might not be able to detect. And so my method can take into consideration is the, the relationship between one parameter, certain parameters to the other parameters. For example, the speed is related to the engine rotational speed and is related to the current gear, the transmission is, and is related to other, like the acceleration pedal, like the engine intake, the air speed, the, the air mass flow. So all those parameters are related. My method is to exploit those, the relationship to detect the attacks. For example, you, you, if the attacker want to, if the real speed, current speed is like 10 kilometers per hour, and the attacker tried to spoof the speed to show, he want to fall in the other, the other issue into thinking that the speed is like 60 miles, 60 kilometers per hour. The relationship between the RPM and other parameters will change. So for example, I have explained that the relationship can be used to detect this kind of attack, but how to implement it? For example, like this equation, just a linear equation of RPM multiplied by gear and multiplied by another constant, is the speed, is it, is it, is it, is it that simple? But it's not that simple, right? So you have this, this is like a typical transmission and it has many clutches. So when you have clutches, much you introduce nonlinearity. So the, the equation I've just shown is not practical. You cannot use a simple, mathematical model to describe the relationship between those, those two parameters. So, so this is, you can tell from the speed, this is a graph that I grabbed from, directly from the internet, that you can see the speed and the, the speed and the, and the gear ratio. They are not even like that linear, the relationship between, between them is not linear. And you can tell that the speed is also, it's a continuous variable. So these features are, you need to consider, we take consideration, take into consideration those, those features to detect the attack. So this is the real data that I, I received from the, my experiment, the car, the, my, my car, my experiment car. And you can see there, I, I, these are the parameters relationship. You can see the purple one is the acceleration pedal, the, the gas pedal. So you can see that when the gas pedal goes off, the, the general trend of other, other parameters also goes off, but you cannot use it as simple function or to like describe this relationship. So how to do, do this. And, and now I know that all the parameters are related. I can use the other parameters of current value to, to see that if another parameter is anomalous, anomalous or not, but how to implement that. So let's, let's assume that we already had, had like a mathematical model to describe the relationship of those parameters. So how could we use that, uh, mathematical model to detect attack? You, you, you'll receive the, the live data stream from the CAN bus. The CAN bus, uh, like for example, the speed gear, RPM and the acceleration pedal. And you use this par, these parameters as input to the mathematical model you have just built. And, uh, uh, and use, for example, you know the current gear, current RPM and acceleration pedal position. You use those parameters to, uh, to as input of, uh, of the system model, the mathematical model we just built. And you make a calculation, uh, you, you can say it's prediction of another parameter, for example, the speed. So, and the, the, the speed you received on the network, if it's deviate very much, too much from the speed you just calculated based on the other parameters, you, you can say that's an anomaly, that's an attack. So the basic idea is like this, to increase the accuracy, to increase, uh, uh, um, to, to reduce the error or force positive. You could use like, for example, the, uh, cross, cross validation you use. For example, the attacker is currently spoofing the speed. You use other parameters to, uh, calculate the, the, the speed. And you, this is one, one, uh, one result you can get. And, uh, and you use that speed, whether it is, uh, be the real speed or the spoofed speed, you use that speed, uh, other parameters to ca, to calculate the, the current gear. So, uh, it goes on like, and, and then you use, uh, uh, you, you calculate RPM based on the other, the rest of the parameters and then use that to do like a pulling. So, finally you get a high accurate, high, very accurate anomaly detection model. And now let's get in, uh, to talk about, let me, uh, introduce how to build this system model. Uh, the method I employed is like deep learning. Deep learning is just, uh, although there are so many, so much, many, or hype about them, deep learning, but deep learning is basically just a, a, a method to build a mathematical model. So that's, uh, how the machine learning or deep learning work. So we, we, we now want as, uh, to build a mathematical model, but no, so we, the deep learning is using like a neural network. The neural network, you can, you can just say that it's, uh, it's, it's just a like universal function approximator. You could say that it's just a template of the, uh, mathematical model. So you have to use the live data stream from the car to train the model. The training process is like you, you give the, uh, um, this, this model some example and you, uh, you, the machine learning technique is used to automatically figure out the relationship of the, the, the parameters. So, uh, I don't know if anybody can have questions here. Uh, and I haven't, uh, deployed this, but you could do that. For example, uh, when you deploy the system on a, on a real car, you, you can, but you, you know, the car wear out all the parts, the relationship between those, relationship between those parts will change. So the relationship between the parameters will change. For example, yeah, a new, a new car, you press the X-ray paddle pedal slightly, the car will travel, travel very, very fast for an older car. It's not the case, right? So it's, so you could use, uh, so continuous, the, continuously updating the model is the best way, right? So, so you could deploy the system like this. The, after the training, this, this network is just a bunch of weights and the network configuration. So this parameters you can, like for example, uh, you, you could use a server so all the cars can grab the trend, trend, already trained model and deploy it on the car to detect, detect the anomalies. Um, another case is that if the car, uh, has, has wear out, the, the relationship has changed, you may retrain those model using this specific, specific parameters, the specific parameter of that specific car. So does that make sense? Okay, thank you. Okay, let me, uh, go on. So, uh, you can see this is just like a template of the mathematical model. All the, uh, let me use here. So this is a specific neural that in this network, you can see that, uh, there is the input x, x one to x n. Those input, uh, as a, as a parameters I just mentioned, you use these parameters and, uh, uh, I'll, I will introduce the, the input out of, under the input vector and the output vector of my system, uh, uh, first, uh, I'm wondering how, how many of you had a, a knowledge about the deep learning or machine learning? Okay. Oh, great. So, uh, you use the normal data of the car to train the model. So, uh, after trend, uh, after trend, the model can wear, uh, let me introduce the training process. Uh, this is a neural, you give it, uh, some training data set, some example to make the new neural network to figure out the relationship between the input and output. For example, I, I don't, I don't know the current, uh, uh, the previously mentioned that, uh, uh, the function is just a simple mathematical model. For example, the C, the constant is not initialized. You don't know what the constant is. So you, you give the some, some example. For example, there is, you can imagine that there, there is a curve, uh, on the x, y plane. The curve, uh, there, there, the, the part, the mathematical model can be used to describe that curve, but you don't know the mathematical model yet. So I, I gave you a bunch of points on the, uh, on the curve. And you, based on those curve, uh, those points, those, those, those x, y points, you can figure out what the, what the curve, the mathematical model that describes that curve is. Uh, I, I just, uh, uh, make, uh, take, take the example to explain what the machine learning training process is like. It's just like you give it, uh, uh, there's a curve, you give a bunch of points on the curve and you'll, based on the points, you'll figure out what's the mathematical model that describes that curve. So that's the training process. It's, it's, it's not exactly describing, uh, description of that curve because, uh, you have errors, but it's approximate, uh, approximate need, uh, uh, approximation of that curve, the curve of the mathematical model. So this is very important that has anybody understand this? Uh, no, you're training the, you're training the model using the, the, the normal data. Yeah, we'll accept the anomalous, anomalous traffic as well. Uh, so let me go on. So if you have any question you could ask me later. So you can see that, uh, uh, this is a training process. You can see you, you use the examples to, uh, uh, train the model. It's, it's like a first thing and you use the first example and you, uh, output, the, the output is zero. You can see that there's a threshold if it's larger than, uh, zero point five, you give it, uh, one. If it's lesser than zero point five, you give it a zero. So the first, the first example, you had this three input, you got zero output, but there's, there is, uh, zero point eight. So you, you actually get the output is one. So you have to modify the weights. The, the, the hammer is like the modifying the weights of the network. And then you'll get that using another example, the training data. You can get this, this resulted here is right. So I just skip a little bit of this. And if you have an expression, uh, this is the experience card that I used. It's, uh, has many cyber physical features. You can, it's a hybrid vehicle. It was used battery and gasoline and it's, uh, it has electronic brake, uh, steering and the electronic throttle. So you can, and they have rich internet, uh, based features. For example, you can use a cellular connection. You can use their cloud service. You can use, uh, a mobile phone to remotely open the car for your friends. And you can use Bluetooth KVC. You can use, uh, uh, phone as the car's key fob. And it's network architecture is like this. So I mentioned, uh, previously that I acquired, acquired the data from the engine canvas. So the, the, the, the, this is exactly the same thing. Why I'm, uh, I would like acquire the data from the engine canvas because the engine canvas has all the parameters I just mentioned. Those related parameters are just mentioned. So I acquired the data from there because they're the gateway. So, uh, not all the parameters are transmitted on all the canvas. Otherwise it won't be, uh, lead for the gateway. Uh, so, uh, the canvas, uh, they can't data, they can't package. Uh, what can, can't, can't package with what ID, uh, is, uh, carrying what kind of parameters is kept highly confidential by the manufacturers. You have to do some reverse engineer to understand what's, uh, what's in the payload of those packets. So this is like the result of my reverse engineering, uh, work. And you can see there are many parameters. For example, if you turn, uh, uh, single nights, there will be a packet on the canvas. And these parameters, uh, after your reverse engineering, you get a canned database that's highly confidential. Okay, this is an example just to show you how, how to, uh, how those, uh, reverse engineered, uh, signals look like. You can see that the, the canned payload is actually eight, typically eight bytes. So what, uh, the signals like the right turn signal is, uh, only like, uh, one byte. So that one, one bit, one bit, that one bit indicated if, if you are the right turn or left turn. So that's the signal. And, uh, so the whole process of building the mathematical model is like this. You acquire the data from the canvas and then you'll do some deep process, prep process on the data and then you analyze the data, the reverse engineer. And then you'll select the features that are related to each other. And uh, you're training the model and then finally you're testing. So this is the procedure. So this is data acquisition from the engine can. I use the computer and the interface device to acquire the data. This is, uh, reverse engineered, uh, that I, uh, so I, I, I masked this to avoid the, the some, the problem that my, the manufacturers might, you know, enforce me. Uh, uh, this is a raw traffic, uh, that the can package capture tool called the bus master is open source. You could try to try this at home. And uh, so the data, uh, at, at different scales. For example, the RPM might range from zero or ranging, might range from zero to like 1,000 the RPM. You have to do the conversion to get the real RPM and the speed also like this. So you have to, before you use the data to train, train, train the model, you have to normalize the data to convert to make them ranging from zero to one. So that's normalization and interpolation. Our experience is I use this simple, uh, um, um, simple like equation to convert to, to normalize the, the, the input and the interpolation because, uh, I, I just mentioned that the can package have different privileges. For example, the, the, the package with lower KID value has highest privilege. So it gets to send more often. Yes. It's often that's part of it. Those parameters are important. They, they are transmitting at high, higher frequency and those and, and other parameters may, may, may have lower privilege so that you can see that, uh, uh, you can see that when mini-map package may transmit like twice once per second, but other parameters might appear only like once per second. So you have to interpolate to do, derive the, the, the, the, the value of that parameters. Uh, uh, as those, uh, when, when the parameters are not transmitted on the network. So you have to interpolate. I'll stop sampling. After you interpolation certain parameters are not, you will change very, very, very slowly. You can see this, the parameters, uh, that, that, that as the most left side, uh, column is, uh, is, uh, uh, time in, in microseconds. So you can see the parameters, uh, the parameters, the intake pressure, uh, is not changing that very fast. So you, you could go to some sub sampling to, you know, to make the, uh, trend in set, trend in data set smaller. Okay. Uh, after the sub sampling is like this is change. You can see the changes of the, okay. Uh, this is very important, very important. How do you, uh, uh, get the input vector and output of, of vector of those parameters? Um, you, you can see that, uh, because, uh, most of the machine learning techniques, they are supervised learning. And now I only have the, uh, normal data of the car. I don't have the attack traffic on the car. How do I try this model? So I, I used this. I used, uh, because I mentioned this, this, these parameters are all, uh, continuously variable. So I used the, uh, the value of those variables, those parameters, uh, the, the value of those parameters that form like time, uh, use the previous values to predict the values of, uh, uh, and use like, uh, how to explain this. I used like, uh, the value of those parameters from like time minus, uh, 10 to time t minus 1. I used those values of, between the, that time to predict the values at time t. Does that make sense? So, okay. I used, uh, so, you can see that I used like, um, seven parameters, seven parameters, the intake pressure, the RPM, the speed. Those parameters, I used the, uh, values, values of those parameters from time to minus like, 10 to, uh, to, to time minus 1. And to use those values to predict the value at time t. Uh, okay. This is how I, uh, how the input vector and output vector of the neural network is formed. How this, uh, I don't know if everybody can, can see this code. Uh, actually this code is used, uh, is, I used the open, open source, um, deep learning, uh, uh, software called Keras. This software is very simple to use. You just, uh, preprocess this data. It's in Python. It preprocesses this data and, uh, you, you, you'll split the input vector and output vector and use those data to train the model. After train, then, then, then we show you the result. So, uh, the, the, this is the, the speed. The, the one, one of the curve is, uh, curve is the prediction, the, the, the result calculate from the mathematical model. Uh, okay, okay. Output vector, I think I, I talked about that here. And you have this, uh, you can, you can see that the only data I have here is this kind of data. Those curves, those curves, the only data I have, and these curves are normal data. I don't have the, uh, don't have the attack traffic. For example, you, you, you'll slip the traffic when the attack, attack is happening. I don't have that data. I only have normal data. Yeah. Yeah. Yeah. Yeah. And, uh, not only I think it's not that simple. You, you, because you, you take input as multiple, the, the values of multiple parameters. So, you're not based on the decision. Uh, you're not predicting the, the, the speed. Only using the speed, the past speed. So, you are also using the past values of those RPM intake pressure or, or can you understand that? Okay, great. So, for example, you, you, you use, uh, okay. You use, uh, uh, the curve. This, this section of the curve to predict a single, single point at this place. So, okay. Okay, this is the result. You can see that, uh, uh, it looks, uh, pretty, pretty good. Okay. There's the speed. You use, uh, you use, uh, all the parameters for, uh, to predict the speed. It's pretty, pretty good. And I used, uh, measure to, to measure the, the, the, used, uh, the mean square error as a measurement to, of the deviation between the predicted speed and the real speed I received on the network. Uh, because, because, actually I have plotted the, the mean square error on this graph as well because it's not on the same scale. So it, it cannot see it. So I reached, plotted it here. So you can see this, uh, this is, uh, point zero, zero four that the scale. Okay. It's much smaller. So let me try to, uh, try this on, uh, like attack, uh, uh, attack. This attack is like the RPM is, is the, the, the attacker is replaying the RPM. It's like spoofing the RPM with the original, the, the right RPM is also transmitting on the network. So you can see that the RPM when you plot it is oscillating because they are sending it the same, like the same frequency. So you can see the curve is oscillating on the top right left corner. And, uh, this is the abnormal RPM. You, if you zoom in those, uh, black and black sections, those are oscillating curves. And, uh, you can see, uh, below is the attack and above is a mean square error. You can see when there is an attack happening, you can see the prediction. The prediction is deviated very much from the actual value you're, uh, you're acquired from the network. Okay. Uh, I think this is, uh, all, all I got to have any, does anyone have a problem? Okay. No problem. Oh, the threshold. Yeah. This is, the lower the threshold, uh, the fourth more false positive you may get. Yes. How do you do it? Okay. You have to, like, uh, next stage you use the real attack and may, and there may, you may create some attack traffic and test that and then based on those test results you'll get the threshold. Or maybe you can make this threshold adjustable. That's based on, yeah, that's based on your, uh, anyone has a problem? Question? Okay. Okay. Sorry, sorry. My English is very not accurate. Oh, that's what would be a big, big problem for him. Yeah. If you just try to speak, uh, I used actually, I, this is like a prototype. It's a preliminary research. I used like seven parameters. But if you are an attacker tries to send all this kind of parameters, I think it's very hard because you only can control of one ECU on the network. You should, uh, one ECU on the network usually. So you can, canvas the speed of the canvas is, uh, not only the speed is limited, but also if you, through all this, you have to first make this kind of model in order to spoof all the parameters, right? Because, for example, if you want to spoof all the parameters, you have to build a model that describes this relationship between those parameters in order to decide what value to send, right? Yeah. Yeah. Yeah. Even, even, even if you do an attack like this, that you spoof all the parameters, I think this, this model can still detect the attack because when you spoof the traffic, you have, if you're not, you don't have this kind of model. This model, the, this, the model, because I use the data from a specific car to try the model, even if you spoof all the parameters, I think I would still be able to detect that attack. Do you remember the polling? You cannot, well, for, for example, an attacker usually don't have the capability to, like, uh, spoof the, the, the, the, those values so perfectly, I think. Playback the data. Playback the data is, is still can be, is still can be detected. You see the oscillation, the oscillation, you mean, uh, spoof, as you mean, replace the data traffic while blocking all the normal traffic. The car, I think the car would have already stopped, right? I can't differentiate between this, this, this two scenarios because, uh, if there's a mechanic finier, I think, uh, it's, it is, I think the information that, like, for example, you, you could, the system would opt to, opt for this kind of scenario as an attack, but I think it's still informative to know that. Uh, I beg your pardon, sir? Oh, you mean just like, uh, differentiate. I think your question is, uh, is very familiar with the, the, he asked. And I think, uh, with more work, you can do that. Once you detect, uh, detect an attack, you add another stage between the final decision and this detected attack. You could use, like, a database that, uh, describe in the situation you just mentioned and, you know, do, uh, add a filter to that, maybe. But I, yeah, I, it's just, um, like, sir? Oh, you mean that those parameters are not related to, like, the, those packets not, not related to other parameters. For example, a random human input, right? For example, the break, uh, you don't know, you cannot say, predict how the user are going to press down the break. That kind of attack, um, this model cannot be used to detect, detect that kind of attack. For example, if the attacker want to, like, send a packet that only gets sent when a user press certain button, button, you cannot use this to detect that kind of attack. But I think there's more, I think the, the, this model covers, uh, the most important attacking scenario already. For example, the speed, RPM, acceleration, this, this, this attack can all be detected. So, uh, next, uh, you open the, for example, you turn on, you, you send a packet to turn on the, turn, turn, turning signal. This kind of attack is, like, based on the user's random input. I, I, I assume it's random because I don't, don't have sensors to take the current, uh, environment where, where, where it is, uh, across in front. I cannot detect. So I, I assume it's random. So I cannot, cannot detect this kind of attack. Oh, I haven't considered that, but I think it's necessary to consider that, you know, because you, the system is certainly going, going to get many, like, false positives. And I, I, I said, suddenly it happened to me that the question that, that gentleman has asked, I, I think I got, uh, another, uh, had to further answer. You, you said that, uh, uh, certain parameters for, for them, the, the, the charimeters work. And uh, uh, recently the cement, cementics company has released their, their product. Those products are based on like the, uh, they also use machine learning. Uh, uh, but they use machine learning to learn the frequency of certain parameters. Uh, or the frequency of certain packets. So, uh, I think, uh, their system might be, uh, I don't think their system will show the problem you just mentioned, but I think it's, uh, uh, would, would add, uh, add capabilities to the, the system. And I, I heard that, uh, uh, some, some academic researchers also had the idea of, like, um, measure the, uh, subtle deviation, subtle, subtle error of the certain packet, because the, the ECUs, they all have the, the, the microchip. This chip, the, the, the timers, the, the, the accuracy of the timers, uh, very high. So the, the certain packets you can see that it's, uh, the periodic messages, the packets are, uh, different packet, the time, timing, what, what exhibits anomalous behaviors. Okay. Have anybody still have questions? Okay, sir. I, I, I wish, I wish I could, uh, add more parameters to the system, but, uh, a typical car only have the, the, the, the, that many parameters, uh, like related to each other. So if you are able to, like, mount all more sensors on the car to monitor in more parameters, and those parameters are also, uh, like, related to each other, I think it's what dramatically reduce the force positive rate. Uh, I beg your pardon? Okay. Okay. I don't know how much time I have to spend here on this. Okay. And the next talk is also from the research done by my, uh, by Unicorn team, my team. And, uh, oh, sorry. Uh, I think, uh, uh, there's a client man. So, okay, I am not very that, uh, machine, uh, machine learning savvy. So I asked them, I had help from those, those folks from my company. There's Professor Shui Cheng Yan from the Shihu Shui 60 Institute of Artificial Intelligence. They are very good at the deep learning. And, uh, I have, uh, uh, Dr. Min Lin from the Institute of Artificial Intelligence there, which is a student of, of Professor Shui Cheng Yan. And I also had, had, uh, from my colleague, Dr. Lin Huang from, uh, my team. So, uh, hereby, I acknowledge, uh, appreciate their help. Uh, these are the references. So if anybody who is interested in, uh, cash security research, you can, you know, reference, you create those papers to take a photo. Uh, it's, uh, I think the, the carol, uh, carol cautious, uh, research on the Chinese research are, are very good. And, uh, okay, other research. Okay. Um, so the, the, the, the, okay, fine. The, the last page. Okay. Many, many paper to read. Okay. Okay. Thank you. Thank you everybody.