 This is a quick introduction. Most of us, his specialty is in quick cyber security. That's almost the only thing he works on. So he finished his PhD in 2018 under a joint program between Shiraz University in Iran and Horizontal State University. He's now a post-doc at Horizontal State, working primary on cyber security and AI and machine learning applications in smart grid. He has published a lot of papers for post-docs, a lot of papers. So I'll let him start his presentation. This is Mustafa. Today, we will talk about cyber security of a smart grid, offensive security and defensive security. I am a post-doctoral researcher at ASU. I hold a computer science PhD, specifically working on cyber security of a smart grid for 10 years, almost 10 years. So first, we will start with the traditional power grid. As you already know, in traditional power grids, you have bulk generation delivered to the end user. And the most important feature is having one bay of electricity flow. End user has no share about their end generation. End user doesn't get end user. You have bulk generation delivered to users. So what are the most important differences between smart grid and traditional grid? And why do we call it smart grid? The smart grid, the most important factor that has changed it to be intelligent grid or future grid or smart grid is the fact that we have the power of distributed energy resources, including solar panels, wind farms, and et cetera. Why we have those? Because we want also users to have the share in generation. The main difference between traditional grid and smart grid is that the end user now has share in generation. It is not pure end user. The end user can also generate electricity, sell it. So once the end user can generate electricity and sell it, we need to have more entities compared to traditional grids, like operations, like markets, where I should sell my electricity as an end user in market. So the most important difference between traditional grid and smart grid is two-way information flow. We will talk about this. But let's also review some other features. In existing units, we have different features which have been revolutionized into a smart grid. What's one of the features that we will talk about in this presentation? It's about self-healing. A smart grid should be equipped with a feature called self-healing. We will see how this self-healing feature works on cybersecurity and how researchers in cybersecurity overlook this feature self-healing. Why cybersecurity on a smart grid is important. There are many reasons that cybersecurity on a smart grid is important because of the increased targets, increased attacks. But to cut the long story short, whoever controls the grid controls the community infrastructure. Whoever controls the grid controls the community infrastructure. Without electricity, we even can't hold this meeting. So it is a base for everyone. And that's why recently attackers and hackers have targeted many electricity companies, distribution companies, and smart grid is a very interesting target for hackers. There are some security policies and aims in cybersecurity and IT security we should follow. One is confidentiality, the second is integrity, the third is availability. We call it CIA, confidentiality, integrity, availability. These are usually well-accepted policies that we should follow in IT security. The priority is confidentiality first in IT security, integrity second, availability third. Confidentiality means that our communication should not disclose to unauthorized parties. Integrity means that the message that the hats have sent to you should not be changed or manipulated in communication line. The receiver should receive the same message sent by sender. Availability means that the service should be online and available all the time. What do you think that the priority in critical infrastructure and in the smart grid is the same as IT security? In critical infrastructure, we have the reverse priority. Availability is more important than confidentiality. Availability is more important than integrity. So the priority is reverse. It is not like IT security, saying that confidentiality, integrity, availability. In smart grid context, generally critical infrastructure context, we say that availability is important, second integrity, and third confidentiality. Let's have some examples about things. Confidentiality in smart grid, what could be the example of confidentiality attached to confidentiality? For example, consumption data, low data consumption data by users could be an example of confidentiality. No one should eavesdrop my consumption data. What could happen? For example, a robber, see, burglar, could eavesdrop for the sleep, the traffic to understand which houses are empty in the neighborhood. So they will spot the empty houses. Or more important, industrial espionage. Industrial competitors could eavesdrop the consumption data, the low data, to understand what the opponents do. Especially when they are going to announce a new product, there might be an increase in the consumption data. So the confidentiality is important, but I already told the priority. Integrity. Integrity in smart grid means that measurements should not be manipulated. The measurements that are sent by sensors and devices, meters, to control center and control center receives sensors, receive measurements from sensors to do some functions in analytics, to report some information, to make it for making decisions. Those measurements should not be changed. It means integrity. Availability. What is the service that we provide in smart grid? Electricity. It shall be always available. So it is not important to have confidentiality. It is important to have always availability. It is not important for me to actually steal consumption data while I don't have electricity. So the priority is reverse. But sometimes we do not need to target availability. For each type of category, we have a specific type of cyber ethics. But sometimes we can conduct integrity attacks, specifically I'm talking about smart grid. We can conduct integrity attacks leading to availability attacks. We will see that. So you need to have a solid understanding of smart grid and its functions to know that how you can launch integrity attacks leading to availability attacks. What are the overall general attacks that could happen in cyber security of smart grid? Cyber attacks like malware phishing that we will cover in this presentation. In other important categories in cyber attacks, in other one is vulnerability. Why I have brought this? Because usually hackers exploit vulnerabilities to conduct attack. So one might ask that vulnerability is the same as cyber attack. By vulnerability, I don't mean regular vulnerabilities. It's a question of critical impress structure. The question of advanced persistence threats. We are buying some stuff from other countries. Inviting a question of advanced persistence threats, sometimes there might be a nation, a country behind of the attack. Be like what happened in Venezuela. We buy some stuff, electricity, transformers, device, deep phone, smart cards from another country. How do you make sure that there aren't any malware embedded in those devices? You have done all things to protect your cyber smart grid, but you have bought some stuff devices that already are equipped with malwares. What is your solution for those kind of attacks? So by a vulnerability, I mean those kind of attacks. So we can classify into three groups, regular cyber attacks, malware, species attacks that we will cover in this presentation, insider attack, and also vulnerabilities. By vulnerabilities, I mean nationwide attacks. I mean attack by other countries. You should care about what you buy, where you buy, and you should also evaluate, sanitize the devices that are going to use in your grid. In this presentation, we will talk about different cyber attacks. As I mentioned, cyber attacks on energy storage, cyber attacks on distributed energy resources, cyber attacks on distribution grid, cyber attacks on transmission grid. So we will cover a wide variety of cyber attacks. But let's dive into one example. One of the example, one of the attacks that already have discussed a lot in literature of a smart grid cyber security. Estate estimation is a function in a smart grid in energy management center in control centers. It's a very essential function. Why it's essential function? Because the output of a state estimation is used by many functions in energy management system and central center. Estate estimation tries to minimize the loss of their measurements received by sensors because we don't receive very accurate measurements. We do need to minimize the loss in received measurements. And also we don't have sensors by metering devices all around the grid. It's very expensive to have sensors and meters all around the grid. So what is another task of responsibility of state estimation? To estimate the unmeasured variables. To estimate the measurements that we do not have any send maintenance over there. So a state estimation is a very important function of the control center. Why? Because the output of a state estimation is the input of many other important functions. The state estimation receives measurements, reactive power, power injection. And the output of state estimation system states called phase angles and voltage magnitudes. Since it's a very important function, it has been targeted by attackers and attackers. So it is very interesting to attack the system states because the output of state estimation which are system states are used by many other functions. But the state estimation also is equipped with a function called bad data ejector, BDD, Bad Data Detector. It drops the measurements faulty measurements. If I receive a faulty measurement, that data will drop it will eliminate it. Will not feed that measurement to the control center. Recently, I mean a couple of years ago, a cyber attack has been discussing the literature that targets the state estimation in such a way that I change the measurement that are fed into state estimators. Our input to the state estimation function. The hacker changed those measurements in such a way that the output of the state estimation which is system state will be affected negatively and attacker inject arbitrary errors to output of system states while the hacker or attacker will be able to bypass the bad data detector. The bad data detector will not raise any alarms. Why it happens? Because the error here, if you see, the attacker inject the attack vector to the normal measurement received by meters, remote terminal units, and PNUs. The attack vector is injected to normal measurements. But the attack vector needs to actually meet some conditions. What is it here? It's that could be a matrix and typology matrix. Information about the typology of grades. Attacker in FDIA, attack to the system in such a way that he will change the measurement while he will inject arbitrary errors to system states. At the same time, the residual error of the manipulated measurements computed by bad data detector will be the same as normal measurement. So nothing happens. Normal measurement residual error is the same as manipulated measurement error. But you need to meet this condition. Attack vector that you are adding to the measurements should meet this condition. What is C? These attack costs. How much do you want to change the system state? 10%, for example, whole storage magnitude is one. 10% means 1.1. 20%, 5%, 1%. The attacker controls the arbitrary errors that injected to system states. And it's matrix is typology information. So another important point, I don't need to change all the measurements to the system state. Imagine here, see here, this is IEEE 14 bus system. As an attacker, I want to inject or change the system state of bus 13. I don't need to change all measurements. I just need to find the dependent measurements, these three lines, dependent measurements to that bus and change those measurements. So a question, how can attacker just notice dependent measurements by this it's matrix? You might see that not always the attacker has informed inside information, not always the attacker has information about the quality of the grades. People later discuss about this limitation and how we can relax it. How already has been relaxed. So by attacking one bus, I don't need to change all measurements. I just need to find dependent measurements to that bus and change those measurements. For here, three lines. I need to take the measurements of three lines. What could be the impacts of FDA and smart grid? Financial loss, because right information, right decisions could lead to tripping the line, line out test to pull the details. Financial loss, lack of brownout and also since it is a question of critical infrastructure, also safety concerns. But our security mechanism that could be used in smart grid, not only to defend against the CIA, defend all types of cyber attacks happening in smart grid. In IT security, there is a concept called defense in depth. Defense in depth. The first layer is prevention. The second layer is detection. The third layer is the cover. In prevention, it is clear. We don't want to have any cyber attacks. We prevent cyber attacks. In detection, we imagine the cyber attack happened. We weren't able to protect the system, prevent the cyber attack. Cyber attack has happened. But we need to initial phases. In the initial phases, we need to take to cyber attacks before having catastrophic consequences. The final step, we say that we weren't able to prevent that from attack. We weren't able to detect the attacks. So attack has happened. How we can recover from cyber attack in smart, specifically I'm talking about smart grid. How we can recover from cyber attack because we don't have nothing named absolute security. We do not have absolute security. We are just increasing the attack costs. Whatever we do, we increase the attack costs. So this presentation, we will cover all aspects, prevention, detection and recovering. What can be done in prevention phase? What has been done? I mean, usually in regular literature review, we can use encryption, but not blindly encrypt all measurements because it's very expensive. We can spot most important measurements, most important sensors, I mean, identifying critical sensors. They're using tamper proof devices or like SSM, harbor security modules or do encryption on those lines, on those important fund security meters. The second step that is used in prevention phase is redundancy. What redundancy means? Attack, I mentioned it almost as if I mentioned it for attacking just one boss, I need to take the dependent measurement. If as a defender of a system, I increase the dependent measurements of each boss, I install more sensors on the system, on the grid. So what would happen? The attacker needs to attack more measurements, change more measurements. So it increases the attack cost. One and other ways to prevent from attacks is redundancy. The third one, which is very important, the recent topic and I will discuss about this more is moving target defense, MTD, moving target defense. In moving target defense, the defender of the system, the owner of the system, regularly change the attack service. What it means attack service, I will discuss about this. But before diving into the details of moving target defense, let's see what could be the drawbacks of prevention methods. First, they are dependent to specific assumptions about the system. The prevention methods, the papers, various projects that have been done in prevention, they assume that I have a static smart grid. There is no line contingency, there is no generation out there, there is no line out there, there is no integration of renewable energies, there is no integration of distributed energy as solar panels. They imagine we have a static network, why the smart grid is a dynamic network that can be used. To cut a long story short, they are also expensive because using encryption, using hardware secure modules, using secure sensor is not that cheap. For example, if you buy a sensor, for example, a simple sensor for $1, secure sensor will cost you $100. So it is also expensive. But let's dive into MTD, which is a very, very promising idea talking for smart grid cyber security prevention techniques. Moving target defense. As I mentioned in moving target defense as a prevention technique, attack service, the defender of the system change the attack surface of the system. What always happens in smart grid. In FDIA, for example, I told that the attacker needs to have some information about the topology of the grid. And what we call it, the Kubian matrix. What did happen? If I chain as a defender of the system, I tune the Kubian matrix of the system regularly. If you remember in FDIA, I told that attack should meet a condition. Attack vector should equal to, it's multiplied by C. Next is the Kubian matrix information about the topology of the grid. If I change this topology of the grid regularly as a defender of the system, what would happen? The attacker spent one week to attack the system. At the last step that we want to conduct the attack and finalize the attack, a day before or two days before, three days before, I have seen the topology of the grid, how? Usually what techniques is used in smart grid for FDIA? Changing line reactances of the system. The defender of the system changed line reactances of the system. What would happen? When I change the line reactances of the system, I indirectly will change the Kubian matrix, the topology of the grid. Active power, reactive power, all will be intact. So I change the topology of the grid by changing line reactances. How I change line reactances by defects. I change my reactances. I change the quality of the grid. Attacker wants to attack the system. With all, what would happen? The attacker will attack the system with all the information. So once he injects the attack vector, the normal measurement, a really dual error will not be the same as the normal measurement. And the measurement will be dropped by a bad data detector. So without using any detection method, without using any machine learning technique, the bad data detector will detect almost 99% of them, manipulated measurements. Because bad data detector has a threshold. It computes a reasonable error. If the normal manipulated measurement is less than the threshold, it will bypass. But if it is not, it will be dropped. And then the attacker attacks to the system by all the information, by all the Jacobian metrics that I already have changed in my system by MTD, there is a dual error of the attacker's reasonable being higher than this dual error of the threshold of MTD. For example, for ICP-14 bus system, it is 6.5. Is that a question? Yes, that's a question. Just how long does that take for you to do as the person protecting the grid and kind of what is the trade-off that you face? Like why would you not just do that all the time? Exactly, I will talk about the trade-off. But how regularly happens three or five days? Regularly, this is three five days, okay, for research papers, three or five days. But it depends on the defender of the system and exactly the point that you mentioned, trade-off. I changed the line reactances. The Jacobian metrics become old. I detect the old measure attacks, manipulated measurements by, that's a simple BDD, already all grids have almost. I don't need any mission learning. I don't need anything. So line reactances is, for example, measure numbers is X is one. I make it 1,000. The more change you have in line reactance, the better detection rate you will have. But does it make sense? Line reactances? I don't have any public grid limitations. What's the trade-off? What should I consider also when I change line reactances? Because if there is no trade-off, I can change the more I 100 to 100, 200. So the more I check, the better to take sure I will get 100% detection rate. Costs. OPA costs. Because poverty is also about costs. It's a question of cost. And later I will discuss that the limitation of the current research papers could be a new research direction. Costs. I can't change whatever I want. I also need to have a trade-off between costs when I change line reactances. They call it so. Costs benefit moving target difference. The new concept. Costs benefit emptying. Because I need to change topology metrics in a way that the OPA of the cost will be in an acceptable range because the generation differently, the generation cost will change from $1 to 1.1. But I can't change the system in such a way that, for example, if the generation cost for me is $100, to make it $1,000. So what's the point of it? The defender of system will say that if I attack or attack the system, the cost would be $500. But you implement the system for me that change the cost from $1 to $500, $1,000. It's better to have cyber lessons. So there is a trade-off between costs and MTD. There is a trade-off between costs and MTD. But I will talk about this cost again and our old submitted paper on this issue. But overall MTD, one of the most important parts of MTD's dynamic threat response, we regularly maintain the attack service. So the attacker wouldn't be able to attack the system. And we enhance the resiliency of, finally we enhance the resiliency of your grid without any mutual learning, without any extra stuff. What are the challenges of multi-tasking? One is cost. The other is cost. The only trade-off that have been discussed on literature, I mean all the published papers on MTD, the smart grid cyber security, it's cost. I need to change the essential way that I also will have acceptable costs for my generation. I raise a question here. Is there, do you think that, please respond to this question, do you think that there are other limitations also that researchers overlook it? What could be the other researchers over there? We talked about one item, cost. And already have been considered. There should be a trade-off between cost and MTD. Cost of generation. Because attack cyber security is a question of cost. If I attack the system, it costs me $500. But they have implemented a system that costs me $10,000. What's the point of having provision thinking? What are other limits? This could lead to other resistance directions. What are other limitations that MTD systems exactly? Power system is not summarized in just cost. Basically it makes sense to summarize the whole power grid in just cost. Does it make sense? We have system stability, system security indexes. We have thousands of indexes in power grid. It doesn't make sense to just grab one index to say that we need to have a trade-off between cost and MTD. What about other limitations? So in other words, what could be another question? If I have a situation that transient system stability and security indexes imposes to MTD, say that I can't reach detection rate higher than 60%. I mean, I consider all of the power grid indexes, stability indexes, security indexes, and say that if we want to meet those indexes, not only cost, I can't reach detection rate higher than 60%. What's another problem of MTD is that we see the problem from cyber side to the power side. Why? Here, power side is much more stronger than cyber side. Why? Availability is more important than everything I discussed. The priority is reverse, not cyber security. We should have system always available, electricity available. So the limitation should be imposed from power side, not cyber side. The current focus is on cyber side. I have detection rate that have cost. I should reach higher detection rate and also sacrifice costs a little bit. But what about other limitations? If my power grid stability indexes, security indexes, tell me that I can't reach higher detection rate higher than 60%. How can I implement MTD in those systems? This is a reality working on that. This is another risk-based direction. If you are interested, you could also discuss after the meeting or you could email me. How we can handle this limitation? I have brought some results of our old submitted papers. So I put the transactions on power systems in regard to this. This is about the submitted paper and it has brought a little bit of a portion of results. I can present all paper. But here, we have discussed about the detection rate of 118 bus system and compared with other two, the blue one is our proposed method and the two other ones are published in other journal, I mean literature papers. We have risk-based actually higher detection rate but the main, and also we have costs better or less cost than other methods. But the main important of this result is this table. Where I have brought this result. In this paper, let me ask you another question then review this slide. What could be another limitation of the entity? That already we discussed about one. Stability indexes and we raise another research direction that could be followed. If this is security indexes, we see stability indexes and we can reach detection rate higher than 60% what could be the solution? Second limitation, what are other limitations? Do you? Accuracies? Accuracy of what? Measurements. That could be great but this is not our case. Yeah, that could be great but what I emphasized in the differences of traditional governance market in the table, self-healing. I said the self-healing is also important. What does self-healing mean? It means that when I have a content that's innocent, when I have a line out that's in the power grid, how my prevention mechanism, how my detector, how my recovery method to have that self-healing feature. Imagine that I have a line outage. How moving target defense methods slide that line outage. Because this market is a dynamic network, it is not a static network. I could have line outage, generator outage, integration of renewable energies, integration of solar panels, et cetera. Here, in this case of study, we have simulated a simple line outage and you see one of the methods because for AC power grids, this one, why it has this one, this is our method, convergence rate of one. In the previous slide, I do not have that convergence rate. But in this table, I have convergence rate. Why? Because this result is for having a line outage in the system. I implement the MTD technique after having the line outage. When I have a line outage in my system, what happens? How my proposed MTD methods can manage that line outage. This method cost-benefit method, likes one of my previous co-authors, collaborators. Grids one, why it has risk one, why it has not discussed about self-healing, because it is for DC grid. Because it is for DC grid and it has, it yields actually lower detection rate than cost, you see this one. Since it is for DC one, when it's implemented for AC one, the cost is substantial increase. But it is able to yield convergence rate one and then we have a content as you see. But this method, published in 2020, Surin, United Transactions and Smart Grid, and EMTD extended moving target defense. Convergence rate is not one. Why? Because they have not considered the dynamic nature of power grid. The method is not converged after a line outage. If we have 10 line outage, what happens then? Probably this convergence rate will be reduced to 30%, 30%. What's what happened to the convergence rate? We have 100% penetration rate of renewable energy. So self-healing is also very important and another limitation of EMTD. Let's talk about detection site. In detection site, we say that cyber attack happened. I wasn't able to prevent it. So I need to detect it in initial phase. What are used for detection of cyber attacks? Mission learning is statistical methods. In mission learning, many methods, traditional classifiers, k-year's neighbor, sub-perfection machines, LSTM, deep learning methods, different deep learning methods are used. Many, many methods. Usually how they work, they gather real-time information, usually they simulate it or get from utility companies. Then they do some kind of normalizations and feature selection. Then they run the algorithm, whatever it is, traditional classifier or it is deep learning-based methods. Usually the evaluation metric that's used in literature for those kind of detectors is precision, recall, and if-measure. But let's talk about the talents of the detectors. Against self-healing, dependent to specific assumptions about the system configuration, they train an algorithm, they train a machine learning method on a fixed dataset. So based on what we learned, is it correct? They imagine that I have a system for example, I can fix a system, a network, there would be no change in the system. There would be absolutely no change in the system. No line outage, no generation outage, no cascading events, due to cyber attacks, due to maintenance, no renewable energies, even one person's penetration rate. Is it correct about smart grid? Smart grid is a dynamic network. We do have content agencies, we do have line outage, we do have generation, due to maintenance or exhaustion and adjustment, or we do have integration of renewable energies, solar panels, they will discuss about solar panels and renewable energies more. So also another limitation, for specifically for supervisor in machine learning, they require labels. Do I have labels for machine learning in real world? In real world, do I have tech samples in real world? I really can't find the substation I've been attacked, they provide the information to the label. So I do simulate, I simulate the data, but in real world that might not be the case. Another one, considering all combinations, I discussed about attacking to bus 13 with specific bus. Imagine that I have 3000 bus system. I can attack to each bus. So for successful machine learning methods, I need to consider all combination of possible effects. Attack to bus one, two, three, four, five. One, two, three, ten. I should consider all combination of possible effects. Because attacker selects to attack to bus one, 100. Two, simple two. Two, five, 2004, 3000. I need to consider all combinations in my machine learning method to build as robust and solid training they deserve to be able to predict future samples and attacks. Is it possible really? Is it possible? It is very difficult. So, something for a major drawback is that they consider aesthetic network. For example, here we have line outs. Line to four to five, 62 megawatts. We have imagine you will have a line outage. And we will have integration of renewable energies that's 10% penetration rate. What happens? I have this line outage here. This IPP14 bus system. This is a line outage line four to five. Transplant 62 megawatts. Not 10 lines, that's one line outage. The blue one, the distribution before a line outage. The distribution of phase angle system states before a line outage. The red one is a distribution after a line outage. You see the tremendous shift in the distribution? What's the problem here? I train my machine learning model based on follow distribution. I train my machine learning algorithm based on follow line, follow distribution. What is the red one? Red one is a test distribution. Is it possible for a model built on follow lines, follow distribution to correctly predict and classify samples, label samples in future for red distribution? Train it, test it. What it means? I train my model based on a follow line, follow distribution. It works in here, even a year. After a year, a line outage happens. And after a year, I will have that line outage for example, forever. Because it might be because of maintenance and new adjustment. So what's what's happened? Cyber attacks happened after that line outage. For a year, nothing happened, no cyber attack. It's everything works. Line outage happens, new distribution comes. I attack to this distribution, to these measurements. Attack is here, attack distribution is here. Old observations become irrelevant to the new ones. Old observations become irrelevant to the new observations. We call it conceptory, covariate shifts. The question, how do you expect the machine learning to perform in this situation? Because I will show in this time. But how do you, to me, what will be the performance of machine learning? Working on this, training on this one, test on this one. What could be the result? No estimation, I guess? Here, very, very simple case, very, very simple case. I have used three algorithms. Canaries, labor, different laser learner, sub-perfector machine, and this decision tree. I have trained my algorithms on this data, on this network. I have generated some sample attacks. I have normal samples. I have text samples. I have trained an algorithm. What's the accuracy of the data? Detection rate, almost one. J1 models reach the one detection rate. Please, almost all methods are able to detect all the text samples. What's the second and third one? Second one. I imagine line-outage happens here. And after that line-outage, cyber-attacks happens. I mean, I have this distribution. Attacks happen on this distribution. And I have tested this train model on that data. You see, the substantial reduction back there, is there anything? Again, I emphasize, this is a very simple one. I could make it zero. I could make a scenario that the detection rate will be greater than zero if I have cascading events. The third one, that's 10% penetration rate. And after integration of radio-minerities, I have conducted a tag. Then I have used this train model, and I have used this one as a test. You see almost 20 or 30% reduction. Again, 10%. We can have more than 50%. You will see more reduction. I would not bring my results from my paper, because if you search in my papers, I have talked a lot about this issue, concept rate, type of detection with radio-minerities, not in the dynamic scale smart grid. So I have talked a lot about this issue. Another problem with the current method of detection. I detect the attack. What's the problem? You see, this is not done. I detect that. In an I3P4 team bus, I spot this measurement as an attack, measurement X. Is it done? Once we are missing here, by the real world, I am the owner of the system and defender of the system, what should happen after the detection? You can handle any impacts, especially on availability. Like if you talk about impact availability, then you have to counteract that. But for that one, what should I do after detection? I need to spot which location, where that's your proper ride. What I need to spot? Where has been attacked? Where? Which measurement? Which bus has been attacked? I need to localize. We call it localization. In smart cyber security, we call this concept attack localization. Many papers don't consider attack localization. You know why? Attack detection is a binary classification. I have normal samples. I have attack samples. What happens? Binary classification. What many have to attack bus one? Attack to bus two, three, four, 3,000. How many classes I should have? 3,000 classes I would have. Do you see that it is easy to manage that 3,000 class classification problem? 3,000 X and 1, 3,000 attack and 1 normal, 3,001 class classification. Is it easy? You see, in recess, we need also the real world. It is called multi-class classification problem. Multi-class classification. For each attack, I have a new attack. Attack to bus one. Label, this label is attack to bus one. Or attack on bus one. Bus two, bus three, bus five. I detect, then I say not only only detect, I say also that this measurement is detected as spotted as an attack, but attacking where? To location one, location X, to location Y. So it is very difficult. Why it is very difficult that we'll need to degrade our accuracy, you see? Why we'll see reduction if we see multi-class classification? If I see the localization, why the accuracy of the mission learning, the overall mission learning model, my decrease? I say that I detect an attack bus two. I detect an attack, is it correct? An attack bus five, when I have two class classification, one classification. I detect also attack to bus 100 as an attack. Decreate. Both are attack. But when I have multi-class classification, G-powder buses, I say that I detect this an attack. My model said that I detect this an attack. But instead of detecting it as an attack to bus one, I have detected it as an attack bus 100. So it is misclasification. It is correct that I have detected the spot of the attack sample, but that I have wrongly located it. Instead of saying that attack has happened on bus five, I say that my model says that attack has happened to bus 100. So if measure will be reduced, accuracy will be reduced. That's also why we need to select event correct evaluation metrics. In some research papers, you will see that they have 100% accuracy. But when you see the evaluation metrics, when you see the samples, or some they might actually receive 95%. You see that okay, 95% accuracy is good. But I had 100 samples. I had five attack samples. I have 95, I imagine this one. Paper some papers. Say that I have reached 95% accuracy. What has happened if you drive into the paper? It has 100 samples. In 100 samples, five samples were attacked, 95 samples were normals. The algorithm has detected all samples and a normal. Missed all five samples. It means whatever I run, it will receive that accuracy because I reached 95% percent. It's important for me to spot those attacks. Whatever I run, I will reach that accuracy, 95%. If I don't have any algorithm, I say that all incoming measurements are normal. What would be the accuracy? 95%. Because 95 samples are the one of the samples are normal measurements. I miss five samples. I will miss five samples. So it's also important in your research to focus on evaluation metrics and also dataset and sample. And that's why multi-classification is difficult because it's putting one attack to another. It's stopping all of it. It is attacked to bus five. It is stopped saying that to bus 10, we degrade the measure. So that's why researchers usually try to not talk about localization. Another one, reconstruction. I will come back to my sense about integrity, attack, and evaluation. As an attacker, you are the owner of the system and the defender of the system. I attack the system. You use it well to mission learning algorithm. You detect the attack. I attack the system. You keep dropping the measure. I attack the system. You keep dropping the measure. What's the attack? Just take it. Sorry. I attack the system. You successfully spot the attack. I just even localize that. What's the attack? But I keep attacking your chip dropping. I keep attacking your chip dropping. What's the, it is the integrity. I decide it is the integrity attack. I take the measurements. You spot the manipulation. So what's the attack? I keep attacking your chip dropping. I keep attacking your chip dropping. Remember, confidentiality is completely valid. What happens? Eventually you don't have any measurements. Exactly. What this means? No availability. I have lost integrity attack. But I also targeted availability. You successfully keep dropping measurements. You block attack measurements. But you don't have any measurements to control to secure, to report to control center. So control center has no measurements to report to make a decision. So it means availability attack. So here reconstructions plays a key role. Because in real world, I wouldn't be able to immediately spot the attack and find the attack and recover from cyber attack. To find the vulnerable points. I think might happen duration of the attack. Why here? Two weeks, three weeks. So here reconstructions play a key role. How I can reconstruct a measurement. It is right that it will lead, I don't want the DOS. I don't want the denial of service attack. But what happens? Finally, my attack needs to denial of service attack and also availability attack. This means that we need to have also correction in our reconstruction. Sorry, reconstruction in our proposed methods. To reconstruct the falsified measurements. I spot the attack and I need to reconstruct the measurement. This was another important point about detection. So we are closing detection. We talked about prevention. We talked about detection. So any question? We will dive into offensive security. Any question about prevention, detection? No, okay. Offensive methods in smart grid. I have classified into two groups. Sophisticated attacks and study agents. This is my own, I don't say this is great. This is standard. This is my own definition of, this is these words are my own definition. Sophisticated attacks and study agents. I will tell you why these are my own definitions. We can have two offensive methods. Sophisticated attacks and also study agents. What I mean by sophisticated attacks? Sophisticated attacks, I will show you. I mean that you don't need to have very skill set, specific skill set about cybersecurity and missionary. You do need to also have a robust understanding about smart grid. What's happening in smart grid? I am a computer scientist. But I work on sophisticated smart grid. So definitely you know more about smart grid. So you need to utilize that smart grid knowledge to use agonist smart grid. By sophisticated attacks, I will review two sophisticated attacks that I already have published. And in associate attacks, I will talk about next generation of in data integrity attacks or false data utilization attacks. It's still zero attacks. What I mean by sophisticated attacks? Maybe try to grab on you. A couple on this? Yeah, yeah, yeah. Okay. You could read that our this paper is published on IEEE transactions and control and network systems, revealing a new vulnerability of distributed state estimation. What I mean by sophisticated attacks? If you notice the vulnerability of the system, usually we do have clustering zoning in smart grid. Why it's the AMA clustering? Because we have big data, we need to split the area, the poverty rate into different zones so we can manage better. The smaller it is, the better management we will have. But we notice that when we cluster the network, we will have a vulnerability of boundary bosses. And we could attack boundary bosses easier. We spotted that. We attacked the system. We mathematically also, not also numerically, we also mathematically showed that our bond vulnerability script and we attack the system with less cost. So when sophisticated attacks I mean this. You have a good understanding of all poverty and a little bit of smart grid cybersecurity and general IT security. You find and spot the vulnerable points based on functions in smart grid, not general cybersecurity knowledge. Another one. This is since it is a submitted paper, I will not discuss about the details. But a question about substrate attacks. We have renewable energies. What is the renewable energy consequence? Uncertainty. When I have integration of renewable energies, I do increase uncertainty in the systems because I can say that renewable energy print output exactly, it depends on the natural parameters. If it is really being solved and et cetera, different parameters that I don't have any control over them. But I can say that adding renewable energies increase the uncertainty in the system. So I see from this cybersecurity aspect not to detect the system as an hacker. Because in cybersecurity we have threat modeling. I need to imagine myself as an attacker. How an attacker could use this function to attack the system to be able to prevent attacks or protect the system? This blue line, before integration of renewable energies, the system works. This is on phase angles. The system works well. No uncertainties. I have integration of renewable energies, again, just 10%. You see the uncertainty of the measurements. The interval, the range of the measurements for the same phase angle for one bus has been changed. So the question, how an attacker could use this, oh, sorry, how an attacker could use this uncertainty to attack the system as an offensive model. Attacker could be wise and select measurements in such a way that the attack will be concealed, it'll be concealed between normal range of measurements. In the past I didn't have normal range. It's in line. There is no normal range. But here I have a range. I have a range. Uncertainty increase. The attacker could conceal this attack between normal range because variance of data has been increased. It has definitely negative impact on cybersecurity. If I am an hacker, I will see how I can use that. And I have used, we have used in one paper and this is just final result. And I show that, we showed in the paper that if I use the measurements, then this uncertainty, against the system, I will be able to bypass the detectors. The detection rate of the algorithms with established algorithms will be decreased. If it is a traditional FDIA, or if I use this uncertainty, the detection rate is lower if I use this uncertainty to attack the system. So, another attack, it's still your attack. This one is specifically about constate prediction effect being discussed. What the next generation of constate prediction effect. How we can make it more complex and fast and associated. The focus on the FDIA is on detection. So, they have paid attention to generation side. There are some work that have paid attention to, have worked on generations of FDIA and new FDIAs, but there are some limitations. There are some limitations. First, they relaxed a something, focus on relaxing assumption about having knowledge of the system that Zikubian mushrooms. They have proposed new FDIAs that they don't consider to have full of information. Or they have used simple GAN. In 2020, I also use that one, to put a healthy FDIA. Simple GAN, simple methods. But they overlooked the complexity of measurements and power system. Search, they do not have strong evaluation metrics. They use a speedy. They say that I generate this attack. It is still here, why? Because I wipe a speedy. For this, we also have, because it's also submitted paper to attribute transactions on industrial applications, I can't discuss about the details. What's the result? We have proposed a new FDIA, a method to generate complex constate prediction attack. I just present some results here. This is traditional FDIA for team boss system. Detection rate of LSEM is 98% for traditional FDIA. Our proposed method is 92%. It shows that our proposed method is able to await the detection system. Since it is submitted one, I had to actually present us proportion of the results for 157 Boston. LSEM is 95% for traditional FDIA, not traditional FDIA, for ours is 88%. But we do have used, we have used many other evaluation metrics. Just to mention, for example, KLB. We have used KLB to show how similar our the rate of distribution is similar to real normal measurement. Over second, B. I just can name some of them because it's submitted papers. We have used many, many evaluation metrics to show the performance of the proposed attack. So since the time, I have also some future resist directions, but since the time is limited, I wouldn't be able to talk about distribution grid, Ukrainian poverty rate attack, smart meters attack and other stuff. So if you have airness, but if there's any question, I would be happy to respond. At the time of the speech, don't we have? Yeah. We will be able to share this time. I hope you're saying no. I will think a lot then. Any questions? Yeah, two questions. No, we're not. Okay. Sure. We'll have a minute. No, here. Here are questions and answers. Yeah. What are the practical chances and how to react if this is okay? This is a good question. I will raise another open question and resist direction on this. In defects, Ravi, Mr. Ravi has asked this question. I am asking this question. I am responding your question with a question. I said that what defects? There is another limitation. Imagine in a system, I don't have defects, specifically for distribution grid. How I can, this is the work that we are making with USPAP and I'm working on this topic exactly. How I can propose an MTD method, propose moving target mechanism, defense mechanism that don't require defects. Because defects location has impact on the detection rate of your proposed MTD. That's why if you read papers on MTD, they always discuss about the location of defects because it affects the detection rate and cost. So this is my question. How do you have some practical challenges? Imagine that I don't have defects. This is the most important challenge. Why? Because when you are talking to industry folks, one important problem is that legacy devices or imposing extra cost to them, they do not have that those devices. What if they do not have those devices or they do not have enough devices to reach, to provide, to present a successful MTD attack? And specifically in distribution rate, imagine how you can, I can't respond to this question, imagine how I can implement moving target defense in distribution rate where there is no defects. It is somehow revolutionizing MTD strategies because all MTD strategies rely on defects because when we say that we change line reactances, how we change line reactances by defects as I mentioned in my presentation. So if I don't have defects, what would happens if I don't have any defects? And what functions of distribution greed I could use instead of defects to gain the same performance, I mean to reach that goal? I don't know, is there is another question? I believe there is another question. Are utilities or IT industry implementing MTD moves? Yes, IT industries also, this is where the established topics in IT industries. Utilities in the smart grid section, no, IT industries, yes, because MTD is not just limited to line reactances. We also have dynamic research allocation. We can also, in MTD, we can also work on IP change, network configuration change. MTD has three methods, one is line reactances. In IT sector, it is different. It is a well-established topic, but in smart grid, it's a recent new topic that is not that will establish one because I mentioned one limitation is not considering security and stability indexes one another is not considering dynamic nature. And third limitation is if I don't have defects, what should I do? The point is that, I talked about here, in traditional FDA, we do need to have access to topology metrics, Jacobian metrics, but in that assumption is not that correct anymore. I mean to say that FDA has information access, access to information inside information because the recent generations of FDA, many existing FDA generations, have relaxed that assumption. So for generating FDA, we do not need to have access to information. For example, this method that we have proposed also, it says here FDA is a completely unsupervised method. We do not have any access to any information. Our proposed method doesn't have any access to information, has not bypass encryption, has not anything between encryption. We do receive just normal measurements and we generate a sincere FDA. Not only our word, but also other methods have relaxed that assumptions. For example, if you search blind FDA, blind, the word blind FDA, they have proposed many FDA methods that they don't require any access to inside the information, like bypassing encryption. And as I mentioned, our proposed method is a hybrid deep learning method, but it's completely unsupervised. We do not need to have any access to information, thus normal measurement. It means that that assumption is not true anymore and it has been relaxed by other generations of FDAs. If you search the word blind FDA, you will find the trend on FDA that doesn't have any access to Jacobian metrics. And if you search deep learning FDA, you will see that there are other published papers that also they don't have any access to information, but they generate sincere FDA, but I said the limitations. In regarding the deep learning methods, they do not have mathematical proof. Also they, in our paper, we mathematically have shown the single GAN, single otane coder will not be able to bypass the other detectors, like deep learning classifiers, like JST, KLD. So that's not correct anymore. Okay, I think we're past time, so we will just end here. Okay, thank you very much.