 Thank you very much for the invitation and it's a very good opportunity to share my recent research on AI and energy systems. The topic is AI assisted the power of great disconnection and control, so having also been on the various perspective of this research ever since 2016. And today's talk, I will talk about the idea of how I started this idea and the real world implementations we have developed and also we will start up based on some of the technology developed here and share some insight we obtained from field. A quick outline, I will start with some operational challenges, the power grid facing and a little bit about the basics about machine learning and AI, just to start digging into the idea. And then I will talk about how AI can potentially help to make the life of our system dispatch easier. Followed by some studies we have done with the real grid and the transition to a very important topic on the safety of AI, especially for critical infrastructure like energy systems. Lastly, I will share some research on transformer and LLMS in energy systems. So the power grid is ended through a profound transformation that you can see from this picture. This is what the power grid used to be and what is right now. So we have generations and the power are generated from remote areas transmitted to low centers and customers. We have unidirectional energy flow. Now the power grid is basically becoming decentralized and distributed where we have generation and load everywhere in the system. So it's essentially energy internet. But why is the challenge this cost to the power grid is dynamics and aesthetics. We have so many stochastic variables over a big grid. So that is making the operation of the grid very challenging. We have several backups and one of the grand challenges is the increasing dynamics and the stochastic in the model power grid. So this does not leave our system dispatch a lot of time to make control and dispatch decisions. And we really need to make this process fully automated and fast. So this is a distribution rate. We used to have unidirectional voltage profile now becomes because of the multi directional power flow. And we are having fluctuating voltage over the grid and that fundamentally change how the grid can be operated. And one of the major reason was this was the majority of the generators synchronous rotating. So we're talking about time steps of minutes, tens of minutes, or tens of seconds. Now everything is inverter, the IBR inverter based resources. We're talking about milliseconds. So that's a challenge. And we also have a big opportunity. The big opportunity is the digitization of the grid. So this one shows the prediction of IoT devices across a grid over the next few years. And this was predicted a few years ago. And now even this is very conservative. So I see some numbers triple the prediction here. So in six years, we're going to have 15 billion or even 100 billion IoT devices across a grid. And all devices use electricity, right? We generate measurement and we send the measurement to the control center or grid controllers for operation. And that's a internet traffic explosive. And then we have smart meters, PMUs synchronized by GPS across all stage of the power and energy generation transmission distribution and consumption. So what's in the pretty clear the digitization is going to be everywhere and how we can utilize that to address the fundamental challenges. And by the renewable energy resources is the key. And this is what I have been working on since my master research. So we have a lot of a face amendment unit across a grid across a wide area and the grid. Those sensors are synchronized by GPS. And over the past decade, people have been working on how to analyze or how to understand what is going on across the grid. Like saying what is happening, although it might be far from you. So something like a grid I situation awareness. And I was doing this research. The one thing challenged me. I talked to system dispatchers. This is like cities, right? Previously, I don't see the problem. Now you have very accurate high sampling rate devices telling me. So this is a problem of the grid. It's very painful because you know what's going on, but you don't know how to make it right. And that research can be here for a pretty long time. So if you check this loop, we are power grade. We have a lot of sensors and the sensors generate data. We use that data to create a situational awareness where we do perception comprehension and projection. So everything up here has been fully automated. But after that, we need to see some dispatcher to make a decision. The last human is in a loop. It becomes slow because you break them. And actions will take some time. And we talk about we are on a faster and a faster system dynamics. Don't have too much time to address the challenges. So the idea was if I can fully automate this loop so that you give me data. I can tell you what is going on. In the meanwhile, I can tell you how to make it right. So that is the goal. And a brief overview of AI machine learning. So AI is a broad term and the subcategory is machine learning where we focus on telling stories or learn from data without being explicitly the rules and the model. And a further subset is deep learning where we use neural network. And in particular multiple layers of neural network to address a challenge. And this is what happened over the past decade. So I think those stories are very familiar. And recently we have the challenge everybody is using that. So several observations from this event were previously we were generating labors. When I was working at NEC, we hired many grad and grad students to label the images in order to learn. And with AlphaGo, AlphaZero, they basically learn from interacting with the environment. So the entire research or application is heading toward self-supervised or weekly supervised learning. So then the idea was, can we do something similar for the power grid? So which is a lax system, a lot of opportunities. But in the meanwhile, a lot of security safety concerns, which is very difficult to change. And we have developed decades of classical computational tools, software and models to control and understand the power grid. So the idea was, can we sort of combine the most advanced machine learning AI ideas with our classical long time developed tools to create AI agents that can learn on its own with little or no supervision by human beings. And a quick review of different categories of machine learning. I think some of you may already know this, but this is for the unknown. So supervised learning in a simple sentence would be, I have data, I have a label. I'm trying to learn some model and the rules based on labeled data. So unsupervised learning, I have data without labels. If I give you the major US city from the west coast, east coast and the central part of US housing price. I don't have to tell you, this is San Francisco, but from data you can tell, right? So learning without label and reinforcement learning. Reinforcement learning is very much like the control system where you try to do something and you observe the result. And based on the observation, you try to determine what you need to do next. It's pretty much useful in a very dynamic and a stochastic environment where a computer program can try things out, watch the errors and determine what to do. And the deep learning is actually where we use multiple layers of a neural network. It can be supervised unsupervised. And my research mainly focusing on the combination of deep learning and reinforcement learning. So AI has a lot of applications in energy systems. So from the power generation, we use AI to predict the solar power generation for the next day for the next hour. In cameras, in past data, in the transmission grid, we're trying to locate where our thought is, whether our transformer has some issues, distribution. So we look at intelligent reasoning and monitoring of equipment and the self-quitting control capabilities of the grid. And from the end-user, we do non-intrusive load monitoring, our commercial product. We also try to predict the load. And with the deep reinforcement learning and enlightened language models, a new trend is we were focusing on monitoring, diagnosis, forecasting while moving forward to reasoning, planning, both short-term, long-term, control, online decision-making. So the ultimate goal is to make this fully autonomous. And this is a comparison of the classical way. So we learn power systems, energy systems, rules. We also set up models. So when you give the rules and the data, the computer program, try to figure out what is going to happen, what can you do. So the new way is data-centric or hybrid, where we feed data and then we feed the observations. So sort of answers. The program is set to figure out those rules, which can be very complicated. But once we figure out those rules, we can use that to guide our future research. So that's the difference of automatic and autonomous, where autonomous you learn, you improve, and you adapt. So over the past years, whenever I talk to system dispatcher or power system engineers, one question I got asked often was, why do we want to use AI? The grade has been running OK. What is the key advantage of having AI and machine learning for the energy system? So I've been trying to answer that question along with my research and field deployment project. So here are three reasons. One is flexibility. Previously you have fixed the rules. Now we have statistical variables across the system. We really want the system to be flexible and continuous learning. We are adding ERs, renewables across the grid every day, every single day. So the grid is evolving pretty fast. If you have fixed the rule, you'll have to update that rule frequently. And machine learning, we can keep learning and adapt our program to handle the challenges. And another benefit is there is no need for explicit models. Although we can still use it, but it's not a must. And that can bring a lot of opportunities. For example, people working on communication, they can now work on power grid and other majors as well because of the power, representation power of the machine learning and AI. So the gap in one sentence would be we are in lack of approaches, which can synthesize a massive amount of data and measurement across a wide area to best allocate the resources. And that's what my research is trying to address. So here the goal is trying to close the loop. I'm basically using deep reinforcement learning as the core and combine that with models, with other machine learning approaches to make this process automated. And the idea is you give me a data, I'll tell you what to do within a second. I'll skip this part about deep learning because of the time constraint and the reinforcement learning my dog. So the training process of my dog is really the reinforcement learning process. So initially I'm trying to ask my dog to enter the grid. Very difficult, tried mass. One time he succeeded, even milkable. So that's a sort of reinforcement. And after that, he may still have no idea, but somehow he realized if I do something and if you say something, I do it, I'll get a reward. And next time, still, but better, so I'm giving a reward every time she can succeed successfully finish my task. So over a long time you can expect that she's doing very well. So now, even she just saw me, I don't have to really pronounce the word, she will enter the grid. That's the power of reinforcement. Not going to talk about the technical details. But the basic idea is a computer program, typically called agent can talk to interact with the environment, try something, receive a feedback and over and over again become pretty well. And this is one simple example to illustrate the idea. This is the pretty easy mouse and the clip, the game. So a mouse start from here trying to catch the cheese without touching the clip in shortest distance. So you can use various ways, algorithm to fulfill this task. And the one way would be if I can create a table on one dimension, that's the location of the mouse. The other dimension, that's the action. And the numbers, the numbers is the cash you're going to get at certain location by taking an action. And the game becomes very easy. You start from here, you pick the most rewarding. And it will finish the job. And the real world is much more complicated than this. So, thanks to the representation power of deep neural network, we can create a deep neural network to approximate this table. And we can keep doing experiment to train this deep neural network. So wherever you are, the deep neural network is going to tell you, if you do this, you will get the biggest reward. And then you'll finish the job. So that's the major idea. So, essentially, we are trying to develop an agent that can achieve a human level task. So where the reinforcement learning give you the framework, those loops of training and the deep learning. Well, and then we combine these two. We can do quite some good things. So this is the first example, myself and my team try to develop. It's a very classical problem in smart grade. You have voltage fluctuations and we know if the voltage is too high, your life will get burned. Too low, you will be able to see anything. So one of the critical function of power grade dispatcher would be how to maintain the voltage at all nodes. Within a certain range. And you have some resources to do that. You can adjust the generators. You have shunt elements, shunt capacitors, reactors. You also have some transformers. Those are the resources you can do. Remember the goal is within a wide area to give me data. I will tell you what is the best way of allocating those resources to solve your system level problems. So that's the goal. So this can be actually map to the framework I just talked about. And we did that using this loop. And we find it works very well. So one thing about the reward, because it's very important. So you can imagine that the reward will guide the agent to do something that is right. So here, this is the simplest way of defining the reward where if the agent takes some action and make the voltage right. I'll give a big reward. If it does something that partially solve the problem, I'm going to give a smaller reward or small penalty. And if it does something really crazy, I will just penalize the agent once so that you can remember for a long time. So with that guidance, the agent is going to master the voltage control pretty well. Not going to dig into the technical details. So one thing about a very good feature about this is we started from scratch. So initially the agent has no idea about what resources, if I dispatch, will make the voltage right. I mean, from a power grid, our system point of view, we have been learning the modeling and the control analysis approaches so that our engineers know what we do well cause, what effect to the grid. But the agent does not. So the agent starts purely random resources, but it observe, it grow, adapt, and finally master the problem pretty well. So this is one set of the very first result when I started this research. So the x axis is the data samples I saw and the y axis is the total return. So basically, if you get a hundred, that's perfect, that's a plus. If you don't do so well, you get a 90 something that's still a right. So and it's a great as bonus. So from here you can see that's a group of intelligence. It basically grows. And once I pass the agent to see 10,000 samples, I start fixing all the parameters of the agent and we try testing. So out of 10,000, 9,998 samples, I solve the problem in one iteration and for two iterations. So that's like, you give me data, I tell the problem, you do it to solve it. And that's the growth of intelligence. So this is a numerical example on how this works. This was at very initial stage. So I pass this data. So the yellow ones indicates the node with voltage problems. I pass this to the agent. The agent will tell, okay, please do this, adjust. And that's a system state. So some problem get solved. Right. This new problem appear. Okay, this is nice. So new problems start our path that data to the agent agent assess trying this. So boom, our problem solved. No more yellow. So, and as time goes by, I go to here. So that's the data passed to the agent that has to do this. I directly solve the problem. That's very nice. And you can see I only pass a vector, right? Agent has no knowledge of the topology of the grade, the parameters, the generators, location of your resources. We don't need that. And a lot of times in power, we will have to deal with hybrid categories of variables. So integers and continuous ones. So there are different algorithms we can utilize to solve this. So one approach is called the DDPG. So DDPG determines the policy gradient. So it can handle continuous control variables. So I do the same thing. And you can see in the testing phase, all the samples are probably more perfectly solved. And if you compare this, this is an Illinois portion of the US grade. And we try this grade and continuous control. So this is a very interesting one. So you can see in the training phase. So the first algorithm as well, because there's not so many negative reward. But here you can see many negative reward, right? But in the inference, in the online testing phase, this one does much better. And the reason is that this algorithm has more exploration. So you don't hurry to push the agent to grow. So it's better to learn more in the training phase so that you can do much better in the future. And the power grid that you can see is a big graph. We have transmission lines, lightnings, operations we open in, and the grid can be isolated. So it's a changing topology. So one of the major concern behind AI where the power engineer has is if I have a changing grid with varying topologies, where your AI be able to generalize well enough to handle the problem. So we have done a lot of work. So we learned, well, we are training the AI. They randomly add topology changes and the AI can pretty well. So topology problem can be handled. So this is another application. We have a grid and a certain portion of the grid is overloaded. The system is very simple. So our grid is better as you want to know in order to reduce the power flow, the current. You reduce here, you increase here, and you will be able to do that. And even if you have a thought here, you'll do the same. So now we get this grid, right? And if we have overflow here, it's not so obvious. Traditionally, we have approaches, sensitivity factors, but the grid is going more decentralized. The control decision is more difficult. And with AI, we can do that the same framework. So we actually deployed one of this to assist the system dispatcher to relieve overflows wherever you have in 20 minutes. And one of the fundamental problems in power grid, and it can be extended to many applications, is the AC. So AC is alternating current, AC optimal power flow. I have this many resources. How could I best plan my grid so that I either run the cheapest cost, lowest loss. So it's optimization problem. And you have to do that online, right? So it's a non-combat. So there is no guarantee that you will get a solution in minutes. And now everything is moving so fast. We need to do that very well in a faster period, a short amount of time. So this is where I combine the other learning. This is like a teacher, the imitation learning. I combine the imitation learning with the different enforcement learning to hard start the learning process. And I try that on different systems. So this is the process, how the AI agent is adjusting the generation from the generators in order to reduce the system cost. And so this is the result of this idea with California way. As you can see, it's very dense. So every few seconds you get a data sample and it fluctuates. So the idea is traditionally we do every five minutes, 10 minutes, 15 minutes. Now I do second by second. Whether you use it or not, that's another story, but the decision is there. So I always give you a reference. So this is what your grades can do to achieve the best performance. And that happens within a second. So this is the one AI computation. So the idea is that we can change the topology of the grade without touching any other element. We don't need to do demand response. We don't need to ask people to stop using the load to shut off your AC. We don't need to really dispatch our generation just by changing the topology of the power grid. That's one of the biggest results we have in the energy system. So here are the same thing. This one, we have a cascading outage, blackout. Just by opening two lines, the same operating condition, the system running smooth without any problem. So this is great. However, the challenge is where to open, where to change the topology. And if you check, the solution space explodes. It's a very difficult decision problem. We have tons of constraints. And we write the equations just for this problem. This is something you only want to look at. The AI and this, where's the modifications? This runs half an hour. So by the time you figure out, oh, this is what I need to do. People already move on with this. So we can get a solution in roughly 18 milliseconds. And without human intervention, we run a system for half a month without even touching the system. Everything runs autonomously. Open source of code if you're interested, you can check. For time considerations, I will not talk about details. So we have a lot of elements in the power grid and we have more data. So here the idea is, can we design algorithms that looking at the data trying to figure out what the right model of the grid is. Used to be very challenging. But AI has that potential, a similar framework and a load. So this is one example where we collected from the field. And the learning process, the AI is trying to figure out a set of differential equations. What are the right parameters? So this is for the load. And the real-time control can also do that. And it should show some benefits. And at a certain condition that we can even prove that. It can always guarantee stability of a dynamic system. And when the system grows, we can use multi-agent. So each agent communicates a little information. Whether they can make the grid better. Developed software, which I don't have time. So my partner was pretty brave to try on this critical infrastructure, AI power. So this was the first system, real system, where we got a chance to deploy. That's AI assisted software. So the system peak load is 3.5 big wire. So that's like a load of maybe several big cities. And our AI needs to communicate with the production system in the real world. So this is where we train, deploy AI agent to interact with the existing energy management system of the power grid. And then we try to compare AI power solution with the traditional. So this is a result. I remember I was sitting in the control room right before the pandemic. So this is a result. It's very promising. I don't have time to cover everything, but the idea is on top of the existing optimization. The AI can bring in 6% improvement on a system loss while mitigating voltage violations and a powerful violations. So that translates to minutes. So then my focus was how big a system can the AI. So can the AI skill. So this is by far the biggest system. I've done it based on this Richard to see the biggest system. So this is a demo offline demo of an online system. So the idea is I have a power grid. And this power grid, as you can see, has 4000 nodes. So about 600 generators. Think about the dimension of the problem. And the system is running online. This is offline version. So created the four steps. So I get a snapshot of this system. So choose one system snapshot. And try to add some system of thought. Consider that as a game. You're trying to break the system down. And when you do that, you will have system problem, a lack of seven giga power. Right. There's a multiple system constraint violated. And even for the system dispatcher, they want to be able to do anything to resolve this in faster time. So in 244 millisecond, I tell the safe hand of the generators. This is what you need to do. And if you do that, the problem is solved. So this is by far before and after. So this is by far this technology. The biggest system this technology can do. So just to give you an idea. You can handle the complexity of a major regional power grid. And then deploy this to distribution. So much lower voltage where we have more renewables. Right. Then the challenge changed. Challenge changed because the agent does not have an environment. So how do we interact? So how do we do that? In a year, we developed a surrogate model. So we collected real data from the fuel, a training, a surrogate model, and a surrogate model interact with the AI agent. And in 2021, we closed for the real system. We actually let AI to control. And we have several observatory engineers sitting in a control room, catching the difference. And it's a tremendous amount of work. It works pretty well. So comparison, I'm not going to cover two major conclusions. One is in terms of optimization. AI does badly because of the lack of a good model. The existing system, try an error, try an error, though you don't know what will happen if you do something. So the only thing is you'll be more conservative. You try, you observe, then you go further. But the AI, before I'm doing this, I can predict this is exactly what I'm going to happen. So now the software runs at 13 distribution rate in the control room. So safety is very, very critical. So a power grid is such a mission critical infrastructure that we can't tolerate blackout. And if you do something improper, something wrong, a power grid can be done affecting 10,000, 10,000,000, even millions of people. So that's why it's moving slow, because security, reliability is the key. So at the AI, I talk about AI, when we try to do exploration, we really don't know if this can actually break. If you are communicating with a simulation software that's a five-year driving a car, you cannot tolerate failure. You cannot hit something, but there is no guarantee unless you do something. So for example, so this is, this region is called undesired. A lot of time is the system is running on desired mode. Outside is unsafe, which you don't want to want the system to run. And here is the perfect. So a lot of times we want to bring the system state from undesired, but safe to perfect. But there is no guarantee that you jump out of the circle and then come back. And we definitely need to guarantee that. So this is one example I gave during the learning phase of such landing on the IEEE system. So as you can see, there really is no guarantee unless you do something. So there can be several ways, constrain optimization, control barrier functions, constrain the markup precision process. So the previous demo I showed you was using the constrain markup precision process. So this is something borrowed from the control, so control barrier function. The advantage, this is a faster growing research area. And the key idea is I can train a neural control barrier function so that you give me the state and control. And I can tell you whether the system can go safe or unsafe. And this is a one to three bus system. And the x-axis is the number of the ID of the buses. And this is a safe range. Wherever you see a red one that's attacked during the training phase, this node becomes unsafe. You can see there are many ones safe. And then when I deploy it, safety algorithms, you don't see red. So safety guaranteed. It's a growing research and very important for a mission critical system, like power grade. And the edge intelligence. Right now we have a lot of data. Data are transmitted to control center for processing. But data is exploding everywhere. And that's obviously not the best way. So the best way as I can see is that we want to handle data at the places where they are generated. But right now the infrastructure is not there. And my research tries to do that. So from hardware, software, co-design perspective. So the idea is that we want to make AI runable, running on edge devices, cheap, portable, low cost. So that is one research direction. Time will not cover. So blockchain plus AI. So that's really AI over the edge. We can do demand control and transactive energy. So one critical issue for deployment of AI in the power grid is there is always a hidden assumption in deployment and learning process, which is we assume the distribution of the data does not change much. And we assume that the model does not change much. But when those assumptions are violated, so the EM model will fail. How to make sure you have a maintenance free software that can does it autonomously. So the basic idea is that you probably know the concept drift, the model drift, data drift. So change is the only constant. Right. Whatever you learn today may not work tomorrow and less likely in a month, in a week. So the idea how we deploy this is we deploy two agent in parallel. So one agent handles the influence, the online decision. And the one agent consistently learning with the new data. And once in a while, for example, pricing a day, I will check the performance of the learning one, evaluate the performance of the inferencing one. Either switch or replace that goes hand in hand. And that's ready. Through the practice to be quite successful. And my team also started doing some work on that language model. This is probably the second and last slide. So, which is really hot research area. So this is one invited paper where I did. So the idea is that transformer and pretty helpful, powerful in handling natural language processing. Right. So how about a power or a low hand from fruit was, we can use this model to predict the load. All the renewable generation and that can help the decision. And we tried and we compared the transformer model with the other machine learning model and algorithms. And the result was a little surprise. And I haven't defined the one application that it beat the rest of the algorithm. And we try to dig into the reason why. So one obvious reason is that it's very powerful for multi model. So when you have data in text, in image, and they use that data to do the prediction or training, it may work well. But here is a purely numerical data. And I haven't seen this working better than the other algorithms. So it's still ongoing research. So now one other goal is that we want to train an agent that can be multitasking to talk about several applications. Why I can handle a few application or challenges. So not a slide. So where the research and my vision goes from trying essentially trying to combine the eyes with the mind of a great eyes with a great mind. And I'm reaching that in the future power grid and energy system is going to be running controlled by robots or. Operative robots that will make a great fully autonomous. And my research is trying to fulfill that vision. Thank you.