 Thank you for inviting me, Steve. I really enjoyed Balaji's talk. And I'll try to follow up what he presented, presenting what is the kind of a similar view for electric grids. And I want to show you some academic projects we have here at Stanford as well as part of that. So here's the list of students and faculty as well as external collaborators that provided either data or their time to make this happen. So a sustainable grid in my definition and the broader definition out there is a grid that is mainly powered through renewables. The problem with such a grid is first, your supply side starts to become a lot more variable. And what do you need to do to manage that? Well, you need to do better forecasts. We all hear that all the time. What is the limitation for forecasting? So we went out there and met with operators from Kaizo as well as PJM. The limitation for forecasting is that in the medium, short term, you don't have enough sensors in the right places to take measurements and enable accuracy in the forecast. And you also want to be able to understand some impacts of this variability. So a lot of conversations go around, wind farms combining together and operating as one. And is there any consequence? So you need to take market data, the data from wind farms and some economic model to try to understand that impact. That's on the supply side. Well, one opportunity is that in the demand side, we have some flexibility that currently it's tapped but only at a very large loads and very large systems. And if we want to scale that up and really match it up with the increase in variable supply, we need to start understanding how we consume electricity not at the scale of a big bus or a campus but really at the scale of a consumer. And that is what I call consumer modeling. Once you understand that, then we can try to do some type of management of this consumption and even design better schemes. And the last piece here is that in this new grid because of all these variability renewables and so on, the operations themselves are changing. So a lot of the principles for which the grid was designed early on don't apply now. And to update these operations, the fact is you're going to need a lot more sensing, also new strategies to do things like frequency control or faults or so on. I'm not going to have time to address some of the projects that were done here at Stanford and this last item, but I want to show you some examples on the other ones. The one thing that I wanted to say is that I believe it's not just a challenge of data analysis. I think you have to do data analytics and analysis. And by analysis, I mean incorporate power systems and an understanding of economic models or the appropriate models for whatever problem you're looking at. So that's where we can do those type of things here in the university as well. So first thing where my journey started on this is through a paper I wrote which is called risk limiting dispatch. And the idea was to build a simple dispatch model for a whole system, a state system like California and come up with very simple formulas that explain how uncertainty impacts the cost of dispatch. And we created this metric that we called cost of integration and we were able to show some very simple formula actually where we took into account networks and everything that you need to take into account. And it says that your cost increases roughly linearly with the forecast error and linearly with this variability term which is like the intra hour tracking error. And typically you can think of this if I added storage in the way that you think about storage, a fast storage you will try to erase that. And if I add better forecast I will reduce this. And in a way these two are interchangeable. So the model allows you to model a lot of the details of the system. I'm kind of summarizing here with one of the results. But the point I wanted to make is if we improve the forecast by 10% of your wind it's almost like you're adding 10% storage on your system off the capacity of that wind. So except the cost for improving forecast may be very different than the cost for improving wind. So this model here was done for 20% penetration of wind here. So the one project we did was that we went out there and tried to find what is the limitation for forecasting and as I mentioned is really the ability to sense. So we built some sensors here at Stanford that allow you to measure wind speeds at hub height and above autonomously. And here's just an experiment we did back on the campus here. We have a kite and this little robotic sensor climbs up the line and just stays there and it can measure wind speed for hours in an end. So what are we doing with this? We partner up with a company called Sailors Energy which is an ex student from Stanford from our program. He has built a high resolution weather forecasting model that takes inputs from satellite data and all kinds of sensors and he's doing predictions for sailors and he's one. And one of the things we are doing is providing him with these sensors so that he can do a much more accurate forecast for the Bay Area. So one of the things we learned about building these systems one, the accuracy of our sensors and this kind of we have worked on this for about a year and a half. It's only between one and 5% but these sensors here, the kite sensor costed in materials maybe $300. This one a little bit more fancy maybe a thousand bucks. The Met Tower is $15,000 to $80,000 if you wanna measure the middle of the bay. Those are actual quotes we got of the installed cost of these systems. So that is one important piece when you think about data and grid is sensing. So as I said the next question you can ask is about aggregation. So a clever observation is that I can add a bunch of wind farms spatially separated and I start to smooth out the wind profiles. Therefore I'm smoothing out the forecast errors. What is the challenge when I do that? So we took data from the PJM market and we just plot kind of the supply curve it's turned around here. So in this axis I have price of electricity in one of the buses in the market and here I have the fraction of the total demand. So you can think of average price and total demand. And you can notice a very interesting fact. Your last 20% of demand takes your price from $20 per megawatt to $60. And we have verified this as well in ERCOT and Kaizo. And if you think about it, wind let's say you're in a situation of 20% wind that's 20% of the demand. And if they all aggregate together they can game the market by deciding I'm not going to supply all the wind I produce. So I can get more profits off of decreasing the supply because wind is free. So if it's 20% wind typically my price would have been here. So maybe I want to provide a little less and put it there. The challenge though when they do that is that for themselves they have a benefit from aggregation which is coming through the forecast. So we did a study and unfortunately the graph here is not showing correctly. But what we found is there is actually an optimal group size that you should allow which balances in one side the benefit of having spatially aggregated wind farms and in the other side it balances out this market power that they can have. And to design this it was a combination of data from actual markets, data from wind farms and we had to do some analysis based on games and economics, okay. So it was an interesting insight and it tells you something about how do you manage the system with groups based on data. So now I want to do jump into the demand side. As I said the first thing about demand is we need to start understanding consumers at a higher resolution like Balaji was doing for travelers. So instead of understanding the big system you're starting to see individual buses and individual passengers and so on. And here one of the things was we went out there and learned okay how are these individual consumers understanding built today? Well it's built kind of like this. They take a small set maybe about a thousand customers, your utility and let's say you want to study the impact of a program or understand demand response potential. So you take select a thousand people, you flood them with questionnaires. I saw one of them had something like 400 or 600 questions so I would not answer that. And then you say okay for these thousand people I apply a program, whatever I want to do. Maybe it's pricing, maybe I ask you to come early home or maybe I say take a vacation and you can measure the impact of that. You have a before and after and you can do a lot of different things with that. You build a response model for that population, you group them together to say these similar characteristics of customers allows me to predict the response and I can even start selecting customers into whether I should make this offer for them or not that's called targeting or even scheduling when they should do certain things. The problem with this though, thousand customers not representative for a million when you're looking at the smaller customers. You're not talking about large campuses where there is a very good understanding of loads and demands and things are much more predictable. But you have an alternative. What is the alternative? Well at least in the US we have smart meters which means we can take a million customers, have the sensor ask the questions basically because it's measuring what you're doing in your home and you're not actually seeing all your actions but you do see the reflection of your actions and based on that we can learn some features which are trying to understand what is your behavior given your data and I'll show you in one example of a feature right after the slide. Now using these features which were built across this large number of customers I can do various things. Before the program itself I can postulate the response model after the program and I can learn that model. I can do segmentation, targeting, et cetera. The difference, I have a much larger customer base. So that is the idea we had, we said okay maybe you don't need to answer a thousand questions and obviously here I had before the program you run blind but once you get the results you can update your model. We have implemented a lot of these things as a set of tools and as well as a website that we are building now and if you're interested in that you can contact Chinwoo and we will make all of that available. So it is really to promote the understanding of this in the broader community. So methods are not hidden, they're open. Okay so here is a simple behavior model. This was one of the first ones that we built. We just took your consumption patterns every day normalized out the total energy and I'm saying this reflects what you did on that particular day and I asked the question how many such patterns I observed in a very large data set. So we got a hundred thousand customers but now we have maybe six times that number but and that corresponded to 66 million load patterns over a year, okay. And we found you only needed about 270 to capture these behaviors within a certain error of course but that was amazing to me that in this big population you only needed 270. First thing that we also did was I took the average of all of this for a day and they all look like this if I take the whole of neighborhood it looks typically like that, a double peak which is also the model used on every planning of the utility and based on this model you can say things like well there is no opportunity for any demand management during the peak time on a household because they're not consuming any electricity, okay. But in fact when you look here you can see there are guys consuming during the peak hours there are guys who only consume at night maybe graduate students at home at the time and there is even homes that are kind of flat turned off. I asked the question of how many of these homes or of these load shapes actually follow that? So if you think of this as a behavior so you think a hundred percent does it and I said okay maybe that was based you know in ideas that are somewhat old maybe 40% or 50% will do it. Surprisingly the number was 14%. So the data is telling you a story that was not there before. Now the question is what can I do with this? One of the things I already gave as an example was use these load shapes as a simple example and identify folks who consume during the peak and you can incentivize them using the idea biology hat or you can do different programs that would work for them during that time. But you can also ask questions about how reliable this customer is. Does he consume many different load shapes or is he more I don't know like a German guy every day doing the same thing one load shape fellow. So unlike me I'm from Brazil maybe 10 load shapes. So different people might be suitable for different programs. Well there is another side of the coin here which is if I want to start designing programs and optimizing I need to do some type of predictions. And I put one of my graduate students through the task of build a predictor for all of these hundred thousand customers you know it's kind of a fun project they have machine learning classes and all that. And I thought okay we are going to be able to meet the standard of literature which was about 70, 50 to 70% error on your hour ahead prediction for an individual household. He spent some time on that, couldn't beat that and we felt frustrated because this number is so high. Here's a feeder that's about 5,000 customers or more 100 megawatts so the error there is below 1%. And the question that popped up was okay I had the weather data I really used all pieces of data available. Information from the map, characteristics of the customers for whoever we had questions we tried everything. And then I realized maybe there is some trade off here and actually I didn't realize my student gave up on the forecasting and said no I'm going to do something different. Let me actually try to predict sums of these customers, groups of these customers. Let's find what is the curve here that links these two? So a single home is here, your Starbucks or building from Stanford is sitting somewhere here 10 kilowatt hours in an hour average. And here's the curve. And the surprising thing for me was well if I go around 500 kilowatt hours average I'm getting about 3% error. And the predictor I used here is not a fancy neural net or anything like that. Just a standard AR model with seasonal effects that was taken from an R library. So it's not that we are inventing a new forecaster but we saw that curve which made me think first of all maybe you don't need to manage this big grid in all of these huge parcels of 20,000 customers or 100,000 customers. But also that whenever I design a program pricing or demand response or something like that I cannot do it for an individual but I can do it for a small group. And what does that mean? What is the definition of a small group? There is many definitions. Maybe people who are geographically close or maybe you can create segments of people whose load shapes are similar. So that is a small group of folks who behave similarly. And you can predict them very accurately. So that is the idea here that segmentation is important. One fun exercise we did was we found hundreds of papers doing forecasting and we created a plot where we just plotted the performance of all these papers against this curve here. This axis was the log of the size of the data set they used. Here's the error performance. We had a non-fancy forecaster. These papers have very fancy methods and you can see it's pretty much very close to that curve. And in these papers you always see a tendency for people to say, well, here I spent all this time I improved over this guy because I used a really powerful neural network. But maybe it is just an effect of the behavior. This is why it's so important to understand behavior. One of the things now, yeah, I only have two more slides. So the last thing that we wanted to do was, okay, I have these ideas. What do we do? Well, we came up with algorithms that use these two contrasting things. One side, I have your individual behavior, which is very unique. I need to incentivize good guys and convince bad guys to change in some way. And the other side, unfortunately for you and for me, I can't really predict what you're gonna do if I look at you alone. So these algorithms always try to balance the group predictive performance and the individuality of what the group is. One idea is to do, for example, pricing. So we design optimal rates for places like Texas where you can offer retail rates for customers. And these optimal rates have a very nice property. They have this property that any group here, so we take this group one is a subgroup of customers that was selected from this very large set. They will never find a better rate no matter what is done. How do we do this design? It's an algorithm that takes as inputs all the market prices, your data, weather, and prepares forecasts and then optimizes the combinations of customers so that the cost of purchasing electricity for them through the market is cheapest, okay? So you can design these groups and form portfolios. Maybe you are selling energy to them so you have some risk. The other thing you can do is actually target people with programs so we have algorithms that if you specify what you want to do, the algorithm goes there and finds the combination that achieves a certain reliability and it can give you curves like this one. You can extract this many kilowatt hours from this program with this many users. For all of these algorithms, we always seek a few things mostly because we are interested in understanding. One is first, of course, they need to scale, but you want to understand the performance of the algorithm because typically they have some approximations and we want to have some type of economic guarantees, some type of customer who will not gain or they will not find a better price or things like that, okay? And just to finish here, in the operations side, I think there's a lot of questions as well because of this increased variability, you want to be able to place sensors in the system, learn the system state, identify topology, detect faults optimally and do optimal control. We have done some algorithms that address some of these issues and presented and so on and they are very applied. Just wanted to plug in one last thing. To do some of our tests with this last part, we developed a freeware that is available online that's basically an open sensor network that's completely end to end. So you go, there's this website from SparkFun or so on, you buy the sensors, download the software, turn the sensors on, your data appears on Amazon Cloud and you can have hundreds of these sensors spread around. And my idea here was to use a lot of the things I used in my other life in startups and so on, building hardware to make something really accessible to researchers and academics to do real experiments. We're always saying we don't have access to data. So all you need to do, send an email to EZeng and he can give you this. Okay, thank you.