 Hi everyone, welcome to the first smart resummon of this quarter. Our speaker today is Professor John Burge from the University of Chicago. And you talk about how to increase electricity market efficiency with flexible mechanisms. And before we start, I would like to go through some of these basic instructions for this webinar, which I think are particularly useful for those who are attending this seminar for the first time. Everyone will be muted upon entry to reduce background noise. And there are a few features on your screen that you can use to communicate with the speaker and the panel. There's this check feature. If you have any, if you encounter any technical difficulties such as you cannot hear the speaker, you can use that to let us know. And there is this raise hand feature. You can use that to, if you need a simple question to be clarified. For example, a definition or something like that. And if you have any in depth questions, you should save it to the user Q&A feature. And those questions will be answered after the end of the seminar. Now, I would like to know this is a schedule for this quarter. There will be five seminars. And I would like to remind you that the next seminar is in two weeks. And the speaker is Manini Bhattanu from VMware. Charlie, I will let you introduce the speaker. Thank you very much, Chen Wu. My name is Charlie Colstead. I'm the faculty co-director of Bits and Watts and welcome to the smart grid seminar for the fall. It's my pleasure today and privileged to introduce a distinguished Stanford alumnus, Professor John Burge of the University of Chicago. John's specialty is mathematical modeling of systems under uncertainty, particularly stochastic programming and large scale optimization. This is an area relevant to my research in economics and highly relevant from so many modern problems. As signs of his stature in the profession, John is editor-in-chief of the journal Operations Research and was elected to the National Academy of Engineering. John received his PhD in 1980 from the legendary Stanford Operations Research Department, which is now part of MS&E. He went on to a career at the University of Michigan, Northwestern and now the University of Chicago. Northwestern, he was dean of the School of Engineering and Applied Sciences. We're lucky to have John Burge with us, sort of with us today speaking on the highly relevant topic of increasing electricity market efficiency with the flexible mechanisms. John, I'll turn things over to you. Great. Thanks, Charlie. Okay. Now let me share my screen. Great. Well, thank you, Charlie, for the introduction and for inviting me. It's great to be virtually at Stanford. I usually like to, when I go to Stanford, I usually like to go for a run around the dish. So I didn't get a chance to do that today. But I thought about it while I was doing my morning job. I want to thank you all for coming. What I'll be talking about is increasing market efficiency with different kinds of mechanisms, so called flexible mechanisms. You can think of this as being somewhat synonymous with increasing renewable penetration. For the most part, renewables have very low marginal costs, and I'll be basically talking about marginal costs. So when I talk about efficiency, even though it's minimizing costs or minimizing resources, you can also think about it in terms of renewables and I'll mention that as we go through. So this is joint work with colleagues at Wisconsin as well as also at the University of Chicago, Bernie Lissuter, Lena Rolt and Victor Zavala in Madison and then Andrew Chen, my colleague in computer science and Arnab Sir, who's postdoc with me. Okay, so the big theme of what I'm going to talk about is that electricity, forward and spot markets are very complicated things. There are things like startup costs and minimum run levels and times, all of which make operating these markets quite complex. There's also a great deal of uncertainty. There's uncertainty and demand, for example, it depends a lot on the weather. There's also uncertainty in supply and particularly with renewables, wind and solar and in terms of how much they're going to be providing. And the way markets are set up now, there are inherent inefficiencies and that uncertainty in the supply and the intermittency makes those inefficiencies, I think even more salient and perhaps even greater. But there are ways in which you can redesign these markets that can increase the consistency, making incentives for efficiency, and that you can do that by allowing some kinds of flexibility so that's the basic bottom line of what I want to get to. So what I'll do is I'll go through talk a little bit about renewable sources and their impact. I'll give you, if you're not familiar with the way most of the markets in the world work, I'll give you a little bit of an overview of how these markets work. And then I'll talk about some of the issues in terms of market power, which you can think of as not being efficient issues that arise because of these fixed costs which create non convexities, which markets have some difficulty dealing with. And then a particular property of the way markets are operated today, which means that you can never match both prices and quantities in the right way, the way you'd like to. And that there lack, there are, there is a lack of incentives for things that might be good for the market like pooling or like relocating to me. And so I'll talk about ways to resolve it. As Charlie mentioned my introduction level what I do is is stochastic optimizations because of programming. So it's not surprising that I'll talk about how stochastic programming can help resolving these. But it requires some additional mechanisms and I'll talk about those in terms of flexible bids and integration of markets in particular here I mean the market for energy and the market for transmission. Okay, so you've probably all seen something about the growth of wind and solar. Probably around 1500 gigawatts globally right now. To put that in perspective the US has capacity of around 1100 gigawatts. So, the amount of global wind and solar power is effectively greater than total us electric power generation capability. It's been predominantly wind, but solar has been increasing at a very high rate somewhat recently, and now wind and solar globally have about the same installed capacity. And as I mentioned it's highly variable. I just took a snapshot here from my so, which is the mid continent independent system operator. Which operates in the central part of North America from Manitoba at the north to Louisiana in the south. And my so has a lot of wind capacity. But it's highly variable. So this is the snapshot several years ago it's even more so now since the my so has increased its wind generation capabilities, quite significantly. But you see fluctuations that are basically, you know, from less than 1000 megawatts available to 10,000. So, and in very short amounts of time these are ours down here. So in very, very short amounts of time. You see wide fluctuations in terms of wind variability. Solar same thing. It's variable. Obviously, there's not too much solar production at night. But even during the day, there's there can be very high variability in total solar availability, depending on particular weather conditions. So, so solar, solar and wind both have high variability. That sort of led in California to this sort of the famous duck curve and I took an example from just August of this year when when you had I think the highest net demand for this year although it's actually much lower than it was several years ago. But here's that this the green curve is the total net demand, and the purple curve down here, or the total demand is is is what's here in green. And then the lower curve is the net demand after I take out the solar generation. And, as you would imagine a solar generation goes when the sun sets. Here's some time around seven o'clock. It looks like was around seven or eight o'clock on that day. So we have to match. And I guess there's a little bit of winds and probably have some wind in here as well. So, what that means is there has to be this ramping up of generation in order to be able to match these, these two curves at this point in time once the sun sets. That's what that required a ramping rate of 9000 megawatts in three hours here on that day in August. So it's it's a big challenge. It's even more of a challenge because this is an aggregate. But, but the grid actually is a network. And that means that in some places the grid we have to ramp up very very rapidly but there's, there may not be any generation capability there we may not be able to get the generation from one spot to another. And that can lead to really wide variation in the prices across different regions. And I'll show some examples here just these are examples actually just from today, I thought I'd just take a snapshot. This is the price of electricity in them, and two parts the myso grid. Here this is, this is Iowa and Illinois and Missouri. And here this line the border between looks like Minneapolis or in Minnesota and Wisconsin. And you see wide variation in prices in very close proximity locations. So here, for example, and this this was persisting a lot today I'm not sure exactly why, but here this dislocation had a price of $105 per megawatt hour. And the dislocation over here had a price of minus $30 per megawatt hour. I wonder why, why is electricity selling at a price of minus $30 per megawatt hour. Well, $30 per megawatt hour is about what is earned on the production tax regulation that allows you to recoup taxes by providing wind power. And so by taking that production tax credit, which amounts to about $30 megawatt hour, it's still profitable to be producing, even though there's actually no place for the generation to go. So these wind farms, there's a wind farm located here. These wind farms are generating power just to get the production tax credit. It's not productive in any means. I mean, you could earn money just by parking a truck full of electric toasters here and getting paid $3 per megawatt hour for what you were producing or for where you were consuming. And that's not a very effective use. Now I'm not going to talk about that specifically, but what I would like to talk about are ways in which we can try to eliminate this kind of inefficiency. So that, you know, we don't have circumstances like this where a distance of only about 20 miles has price differences. There are more than $100 megawatt hour and and we have either what we call stranded power or negative prices. Either you could just sort of spill the wind, not not generate any power or you generate power and have negative prices. And that's a function of the network. So the question is, how can we come up with some mechanisms. But before I talk about the mechanisms to try to address these problems I thought I'd go through and talk a little bit about how these markets are constructed. Let's talk about this if you know everything about the electricity market. I apologize, but I'll try to go through and give everybody someone of an introduction. So in most places, it's about 60% of North America. Most of Europe. Many other countries around the world, South America, Asia, Australia, New Zealand, have something called an independent system operator. That example of those prices was this independent system operator called MISA or mid-continent. They coordinate the use of the transmission system, basically the use of the network. And they provide service to the customers, customers being both consumers as well as industrial customers and then securing the power from the generators. So they run this wholesale market for energy for distributors usually here buying electricity from the generators, but going through the independent system operator as operating the market. I'll talk a little bit more about the role of transmission in this but basically the independent system operator is organizing the use of these transmission lines and deciding what generators are going to be generating and how much and ensuring that there's enough generation for each of the consumers that's associated with the network. Okay, so the participants in the network are in the market are the generators and what in electricity is usually called the load entities or the demand. Those would be the consumers or the distributors. And there are also financial participants and the financial participants are there as a means to try to lessen the potential market power that either generators or the load entities might try to exercise. The way the market works is that the buyers and sellers submit bids that the bid is both a quantity and a price, either to buy or sell depending on which side of the market you're on at particular notes or locations along the network. And the independent system operator clears the market by considering these bids but also considering the capacity of the transmission network, as well as the energy that's lost in transmission. The energy loss and transmission is not very great, but the independent system operator still has to consider that and basically balancing the market. But because of the transmission network, it's, it's not just a matter of summing up the supply and demand make sure we're finding out where they cross. They have to solve an optimization model actually to make these determinations because of all the constraints from the transmission. And then they come up with prices. The prices. If you're familiar with these authorization models, they're basically the grunge multipliers in the authorization model. And they are the marginal costs of supplying electricity. An additional amount of electricity at each location. As I said, and my zone particular the many transmission lines are congested and you get these, these wide variations in prices. So the, the way the market works is there, there are two instantiations there's a forward market which is called the day ahead market, because operates 24 hours in advance. And that schedules everything and it basically tells the generators, oh, we want you to produce tomorrow. So what that means is the generators are then able to start up their plants since many of the plants requires some time before the thermal generators in particular, requires some time to be able to actually get to full generation capacity. And then in the real time market they make adjustments. So there's a real time market to make adjustments. And, and then the day ahead market to make commitments. It's also called unit commitment. Ideally, what should happen is that the day ahead market has prices that are the expected value of what you would see in the real time market. And that the only thing the real time market is doing is adjusting in real time. The idea of having the day ahead market is because it's cheaper if you've planned exactly when these plants are going to be operating that it's less expensive if you can plan on it in advance. But because of the network, the congestion in the network actually creates areas in which a single producer or a single consumer. But in particular in these networks, producers, a single producer actually can exercise market power and affect what the prices. And in fact, there's a big incentive for generators to withhold the amount that they put into that market. In the day and relative to the real time I'll show you basically what that means. So here's the quantity say that they're offering that's on this y axis. And here might be the price that they would receive in the market. And this, what I call the residual demand curve here, this this line. They call the residual demand curve represents what price they would receive if they produced or offered a given quantity. There's one incentive to offer a lower quantity. So this, this red line here first month's a lower quantity in the day ahead market. And then what they'll receive in the day ahead market is this price that's here so they'll receive a high, high price in the day ahead market. However, they're actually able to produce a lot more. So by offering this, this smaller quantity in the day ahead market. And what they can do is then later in the real time market. They offer a higher quantity. And now they'll receive this different price in the real time market. They received in the, in the day ahead market. So by offering this additional, this additional quantity, they'll receive this price, they'll be able to get this amount of revenue. So they'll get their revenue from day at market plus extra revenue from the real time market. So there's an incentive again for them to withhold in the day ahead market, get a higher price and then have higher quantities in the real time market when the real time market has lower prices. Now, we might, you might wonder, well, can that really process if everyone's, if they're not able to exercise market power, then this disadvantage should go away and these two prices should end up the same. But what we see is actually they, they don't. This is the cumulative difference between day ahead price and real time price in the my so market. Again, this was some years ago, before they changed their, their market rules. But effectively what this says is, if you sold power in the day ahead market, got the high price, and bought it back in the real time market, you would make the difference in these two between those two prices. And your portfolio would be growing like this curve. This, if this was something like a stock investment, I think it has a sharp ratio of something like 150 where the market generally has a sharp ratio of 0.4. So this, this was, it looks like kind of a no lose bet if you could enter into these markets. In fact, there was a way to enter in these markets, they allowed financial participation. So, a pure financial participant in this market should have been able to realize a curve like this and as I said, that's like an investment with this enormous return relative to very little risk. So that seems like almost, well, it seems like too good to be true. If this money is sort of sitting on the table, why aren't people entering into this market. Yeah, so as I said that what you might expect is maybe that the consumer side that the demand side should be adjusting their load but they're often, they're often regulated. And they have steep penalties if they don't meet the demand. In other words, they're representing consumers like retail consumers. And there may be very high, high penalties if they're not able to meet the demand. So, so there's little fact in terms of actual physical adjustments in terms of demand. But as I said, there are these financial bidders, they could enter the market either on the demand or the supply side. And it seems like what they should be doing is entering on the supply side. In other words, trying to sell electricity at the high price in the day had market, and then buying it back in the real time market and making that difference. As I showed in that curve. However, I'll just mention this now and then then I'll talk about why it's problematic. Not only can they enter the energy market, there's another market for transmission and the transmission market behaves a little bit differently. But let's look at what actually happened with these markets. So here are the number of bids that were placed by virtual bidders by these financial traders, either on the supply side, or on the demand side. So the blue dots correspond to the number of bids on the supply side. And the red dots correspond to the amounts that are bid on the demand side over this over that same period of time. So as I said what what we might think should happen is that people should be bidding supply. They're going to trade in the day had market and then buying back in the real time market, but instead they're doing the opposite. The red dots are a lot higher than the, than the blue dots. It looks like the financial participants are doing the opposite of what would be expected to try to make this market more efficient and decrease the market power of these generators. Okay, so. So the first thing is to try to understand why, why this might be going on. And the least, to some extent, I think it's explained by what are called financial transmission right so as I said there's this other market for transmission. And it's usually represented here by these financial transmission rights and what this financial transmission right, let's say for a line that can has a capacity of 100 megawatts. And when you, when you own that right or when, when you buy it, then you're paid the difference in the prices between the two locations because the my so is collecting additional funds in place a but it's paying for it maybe through the transmission in place B. So my so is collecting an additional amount, and this infant system operators, and this owning the transmission rights means that you receive that that difference. So let's let's imagine a situation in which the transmission is binding and let's say that prices are in a or $20 megawatt hour and in be their $30 megawatt hour. So we expect the real time prices to be lower. So we would expect someone to try to offer additional supply here to bring price down to try to take advantage of that, let's say that $30 to $28 swing in place be but instead what happens is we see an additional demand bed at, at be so in other words someone says well I'd like to purchase additional power at be was you purchase additional power that's going to require another resource which is more expensive. And that raises the price. Let's say that raised the price from $30 megawatt hour to $40 megawatt hour. What that means is the owner of the financial transmission right now earns an additional $40 at be compared to $30 at be, and they earn it on all 100 of the megawatts of transmission that they have the right for. So they'll earn it an additional $1000. That bed only loses on this one megawatt. What you lose on the bed is the difference between the real time price and the day ahead price. So they lose $12, but they gain $1000. So, there's a huge incentive then for the virtual traders. To try to manipulate the energy market, because of their investments that they have in the transmission. So, the fact that these marks are not integrated is, is again, one of the issues that creates some kind of inefficiency. The other thing that happens is this, what what can happen because of that is that in the day and mark we have commitment and the commitment involves minimum run times. And having to deal with turning units on and off. So you'd like to run that that initial auction in such a way that that's done in an optimal manner. But if there are these additional forces that are increasing demand artificially say that can lead to inefficiency by having more generators on. The other thing it leads to is non convexity and I'll just mention that briefly. And this is a little example of say three generators. Here's generator one every time I turn on it costs 50 this one costs 300 this one costs 100. These two have some minimum one capability so many many thermal generators have to run it at some minimum level. And then different costs that are associated with it. Because of that, you're showing, as you increase the amount of generation. What's the total cost and then what's the marginal cost. And what happens is that the marginal cost actually changes substantially. So if I marginal cost that low as another generator comes on, then maybe it goes back to high for the next one, then low again as as an additional generator comes on. And then it has these periods in which it can go from very low to very high and back again there are ways in which we might be able to get around it I've sort of drawn in possible way here. In general, this has these these non monotonic issues, which also can create inefficiencies. In this case, the marginal cost themselves might not cover all of the cause for production and so there have to be some sort of charges that are spread through the market and so the market has to adjust by having additional charges for everybody who's generating. And then there's additional incentives for for generators to exercise market power to to reduce their outputs that they can increase their marginal costs and increase and increase the marginal costs and then increase the amount that they receive from the market. Now the other issue which I want to try to deal with is that there is a lot of uncertainty associated with supplies I said is also associated with demand. So if I think of the residual demand is like the load minus whatever is provided by the renewables. There's there's absolutely no way that this day ahead market that's just based on bids and then run like traditional auction assuming that production will be exactly equal to those quantities. There's no way that that market actually can reproduce what will happen in terms of both price and quantity. Because the price the marginal cost basically is convex it's it's increasing as the generation or the output increases. That means that if I look at the price as a function of the quantity it's this convex curve here. And then I fix the quantity and look at the expected price is going to be higher than whatever is right along the curve. So the, the way the market is cleared is going to be along, let's say that blue line. But the expectation is somewhere inside. It's inside the convex hall of that blue line. It's somewhere inside. And so if I try to match the quantity, the expected price is always going to be higher than whatever I have in, let's say the day ahead market. If I try to match the expected price, then the quantity will be higher than whatever I have in the day ahead market. These, these markets actually can never match both the price and quantity. And the main reason for having them was that we wanted to have sufficient quantity. If we end up matching on price well we'll have sufficient but we're actually going to be paying more than than we actually need. If I match in terms of quantity, then the price is going to be lower than, than what it should be. And that's not going to get enough people to supply. So there's, there's additional kinds of inefficiency because of this uncertainty. I'm just repeating what I said the nonlinearity and the cost implies that matching these expected price and quantities is just not possible in these deterministic models can't do it. So we can include that. And as I said, that's the kind of thing that I've done in my work. Actually run the process assuming that there are generators like wind and solar which are going to be random and assuming that I can represent that randomness effectively than that we can run. We can run that auction with this uncertainty actually included into it. And then the other thing is to allow some kind of flexibility. Particularly for certain kinds of loads, such as data centers. So date data centers are actually represent a large amount of load and someplace that can be about 5% globally it's maybe one or 2% of all electric power consumption. And that basically can be done anywhere. A lot of it for example gets done above the Arctic Circle. Particularly for mining crypto currencies. But, but there are many data centers that are performing tasks for like Google and Amazon or Facebook that are located in the main grid. But those tasks can be performed virtually anywhere and sometimes they can be performed at different times. So the load can be shifted. So allowing flexibility and shifting that load actually can improve these markets substantially as well. And then the third point is to integrate it so that we don't have that incentive to try to manipulate one market to receive additional profits in another. Okay, so I'll just mention about the stochastic unit commitment. This is something I've worked on for quite a while. You can solve these models somewhat efficiently now, at least to the extent that could be used in these bidding processes. We have required generation of many scenarios. We can fairly robust ways to generate scenarios not exactly it's hard to predict wind in a very specific location where there's a wind turbine. But, but we can generate many scenarios of what the wind might be in different locations. So this is the way these models are set up. I minimize the total costs, includes the fuel, the startup, any penalties, I'm not able to actually meet the load. I balance the low balance it or make sure that I don't exceed transmission. I respect the physics and make sure that I stay within the limits of all the different generations. This is mathematically what it would look like. Here's some wind scenarios as I'll just show you this. This is for a small example using one of the toy grids. That's available. This is for the wind generators and then allowing randomness in the wind that's being generated. And then looking at the effect. I'm going to look at the effect actually of doing this two ways. One is how much is it worth if I was able to precisely predict the actual wind generation. But it's worth something in this little model. It's sort of 500 out of a total of 60,000. It's not very much about what less than 1%. But it's certainly certainly a value to know something about the future. But what's interesting is this other concept which I have coined as the value of a stochastic solution, which is how much is it worth to solve this stochastic model versus the deterministic model that's used today. And that value is actually a little bit more than 1%. It's something like when her close to 1.5%. So that's, that's actually quite substantial if you could save 1.5% just doing this. That certainly that's it's worth the computational cost. So what one of the efficiencies could just be just including this model just what how much is the model worth that's it's worth substantially. What happens if I also include these, these flexible loads. And in this case, let's say I have this data center, and it could bid and one of two locations either a or B, let's say there's a load and a load and be there's a transmission capacity between a and B. So I'll allow the data provider to say, Well, I'd like to get this served this delta, but you, the ISO can determine where to place that place that book. So it could either be over here so if the wind let's say the wind is concentrated in a but as higher costs, or it can be over here. So that load actually to be flexible. Then we can get sort of substantial savings. So the way I've worked it out here. If I did the fixed bids, this would be the cost. If I include uncertainty and include the flexible bids, then I get this cost here the gain is I don't need to turn on that expensive plant that was in part a I get some marginal advantage over whatever the minimum run capacity was for the generator in region a and I get additional amounts of wind that I couldn't have used before because I was limited by the transmission. And as well as having already this minimum run capability for the generator in a. So the load at a can can help share the savings and actually it reduces it reduces costs but it also reduces carbon emissions because I'm able to use up all the wind whereas I wouldn't have been able to use all the wind. If I didn't have this. So, these incentives can include some kind of sharing so that there's an incentive there's an additional incentive for like the data center providers to offer flexible loads. The uncertainty. We can can try to ensure that those resources again don't don't try to exercise market power in any way by saying that they face penalties for example then have some charges if they are not able to produce what they what they bid into the market. And I could also include similar charges for those who enter into this market and impose some kind of additional costs on the financial transmission market. So by including surcharges for those as well. I can, I can even increase the amount that I get in terms of efficiency. So there are many things about making this actually work in practice. Particularly if if a generator like one generator has additional information that the ISO doesn't have. How can we ensure that it's incentive compatible for the generator to provide that information. How can we determine exactly how these, these savings might be shared by having these flexible bids, how this could even be greater if you could share across ISOs, and particularly because a lot of these data centers. They don't, they're not physically restricted really in any manner. And so you, if you were able to go across ISOs throughout North America or not even in North America outside. How would you be able to share to make this even more capable. And how can we test this, but so I haven't, haven't done empirical testing. That's one of the things I'm quite interested in, in trying to be able to do. So, to summarize, electricity markets present a lot of challenges. The current market designs create some efficient inefficiencies, and by explicitly including uncertainty and these flexible bids and integrated markets. These inefficiencies can be eliminated and we can increase the use of renewable resources. Okay, so thank you for your attention and hopefully I'll have time for a couple of questions. So put, put your questions in the Q&A. There is already one question there. Yeah, can you see, John, can you see it? I do. Can you see it, John? Yes, okay, it's when you discuss the effects of transmission cans constraints. So it's your slide we have the negative $50 for wind or never negative 30 next to $100. Is this a situation where there's no transmission line? Yeah, so, so effectively, there is transmission. The promise it's all congested. So, so what's happening is where the electricity is being generated. It's already using up all of the transmission, and you're not able to get additional power into that location where the price was spiking more than $100 a megawatt hour. Okay, let's see. Oh, what would be the impact. Yeah, I, I think it's possible. Like my, my little toy example was something like, you know, one 1%. I think the total cost impact could be 2%, which is something in excess of $100 billion a year. So, yeah, and that will be in the United States. So, yeah, the impact I think could be something like 2%, maybe of total electric power cost. And, but a bigger impact in terms of renewables. That's something on the order of 10%, maybe greater renewable usage. Okay, next was, don't we expect a day ahead real time premium due to risk aversion. Yeah. So, there is, there is a little bit of risk aversion and actually when I started to look at that. I thought, oh, that must be yet. It must be risk aversion that's, that's causing this, but the differences are so huge it was $2 over a day. And to have that level of risk aversion. Like 10%. It just didn't make it, it didn't make any sense at all. So it, it really can't be easily explained by risk aversion. So the fundamental role of FDR is hedging against congestion. That people often lose on FDR doesn't seem surprising. Can you explain how this is evidence of market power. So the evidence of market power is more about about how those the day ahead prices are being pushed in the opposite direction from real time prices. And, and how that's benefiting the people with financial transmission rights. We in a study, I have a paper that talks about this with Ignacio Mercadal, Holly Hortaciu and Mike Pavlin. And what we found and looking at the myso data are some of these examples where actually the Federal Energy Regulatory Commission for actually did discover that people were doing this deliberately. So that paper goes into about kind of the evidence for this. And the model. Okay, so that the, the generation scenarios that I showed. Those are, those were predictions from models from Argonne. Some of my colleagues at Argonne National Laboratories. Now they're very precise so they, they will model when conditions within, you know, sort of the few square meter kind of areas so that they can get precise when condition predictions at different points in time. Exactly at a wind turbine. And then they generate these, these scenarios. So, so those were scenarios that were generated by that, by that Argonne model. After you are be synthesized. You could the by buying and selling power in different locations with virtual bits and thank you Bob for the question. So, yeah, that's, yeah, that it's possible but you have to you'd have to leverage to the, to the total capacity of the FTR, which is, which is, what to do so it, it's because the price is changing in a nonlinear manner. That's that's a little bit harder to do. And then the next one. If power generation forecast include information on their own uncertainty, could that be useful in classifying the magnitude of certainty. Yeah, so, yeah, that's kind of I was leading to at the end. How can you extract that uncertainty and what the wind generators might might be might be considering. I mean they they know something about local conditions that that the ISO might might not know. That could be a value so yes that could help in terms of in terms of classification here. Okay. I don't know if we have a the ability to have an applause can to pause here but we should. Thank you very much. Thank you Charlie think thank you everyone for for having me. And addressing other issues. I don't have any other issues. Yeah, and I would like to thank the Professor Jones for this wonderful presentation. Does anyone have any last minute question. Thank you so much for taking time out of your day and being here to offer this great seminar. Thank you. Thanks for here. All right. Okay, thank you very much. Thank you everyone. Thank you.