 Alright, so we're almost at the end of the workshop. And so the goal of this final panel is to synthesize what we've learned over the last two days. So we have three distinguished panelists who will engage in discussion and for next steps to enhance the corporation of climate and to macroeconomic modeling. Hi, we're going to start with some prepared. And then we won't open it up later and as usual, please use the online raise hand function to regions. So we're going to start with me. So can we get Do you have slides on me? Oh, okay, great. So I think people, well I would say been introduced, but Emmy Nakamura is the chancellor's professor of economics at Berkeley and her research focuses on monetary and fiscal policy business cycles and macroeconomic measurement. Thank you. Thank you for the great conference. I learned a lot. I wanted to go back for a moment to the first session just to sort of remind us all where we've been and where we're going. We've been thinking about macro policy models, the ones that are used by the government, for example, the CBO model, or the mouse model and other related sort of simple macro policy models. Their primary use case are things like forecasting growth, deficits, fiscal policy scenarios, things like this. They are a very parsimonious representation of the economy. The first order thing that's included is accounting identities of various kinds. Other parts of the model, you know, there, there are a lot of aspects which are, you know, simple statistical relationships. But as has been emphasized, many of these channels are very simple in nature. So, to a large extent these models take climate change science as inputs, as opposed to saying something substantive about, you know, what climate change will do. And what, you know, the framers of these models are asking for to some extent is how to best adapt their models to use them as an approximation for how climate change should inform their policy projections for these use cases such as forecasting deficits and things like that. One thing that I wanted to emphasize is that it's not just the models that matter in terms of the discussion that we just had there was a lot of emphasis on the notion of having new models and you know that's certainly a relevant thing to do. But I also wanted to emphasize that the parameters really matter. So, with these simple models, the outputs are really only as good as the inputs, and depending on how you choose the parameters you can really kind of get anything out of the sun. So, I think that's important to recognize that it's one thing to hope for having, you know, all the channels there. It's another thing to to actually have the right parameters for all these channels and it is a real challenge even taking climate out of the picture to know how various, you know, changes in the economy will will affect outcomes. You know macroeconomic forecasting is really difficult in this regard and I think the fundamental reason is that macroeconomics is really a strong case of small data. In the sense that we only have a very small number of recessions and recoveries and disinflationary episodes to learn from so we're often in a situation of trying to draw analogies between things that are not very really very similar. So like in US history, you know recent US history really really had one major disinflationary episode, we can look at other countries but naturally there are big differences versus these other countries. So it's a situation where where it's often a challenge, even if you think a channel exists to know how big it is, and that's even before you get to the climate so I think that's kind of important to be realistic about that. We can think of adding many different channels but in the history of macroeconomic forecasting. It's well understood that more complicated models often forecast much worse. And I think that's one of the tensions here that that we're kind of up against. I think that against this backdrop of the challenges of macroeconomic forecasting even before you get to the climate. I want to emphasize an important caveats that have come up. You know important things that are kind of missing from the macroeconomic models, or you know maybe not, maybe missing partly in emphasis. So to some extent economists know some of these limitations but perhaps we don't always emphasize them as much as we should. So one is the distinction between GDP and welfare that's clearly an important issue. A second thing is aggregate versus sexual or regional outcomes that's come up a number of times. There's the role of trade and why that means that you know things that happen to the rest of the world could also affect the United States. There's the short versus the long term. And in fact what even the words short and long term mean, which I think are actually quite different in the context of this macro forecasting world where long turn off means more than 10 years, which you know it's very different than the word is used in the climate literature. There's, you know, two way feedback versus only a one way causal chain for the climate effects so in these super simple models, there's there's no two way feedback, which is clearly a simplification. But you could, you know, the simplest way of introducing climate models is to have a one way causal chain from from some climate effect that's viewed as exogenous into the model. And then there's of course the role of uncertainty that many of the things we think about with regard to, to climate are, you know, bad case scenarios that we want to focus on even if they don't affect the mean very much and that's, that's not the way that these macro forecasts are currently done they tend to focus on some kind of baseline case and that may be very problematic for this particular application. And just to bring us back to sort of, you know, what, what is the sort of simple use case that one might imagine with these simple models, the Troika models. You know, think of them as a one sector model with a, you know, what I think the simple case would be a one way causal chain from climate into macro outcomes where climate enters through productivity. I think one question would be suppose you're going to use that model as an approximation clearly it's an approximation in all kinds of ways. What would be the best, you know, parameters to use is there sort of like a gold standard that we can take from the climate science literature for, for what parameters you should use. And, and, and for what parameters at what horizons and what sort of cases so presumably given that you know, risk and uncertainty is going to play a big role. Are there particular horizons that macro modelers should be focusing on when they're contemplating these effects. Are there particular time horizons that they should be focusing on and thinking about this, you know, these I think are some practical questions that might be asked by people trying to do this. You know, could there be several, you know, realistic scenarios that that should be considered. And finally, is productivity, the primary channel through one would, through which one we want to think about these effects because there are several different channels one could be given in this simplest of models, productivity, productivity is one of them so so these are some of the questions that sort of struck me as you know, we made some progress on and and hopefully, you know this this round table will continue to be a forum for us thinking about communicating on these on these fronts. So, our next respondent. James rising James is an assistant professor at the University of Delaware in the school of marine sciences and policy, who works on the impacts of climate change and a variety of forms bringing together empirical estimates and integrated assessment models models of socio environmental systems focusing on complex systems, food fisheries and resource management. James. Thanks so much for the pleasure to be here. And I think that one of the reasons I was invited was because I helped organize a workshop to dissimilar from this with the Royal Society in London, three months ago. That was called the new horizons for increasing the understanding of economic consequences of climate change so very much focused on the damages side of this but there's a lot of overlap between the discussions that are happening here and what we discussed there. So I'm going to be bringing in a bit of that. There's also one of the things I think about a lot is is what's the research agenda care moving forward. I think that the research agenda necessary for incorporating these climate feedbacks into macro economic models really needs to include our better understanding of both transition costs and damages. So I want to sort of open open that up and say, they're, they're still first order questions that we have about the scale of transition costs. So we have a lot of of what the scale of the costs that we're looking at are coming from optimized energy system models, or engineering estimates. There are massive amounts of emission reductions that are available in these models at negative or zero cost. As economists, we think that there probably isn't a free luncheon that if we want to take advantage of those emission reductions we're probably going to have to nudge or push or incentivize people to do it and figuring out what those inputs are is it's like I said a first order question on the damages side just thinking about the the GDP damages like you saw from Marshall Burke. There's a there's a very robust literature that's emerged from some of the work that the Marshalls contributed to Francis more had five different models that were incorporated into the the OMB model. These all have different assumptions on how temperature and precipitation are included different assumptions about the persistence of damages about differential vulnerability about the role of adaptation. And so these decisions actually have order of magnitude differences we don't know on an order of magnitude what the damages that we should be putting into the macro models are we don't we don't know even which are the most important channels when you think of a bottom up perspective that will affect the macro economy. If you look at a review of the different literature. We see that our culture is the most important some say that that health is the most important there there are many different things that come up as just the most important one, much less knowing where the rest, all lying there. We have this. Just to give a sense being formed by the world society event of the scale of the challenge for understanding some of these damages, we're talking about a globally correlated disruption with potentially large scale spill overs large scale. Discontinuities in the structure of the economic system, where both local impacts feeding up international impacts and global impacts in down international impacts are going to play a really important role. I think that do we after the panel that proposed that the general scale of these would be on the order of one 1.3% that's that sort of been a point of knowledge here. I think, I think that we should expect that that's a significant under estimate. Based on what I've done in the UK. The moment the include spill overs and central for large scale disruptions. You end up more look at more like, well, in the UK, it came out to 5.5%. I would expect the US to be at risk of higher percentage losses in 2050 than that and inequality, heterogeneity and temporal variability all matter. So, we, we don't have the transition cost inputs to the level that we need, we don't have the damage estimate inputs level that we need these are research dimensions, and I would say that we don't have models adequate to the task of properly capturing the feedback. loops that we need to understand how these damages and transition costs will interact. We don't have models that probably capture non stationarity and disruptions. We don't have a good connection between investment under uncertainty, which is a core question within all the climate economic literature, including and beauty and models fitness specification which Lars mentioned. Or the role again of heterogeneity and variability, which are going to have significant macro economic consequences. So, what do we do, knowing that that these are all things that that we expect to see in the future as we continue to work on this research. I think that there's, there's one point of light which is that this workshop is actually part of a global effort to update these models. I know that the World Bank is figuring out how to incorporate climate into its model and particular looking at structural changes. I know that the IMF is looking at how to incorporate climate into its models, looking at environmental capital and sustainability. The integrated system models of of pick and other energy system models are now trying to incorporate damages so they can have both sides of this equation. There's, there's an incredible amount of really exciting work going on right now that they think this workshop should see itself as as being part of my last. I mean, let me make two more quick comments and then we'll finish. The first another point from the row society event is that GP is likely inadequate even for evidence metric for the macro economic that these these models need to inform. Certainly it's a it's a poor incomplete metric, but that's not that's actually my focus. The GDP is we know is a is a flow based on various underlying stocks including natural stocks, including wealth stocks and thinking about the inclusive wealth natural stock features that result in GDP. I think is is going to be a feature of some of these future models. And also, we tend to think within the macro context that mainly we just have to worry about the market scale, the market impacts of climate damages, but at the scale that the non market impacts are likely to have the scale that increased mortality risk is expected to have or or or labor to see totally is expected to have those are going to have significant market consequences there's going to be a strong channel between non market impacts and market impacts into things like GDP and tax receipts. The work from the lab on adaptation costs mortality shows this. There's some interesting work I saw recently on the mental health consequences of climate change where 45% of young people say that they are impaired because of the mental health consequences of climate change. I don't know quite how to incorporate that kind of information but I think that that we need to engage with it. Just a quick list of the top line takeaways from the world society meaning which I think I've mostly already mentioned. There was interdisciplinarity in these conversations things incredibly important. Actually, it ended up being quite contentious in our meeting. There's a lot more shouting each other there than there was here. But we still thought it was a good idea. And in particular, on the mitigation side also learning and the role of changing development and so at the industry and sector level was important ethics, including inequality and on the mitigation side justice matters. The future is not going to look the same as the past is was was sort of a core message from that meaning that stream events and tipping points are an important feature of where we need to be looking at for research, and that migration and replacement and climate induced conflict are sort of going to be shaping damages in the future are all things that we need to grapple with and and really are just starting to. Thanks. Thanks James. Our final analyst is Chris. Who is the co head of us economics at H market, which is part of SMP global and has nearly 40 years of experience in macro modeling forecasting and policy and elements has co head of US economics and IHS market and as previous roles principle macro economic advisors, which we learned as a key model within the framework and a member of the staff at CA in 1981. Thanks very much. It, I'm honored to be here. I feel like one of the kids who finally was invited to sit at the adults Taylor Thanksgiving. So kudos to the folks who can see the need for this meeting, and I think, hopefully you'll continue to promote the cross pollination and sharing of information I think it's been incredibly insightful and stimulating. This leads me to my first takeaway that I, you know this is really stating the obvious but sometimes it's worthwhile to state the obvious. In this complex subject that we're studying, it's going to take a village of analysts and models to do a good job at this task so we're modeling complex systems where our understanding remains limited. We should be careful that we're laced with layers of uncertainty as as was pointed out in the very first session, and we should be careful to guard against the natural tendency to believe that just because we can print out numbers to as many decimals as we we care to that that in any way we can actually project with much accuracy. There are many projections but we need to be humble about our ability to be accurate as someone who's made a lot of bad for no good forecast has turned out to be wrong. You know, we've learned to be home. The interactions between the natural systems and economic systems will continue to intensify on a business as usual basis so the non stationarity is a complexity that be face. The portfolio of models that each of which is optimized to describe a given system and it is cared built for care cared by and run by subject matter experts who are intimately familiar with its functioning and its shortcomings really needs to be integrated right to as part of this process, and then there needs to be an integration of the results of these models so sorry we're not going to build the grand unifying theory of climate change and macroeconomics is not going to happen in my lifetime. So, but I think we can each ask what it is we need from the other and hopefully get the inputs that are needed to build systems that are not perfect but are useful. I believe this really was the message also from the excellent CEA white and I want to be white paper that came out in March it was really what very well done. That's first takeaway second takeaway Jim stock mentioned the deep uncertainties associated with this whole topic. We didn't really address those deep uncertainties head on much but as each speaker came up and presented really insightful material and in fact, part of my brain just kept spinning about, you know what are these deep uncertainties so I have my own list and you probably have yours. So, and so this is by no means complete but it's just some that that that come up. So deep uncertainty number one, will the cost of fossil fuels rise or fall over time as we approach net zero. You're formulating an answer in your own head. And I bet that was there's not a consensus on this true demand for these fuels will fall, but some argue that investments will decline faster. So that oil prices won't fall. But you know, it seems to me my simple intuition is that they need to go to the lowest lift cost over time. And or to the nearest alternative natural gas. So we think fossil fuel price. I'm on the prices will decline but not everybody agrees. Second question will the equilibrium real risk free interest rate rise or fall. And by how much we just don't know. Okay, increase in uncertainty and risk premia associated with the transition policies suggest that we'll have higher risky rates, and that will put downward pressure on risk free rates. Large investments in energy transitions driven in part by subsidies and public infrastructure investments aimed at climate hardening and remediation. Higher pace of depreciation of the capital stock due to damage from climate means the gross investment has to be higher to follow the path of desired capital stocks. And this is coming at the same time that we're dealing with the increase investments for chips and, and aija as some call it. So there's lots of an upward pressure on interest rates from this increase in investment. Okay, now this upward pressure on rates is going to be reinforced by the decline in government saving. If subsidies are debt financed which because based on the elevated take up rates a good part of these subsidies in the IRA will be dead financed right. And you've got that so then what about private savings. Private savings is not very impressive. And in any case, and when our buck reminds us that private saving rate depends on the risk free rate, which we just concluded might decline, even if the risky rate rises. So, to just personal saving could actually decline relative to the baseline rather than increase. Remember, investment equals savings so we're stuck now with being bailed out by foreign savings rising to finance an increase in investment. The whole world's in the same boat foreign savings is like not likely to increase. Okay, which means you get a crowding out not a crowding in of investment of non green non subsidized capital. That means potential output in the whole rest of the economy will be weaker because the path of the capital stock will be lower. I'm, these are just assertions, and these are issues I but that's one one interpretation. So anyway on balance of savings is not increase and investment will will not will not rise. Okay, next, next question, are there sufficient reserves and will there be sufficient supplies of critical minerals to achieve the wholesale remaking of our energy economy. So today we economists agree that quantity supply with quantity demanded but at what price will political or geopolitical frictions imply absolute limits to the quantities of some of these minerals, and as an example projections we suggest a global copper demand will nearly double by 2035 in in about a decade with half of that accounted for by increased energy transition technologies. Pebble mines off the table right. Right. So what if supplies don't come online sufficiently fast to prevent increases in prices that will indeed short circuit the energy transition could happen. And face extreme volatility in these prices is probably likely and that being a new fact of life that's not helpful to economic growth. So that's another impediment will inflation go up or down. The easy answer is inflation goes where the Fed wants it to go. Yeah, right. And then increases in agricultural prices and investment increases straining capacity and labor force labor force perhaps reduced as a result of the big inter inter industry transitions, there's likely pressure for inflation to rise. And if the Fed attempts to offset that it does it of course by raising interest rates, and that's not helpful for capital accumulation and potential GDP. What's a reasonable path for the economy wide energy intensity parameter or the inverse the energy efficiency parameter to me this is huge because it tells us how much we have to do to actually accomplish our goals I mean it's one, I think it's one of the two big, big things. Okay, but, you know, the lack of adaptations seen in Marshall Burke's work does not make me optimistic, especially given the fact that we're not doing anything to raise the price of energy. So how, what's the incentive to conserve on energy. So, you know wishful thinking will not cause that energy efficiency to bend in the direction that we want, which leads me to the next deep uncertain. Have we put in place the incentives needed to push energy efficiency along the required past my sense is not even close and I think some of the results the simulations we saw we fall short of the goals. And how much larger increase in energy prices or as an alternative carbon prices is doubtful the improvement in energy efficiency will proceed as needed. And the current emphasis and policies on us regulatory mandates, demand subsidies production investment subsidies, and the fourth and the most arguably the most important lever is not on the agenda. So how do we further incentivize carbon reduction if we can't use that four letter word Tax. Right. Now we can encourage as a substitute maybe encourage additional regional cap and trade systems as a way to boost carbon prices. I have this inkling, and maybe it's more hope than it's certainly not an analysis but it's something I think worth investigating is whether European sea bams the carbon border adjustment mechanisms will in fact export higher carbon prices to the United States because US goods imported in European market will be subject to a tariff of sorts. Okay, and then related will the magnitude of stranded assets whose value will quickly fall to zero this is my next point. Will the magnitude of stranded assets whose value will quickly fall to nearly zero scrappage value basically will this be a material hit to the household sector net worth and will that have implications for consumption. We tend to think about fossil fuel related assets but it extends to whether a sea level rise will lead us to abandon parts of cities as well. Next. I have a question how will we finance the massive investments needed globally. Even if advanced economies can facilitate a reallocation of investments from fossil to green investments and perhaps increase capital flows to manage this task. How will the global south managed to do the same. Okay we we took a teaspoon right and we came in in the last cop, you know, a little bit of a little bit of help. That's big uncertainty how will a remarks respond if there are frictions and we know there will be similar to what happened after the great recession and the pandemic to some extent industry or industrial geographical occupational mismatch arising from the decline in fossil fuel usage will lead to decline in labor market matching efficiency and presumably a temporary rise in the natural rate of unemployment or the Nairu. And that, you know, could, if it occurs slowly it won't be huge but it likely will happen and that will complicate the feds job. And could adversely impact. I mean it will adversely impact potential GDP as folks are simply dropping out of the labor force or not, you know, effectively, not being able to be reemployed. And that means to facilitate the transition for employees and fossil fuel industries, similar to trade readjustment assistance, maybe could be useful. But we have to actually fund it not just say we're funding it. And then finally, immigration, you know, big, big question. It's not clear the extent to which current immigration flows from central South America are due to climate, more likely due to just societal breakdown but you know will it be a trickle of flood tsunami. We don't know. So that's another big policy issue and uncertainty. And that's it. Thank you, Chris. And so I'm going to have some questions for each of you and then we're going to open it up to the audience so I guess I'll start with you, Chris since we just started. So you came up with a great list of deep uncertainties what is your guidance to our colleagues at places like OMB and CBO who have to make forecasts in light of these deep uncertainties how would you suggest they go about thinking about them. How would you suggest they go about communicating about their forecast in light of these deep uncertainties. So I want Wendy to put on a series of seminars for each of these where we have Blanchard and Summers or stock or some other luminaries of the profession. Think about these very carefully and give us some guidance. I think, I think that's, that would be helpful. Additional conferences like this to gather, you know, some of our best minds to address these issues I think would be would be helpful. So, Emmy, I think you sort of addressed your comments towards sort of the sort of models that that OMB currently uses so I'm going to ask you to go sort of the other end of our audience so use cases for their roundtable so in light of your questions what would you tell our colleagues at an FF and I think Anjuli stepped out but and that Noah and others who might fund research to advance the frontier what would you be your priorities for them. I think probably making a connection, you know, finding some connection point between, you know, all of the important things they've learned from their modeling approaches and these simple macroeconomic frameworks would be very valuable and I think that's what this group is trying to do and part of it is about just speaking in a, in a common language. Thanks, you were very much on the NSF Noah research agenda side so what is your guidance to our colleagues at OMB and CBO. I think that there is. tiered series of improvements that need to be mapped out and obviously the work that OMB and CBO have already done to incorporate these simple representations at the front of the models is the first step. But the question for them to ask is when they start to build in some of these feedback when they endogenize the climate damages and the climate emissions which also one thing about how that might be done. What does that mean for the kind of work that they need the groundwork that they need to be putting together now, and I think in particular, it means having a big framework of all the different pieces that need to be filled in recognizing that we only have a couple pieces. And I think that I want to respond to Chris's comments that we're going to have. I mean, who said we're going to have a bunch of individual models hyper focused on different tasks. You know I would have. done that before the advent of Chuck GPT. But, but now I think we're in a world where we really can imagine a holistic understanding and that's exactly what we need and so that's why I think OMB and and CBO need to be moving toward just like the NSF son and those of the world. So, if you're interested want to ask a question. Raise your hand on zoom. So, more Eric. Hi, this is just a reflection on on the panel as a whole and kind of pass a question back that I heard asked from the panel. I thought all of that was very interesting. Very interesting input. Me started out by asking, what are the variables, we can use from the climate models. And since I kind of work in that space I have some thoughts but I'm much more interested in hearing from from the panel, how, what is available from the climate models what what should be taken on board. James. The climate models are quite the right place to get this information, we have incredible bio physical impact models. We have, you know, the next stage of that is where bio physical impacts interact with human decision making at the back at the micro level. So I think. I think that the panels of impacts that are being developed by groups like yasa easy Met coach to understand the whole range of how climate will impact different sectors of the economy is the right way to go and it's really exciting to see those start to come together. So I think this is no reflection on anybody in the room, but a quote that stuck with me from a long time ago and I don't know who it's original to is the difference between a good economist and a great economist is a great economist knows what to leave out of the model. So I think that's a challenge that we that we face and we don't want to get to too much complexity maybe chat GBT will change that I don't. But I think that, you know, so at this at the simplest level I think this very sensible steps that CEA has already taken to look at the impact of potential, you know, the damage function if you will, and how that can be incorporated into the projected path of TFP in a in a BA you versus a policy with a run with a different temperature path I think that that totally makes sense that that's the to me a key entry point for climate influencing the the the model and the impact on the capital stock of the increasing severity and frequency of severe of climate events, which destroy capital. Clearly that brings in the whole regional component which is definitely needs to be looked at, although I do I do wonder whether the CEA wants to be an OMB want to be in the business of doing regional analysis of climate policy because of which states it will hit and it probably is just something you want to leave to somebody else. But having said that I think so the impact of the, you know, the on the physical capital stock because that that will influence both the cost of capital and the level of gross investment that is needed for for firms in the private sector to to achieve their desired capital stock and so to the extent that it spills over into gross investment that has lots of implications. And then the other things there's lots of ways to transition risk and policy changes can can enter into the model but yeah and you know the other ones that are not really, you know they're important, which would but but I think difficult forecast would be the impact on immigration and you know how that would influence labor force working age population and so on and then ag prices. Yeah, that's what I can think of. I had one reaction to I guess both James and Chris's comments, which is that maybe it's useful to translate some of these ideas about simplicity into the language of machine learning so one of the first things that you learn. When you learn about machine learning is about the fundamental importance of regularization. To avoid overfitting. You know you choose a simple model often, even though you know the model is wrong. And I think, you know, the challenge that we have is that you know macroeconomics really is small data in the sense of the number of years we have. So there may be many variables in each year there may be many countries. But you know fundamentally you only see one history. It's not clear we can go back to the pre industrial revolution period and you know use those data to kind of think about investment dynamics today. So I think that is kind of a fundamental challenge that that we face and is actually, you know fairly well captured in machine learning ideas it's just that, you know, the macroeconomic data is not the typical use case for those methods. So that's one of the, one of the trade offs that we're thinking about here. Sorry, it wouldn't let me unmute Bob. Thank you panel. Thanks everyone for a really stimulating workshop. As a climate modeler. I think that would kind of help me understand where we're at at least which I didn't see in the last couple of days is how good or bad are the macroeconomic models in forecast mode. And on which variables are they better and on which are they worse. In climate you can go back to the climate models of the 1990s and see that even those were even back then we're going to be okay at predicting global temperature and put it down all right on the sea level rise but we've been really poor on, you know, the water cycle or something. It's not better on that over time, but I'd love to get a sense in the panel or anybody else. You know if you go back to the 2013 tenure projection or whatever you called it and compare it to unfolded. Or go back to 2003 and look at those 10 years up to 2013. I really haven't got a proper sense yet of, you know, what can be reliably forecasted with what skill and what can't. Because for me that would be really helpful as a scientist to understand, well are we going to be able to pick out you know the climate or the transition signal from the noise or the uncertainty. So if it's a panel or anyone can help guide on that that would be great because my my kind of outside as hunches are, I've got this vague sense that there are some things that are more deterministic and predictable than others but I'd love to know a bit more about that. Yeah, I'll respond to that I think it all depends on what you're comparing to. So I think they're the CBO for example predictions which tend to be very similar to professional forecasters are much better than a lot of crazy things that people say. Okay, so I think they're very helpful in cutting off tails, you know things that have not been seen in history, and so on which which really is out there in the distribution. If you, if you sample, you know, a random group of business people or people who aren't familiar with economic data. On the other hand, if you know you're comparing to a random walk, you know, simple as possible model sort of the no change forecast, then it's, you know, kind of a tighter race. And the fact of the matter is that, you know, there have been many periods of auto correlated forecast errors, you know, I'm sure Chris can painfully remember some of those cases, like for example in the late 1990s, when growth was extremely bad, the CBO among other professional forecasters was repeatedly sort of expecting things to go back to normal. Until finally people start to believe including the CBO that actually this was here to stay. And right then there was the calm crash and so you know, obviously they were burned by that, you know during the great recession, you know no one expected how long it was going to take for the recovery. And this kind of repeated belief that things were going back to normal again back to normal back to normal back to normal and just, just didn't happen. No one expected how long the zero lower bound was going to last, you know we talked about the effect of climate change on longer term interest rates. But you know the fact of the matter is that it's pretty hard to figure out what's happening, it's going to happen to longer term interest rates, even when you don't put in the effects of climate change. I think it's all about what you're, what you're comparing to I mean I think I don't want to, you know, I want to be positive in the sense that I think that these forecasts do a lot better than, than you would do. You know if you're sort of sort of picking randomly or something along those lines, but this is what I was trying to this is what I was trying to say when when I said that I think more complex models have have often not done better. So I think it would sort of be somewhat humble about that fact that, that sometimes, you know, simpler models can can do better because of this issue of overfitting and so on. So, Jim, Jim you want to. Could I just jump in on that one point I think it's also, you know I think it's, you can't really compare across domains. I mean there's a tremendous amounts of uncertainty that just evolves. I think about you think about COVID. Okay, so guess what all of the economists circa 2019 got COVID wrong. And that had the biggest effect on unemployment we've seen since the depression I mean just extraordinary. So there's just like gigantic shocks is kind of like, you know for you guys like Greenland falls into the sea, you know, and then it jumps back out. It's like, it's like, but but that doesn't say that there's other aspects where you can have more confidence which are maybe conditional forecast so conditional on a certain path. What do you think things would be and then if you do a delta say we're going to spend a whole lot more. I think there's better evidence that we can, you know, be able to say things about that so for example if we think there's going to be the most frequency productivity effects of a number of climate variables, then I think one can have a reasonable amount of confidence in those deltas, even that there's a fair amount of uncertainty about what that baseline actually is going to be like chat CBT who knows what the effect of that's going to be on productivity 1015 years ago we just just don't know, but we can still have some confidence in the deltas. You know, speaking as a climate scientist like in the IPCC world we would say that probably a situation we would double low confidence and we would not give a best estimate, we would give potentially a range. The practice and econ is to give it in critically in sort of practical economics of the Torica and CBO is to focus very much on a central scenario, which implies that you know the distribute you have some sense that there is a distribution centered on that central scenario as opposed to a range of options so how do you think about, you know, that that these alternative forms of uncertainty communication and, you know, is is really focusing on the central scenario and appropriate, given what we just talked about or might there be alternatives approach that would fit better. Great point so every month we we produce a baseline forecast, and an optimistic and a pessimistic that are based upon what we think at that particular time, you know might be, you know changes in assumptions so there's basically three different conditional forecasts and then we also do a set of an of an additional five scenarios so we're you know that that's to a smaller set of clients, where again, those are driven by particular narratives that reflect risks that we think are important at that particular time so there's a whole range of this is you know GDP paths look like spaghetti spilled on a plate to represent the uncertainty that's one way. The other way is we know we've built. We call them probability assessment tools where we we actually fit a much smaller macro model with about half a dozen variables, and it's got some nice features about. Growth the GDP is dependent on where you're on the cycle, for example, at the outset, and we do Monte Carlo simulations and create probability surfaces that help to, you know, for select clients help them gauge the degree of uncertainty around the baseline forecast. So would you give any advice to our federal colleagues based on that experience. I think the range does make some sense and you know see, but I don't think Bob mentioned it yesterday but the CBO periodically does produce the fan charts which shows the range of uncertainty. You know which nobody knows what those mean so nobody looks. I mean, very few people know what those mean so they don't look at them but but those are useful. I think I think the question is, what do you want policymakers to do with those ranges. So, if you just produce a GDP range. I don't know that there's much in there that's that's actionable for policymakers and the point of the GDP like that. And frankly I don't think any policymaker is like doing something because GDP is produced by CBO was x y or z. Instead it's meant to be a point of comparison to if you change policy then this is what will happen and it's meant to be an input into the into the position where like the system completely would grind to a halt. If you had a range of budget projections so that's why you need a point estimate on the economic projections. Now, when it comes to actually actual variables in CBO forecast that are particularly uncertain policy makers who are in a position to do something about it are keenly aware of the uncertainty around particular variables, and what to do. There's a huge amount of back and forth so take something like labor force participation of prime age men, where it just kept not going up in the, you know, in the in the teens. And, you know, for a long time CBO then got it wrong because I thought it was going to be higher and blah blah blah like CBO produced report after report about what was happening to labor force participation among prime age men and policymakers who just thought about what policies do we want to address that. Think about interest rates where that's one of the ones where CBO just continuously got it wrong. There is a boat load of analysis that the agency produces of if interest rates are higher or lower this is what would happen to budget projections. So, if you drill down into the variables that are uncertain. This is I mean there is laser focus on how policymakers should care about uncertainty in particular variables that I think is quite actionable whereas arrange around GDP. I mean that's just kind of you know, that's, I don't know what they do with that news. And I think this is really a challenge for how to how micro economist working in climate understand the problem that they're facing, because the we're dealing with multiple layers and kinds of uncertainty right there's there's uncertainty in the way that these impacts will manifest and what the transition will look like. And we want policymakers to help us engage with that we want them to engage with that uncertainty because we think it's quite important and we want to figure translate that into policy action. I don't, I don't have a way forward but I do think that even at the level of basic GDP. This is, this is a manifestation of the deep problems that the climate poses. I just want to react and say I think I mean, at least personally I've learned a ton over the past two days about how to think about uncertainty around climate and the, then the uncertain effects on the economy. And then what that means for risk averse policymakers and maybe separately what that means about making optimal policy that's robust in the face of uncertainty. So, no, no, no, I, I, I, I'm not, I'm not trying to shut this down by any, at all. I'm not actually going to take much more of your time so we just had a Greek. Sorry I should stand in front of the mic. I'm not going to take much more of your time we just. I'm not going to take much more of your time. This has been a great pleasure to have everybody here. We've all learned a lot and we've all had a really good time I think interacting with each other in person, and through the great job of the it folks here at the academy, really integrating the online audience as well so thank you for the online audience. And I look forward to the next version of this workshop. So thanks very much.