 Thanks a lot to all, thanks to all the speakers. We have a panel on basically on the policy dimension of macroprudential stress testing. We have Dianne Perret from the University of Luxembourg, Beverly Hilton, vice president for research, Fed New York. Jesus Saori, now who is the director general financial stability regulation resolution, Banco de España. Martin Chiak was head of stress testing in the IMF. And I'm Kermit Ocelio, head of the stress testing division in the ECB in financial stability. We tried with this panel. We'll try to cover lots of ground. So these are going to be high level short interventions. We'll try to keep some time for questions and answers at the end. I think we'll have Dianne will start with her views on what exactly is the measure you want to look at for vulnerability when you're doing a macroprudential stress test. And then Beverly will give us some views on in what direction we should be developing stress testing to make this an effective policy tool. I will give some perspective on how actually this can be used to calibrate macroprudential policies. Ben Jesus will share some views on how to coordinate the macroprudential and the microprudential stress test. And finally, Martin will tell us about communication challenges of exercises which are complicated. And especially when you move into the policy domain, the translation is not always immediate. And you've heard today also how important communication is and transparency and stress testing. So this is an overarching argument topic. And now without further ado, I'll give Dianne the pointer. And I guess it's on forever. Is this one working? Yes. Well, first of all, let me thank you for inviting me to sit on the panel of such distinguished panelists. I feel very, very impressed here. So here what I want to talk about today, so you asked me my view on what's the right measure of vulnerability of a bank. And I think if there was one, we would be out of business. So probably there's no such a thing as a right measure. I think probably you want to use multiple measures as much as the information as you can have. I'm an econometrician by training, so the information set should increase to inform your view. But why do we need so many measures? And why do we need stress test? Why do we need supervision? In the end, I think what I wanted to talk today is about this trilogy that we have between when we talk about supervision, we have this interaction with capital requirements and regulatory arbitrage. So if there was no rules, there would not be room for checking the rules and having supervisors that make sure that these rules, these requirements, are implemented. And if there was no regulatory arbitrage, probably rules would be enough. And we would be also out of business. So I want to talk about this equilibrium that we have between capital requirements, supervision, and regulatory arbitrage, and see a little bit how, when you move one, what happens to the other two. And my way of thinking of stress test is stress test, to me, is the emblematic example where capital requirements and supervision are intertwined. And what we've seen in most countries in the US, in Europe, is that with the implementation of stress test, you have both higher capital requirements and enhanced supervision at the same time for the same group of banks, the largest banks. So why do we think that supervision is important, is that we actually saw that this was, they still play the role in the US that identifying this direct effect of stress test supervision was actually what was working and reducing risk-taking incentives at banks. And that is because without doing that, we would not actually have seen any effect on risk-taking. So not disentangling between supervision, capital requirement, and the effect that stress test have on capital requirement would not have been able, we didn't see any effect whatsoever of stress test on bank risk-taking. So why is that? Because well, there is an effect of capital requirements. So banks are actually responding to capital requirement. There is no separation between capital structure and investment decision, as I also explained in my discussion. And well, the sale relevance of monthly annual order for banks doesn't only mean that with conventional wisdom would increase capital, banks should be safer. Here banks also respond to that, and they could actually take more risk, such that by taking on more risk, you actually end up with banks that are riskier. And why would they take more risk is because, well, we have in mind this increase in cost of funding that comes when equity is costlier than that, which could very well be the case for banks, knowing that there is deposit insurance and banks don't fully pay for the deposit insurance. So with this increase in the cost of funding, banks could optimally respond by taking on more risk. Could be the opposite channel, more skin in the game that would prevail. But what actually we found in the data is that banks tend to respond to increases in capital requirement by taking on more risk in the short run. And this is something that actually regulation is based on. Basal capital requirements is based on this principle that you should ask for more capital for the riskier exposures in order to prevent these banks to actually, each time, take more risk when they have higher capital requirements. So the idea, since even Basal I, was to link capital requirement to asset riskiness through regulatory risk weights. Now, if this is included in capital requirements, why do we, again, why do we need supervision? If capital requirements reflect, already reflect asset riskiness. And then I go back to the evidence on regulatory arbitrage. So there is, so even if capital requirements reflect risk, they don't fully reflect risk or they don't fully reflect risk in a dynamic manner. And banks are going to always shift the portfolio towards the most underestimated risk weight class, asset class, so for example, market back securities during financial crisis, sovereign bonds during the European sovereign debt crisis. And this is where I think market measures of risk or additional measures that the regulator is not necessarily using to assess the banks on a day-to-day basis, can be useful to flag those episodes where regulatory arbitrage is very important. So why are market measures of risk interesting and useful here is because by definition, they are not subject to regulatory arbitrage. You might say they might be wrong, they might have other, many other caveats, but at least they are not subject to regulatory arbitrage. So they will reflect a ranking of bank riskiness that might not be the same ranking as bank riskiness according to regulatory risk weights. And I think when this can be informative is when you have actually a negative correlation. When the market participants think that the riskiest banks are actually the banks that have the lowest risk weights and have the lowest capital requirements, then that can tell you something about whether risk weights correctly reflect risk and whether there is regulatory arbitrage. It was the case in 2011 for the 2011 stress test in Europe where we saw very negative correlation with market risk weights, market implied risk weights, coming from the fact that many banks actually were loaded on the risky sovereign bonds. But you see that in all the stress tests, this has continued to be negative correlation. Sometimes it's because of low market to book ratio, sometimes it's because of other reasons, but this is a red flag, right? Once you have this flag, then you're like, why is this so? Let's dig a little bit deeper in the data and try to figure out what's going on. Also, in more recent research, we saw that the more the most stringent capital requirement of bank fees depends on risk weights, the more banks are going to respond to increases in capital requirements by taking on more risk. So there is this evidence all over the place there, not only that banks are doing regulatory arbitrage, but they're also doing regulatory arbitrage in stress test. So this is where supervision has the most important roles. When capital requirements are high, this supervision can play an important role. Before the crisis, we know that there was a lot of regulatory arbitrage because there was a lot of opportunities to do so. You had only four different buckets, risk buckets, loose, lee, defined risk weights and the basal one, so banks could very well go for their riskier assets within a risk weight class. Now there are less opportunities to do regulatory arbitrage. You have more precise risk weights. You have also more stringent capital requirements, but still we find a very important role for qualitative supervision. So I'm gonna just leave it with open questions and say, well, maybe the opportunities have decreased, but maybe the incentives have changed too to engage in regulatory arbitrage. And that's maybe where we want to think about what's the role of supervision when banks' capital requirements are high versus low, whether supervision and capital requirements or high capital requirements are substitutes. It seems that there was a little bit of this discussion at least in the US with the Financial Choice Act to see high capital requirements as a substitute to stress-test supervision. I don't think that's going to go through, but I'm saying this has been on the table at some point and these are questions to think about. Thank you. Thanks a lot. And sorry, we'll move on. Where are we? Let's see if we can... So thank you all. The slides come up, I'll say everything I'm gonna say today is from the perspective of the US stress test, which is what I know about, but at the end I'm gonna try to draw some lessons that I think apply more broadly. I hope apply more broadly, so great. So I have to start with a disclaimer, we'll move through this. I was gonna start off by telling you all about the details of the US stress tests that's been fortunately covered already. The key thing for my remarks to pull out of this is that the way we calculate the stress tests, they're calculated for individual banks and they're part of a broader program, the CCAR program, which has implications for the individual banks that are in the program. Let me spend just a moment here because I don't think this has been discussed quite as much today, so what are the key elements of the stress tests that we do for CCAR and for the Dodd-Frank Act, which is stress test, which is what DFAS stands for? The macro scenario that gets fed in gets more stringent as the economy improves. That's sort of built into the way the scenario is designed. The unemployment rate must go up by a certain amount and must hit a certain level. So as the unemployment rate has fallen in the US, the amount by which the unemployment rate increases in the scenario has gone way up. There are similar, or in a different vein, the way it's done, but similar ideas for what happens to housing prices and other asset prices, they will tend to fall more as they've been running above trend. Another thing to know is that we do these series of what you would call top-down models, meaning that we do the estimates, there's also bottom-up modeling as part of the CCAR program. Today, everything I'm gonna talk about is what the Fed does, our top-down models. So basically those results are, we have our own models, they are, the models are the same for everybody, but we get a lot of very, very granular data from the banks, and so same model, same macro scenario, but the differences are in the data that feeds in, that comes from the banks. So I thought it was maybe helpful to take just a moment to think about, if we're talking about macro-prudential stress tests, what do we mean by macro-prudential? Other people have talked about the distinction I'm gonna make here, I'm certainly not the first, but I think there's two different ways of thinking about what macro-prudential means. One is from kind of a structural view that's about identifying important nodes in the system, those institutions where if something happens, the negative externalities associated with distress or failure are the most severe and pervasive in the system. And when you think about the policy implications of that, it has something to do with strengthening the potential requirements and supervision at those systemically important nodes in the banking or financial system. And then there's the view of macro-prudential that I think has been the focus of what we've mostly talked about today, which is the cyclical view, which is how are risk to financial stability changing over time, understanding the cycles and credit, asset prices, leverage, liquidity, all the things that we've been talking about today. And there the policy implications are something about lessening the probability and or the consequences of a turning of the cycle. So in playing those two different thoughts on to what we actually do in the U.S. stress tests, as has already been talked about, what we do is actually project regulatory capital ratios for the banks in the stress tests, you know, sort of nine quarters forward under the scenario, very much regulatory capital and accounting-based calculations. But what I really wanna focus on here is how the stress tests are calculated. So the stress tests are calculated for each of the banks individually on a standalone approach. There's a lot of attention to projecting out the different pieces of net income and then how that impacts capital. But basically, given the scenario, given the models, given the individual bank data, what happens to bank A in the calculations doesn't affect what happens at bank B or vice versa, except as Don Cohn said, the general scenario is one in which things would be pretty bad for banks in general. So in thinking about the structural macro-prudential elements, you know, why do we, how do we come to do this, you know, bank by bank calculations? And I think it's important to remember how the results are used. You know, dating back to the first time we did stress tests in the middle of the crisis in 2009 and now is embedded in CCAR, you know, the results of the stress tests have consequences for the individual banks, both supervisory consequences in terms of the supervisory observations that they will get and then potentially consequences for their ability to increase their dividends and repurchases to shareholders. So because of the sort of micro-prudential implications of the stress tests, the push in the modeling and thinking about it, I would argue, has been towards sort of accuracy and precision at the firm level. Make the models, even though the models are the same for everybody, get more and more detailed data, have the models, you know, reflect particular kinds of lending that, you know, that different banks are doing so that you really get the numbers given the scenario as correct as possible at the individual banks. I would say this, you know, arguably addresses the structural macro-prudential concerns that we talked about a minute ago. The primary cyclical element are the design of the scenario, the scenarios, which as I said, are designed to be more severe when times are good, but as has already been talked about by Don and others, these counter-cyclical elements compete against improvements in asset quality. There's a little bit of a horse race there, and Don's work with Nellie Lange suggests that the horse race is, you know, maybe a tie or that the asset quality, the improvement asset quality is winning, and most of the sort of counter-cyclical elements of the stress test has to do with the fact that dividends and share repurchases, which figure into the ultimate capital ratios, have gone up and increased pretty steadily during the expansion. So, you know, what kind of macro-prudential insights do we get out of the stress tests? I mean, I think there's some limits that it's important to acknowledge. First of all, these are capital stress tests, and Don't Directly Capture Liquidity Runs Fire Sale Risk of the type we just were hearing about. The large US banks are subject to a separate liquidity stress test which is part of the CLAR program, but those stress tests are separate from the capital ones they aren't linked together. The standalone approach means that all the linkages you might like to know about, bank to bank, banks to non-banks, the financial sector back to the real economy, we don't really capture in these stress tests. Except again, from the fact that the scenario is really, really bad and sort of already presumes that there's a lot of trouble in the economy and in the banking sector. And again, relevant to some of what we just heard, the really complex models and the high level of granular data, loan level in many cases, mean that generating the projections is really sort of resource and labor intensive, and it's hard to do more than a handful of scenarios raising questions about whether all the risks are being captured, will the vulnerabilities at all the banks be identified. So what does this imply for some of the design choices? This is what I was actually asked to speak about finally on the last slide here. So I think some of the things to think about is where should the complexity be? In this micro-prudential setting, as I've said, we drove a lot of complexity to get the numbers right at a detailed level for individual banks. Maybe if the concern is the system as a whole, the complexity should be capturing how what happens in one part of the system affects another part of the system. So thinking about the purpose of the test, where do you want to put the complexity in the modeling and the data collection? In terms of data collection, maybe there's a role for more sort of high frequency inter-day transactions that really capture the way funding moves during the day and over the course of days between institutions. And then of course there's the data, sort of the low hanging fruit to mention, the data from the non-banks and from what in the U.S. is a very important, unregulated sector, so how complete a picture do we get by looking at the banking sector alone and then as has already been talked about, the ability to do many scenarios, not just a handful. So I think that's my... Thanks a lot. Okay, so a few thoughts about how one can use macro-prudential stress test to calibrate macro-prudential policies. So basically what is a macro-prudential stress test? In the end, you're projecting along the most likely path of your variables, that's a baseline scenario, and then you have a scenario where you have severe shocks, let's say, as people say, severe but plausible, that would be the adverse scenario, and ideally you have interactions, so institutions react to these shocks and maybe interact with each other and that feeds back into the economy, you know? So that's a macro-prudential stress test and how do you calibrate, on the other hand, when you think of calibrating a macro-prudential policy, how do you look at that? Well, standard approach to policy making is to do cost-benefit analysis. Now in the specific field of macro-prudential policy, there's some challenges because costs are relatively straightforward. We're talking about solvency measures, so increasing capital or borrower-based measures, but anyway, usually what this does, this slows down the economy one way or another. If it's a capital measure, you're restricting a bit the supplier credit. Because you're doing it in normal times, you have to measure the impact along your baseline scenario, so you have a little bit of a slowdown in the economy, but what are the benefits? It's unclear what the benefits are, or there's many ways of looking at it. One way of looking at it, of course, first of all, the benefits should be in some sort of comparable unit. If you say that the benefits are something that is not expressed in the same unit and it becomes very difficult to compare. So what people have been done for a while is to do some sort of macro-analog of the micro-perspective and unexpected loss approach. So you say, well, if banks are more resilient, there's a lower probability of a crisis and or a lower cost of a crisis, and therefore you have a lower expected loss, and then you compare these two magnitudes. The output you lose because you put the policy versus the output you gain in expectation because you have averted a crisis, you've made it less expensive. Now, the problem here beyond the fact that macro is probably not exactly an analog of micro, but without getting into philosophical issues, measuring that is very difficult. First of all, because crisis events are few and they tend to be highly correlated and they're at country level, so you're comparing different countries and these are small samples. And so how meaningful is it if you have in your sample and you are a European country to have a crisis in Mexico? It's meaningful to an extent, but it's not incredibly meaningful all the time, especially if the crisis in Mexico was 30 years ago, which is what you have in your samples. And then when you're estimated, it's unclear what the cost of a crisis is. For some people it's a loss of output in a few years following the crisis. For others it's a changing growth rate. Now, if you express a changing growth rate in GDP levels, you basically have an infinite amount. So you have estimates of the cost of crisis that can go into the hundreds of points of GDPs. And so it's a bit complicated to put this together because you have such a wide variation in what your measurement of benefit is that when you compare it to these costs which are small fractions of GDP, then it becomes very complicated. Now, what would be an alternative approach to look at the benefits of macroprudential policy which relies less on rare events? And basically the idea here is to consider that the macroprudential policy, what it does, it makes your financial system more resilient, not in terms of avoiding a crisis which is an extreme event, but in reducing the probability and the depth of a severe downturn in the economy. Now, that's something that happens a lot more. You have more observations. It's a more, let's say, run-of-the-mill concept that you can see what is the impact of having banks in better shape, a financial system in better shape when you have a severe downturn. And now, where does stress testing come in? Well, stress testing, basically if you think of it, already provides you with infrastructure to compare these things because you can see what is the cost of introducing a macroprudential measure by taking your stress testing infrastructure and going through the baseline without any macroprudential measure. So that's just projecting things as they are versus by introducing your macroprudential measure which means that if you allow banks to react to that in your model, you will have that you're imposing an increase in capital requirements and therefore banks, the cost of, the weighted average cost of credit will increase a little bit and therefore you will have a little bit of a slowdown and that can be done very easily with the models we have. And that's your cost. And that's for the benefits. Well, the adverse scenario is a severe but plausible downturn. So it's something that is not entirely a fully-fledged systemic crisis, but it's still severe enough that it makes sense to want a resilient system because there should be benefits there. And there again, you see what the path of the economy would be without the measure compared to with the measure and hopefully if the measure works, you have introduced, for example, capital buffer that the banks can use and then they will be deleveraging less by using the buffer than they would have without that buffer. And therefore your path of the economy will be slightly less bad than it would have been otherwise. And then you have, in this system, you can directly compare your cost and your benefits. You have estimated it with the same model. You've done it for your own country. You didn't need to do cross-country. You have more observations. You have measures that are consistent with each other and that can give you a perspective on how to calibrate your policies. Now there is also an issue of what is the value to you as a policymaker of a unit of output in good times versus a unit of output in bad times but we won't get there. Let's just assume the risk neutrality and they have the same value but they don't need to be. Now what would be an extension of this consistent approach instead of having one adverse scenario, like Beverly was saying, it would be interesting to have many different scenarios because with many different scenarios, you can cover a wider variety of risks. So if your technology allows you to have multiple scenarios, in the limit, you can basically model a whole distribution of outcomes. The more scenarios you have, the more you have the whole shape of outcomes and then you can see how the distribution changes within and without your macroprudential measure. Then that would be a robust and a complete assessment of the impact of your measure and you would be confident. You could actually also see in what type of scenarios your measure isn't working very well. So this is basically, if you want, an approach that is equivalent to growth at risk but in the policy space. Growth at risk has been thought of as a way to identify risks, how they materialize. So something like loser financial conditions today translate into a different shape of the distribution of output tomorrow and here the idea is the same. An introduction of a policy measure today will change the shape of the distribution of output tomorrow. So here you can see what this would look like. The blue line is your baseline and then you introduce your measure and you have that blue pocket. That's because the economy has slowed down a bit. Now it could be the case that after that it picks up again. So you look at these pockets and you compare them and what is your benefit is that the lower line, that's your path of output in the adverse scenario and you can see the red dotted line is where you would be if you had your policy measure and the gain is a green shaded area and this is just for two cases but imagine this with the whole distribution and you can just compare the distributions. So what are the benefits of this approach compared to other ways of looking at macro-prudential policies? Well compared to standard macro models, here in the stress testing framework you take into account the heterogeneity of institutions in your sample. Now sometimes it doesn't matter but most of the time it matters. There's a big difference in having all banks that have a management buffer of 2% say versus half of your system being very, very close to their thresholds and have the system having a lot of extra capital. The reaction would be different but that you can only capture if you have a model that goes at the institution level. Now compared to the standard growth at risk model what you need for policies, you need to understand the transmission channels. At ACB we have developed a model that is semi-structural for the macro part and this allows us to understand the transmission rather than a reduced form where you're just correlating things. So in the policy space it pays off to have some structure. If you move to multiple scenarios you can also experiment with all sorts of environments and assumptions about your parameters and in the end this kind of approach gives you lots of internally consistent outcomes. You can look at growth at risk or at capital at risk if you want to call it credit at risk, whatever you want. You can look at decide what is a distribution that's relevant for you and look at it in this space. Now of course what are the cons of this? Well this requires large investment and it's quite complicated to run and as Beverly was saying before if you put in all this also interactions among institutions and so on and you don't limit yourself to banks but put the whole financial system this can quickly become completely unmanageable. Now in Europe we are relatively lucky because banks are still quite central so if we confine ourselves to banks we're not too wrong but that's not necessarily the case everywhere. So the bottom line here is that the macro-pudential stress test framework is really if you look at it as a tool and not as an end in itself it's really a natural candidate for policy evaluation because it provides estimates of outcomes in a likely and in a bad outcome which is pretty much what you want in financial stability because what you care about is the left tail of the distribution. Conceptually this can be mapped into growth at risk which is a nice, elegant consistent intellectual framework to look at things which is probably more satisfying than the ad hoc macro-analog to macro-reasoning and of course there's technical challenges that include the integration of system-wide considerations and then also communicating that you have decided on policy measures with this kind of concept it's not necessarily straightforward but I guess Martin will give us some perspectives on communication and with this I pass the ball to Jesus who doesn't have slides so we'll... Thank you very much, Carmelo. Let me start by thanking the ECB and in particular you for inviting me to this panel is really a pleasure and an honor to be here. If there is a prize in this conference for the laziest guy this is for me because as Carmelo mentioned I have now a slide not a single PowerPoint slide. I'm not the odd one out just only for that. I'm just... I'm only the odd one out because this is a conference basically about the frontier of stress test. You, all of you, work in that frontier. My remarks focus on three points that are far, far, far away from that frontier of the knowledge. These three remarks are very simple. No complexity, I think that the whole program of the conference has been to me too complex, probably I'm getting too old for this business, but still I think that they may be useful for focusing your analysis. These three remarks come from more than 15 years of stress testing practical experience starting with 2005-2006 F-SUP IMF stress test, a very nice exercise. Then it came the 2011-2012 F-SUP stress test. Then it came the Troika-led stress test and I still have some friends from that exercise. John is not here. Matias is at the end of the room at work. Then it came the 2014 ECB or SSM stress testing and then it came our annual stress test of all our banks. So I hope that these three remarks are useful for you. The first one has to do with stress testing in bad times. They can be extremely useful to enhance financial stability and to recover confidence in the banking sector. They can also be useless. I think that the stress test in the US in 2009, also the Spanish banks stress test in 2012 are perfect examples of extremely successful stress test to regain confidence and to regain financial stability. This comment, you are not going to like it. The key determinant of the success of stress test in bad times for me is the existence of a backstop. It doesn't matter whether you have a very nice scenario, one scenario, 400 scenarios, 20,000 scenarios, whether you have very good models, whether you have very good methodologies. In bad times, what you need to have a successful stress test is a backstop. If you don't have a backstop, then the stress test becomes much more complex. I know that this is obvious, but I think it's important not to forget it. My second comment has to do with top-down versus bottom-up stress test. Basically, I know that stress tests can be very complex. Some of the papers have shown this, but for me, the stress test conceptually is relatively simple. Basically, you have a starting point of exposures, capital levels and profitability. You have scenarios and you have an engine to transform these scenarios or to apply these scenarios over the initial exposures and then get the final capital depletion. The key variable here is who controls the engine of this exercise, conceptually relatively simple. Top-down for me means that the control is in the hands of the authorities. Bottom-up stress test means control by the banks. If the control is by the banks, then you have a huge amount of resources devoted to make this bottom-up stress test converge with the idea that the supervisor may have. To me, clearly, the top-down is something that needs to be reflected carefully because it means that, like in the US, the control of the engine is in the hands of the competent authorities. I think that this deserves some thoughtful discussion. Finally, the top-down can be very granular. We have a modest stress testing tool in Spain. It's called Flesby. Some of Matias know it very well because it was something that we developed to get out of the programme in 2013. We used it and it was very successful. We have kept increasing and developing it and basically it's a top-down stress test with granular information and there's no complexity there. Basically, you get loan-by-loan information, collateral-by-collateral information, linked to those loans, a price value for each of the collateral you have, and then the haircut from the price value to the real final price. This is a very simple way of doing stress testing, top-down for 90, 95 of our significant institutions that mainly have credit risk and if there is a problem, it would be a credit risk problem. I think that this top-down approach needs some reflection and it's not that difficult. It's true that it may consume some resources, but the other approach also is very intensive in resources. Finally, coming to the initial Carmelo proposal on the micro-macro-prudential usage of stress testing, usually stress tests are performed bank-by-bank. If you have bank-by-bank results, I think that there is immediate micro-prudential usage. Here, to link very tightly the results of the stress test with the capital requirements, I think that needs also careful reflection because you may end up in a place that is not satisfactory all the years that you ran this exercise. I think that the stress test is a tool to set capital requirements among other potential tools and qualitative assessment that may be needed to set the P2G, for instance. I also affirm, believer, after my experience, that stress tests cannot be performed without micro-prudential supervisors. We may have a very nice top-down model, but you need the interaction with the micro-guys, the guys on the other building, down the river or up the river. I don't know which way it flows. I think that this is an important part because you need a reality check of your models with the joint supervisory teams that know whether these models should be properly applied or not. Double check your results on the projections of the P&L, projections of credit risk, and so on. And, of course, there may be a micro-prudential usage for stress testing. To me, this micro-prudential usage is to set the stance of micro-prudential policy, maybe regarding the CCYB, to set the stance. It's not easy to link this directly to a direct micro-prudential tool. I think that we need to be humble of what we can achieve with this stress test. And finally, and I think that Rochelle's paper was a very good insight, we need to reflect about whether the scenarios should or should not be counter-cyclical. I mean, stress testing banking sector in bad times, probably needs a less stringent scenario that in good times. And this is very important because this is a fundamental way of setting this stance of the macro-prudential policy. We also have comments on the governance, but I think that the governance is much easier in bad times than in good times. Thank you. Now, to Martin. Thank you. My name is Martin Chihak. I lead the division in the IMF that runs the financial sector assessments, including stress tests. And it's a pleasure to see so many people listening and talking about stress testing for a whole day. When we started this 20 years ago, it was a very small exercise and has obviously grown up. And in those early years, when the F sub study was 20 years ago, there was no question about publication or communication of stress tests. In fact, the first 12 assessments we've done, you can never see them. We agreed not to ever publish them. So since then, the debate has moved on quite dramatically. And now quite often we see the opposite requests for technical assistance or capacity building on communications of results of stress testing and financial stability reporting. And we often tend to push back and say, this is all very nice, but first make sure that you have a release of solid microprudential stress testing for your own sake before you go out and publish. And so this chart is from an earlier paper on communication where we sort of put out these pros and cons. Here I focus, this is the microprudential stress testing. So it's the kind of ex-ante stress testing as opposed to the crisis management stress testing, which is a bit more tricky. But anyway, even for those stress tests, there are many arguments you can make to be cautious. And one, again, what that comes to mind on the microprudential is this sort of potential for confusion that, again, I think we've seen mentioned a couple times. So you sort of, in some ways, we are victims of the success of the stress testing because it has proliferated into many different uses. So we have the micro and the micro bottom up top down and so you have the communication challenge when you come out with a new stress test to explain, well, this is now a purely microprudential stress test. It's not the one that's just micro with some micro overlay. So that creates an additional challenge. The obvious one I mentioned here is this idea that applies mostly to the micro, which is the attempts to game the tests. And after a while, you have this sort of false sense of confidence that these are very granular micro stress tests, but in a way, you may be missing a whole lot of topics that come with interconnectedness. So anyway, I think there are many examples and as I'm not picking up on these two, but I think that we can go also to IMF reports that oftentimes we have missed in our stress tests, some risks that were incoming. But these are some examples that obviously didn't turn out the way that these stress tests were modeling. And you, of course, may know that in the Icelandic case, it was a very traditional sort of solvency-based stress test and I need the banks to look very solid and they did not really have the liquidity in it. And again, the Latvia test didn't expect that the growth of GDP in the right next quarter will be minus 18 year on year. So this is an older paper that I used back in 2006 where we look at the communications of financial stability in FSRs and we have updated it now. And it's another new methodology. I stole it from an earlier paper that looked at monetary policy communication and then it looked at dispersion of inflation forecasts. And what we found is that it's really the quality of communication that matters. And so we've done a redo of this paper to just look on stress testing in FSRs. And again, we following this earlier paper, we sort of have these three measures. One is the clarity and the consistency of reporting and the coverage. And here when we talk about clarity, it means it's not just you clear what the outcome is, but you also clear on what the assumptions are and what the data are and do you put out so the underlying the metadata on the website. So again, we ran several sets of private regressions looking at properties of crisis. First, just looking at the reporting itself. Do you report or not? But then you also look at the quality of the reporting. Of course, you have to control for the fact that the countries that are report, start to report, are more likely to be confident that their system is stable. There was also a period when we saw just stability reports popping up. So we had to sort of incorporate this Heckman equation to correct for the sort of selection bias. And the results were actually quite striking in the sense that it's the high quality or low quality of the reporting that makes the difference. And again, it's the fact how much duty is close on the metadata of the exercise. And again, it's a bit different from the sort of pure crisis management exercise. Publication has no link whatsoever. Whether you publish the FSR or the results of stress test or not, it has no statistically significant effect. We were probably did for publishing the whole list of the FSRs, but at least for the ones that you see here, this is from an IMF working paper. You see that the quality varies quite widely and it's not, you know, it's still as much to be desired. In this case, the maximum is four for these four components. And why is that? Well, especially when it comes to the macro-prudential stress test, the challenges are quite enormous as we've seen. So this is kind of our standard toolkit, I think that you heard about top down and bottom up each of these components. This is all very nice and dandy, but then once it comes to the macro-prudential stress test, ideally we'd like to measure the feedback effect going back from the stress test to the scenario design. Again, this is something that's really challenging first to do and then to explain why the results, I think, in this way may be quite different from the ones that you just get in the micro-prudential stress test. And so what I'm talking about here is we're using the micro-by-bank data to feed back into the macro-scenario design. And we have really done it on a limited basis and using structural of our models. And again, this is something that's still work in progress. So try communicating this, you know, leaving aside all these other issues. We discussed earlier today. I think there was an interesting discussion earlier on the building of reputation. And again, once you're trying to build a reputation with micro-prudential, it's of course even harder because we're not talking about pass and fail of individual banks. This is about sort of coming up with a bottom line assessment for the system as a whole. So this has been quite practically a challenge and that's why most of the published stress tests are in a way simplified version that really gets into the pass and fail and looks at how many banks will be below a certain threshold. Of course, the challenge of truly micro-prudential stress test is that you need to take into account not only the micro-prudential but also the other elements of the financial stability assessment. So it's the kind of oversight framework and the safety nets you have in place. So here's just an example from a Caribbean country that we've done assessment recently. Again, one, you have very different types of shocking to consider. So the issue is when you're doing micro-prudential stress test, you're not trying to look at just the cycle but also you're looking at shocks that hit from, let's say, the US mainland. So also you have some major disasters that may hit. And then you're trying to interpret the severity of the shocks. And when you look at the results, is 22% of capital aggressive ratio sufficient? Well, you need to know more about what kind of regulatory and supervisory framework you have in place. And again, I think this is one simplified way that this particular exercise was presented. I mentioned we've been doing these assessments for 20 years now. So on the occasion of this anniversary, we've been reviewing the stress test and the whole FSAP. So the review will be finished by summer. Actually, today there's a paper that you can download on the IMF stress testing. But we did a survey of country authorities around the world about what they see as the main priority. Interestingly, the two areas that came most strongly were interconnectedness, including cross-border. And then it was the non-banks, because you need to add up with non-banks and market-based finance. And so when you add those up, those were important. And then we also had quite intense interest in these emerging topics, like cyber stress testing, which we've done so far once in Singapore. Work on fintech stress testing again once. And then push for more analysis on climate. And some of these new topics make the challenges of communicating stress test even larger, because of course with a transition risk on climate, you're not dealing with a stress test that's more counter-cyclical, but something that's more like a structural change in the economy happening over decades. So again, it's a very different type of exercise. We've started in this vein. We have an ongoing one on Norway. So this is something that's quite exciting. By the way, back to the topic. I think we are on communication. I think it's important to make this distinction between the experimental exercises we talked about and the basic exercises that Jesus has highlighted. And oftentimes can be quite insightful, because they are more transparent and easier to explain, as long as you're very clear about the caveats or the assumptions that you're making. Thank you. Thank you. So, well, let's have questions. And maybe if you have in mind someone who should answer specifically your question, then, actually I would have three questions. And let's say maybe we don't cover, I mean, you don't cover all of them. So the first one would be for Jesus. You made this point about top-down, let's say, against bottom-up, clarifying that for you top-down, it means that the machine is in the hands of the policy authority, let's say. And I mean, if I understand correctly, understood that you kind of think that the top-down approach, let's say, has some advantages with respect to the bottom-up. I mean, I would tend to share this view. But my question would be now where you think that kind of a top-down exercise could be really conducted in the SSM, let's say, Euro-area context, where basically, I mean, let's say, I mean, banks are much more heterogeneous that may be in only one country. Basically, we have different business models. The number of banks is much higher than, for example, in the US, the ones that are covered in the fast of C-car. And then basically on top also there is the issue that despite, I mean, the amount of data that we have available is kind of growing and improving year by year, basically. Still, I mean, we don't have such granular data as, for example, they were used in these papers today from the colleagues from the Fed. So I mean, in principle, what I mean is that I tend to agree with your view, but I would like to hear your feedback, basically, whether you think it to be something doable at the current stage, basically, in the SSM context. And then maybe, like, if I might ask a question to Beverly as well. As regards you kind of pointed out to some elements that a macroprudential stress test should feature, and I totally agree with those, something like interconnectingness, et cetera, let's say. I would understand also that at the moment, like, for example, the fast C-car do not cover some of these elements, because maybe they're more used also for microprudential purposes. I was wondering whether, in the Fed, you are planning to conduct, or you are already conducting some similar exercises, like something like more macroprudential exercises in parallel, or if you are planning to do something like that in the future. And then maybe, OK, I would have also a fair question, but maybe let's see. OK, I also have a question for Ms. Hirtl and Mr. Saulina regarding. So you mentioned that, of course, one has to weigh the complexity and where to put the complexity in how to design the stress test, also that it involves a lot of granular data and a lot of resources in working on this data. So I would like to ask you what you think about how also regarding the design of the stress test, where the competitive advantage lies on the side of the supervisor and the regulator to work on the data that you have available in forms of the top-down approach can, of course, being the regulator, having the information on all the banks simultaneously produce much more information that a bank can produce on its own. And what you think about how to factor that into the design of stress test. OK, we'll take one more question, and then we answer this round. Then we'll see if there's a second round. Thanks to everyone for the very interesting presentations. One remark to Jesus. You might be accused of laziness with the slides, not with conceptual laziness. That was a very forceful and clear presentation. Thank you. I want to go back a bit to Diane's presentation. So you were talking about regulatory arbitrage in connection to stress tests. So it was more about playing with risk weights, playing with the allocation, the type of asset. So let's imagine, and the stress tests are mostly conducted with quarterly data, like the one top-down with very granular quarterly data. So let's imagine you solve that problem, you solve that link. There is still the issue of regulatory arbitrage that takes place within the quarter. And this can be very large, like a canonical example, repo window dressing by some European banks. This reverberates through the entire financial system. It's seen in the Fed's balance sheet. It's seen in FX swap bases in currencies, which are not the euro. So how should one think about this? And does it maybe change or increase the role of one of the corners of your triangle, namely the supervisors? Thank you. Maybe you want to start with you, Sendean. All right, so the first question was about whether the Fed is working on other forms of stress testing that build in some of the internal linkages. The answer, I think, is yes. I mean, there's a pretty active research community across the entire Federal Reserve system thinking about and building those models. I don't want to speak for the staff or the Board of Governors. I'm in no position to do so being at the New York Fed, but there is recently begun to release a financial stability report twice a year that looks at, among other things, interconnectedness, leverage, liquidity across a bunch of sectors, including banks and non-banks, so that there is analysis that is done that goes into looking at some of those other factors. We at the New York Fed in our Liberty Street Economic Blog series released the results of some models that looked at Firesale was. I think Fernando, who is sitting over there, is gonna talk about some of the modeling that was underneath some of that you'll hear tomorrow. So we're not, at least at the New York Fed, there are not things that are set in the same supervisory consequences that CCAR is, because that is a supervisory program. But there's a lot of work in analysis that's going on. I'm sorry, what was the second? I didn't have something to write down. The second question was just. You were to put complexity on the advantage of the supervisor. Oh, the advantage of the supervisor has. I think the way I would answer that is that it's not so much about whether from the top down or bottom up, they both, at least in the CCAR program, involve a lot of complexity. A big part of what is done in the supervisory review as part of the CCAR program is looking at the models that the banks use to do their own stress testing. And there's a lot of complexity in both ways. I think the one advantage of the top down, the way the Fed has done it, is it's the same models for everybody. So you know you're getting an apples to apples comparison, same models, same scenario. And with the differences varying, based on the very granular input data, so that in a sense that the exact same loan held at two different banks should generate the same kind of losses at the two different banks. And so it's that it's ability to have a common framework where you can say, well, this is everybody together. What does this look like for the system, even if the system consists of just adding up the pieces? I think that's maybe the advantage, one of the important advantages of having top down estimates. Thank you. Cosimo, well the first obvious answer is called the consultants. They will organize you the top down exercise without any problem. No, no, as they did in the past, right? So more seriously, I think that one way you point to the right problem of too many banks, significant amount of banks, but maybe there is a scope for a true cooperation with national authorities that may have models locally applied and you check the consistency, but in let's say enhanced cooperation with national competent authorities, I think that it's possible to run this exercise. And then time is on your side because an accreditation is building and there are other national authorities that have credit registered with different coverage, but you also can explore this credit register together with the national authorities. So frankly, I don't think that this is a big problem. It is true that you need to have this as a medium term objective. This is not for tomorrow. Cannot be implemented tomorrow, neither next year or in the coming two years, but the important thing is whether you agree on the direction of travel or not. For me, this is a fundamental question. Who controls the exercise? If I may ask your colleague next to you, I think that supervisors have a significant advantage. They can ask for the granular data. I mean, on my right is an example of a supervisor that asks the granular data, loan by loan, right? So collateral by collateral. This is something that it is not for tomorrow, but I think it's a significant potential ability that the supervisors have, micro, demacro, both of them. And you need to develop a modeling teams and so on. And the cooperation with the national authorities, I think that could be really very, very, very useful. And if I may, also, although the question was not for me, but listen, you are not gonna be never happy with this line of reasoning. Because you say, end of year data, they arbitrage. Then you ask for quarterly data, then monthly data, then weekly, then daily, and then intraday data. You are not gonna be happy. The answer is very easy. Send the JSTs to the bank. Thank you. You see what I mean. So about the regulatory arbitrage within the quarter or window dressing, I would call this window dressing. I am well aware of that because I've been working also on US money market funds data. And money market funds in the US have to report on a monthly basis and European banks report on a quarterly basis. So if you look at investments of US money market funds in European banks, it's very funny to see the seasonality. So, and why do we have this a bit less in the US is because in US banks have to also report average capital ratios over the quarter. So, and that's supposed that the supervisor is collecting data at a more frequent basis than the quarterly level. So there are some data actually at the monthly level that makes a lot of sense. Well, market measures can also be a way because market measures are available on a daily basis or even more frequently. So if you want to compare with market measures, you will also have that dimension. But also in general asking, where the supervisor is asking for more and more granular data at the single exposure name or at the borrower name level, you might also think that asking for more frequent data might be, of course, there's a cost for that, but that might be also, there might be some value in that. Of course, with this multiplicity of constraints that we impose on banks, they are more and more innovative in the way of doing regulatory arbitrage. So recently, when I was talking with the Swiss supervisor, they were telling me that there's also regulatory arbitrage when banks have to report to exposures. So, the more you impose constraints and the more severe the constraints are, the little, the room for maneuvers of banks are, the more they are going to be innovative, of course, in the way they do arbitrage. Thank you. Other questions? So, thank you to the panel very much. I thought it was a really interesting discussion. I had, I'm not exactly sure who my question is directed at because most of the time when you're talking about stress tests, there were more sort of stress tests where the scenarios feature very sort of acute business and credit cycle downturn type scenarios. But I guess I had a question to do with like, so everyone who works on financial stability talks a lot about low for long interest rates. And so I suppose my question is on your thoughts for thinking about more of these types of like very slow burn type scenarios as opposed to more acute type scenarios and the challenges, usefulness of using stress tests to actually look at these sorts of scenarios, like low for long type scenarios. You know, what were your thoughts on that? Well, I guess I can answer this one because the EBA exercise that is going to be performed in 2020, the scenario was published a week ago and it is a low for long scenario. So there is the usual downturn and drop in asset prices but there's also a shift downwards in an inversion of the yield curve. Now, I don't know whether you call it with the three years, you call that low for long enough. So maybe there could be a case to try and project that even further. But let's say that this year, the ESRB, which is the body that chooses the relevant risk, decided that this was worth exploring. The current framework is over three years. I think from a top-down perspective, since we will have the starting points, we could extend that for longer. Of course, the question is the more you extend it, the more you would have to model what banks would do because the EBA exercise is a static exercise. So already it's a bit of a stretch to say that nothing happens over three years. But if you were to wonder what happens if we have another 10 years of low interest rates, then I think this is something that in the current framework in Europe where banks are not asked to provide their reactions, this is something we would have to do here as a sort of top-down projection where we do have models that project banks' reactions to such a scenario. And indeed, it would be interesting to try that. But we're going in that direction, okay? That's the short answer to the question. Other questions? Okay, well then, thanks a lot to everyone and we'll see you tomorrow morning.