 We have heard interesting presentations and all of the speakers have indeed raised important points regarding ed effectiveness. From what we have heard so far, the balance of evidence on average actually points to a positive impact of aid on growth. So we have actually learned something to answer what Sam had raised at the beginning. The evidence for the most parts is based on a research outcome from individual studies. But another way of looking at the ed effectiveness question is to ask what the accumulated evidence on average will say if we combine all the existing empirical literature in the ed effectiveness area. So this is, this basically amounts to doing a regression of regression analysis instead of doing a single study based on one regression analysis. So this is where our meta work comes into picture. This is the meta paper we did with Sven. So the meta analysis method is originally comes from the medicine profession and it's used to quantitatively combine empirical results from a range of studies and get a single effect estimate of interest. I would say if it's implemented with the necessary caution that the method requires, meta analysis can also be informative in learning from the accumulated evidence regarding ed effectiveness. I should emphasize that of course the conclusion of meta analysis will be only as valid as the care that we would exercise to code and to analyze the data. So having this in mind, our meta analyst from combining 68 ed growth studies also confirms the positive, modest and yet statistically significant impact of ed on growth which is emerging in recent years and which we also have heard from the presentations in this session. On top of this, our study also shows the importance of taking into account the heterogeneity across countries in terms of the effect of ed on growth. So overall the positive ed effectiveness stories we have heard today both in terms of the growth impact and the return of ed investments at the macro level are very much in line with the uncontested positive impact we observe at the micro level. So this basically means that the micro macro paradox is less of an issue for the contemporaneous ed effectiveness debate but even if this positive assessments enhance our knowledge regarding ed effectiveness, I still believe that more research needs to be done to gain further insight on what works and what should be improved in relation to ed effectiveness. So below I will just outline four major areas where we can do better. The first one relates to understanding the channels. As it was mentioned before, we need more research both quantitative and qualitative in relation to understanding the mechanisms and the channels as to how ed works like it has an impact on growth. For instance, the health and education outcomes that Sam has already mentioned. It doesn't mean that this work is nonexistent. It's emerging but more work will contribute to our understanding in this area. And having an understanding in this area is also crucial to make ed interventions more targeted and focused and also it will enable us to make some like to make our expectations reasonable regarding different outcomes. My second point is the need for formulating a slightly different question. The papers presented here today have also shown how asking a slightly different question can add more to our knowledge regarding the role of ed in development. For instance, the presentation we heard from Henrik this afternoon and also the other outcomes that Sam was referring to that step in the right direction. We have heard that investment on ed has a high return but we still need to know more for instance like what the implication of this for long-term productivity and the impact of ed on sectoral productivity like on agriculture and manufacturing as well. And moving to my third point and as it's already emphasized by the previous speakers, we really need to understand the heterogeneity and we need to be cautious in using case-by-case analysts because in the ed effectiveness literature it's already acknowledged both on individual studies and from the mid analysts as well. There is an inherent heterogeneity in the impact of ed on growth both across countries and over time as well. So this certainly points to the need for country by country analysis but I believe that this is very helpful like in terms of giving a specific policy recommendation but we have to consider this analysis as not as a substitute but rather as a supplement for the ongoing aggregate level ed effectiveness studies. The reason is that it's already pointed out that there are some difficulties of like making generalization also we also have a problem of quality data and consistent data across the different source. So keeping this in mind, we have to be cautious about making inference from these studies. And finally we also need some research on what's ed effectiveness requires from donors side. As Sam already points out in his conclusion, donor action is actually important. In particular donor coordination and need assessment is crucial in terms of making ed more effective. And also predictability and ed flows is another aspect of the effectiveness debate. If the flows are unpredictable and volatile that will create a challenge on fiscal planning and the last but not least is the importance of engaging recipients throughout the process including in the planning implementation and performance evaluations. Just to borrow the saying from Oliver it's recipient behavior requires recipient perspective so it's important to engage them throughout the process. Thank you so much. Thank you very much for this nice, we have 15 minutes for the question and answer from the floor, question from the floor and answer from the panel who wants to take the floor. Adrian, on Bigstern. On the Bigstern University of Gothenburg, it was quite interesting presentation you talked about the quantitative impacts or impacts of quantities. I wonder if or what we know about the impact of different forms of modalities of aid. A mechanism was touched upon by Oliver of course but does it matter how we deliver the aid? It's quite important and it's much harder maybe to figure out whether projects or budget support or whatever form it's transferred, how that affects the effectiveness because we're also interested in what the aid does to the institutions and how they handle the aid when it comes in and the way it's transferred is I think very important. I'm not sure what we know about the quantitative consequences of the choice of modalities. So whatever you know about this I would be interested in hearing. Thank you, Adrian Wood. Thank you for three very stimulating presentations and some excellent comments including by the chair. I'd like to in a way suggest to Sam that he write a different paper, that his question be how much has our research helped policy makers and that really it's you take the same material and you rejig it in a completely different framework. Now of course there are different policy makers in different places but let me just put it to you from the perspective of an ex-chief economist of an aid agency. What I would like to know is three things. How can I justify the total amount of aid that's being given to the particular set of countries that our country has already decided it's going to allocate its aid to? And I argue as it were with the Ministry of Finance, the Treasury, about the aggregate amount of aid. Second thing I want to know is how should we allocate this aid among these countries? How much should each, how should it be shared out among the countries? Third question is how should we use the aid within each country and how should that vary with the characteristics of the country? And those are the three questions, three key questions that I as it were had to grapple with the whole time. And I think that your research does shed light on that but I think it would be very helpful if you as it were re-articulated what you've learned in that light. And I address it to you but in a way the comments apply absolutely to Henrik and to Oliver as well. Thank you very much on this side. Madam? Thank you very much. Thank you for all the speakers for excellent presentations. My name is Kristina Kuvaja. I'm from the Ministry of Foreign Affairs of Finland. I would have a question to Sam. You said in your concluding remarks that donor behavior is critical within the framework of international agreements and commitments. Now from the evidence that you have looked at, would you be able to elaborate any key elements of what elements of donor behavior would be critical for effectiveness of age? Thank you. Thank you. Just one question here and please, can you give your name? Hi, Pibi Anand from University of Bradford. My question is to Oliver. Thank you for bringing out some very important insights about the choice of indicators. If I may just request you to elaborate a little bit more. On two points. One, I do much of my work on country case studies so I'm sad that my career is doomed but how we can learn from country case studies to make judgments about quality of data. And second one is many of your graphs show some degree of convergence or similarity in movement so as long as one is not mixing up between two different indicators is the broader picture of conclusion going to be very significantly different. Thank you. Thank you. Thank you. Three excellent presentations, four excellent, five excellent presentations. My question is to Oliver in addition to all the complications of measurement. I think there's a further one if you do as you suggest sometimes try to adopt the perspective of the recipient and use recipient data. Recipient governments frequently have no idea how much aid is actually flowing into their country. It does not all come through the government budget. In fact the percentage that comes through the government budget is sometimes in the order of 35 to 40 percent. That's a huge difference when you're trying to find aggregate effects from aid flows from the perspective of the recipient. I think that simply has to be factored in. And donors as yet although they have promised have not succeeded in providing data which is usable for recipient governments about the size of actual flows to their country emanating from donor sources. And that includes NGOs as well as governmental sources. Thank you. That was very intriguing indeed but it leaves me with a question. What is aid supposed to finance? That is what is the market imperfection we're trying to address? Partial answer was very clear in Henry's presentation. It's a particular type of capital infrastructure education whatever it is and aid makes it possible to finance that because you don't have the possibility for taxation and apparently there's a international capital market imperfection. That's the familiar case for aid. It could have been phrased not in the same words like that at the beginning of aid in 1950s. People were very much thinking in those terms and I think the evidence which you've now got is supporting that case. Now, let me go back to Patricium's very interesting introduction. He very modestly referred to his own work but I want to highlight that a little bit because that is a very different point. So what comes out of Patric's work is that aid seems to be particularly effective in one group of countries which he calls vulnerable countries and we can argue about what exactly vulnerability is but it seems to me a completely different type of case for aid. So my question to the panel is okay, effective or not, what is it supposed to be financing? I'm still not quite clear on that. Thank you. Thank you. Yes, Samo Luangwe from Tanzania. I just wanted to, first of all there are great presentations. I have learned a lot in the process. But there are two questions which I would like to hear which are actually dominant in the current debate in aid. One is the politics of aid, both from the recipient side and the donor side. What politics is driving the process? So I wonder whether from the insight you have gathered, whether there are any issues we can throw in that regard. Secondly is a challenging question of governance as a conditionality, especially and that's one of the big differences between aid from the OECD and aid from, say, China and other more recent sources. Is it influencing the way OECD aid is behaving? Because with governance, the challenge we are facing is extremely unpredictable. A big story in the newspaper is enough to cut aid, suddenly without notice. So the message conveyed is if there are governance problems, hide them, don't let them come out in the newspapers because the donors will immediately cut until you explain. So how to handle governance in a more predictable way seems to be a continuously problematic issue in the predictability of aid. And after all, there are big questions whether it really influences the quality of governance anyway. So if there are any reflections in that regard. Two last and short questions, just here. And the last one for Richard. Thank you very much, Peter Huote from University of Ghana. I've listened to great presentations, but I just want to throw this to Oliver, your point about case studies. I think that's the norm, journals don't really appreciate case studies. But then if you look at Easterly's book on the white man's bedding, it says we spent too much time planning, we look at a big picture. Whereas the way forward is to look at small projects, small case studies, and that is where all the nuances of aid, effectiveness comes to play. So I think perhaps we have to critically take a look at this. Are journals for academic work or are they meant to inform and also inform policy? Thank you. Thank you. And last question, remark from the post doc, remark Richard Carré. Thank you. Yes, Richard Carré. I was with the Development Assistance Committee for 30 years, so I've been thinking about these issues for a long time. And thank you for very, very interesting contributions. And Oliver, I'm involved at the moment in a set of country case studies in Africa on the management of external resource flows. So all of your material, very, very interesting to us. Now, the remark was made that aid will not transform the developing country. Only a developmental state will do that. That's what Joe Stiglitz told us this morning. But what was coming up this afternoon was the macro-micro paradox. Nobody really asked the question, what about illicit flows? And that illicit flows are on the agenda now of the G20. And is this really something that needs to be much more factored into the kind of analysis that you've been doing? Look at the Lucas paradox, which Justin Lin brought up yesterday. Robert Lucas in 1990 was saying, why doesn't capital flow from rich countries to poor countries? He said that rents are captured by elites in the south and recycled to the north, just as they used to be in the colonial period. And this is really what happens in the world. And the remedy to that that he gave was bringing foreign direct investment to break the grip of the elites and invest in human capital. And then eventually you will get onto a developmental trap. And then in a country like that, there is also no learning by doing process. So you have this capital outflow that is really helping to explain, I think, the macro-micro paradox. And finally, has there been damage to institutions from the aid industry? And I think that there has been damage to institutions. The aid industry has been fragmented, low predictability, high transactions costs, etc. And those issues were addressed by the whole series of high-level forums from Rome, Paris Accra, to Busan with some success. But the whole issue is that the aid industry isn't good at helping to produce systemic outcomes because it's very fragmented, and that's really a very big challenge for us. So thank you again very much for these contributions. Thank you. We have a few minutes for each panelist to react and answer. So many very interesting and relevant questions. Some you begin. We'll keep the same order. Okay, thank you very much. A large number of very interesting questions. And let me apologize in advance if I don't answer individuals specifically. I think that's going to be impossible. So just to, I think, starting with Adrian Wood's comments, how can we reformulate some of the lessons to be more policy useful? And of course the danger is it's very difficult to go from aggregate, average coefficients, if you will, to what should I do for a specific country? And I think that's always a dangerous leap to make, only on the basis of the kind of evidence that is available from these studies. So of course, there is other forms of evidence that one must use. But even so, let me suggest perhaps a way forward. So a way forward with respect to at least total aid volumes and allocations of aid could be more based on rates of return. So what kind of costs are countries facing when they go to raise commercial debt for long periods of time? Can they access these kinds of vehicles and at what cost? And I think that's a starting point for having a sensible conversation about the need for concessional financing to specific kinds of countries. And I mean, for example, large developing or large emerging markets, often, at least when they're growing stably over periods of time, can access commercial debt on quite reasonable terms over long periods, at long durations, whereas a South Sudan might not be able to. So that's an important starting point for, I think, the aid allocation discussion. How should we use aid in individual countries? Well, if we take seriously the issue of long run outcomes, well, I think that's the place to start. I think there's probably good evidence or the best evidence of aid effectiveness is in contributing to accumulation of human capital over long run, so providing public goods, as well as in infrastructure. So these are, again, all the things that sensible governments ideally should be doing and doing more of. So again, that might perhaps provide a starting point. That links a little bit to the question of what is aid trying to finance and perhaps forgive me for being boring, but I think it is accumulation of the critical infrastructure to development human, physical, and institutional. With respect to the questions around donor behavior, there's at least, well, there's three main, the three C's of donor behavior. I mean, one is donor coordination. Are donors significantly or meaningfully coordinating their behavior in a given country? There's quite a lot of reasonable evidence that donor fragmentation, donor fractionalization, donor conflict reduces aid effectiveness. So this links to Arne's question about modalities. So modalities of aid that support donor coordination, sectoral instruments, budget support instruments, I think should be preferred as a starting point, as a point of departure. The evidence for this is not so much. I'll be quick, sorry. The evidence of the differential effectiveness of these kinds of instruments isn't as good as we would like, partly because they've only really been operating for the last 10 years, if that, and inconsistently across different countries. I believe there was actually a poster session paper yesterday that was looking at budget support to the health sector in Malawi. So that might be one to look at. The other two Cs, credibility in partnerships and developing a conversation and consistency in donor behavior. So I think those are kind of the three Cs. I'll stop there. Sorry, an apology is to the other ones. Knowing that Oliver got a lot of good questions, I'll be very brief. Let me just quickly address the... Arne and Adrian had this modalities, justification and allocation. I mean, this panel, if you imagine you would have easterly as the DuFlo and in Japan a G sitting here, it would have been another session. And it would have been more micro-ish, and probably some of what you need could be micro-ish. So what you get from this panel and this research is the justification in the sense that if there is an overall positive return to foreign aid that could justify in your constituency that we actually spend the money. And I think being from Denmark, I can tell you that our aid budget is under pressure. Huge pressure. I think that goes for all Scandinavian countries. We had the Finnish president here this morning, also under pressure. So in that sense, our work is for the Northland in that case. And when it comes to allocation, I was so happy watching O'Connell yesterday because you would see that USAID doing more or less than if it is doing as Danita is doing this growth diagnostics, surveys and a lot of micro-analysis because that's your job. Specialization is important. Our job in this case is justification. And you can look at all these models for the needs if you like. What are the needs? And that will be based on different models. And they have been derived. And as I said, another panel would be explaining how you could use these. For the donor behavior, I fully support Sam for Richard Kerry. I think in this group, what you should do is actually also approach IMF. As far as I know, IMF changed their balance of payment statistics so we can no longer identify the state-to-state transfers, which is a huge problem because one nice addition to the aid flows that all of you are going to talk about are state-to-state transfers we can find or could find in balance of payment statistics. And I've been looking at these and they're much, much lower than the aid flows, of course, right? But this is one new source or different source of information, but we cannot find it with the new definitions. Thank you. That's it. More questions, answering all of those questions will be longer than the presentation. So thanks for very stimulating questions. I'll try to use Adrian's three questions as a framework to hopefully comment on all of the questions to some extent. But the first thing I would say to the minister is that, look, one thing you've got to be aware of, whatever you do, there will be one constituency and perhaps you can identify the newspapers. In the UK you certainly can. There will be one constituency that is against aid. It doesn't matter what you do, they're going to be against it. There will be another constituency, Civil Society, NGOs, that will probably be for it. So recognize that whatever you do, that's the world out there and you might want to pre-empt that. But then you say, right, how can you justify aid to countries? You say, well, look, on balance it makes them better off. They're better off with aid than they are without it. Now you have to introduce an important distinction here which comes into many of the questions. You've got to think of one set of countries where it's a crisis, it's an emergency, it's fragile, it's vulnerable. The mechanics of aid, the arguments for aid are very different in those situations than in a situation where it's continued to support to a particular government. So you get to, and I think you have to make these different arguments, I'm going to focus on the ones where it's continued support to a particular government. How should you allocate it? Well, basically you do what you probably have been doing is concentrate on those countries where you've developed a relationship of trust. Where you think, yeah, I can get on with these, I can relate to these, I understand where they're coming from. Yes, I know there's a leak capture, there are illicit flows, but I know that's the case, but I think these guys are basically okay. Has the aid industry damaged the institutions in these countries? I don't know if it's damaged them. I think up until certainly the 1990s, the lack of coordination of donors and the proliferation of different donors with different requirements, all wanting to meet the officials, that was a huge burden on those countries. Now, I think that has improved not as much as it should have, and that's the donor's fault. They have lots of meetings about coordination, but they never actually do very much in any countries, and that's partly because each donor wants to be able to have a flagship for its aid. The donor policymaker or politician is often not really that concerned about, does aid work, does our aid work? And that's a different question, somewhat. But does it influence governance? And I think there is some evidence that, yes, it does influence governance if the government thinks that the donors will actually respond to changes in governance. So if they think that improvements in institutions or democratization will lead to more aid or more predictable aid, they will be motivated to change. If they think it doesn't matter, the donors aren't going to punish us, then they won't respond. That relates to this issue of convergence. Yes, there is probably a lot of convergence in the data because for a number of things, first of all, data collection has improved in all of these countries, and aid has played a role in that. Aid has played a role in investing and improving the ability to collect data. There's a greater international standardization of how you measure, how you collect things. Computerization at customs has greatly improved trade data, tax data. So yes, over the last 10 to 15 years, there have been significant improvements. There have also been improvements in public financial management, in macro policy that we heard in one of the sessions, and aid and donors have all played a role in that. So yes, over the last 10, 15 years, there have been these improvements, and I think you want to emphasize those to the policymaker. You may not pick these up in GDP, but these are really important. They're probably far more important than something you might pick up in GDP. So as we heard in the session yesterday, you can put the positive spin on it. So you pick the examples that work, but there are a lot of them. And often, the cases that haven't worked, there are reasons for it. Just to comment on Jerry's point, yes, that is particularly if you want to do time series country studies. And particularly if you want to look at the donors. Yes, recently the donors are trying to get better measures of the aid that goes to the country. But you might have that for 10, 15 years. That's no good if you want to do a time series country study. You need 30 to 40 years. And it's really difficult to get reliable, consistent aid data going back. And some countries are better than others. So in Tanzania and Kenya, we were actually able through government statistics or budget statistics to go back and get a measure of the aid they record as having received. But in Uganda and Ghana, we haven't been able to do that because we just couldn't find that data. So it differs from country to country. So then the final one, how should I deliver it? I would say general budget support if possible. What you're trying to finance is public goods and services. If that's not what you're trying to finance, why are you calling it aid? That's the aid. And the reason you're doing it is because you think the circumstances in this country is that at the moment their ability to collect domestic revenue is insufficient to finance their expenditure needs. So that's you're filling that gap. That is the justification. If that's what you believe, and if you trust the country, give them general budget support because that's much more efficient from the recipient's perspective. Now some donors don't like that because, well, then it's not our project and we can't put a plaque on it and we can't sell it to our constituency. Well, tough if what you really want to do is help this country and you trust them, give them general budget support. Thank you. As a conclusion, one short remarks about the data. Nothing has been reflected in the discussion about the new measurement of ODA, agreed by the DAAC, and there may be some prospects for new analysis and measurement of effectiveness with the total effective support. I think there is a new field of research due to the better definitions maybe of ODA and even more than ODA is the total official support for development. My last remark will be to refer to the Adrian Wood topic. This was your first answer. I feel at the end of this discussion that there is still a large gap between the research on ed effectiveness and the support to policy analysis because when you raise a question of the allocation between countries, of course you need to have some consideration of justice of equity but also and it was very important from the donor point of view on effectiveness. If we have no empirical grounds to assess the effectiveness of the different countries from the research analysis, we are a little embarrassed to link the research with policy advice. And it's very clear in the debate which has been developed among the multilateral development banks during the last 15 years about the PBA performance-based allocation and its reform. At the beginning it was supported by the dollar burn side analysis that ed was more effective in countries with better governance and it has been what was wrong in the substance but what was interesting as a way to raise the question of the heterogeneity of the effects depending on some specific characteristics of the country the characteristics involved were not maybe the most relevant but some other are relevant and they are to be investigated to give some indication about where the funds are the more effective and not only distributed according to equity and justice but also according to effectiveness. And I am impressed by the, I agree with Oliver saying we have so many particular case, the fragility and so on but if you are a multilateral institution you need the general rules and the present state is we have a general rule, PBA some kind of PBA but since it's not applicable you multiply the exceptions and there are so many exceptions especially for fragility that the rule is applied to a minority of countries so it's no longer a rule and for that we need some good indication from the research and that's the reason why I raised at the beginning the issue of heterogeneity so that's only for say that the wonderful research which has been done here about ed effectiveness is not the final point and there is still work to do for the researcher about ed effectiveness thank you.