 So thank you for the opportunity of having me as a discussant here. I see a lot of very illustrious faces in the audience, and I'm sure we'd all love to hear their thoughts as well. So I'll keep this short. So thank you to the speakers for giving a really nice overview of where we stand in terms of microeconomics for development. As Carol also started the session that we've been talking about, the bigger picture, there have been some thoughts on how with the bigger picture we are, or rather with the focus on smaller analyses, we are maybe losing sight of the bigger picture. And I think that this session sort of really got all those thoughts together. So I'm just going to keep it a bit more general. So I think this is also going back to the idea of synthesis that was that Stefan Dekon talked about two days ago. And I think this has also been raised very nicely by Alan in his opening presentation today. And I think as somebody who's starting out a career in academic and policy research, I think that with sort of focusing on the bigger picture, we also need to understand what are the mechanisms that make these bigger policies work. So I think that that's where the body of these smaller interventions, whether it's using experimental methods or observational data, we need to get sort of all the analysis together in order to be able to synthesize it to understand what works, what doesn't, where there are positive results, negative results, before we can really come up with smart policy suggestions. In terms of sort of more specific comments, just from the reading of my reading of the literature, I'm not completely convinced that the fact that training has not worked for sort of enterprise performance is a very decided result yet, given the sort of shortcomings of the existing literature that David and Chris highlight in their 2013 paper. There have been a lot of statistical problems with the papers that have done it largely on account of power, in terms of sample size, the sort of time spans that these studies are considering attrition issues, reporting bias and things like that. So I think until we can get a number of really good studies that are mitigating these concerns, we probably shouldn't sort of get to the thing to say that these training programs do not work. Moreover, they also have a recent NBER working paper, which is using a bunch of their own data from previous studies, which actually shows that these management practices that small firms adopt are positively correlated with firm performance. So I think that there's a sort of disconnect between what the observational data shows and what the experimental literature has shown. And I think that we need to sort of reconcile that. Also, there have been these spate of papers on training programs, whether that's at the more personal level. So we now have a bunch of papers that are looking at these skill building programs, cognitive skills, non-cognitive skills. But I think we still don't understand what components of the program work and what doesn't. And I think that we probably need more research in order to understand what aspects of these programs are more successful. And we also know from experimental literature in the lab that framing really matters. So we need to take that into account. And a lot of countries, particularly developing countries, are taking up skill building programs, but we need to know what works before they can be advised on how to take this forward. And I think that this also relates to Karen's point about sort of measurement issues. And just another observation, it seems from what Alan spoke about and also what Chris mentioned, is that mitigation of risk appears to be one of the sort of key avenues for helping households as well as firms and farmers to undertake more productive investments. And I think that we need to think more about that, to think about what would incentivize these people to maybe even invest in these technologies when they have to pay a little price for it, even if it's not the full market price. So these are just a few of my thoughts. Thank you. Thank you once again for being here and thank you for the comments as well. So I'd just like to keep it short and mention a few points that I found exciting in this session and a couple of other sessions as well. And these are things that I would like to work on in the next few years. So the first one was that I think that we can spend some more time thinking about how field experiments can be used to understand spillover effects. So just as an example, we know that there's a tight two way relationship between agriculture and health. So interventions in the health domain would have effects for agricultural outcomes and vice versa. And I think we can still work further on understanding that. And then also from the point of view of availability of data, it's quite exciting to know that there are more carefully collected panel data sets over longer periods of time. So I look forward to thinking more about intergenerational mobility and transmission of various effects. Finally, going back to more measurement issues, there was some talk yesterday, two days ago in the conflict session as well, that I think we need to move beyond just looking at the incidents of conflict at the level of the household and to more thinking about designing new questionnaires to look at the effect of conflict within the household. And finally, there's also been a movement towards measuring individual level behavioral and personality traits. I think there's still a lot more work to be done in measuring them better and also understanding the role of aspirations and expectations. So thank you. So we've time for some questions. So I'll take maybe three questions and then we'll see if we have time for a second round. So I'll start at the back here, would you please? Yes, my question is more specifically addressed to Shannon, but I entirely agree with you that we would like to have longer-term effects of RCT studies. But the question is that it seems that the methodology of RCT is particularly inappropriate for longer-term effect due to attrition and due to spillover effect. So it's indeed a self-contradictory statement. That's what we would like to do, but it's not the proper method because there are very specific conditions that we need to be met for the impact to be properly measured. Okay, I'll take one from the front. I think Alan's presentation put very clear background stories about the development of microeconomics and I entirely agree that microeconomics work had to be put into macro context back when you finish doing micro-works and we shouldn't forget RCT dominated and it brought very good evaluation techniques, but we should understand that development is a long-term process as Philippe mentioned and it is basically you cannot really do it RCT in the long, long-term time dimensions. So we have to be aware whenever you take, make evaluation and conclusions not to be too decisive. But my question is very much to Woodruff. I'm not surprised that microfinance product is not good for scaling from micro to because it's a standardized working capital and basically to get from micro to bigger scaling up enterprises, you need much more firm specific financial product. It's very financial services, very information-intensive things and you cannot do it with a standardized one. So it's not just grace period, two months, changing it, but much more thinking about it. And in that sense, yeah, I will quickly. In that sense, also I have a question, training. You mentioned that actually industries, specific sector, specific things are more important. So if training is too much general training, I'm not surprised it didn't have effect, but if it is training which is more geared to that particular sector they are working, that might make quite different. So again, information. Okay, I think we'll take one more before, I think I saw Louise's hand first. Thank you, I'm Louise Fox from the University of California at Berkeley. I first wanna congratulate the panel members. It was a terrific, this was a phenomenal panel. This was terrific, really good. I really want to focus a bit on Chris's presentation, but I'm gonna pick up on what Alain said about the macro micro linkages. We used to do something called the Micro Foundations of Macro Economics and it seems maybe we need to go back to that at least for development. And I think if we went back to that, Chris I would rephrase the question that you were answering. What do we know about creating wage employment? That's what your graph said that you put up, is that there isn't enough wage employment and that was where Santiago Levy left us at the end of the last session. And I think part of the answer is macroeconomic. It's Roderick's work about deindustrialization and the fact that the manufacturing sector is much more capital intensive due to globalization than it was years ago. And that the service sector is a sector of smaller firms and lower productivity. And so that leaves us then. I'm not sure the right question is, can we get self-employed and small enterprises to grow? And I think that when you talk about the marginal returns to capital in small enterprises, it's important to note that most of those grants may have increased profits, but they did not increase employment fantastically. And I think we need to remember that about your whole discussion about self-employment and these graphs and the shortage of capital. And finally, I would like to... And that's the problem with generalizing, by the way, from these RCTs to the broader generalizing beyond an RCT is that giving a grant is fundamentally different than giving a loan. And the people's behavior to a grant is fundamentally different than a loan because, I mean, Stiglitz has explained this to us many years ago about people's attitude toward risk. Finally... Yeah, I'd just like to say on... Chris, I think one question I would pose and maybe add to your agenda. Do you agree that we have underestimated in survey research on firms? Most of the survey research has been on households and not really on firms. Okay, so I think I'll ask the panel to respond and then we might have time for another round. Yeah, the issue of attrition and the kind of short-term, long-term is not specific to RCTs. We are going to whichever entry point we take, whatever measure we have, whatever way we have used to manage this entry point, we're going to be affected by how do we keep track of the longer-term impacts. I think there are interesting ways of combining natural experiments with RCTs. For example, think of an experiment where you do an RCT, a progressor type, right? And then you want to follow the long-term consequences. You're going to have a lot of attrition to follow specific individuals. Maybe you need to project it at the higher unit of analysis, at the municipality or at the regional level. So you are going to have a kind of dilution of tracking particular individuals, but they are going to be, of course, there's migration and there's going to be attrition across municipalities if you like. But you can recoup some of those spillover effects and some of those longer-term effects by kind of redefining, projecting onto a natural experiment following what you have done at the level of an RCT initially. And then you can do the opposite in terms of what you are proposing, which is start with a natural experiment which allows you to kind of track the broader longer-term impact. And then it raises issues of design, issues of incidents, which then you take on via specific RCT. So you go the other way around. We did, for example, work on the credit bureaus in Guatemala. You look at, first, the rollout and the impact that the credit bureaus have in the general sense, but then it asks very specific questions as to who are the beneficiaries, who are the ones who are hurt. And then you do maybe an encouragement design. You do an RCT. So I think we have to be careful not to put the method before the question. And once we have the question, it eventually suggests ways by which we can reconcile being precise in terms of measuring impact using RCTs, but at the same time following of the broader implications of what those interventions have using other kinds of data. Chris, do you want to? Yeah. So let me quickly go through a couple of things here. So first, let me start with Louise's comments. So fantastic. First of all, I'll point you to the PEDAL website, the program I run for DFID, where there are four research themes, one of which is micro-founded macro model. So we are trying to push in that direction. I can think of a handful of people who I think are doing fantastic work in that area, but it's a very small handful, and I think it's not enough. And we've had trouble getting good proposals on that theme in particular that I think is particularly challenging, but I agree with you completely that that's an area. Second, I would say, I don't think I wanted to say that scaling up micro-interprises is the way to move from the top left to the bottom right. In fact, by bringing up John Sutton's work, I think I'm saying that the evidence we have in large terms, a lot of them start large. There are other reasons to be interested in profit and income from micro-interprises. That tends to be people from the lower half of the income distribution. Even if they don't grow, if you can increase their incomes, there are reasons to think that might be a good thing. Clearly grants work differently than loans. Again, I see this as part of the learning process of trying to understand then what is what we need to do about the loans to make them work more like grants. I think the research itself illuminates an area. The other thing I'd say is it illuminates an area. This is very much the stereotypical, why are you looking there? That's where the light's better. We can do RCTs on micro-enterprises. The samples are big, small grants are enough to get a shock. We can't do that very easily on large firms. That's a problem. I want to highlight in this that I think the focus has been too much on small firms because the methods that have been used have been focused on small firms. We need to think about how we do something in larger firms. I think we're making some progress in that area, but it's come later. I can't point you to a lot of things yet, except things that are in progress. The last thing I want to say is to say that we also, I think David and I are not saying that training for micro-enterprises doesn't work. We're saying that we don't know anything that we haven't learned because of it. Again, I guess when I reflect on that, I want to say maybe we haven't learned because this is just the sort of problem that RCTs are not so good at solving. Perhaps there's a design out there that would actually make more progress in that area. Just to compliment what I'm saying on the long term. I completely agree that the concerns about attrition and general equilibrium are mythological concerns that can be addressed within a RCT or without. It can be costly so that kind of requires and in a certain way the question is more when do we do that extra effort? I think that's one part of the question. The other one is thinking if we're interested in the long term we should be thinking about that when we design those studies and go from there. When we design the studies and we know we want to know the long term then the certain types of information we need and if the general equilibrium is a concern either on the short or on the long term we need randomization at a higher level to take that into account. There is a trade-off there between the short and the long term potentially but that's something that we can consider. The other part and this is coming back to the part that I skipped over I think understanding the context and the household decision making and the theory can also help us understand which parts of the population we expect the long term impact on and that then allows to in a certain way going away a little bit from the average treatment effect on the entire population and thinking about where should we focus or efforts because then it becomes easier in the sense of if we track the population that is of the highest interest and that's a more defined population the ones where we have seen short term impacts for instance because if there's no short term impacts it makes sense that there won't be much long term impacts although I guess you can tell a story where that's the case so I think it's an issue of being more innovative in a certain way and so that's kind of the I didn't explain the methodology behind the Karagba results I presented but that's kind of what we did we focused on the cohort that should have had the highest impact based on the drop-out rates etc and then we did a lot of effort to find them no matter where they went and then we used kind of attrition methodology to bound to whatever we find subsequently so we started a bit late so I'm going to take two more questions John maybe John John Rand University of Copenhagen it's a question for Chris following up on Luisa's comment and referring to the sheer work with the Klinow they also did some work or he did some work with Ben Alton as well about the missing middle but what I took from that work is actually what you also said now that we may be focusing a little bit too much also on research on the smaller medium scale enterprises just let policy makers a little bit to focus their credit schemes on the smaller medium scale sector because researchers are doing a lot of effort in documenting that you have big effects of actually granting capital grants to these firms however the sheer and ultimate work shows that large firms may be more relatively more capital constrained and they are more job creating as Luisa maybe is referred to so are we doing something are we biasing policy makers as well by the research that we are doing on focusing on these experiments on the small and medium scale enterprises Thanks John and then one last question over here if you can make it quick please Thank you I'm from University of Gothenburg I have a question for Professor Goudre very short, you mentioned about capital labour and technology but as I come through to your work there is also one more the surrounding institution of firm may impede their growing bigger because moving bigger may become more visible to the public and you have done job in Vietnam showing that firm realizing on informal arrangement to set to dial the contract and I have done the same research in SME in Vietnam and I felt that paying more bribe likely to have in compulsory industry likely to have environmental certificate on why firm is also likely to use it through surveying route so my question is is this a value for more such on why firm cannot grow in big OK so Miss Miel, so the last question is a great question or biasing policy makers by focusing too much attention on this I have to say I think in most contexts the large firm owners are better connected politically to capital than small enterprises do so maybe there is a sense in which we are offsetting but that is not the right answer because clearly we do need to look at what is happening in large firms as well this is a place where I think there is some work out there Ester and Objit have a paper that has been a working paper forever and I think now it is finally published using changes in policy for slightly larger middle-sized firms in India that looks at marginal returns to capital investments where set-aside programs were changed the thresholds for set-aside programs were changed allows them to look at access to capital for larger enterprises that is the kind of stuff we are going to have to do because you are not going to give grants to firms where it is $100,000 it is going to have to be things like that I know several, I tried myself in Mexico, Santiago is still gone I tried myself in Mexico Santiago is not involved but to do something with a program on credit guarantees governments spend hundreds of millions of dollars a year in subsidies for large-scale firms through credit guarantees large-scale medium size much larger than the small size we have no idea whether these work or not whether it is all inframarginal or anything that happens at extensive margin I tried to put something together in Mexico I know there is an Antoinette shore and Russell Toth I tried to put something together in Indonesia I think that was still alive but barely these are really hard programs to try to assess and I think we have to go back to thinking about how we use the data that is available I agree completely we shouldn't be ignoring that they are much more challenging that is where we need to move surrounding institutions I agree completely I think the microenterprises I worry less about that than I do once firms get to a certain size and the things that have been done on training a few of them go to slightly larger firms and you are going to run into that there are going to be constraints from being visible there are going to be constraints from running past your known network of suppliers and customers that makes it hard to expand more and so forth again I can point to a couple of things that are ongoing Dan Kenison has a project a part of his BRIC project where he wants to do something that is going to shift the cost curve that is going to tell us a lot about the expansion of firms that are slightly larger and what the demand curve at the firm level looks like which is I think one of the big research issues now is I can think about industry demand curves but I want to know about the firm demand curves I look at a place where there are 100 people doing exactly the same thing textbook tells us they all face flat demand curve they don't they don't for a variety of reasons including institutional ones you mentioned and I think we don't have a good sense of how much friction there is in those markets and I think we need to think that out we are ongoing in agricultural markets and ongoing in Dan's work in the BRIC sector and some others that are doing that but I think that's one way to kind of make progress in that area Karen and I maybe to sum up if you have any last comments we have been working with large firms you can have access to administrative data you can work with large banks it's a different set of contacts there are less degrees of freedom but you have a larger database as well so I think the working style is different but I don't see why a priori there is such an impediment to asking questions about large firms that you could ask about small firms okay so I'm afraid that's all we have time for so I mean I agree that it's been one of a very interesting session and we have come a long way but it's a lot further to go in relation to looking at the microeconomics of development so I'll let you all go to your lunch now and I won't say anymore thank you everybody