 At least the way that I framed it in this presentation was to say that there are different communities within the development enterprise. And the way that aid officials and the way that the elite researchers think about aid effectiveness is somewhat different to the way that frontline practitioners think about effectiveness. So for a frontline practitioner, you're trying to be useful on a day-to-day basis and you're trying to make sure that you're a professional, faithful employee doing your job. But for, by the time that range of different activity gets aggregated and it gets talked about in Busan and Korea, for example, then the aid intelligentsia is stressing out about getting to all the wording is right on the international agreements or imagining that and while hoping that in some sense the coherence and coordination powers of within development turn on the extent to which people are responding to these kinds of documents. And so I was just trying to say that those are very different communities of practice and they both have a place. There's, you know, we need to have good research on these things. There's an important role for international actors who want to try and articulate, you know, agreements between organizations and nations to make sure that resourcing and activity is done in a coordinated way. But, you know, when a frontline practitioner is showing up for work every day and they don't think they have in the back of their head am I being faithful to what was agreed on in Busan? They're just trying to survive. They're trying to do their job. So once you have, really once you recognize that there's something of a disconnect often and necessary or was disconnected between what people are doing on a day-to-day basis versus what gets talked about at a higher level of aggregation, then you just need to, that implies that there are very different ways of thinking about what this concept called being an effect, aid effectiveness or development effectiveness might actually mean. So, and then sort of between those two things, then you also have them, you know, the actors that are doing development are organizations or countries trying to enact a policy or enact a program. And those particular actors exist in a political economy within their own country. And in a country like Cambodia at the moment, for example, what does it mean for the World Bank to be effective? Well, as it happens, for various different reasons, the whole lending program is in abeyance at the moment. We're not lending any money because of some disagreements with the government about how a particular program of work had been implemented, right? Now, if you have very narrow or at least have the intelligentsia view of development effectiveness, you'd want to sort of have a checklist of things that you're doing. You want to, are you contributing to millennium development goals? Are you effectively coordinating with other actors? It's kind of, if that's your metric of what constitutes being effective, but your whole lending program is suspended, it's kind of hard then to claim that you're being effective when there's no activity to be done. But there can actually be a way of being effective, and that will mean trying to restart negotiations with the country, trying to ensure that an ongoing dialogue is sustained, and that you won't be able to measure that in any sort of formal accounting way that would then be able to be presented as part of a broader platform of work, but that's what being effective means in that particular context. And so part of the broader way in which we've been trying to think about this stuff is to say there is sort of the big picture, an international level agreements way of thinking about development effectiveness, which is reflected in millennium development goals and these kind of things, and you're effective if you are contributing to these international agreements. And then there's the more super micro level activities of day-to-day employees, but there's sort of a missing middle, and we call that sort of a middle way approach to development, which is sort of trying to think about how can organizations build up their own capacities to be useful, and a lot of that capacity to be useful is about responding to the prevailing problems that exist in the country, but those prevailing problems may or may not be the ones that when you try and respond to them add up to things that the rest of the world would ordinarily recognize as being effectiveness. So even within the intelligentsia space, even when we have sort of researchers in the narrow sense of people that worry about trying to net out the impact of a thing called a project, we find ourselves at the moment in a world where there's a particular dominance way of thinking about what counts as a plausible, defensible answer to the question of whether my project in Ghana, for example, actually works. So in the way in which we've set that debate up right now was a fairly narrow set of criteria for what counts as an effective answer to all of that. And part of what I was trying to say with that was that even with a very neat and tight research design in a strict methodological sense, data doesn't exist in a theory-free zone. Data always has to be interpreted in the context of expectations and with respect to a theory of change that makes sense of that data. So I was basically saying if you took two projects, but in this case they happened to be two seeds and unbeknownst to you, one of them was an oak tree and one of them was a sunflower and you put them in the ground and you had a tight timeline for effectiveness in the narrow sense being put on you and you checked back in in a year's time to see how well those seeds were performing. You'd have beautiful, flowering, sunflower, six-feet tall, beaming away at you and you'd have nothing on the other one. And if you were a strict empiricist, you'd be saying, wow, my sunflower's been to attackally successful, aren't I great horticulturalist? I've done very well. I don't know, isn't this other seed terrible on this other? It's a failure, we have nothing to show for it. But that interpretation would only be correct if you were willfully ignorant of a theory of seed production in which you would know that it actually takes only about six weeks for a sunflower to reach maturity and it takes two years for an oak tree to even begin to pop up through the ground, in which case you've made a very fundamental error of attribution regarding the effectiveness. So the point of that is to say, well, at least in horticulture, we have theories of change that correspond to each of the kind of interventions or seeds that we have. We don't have anything remotely like that in development, it seems to me, we assume implicitly that all projects unfold in a nice little neat straight line. And the challenge is to measure the steepness of that gradient and you're a failure if it's a flat line and you're a spectacular success if it's nice and vertical. But most of the interventions that I happen to worry about and work on unfold what we call a step function, sort of lots of nothingness and a big change. And if you look at what's going on in the Arab Spring, the moment, for example, if you look at the governance indicators over the last 10 years are only heading downhill in terms of their measured effectiveness, but all of a sudden now we're seeing a flourishing of democracy. Or if they're not step functions, they're J-curves. They get worse before they get better. And if you're not aware of that, you can suddenly start to make proclamations about the impact of projects that if you're unaware that you just happen to be in the trough of the J-curve, not only have you not had an impact, you've actually made things worse. So the point of this was to say, well, at least in this one field called horticultural, we make determinations about impact and effectiveness conditional on a theory of change or a theory of seed production that we work with. Whereas in developments, everything has to be unfolding on a straight line and we deploy a very particular kind of instrument to assess the impact of these things. Usually now randomized controlled trials, but my point was simply that even the squeakiest, cleanest, randomized controlled trial didn't give you an accurate interpretation unless you had a good theory to inform that particular thing. And the sun clouds and no trees just seem a nice sort of juxtaposition, but I think a lot of people are starting to recognize that's a more generic problem, but these days at least is on local level justice reform, trying to make sure that dispute resolution procedures and the gradual transfer of rule systems that previously existed in the minds of village elders and chiefs into something that is a bit more universal, that can be adjudicated through professionals called lawyers or whatever. And that's a very different kind of intervention or different kind of problem. So it's important to look at these things historically because the law, as we now understand it, isn't the outworking of a great legal mind that sort of had a very clear goal and everything, all our activities were conditional on moving towards that great aspiration. It was, how do we make an existing situation a little bit better now? And then once we get to that point, how do we make that little situation get a bit better? So it's working iteratively. That's only clear looking back rather than looking forward, which, what came to be effective? So what might be effective? Well, from the point of view of where we are now looking forward, there's a whole range of plausible options that one could choose. Most of the time in development we don't wanna work like that, we wanna work like this. We wanna say, well, there's my goal. That's what I really wanna, that's what I think is the answer to the problem. So everything we do then is conditional on getting to that. Whereas this kind of problem that we're working on is much more over ended with respect to where one could go. And it's only with virtue of 10 years looking back that you say, well, that's what came to be effective. That's what ended up being the answer. But we ended up here with a different set of actors, different set of circumstances we could have ended up over there. And it's not that this one's better than that one, that's how the system evolved over time. It took us over here rather than over there. So to do that effectively is a different kind of research task. And what we've been trying to do with our work on local justice reform is put local evidence, compiled by local researchers into the decision making space where the next steps regarding transformation are being undertaken. So not trying to set ourselves up as the arbiters to sort of give a thumbs up or down verdict on whether what they're doing is working right now as such though that might be part of it. Certainly not in that sense of will this be publishable in an elite journal because we've satisfied all the methodological hoops that we need to satisfy to get ourselves published there. We're just saying at the moment most of these debates about local justice reform are occurring in a data-free zone. Anecdote versus anecdote, it's sort of one power play versus someone else's power play and we are bold enough to think that it might be a little bit better if some evidence was inserted into that conversation to move that forward. But because of the sensitive nature of these questions and because of when you're messing with or changing the rule system in a society those rules are very complicated not just in a technical sense, they're complicated because they're very context-idiosyncratic they're by their nature very much embedded in a constituent of the local culture even. So to have a bunch of white guy experts fly in and sort of deem what should be happening in those situations, even if it's technically the right way of thinking about that is just gonna bounce out of their decision-making space because they are in the same way that we would if a bunch of foreigners came to our country in my case of Australia and started dictating what should be happening. It was like, you have no clue about my country who are you to come and start changing what's going on? So our approach has been to embed local researchers in my case within the World Bank's country team but working as much as possible with local researchers and investing a lot in building up their capacities to do local research so that we produce good enough evidence to be able to move policy debate constructively forward and constructively in the sense of making a little bit more evidence-based than it might have been otherwise. And then the decisions are what the decisions are. We're not sort of saying, our job isn't then to sort of cast judgment necessarily on whether this was a fantastic idea or it will only become apparent over years, decades even how long or whether that kind of approach was working or not. So we're not in the business of trying to say, well, we know what that answer is. We know what good land law is. We know what good administrative law is because it works fantastically in Finland. So we'll hire a bunch of Finns who know all about how to set up telecommunications law, for example, to go and work out this particular problem. So our starting point is ignorance and our second point of moving from that full point of ignorance is to say, let's pay our dues to this country. So let's respect what's already there and let's try and bring what we can by way of best social scientific research practice to bear on this particular issue as it now stands and pay up and just be patient enough to be able to be then part of this ongoing dialogue. And in a very specific case in a country like Cambodia, for example, where the big issue was trying to nurture a labor law, for example, there was no labor law. And in some sense initially that was very attractive to foreign producers because they figured they could kind of just do what they want. And but over time as their production expanded, they just by necessity recognized that actually, you know, having decent labor capital relations was pretty important to being able to respond to international critics who were claiming that they were just exploiting local labor. It was symbolically and substantively important. So they needed a labor law. Now, you know, we have numerous examples around the world of what an effective labor law looks like, but you can't just cut and paste that from one context into another. You have to try and, the challenge in Cambodia was how do you nurture an indigenous labor law that makes sense to Cambodians that has local legitimacy for not just Cambodians, but for international actors. When the Gap and Benetton and all the big companies are doing business there, they want a serious law that they can engage with. So the initial tasks, the initial response and quite an innovative response in Cambodia was not to import an existing system or even just a prototype and then sort of adapt it. It was let's create a space wherein the government, the international actors and labor organizations can negotiate over time and then something that will become labor law and initially it was non-binding and everybody was saying that's how can you, how can, it's not even a law, it's non-binding. How can you possibly even claim that it's a law? The idea was to say in the common law sort of sense, we'll just, we'll let a set of practices emerge. We'll let a set of transparent processes be put in motion whereby something like the beginnings of an agreement can be nurtured and by the time you put 800 cases as now that there are two through that system, you're actually now moving from non-binding to binding. It has become beginnings of a law instead of then and even if 10 years in now, looking back, we said, well, if you'd hired five smart lawyers from Belgium, we could have probably nailed that in five months, why do we have to take 10 years? So then you've nurtured it. It has whole content and credibility by virtue of being through a domestic political process and come out the other end looking like something that Cambodians now own as their own and it has elements of similarity, of course, with how it appears elsewhere. But ultimately it's a law that Cambodians can be proud of. It's not what you'd actually would have designed from scratch, it's evolved and gone this way rather than that way because of this particular space. So I like those kind of examples. So I think it's an instance of where the international community was useful. It wasn't just a completely laissez-faire left on the almost left to emerge in its own time. It was a space that was created and in that space all sorts of weird and wonderful things happened and I like seeing that, I think you can, that's what's replicatable out of all this. Not the form, not the content of the law, it's the process that's the petri dish within which that was allowed to grow and that's when we talk about what's replicatable and what's transferable in development effectiveness, what's transferable is the principles and the decision-making spaces where in these things evolved and that's when I see that happening, that's why I think, yeah, we've actually been useful in development but we haven't been prescriptive. We haven't sort of claimed all mighty expertise on a particular issue. We've just said, well, let's create a space and let a good contest happen and we'll let whatever happens as a result of that good contest be what it is. One of the things that the group of us that work on this have been concerned about for a long time is that you can see quite sustained advances and progress in various different dimensions of development but the one that seems to lag behind and a lot of the others is the capacity of governments to implement things and even implement relatively basic everyday services like schooling and health and those kinds of things. So when you look at the amount of energy and resourcing in development that is done, an awful lot of it goes up front to designing the perfect looking project down to the finest detail and making sure that everything looks like what we think are well designed projects to look like and then we kind of assume implementation and then other bunch of people show up after the fact to assess it and make big claims about the effectiveness of this thing. So it's a very bimodal kind of allocation of resourcing and a lot up front and a bit more at the tail end but there's kind of this missing middle. So both as an observed reality that you have countries that have been independent for 200 years in the case of 80, taking an awfully long time to achieve even some pretty modest demonstrated capacities to do things on the one side and yet from the donors to develop an active side a vacuum almost with respect to worrying about these implementation issues which are kind of seen as a little bit boring, as a bit prosaic or just not really where the serious intellectual work happens, all the brain powers in the, getting the wording and the content of your design or running some nice randomized control trial at the end and puring the guys in white lab coats at the end sort of being the scientists that make the DecoZoo statement about impact. So we're trying to sort of make the implementation side cooler than it has been for its practice but with it not just in its own sake it's because we ultimately think that part of the broader package of modernity is making the state system capable of doing what state systems are supposed to do and a lot of those are non-negotiable tasks pertaining to education health and public finances and justice and so when countries persistently show sort of a flat line with respect to their capacity and capabilities of to implement these things we call that capability trap and our research is dangerous about trying to unpack that a little bit more, explain it at one level but ultimately trying to figure out how to be more, how to be useful with respond to, when we're invited to a particular country to respond to a particular problem how exactly do you help in the case of Lance work on India or how do you help the Indian government to do a better job on getting kids to learn stuff in school. Can't give a truly objective answer to that because of the gaps that I see and the gaps that I happen to think of the salient ones and so I'm trying always and trying to encourage students to put their brains behind these kinds of issues because for me they're the ones that I personally find interesting but from an applied point of view I also think they're interesting by the way of moving the agenda forward so I guess to be consistent with what I've just told you I think this question around how we enhance the capabilities of governments in the first instance but organizations more generally how you figure out how to put, not just in a design sense how you sort of structure that that's how we instinctively want to think about things and I think it's a problem to design is the organizational chart needs to be reconfigured or we need to pay people more money those are relatively pro-zac sort of standard ways in which we do it trying to think about how you put different kinds of pressure I guess or how you try and ensure that what the organizations do rather than what they look like is what we measure is what we ultimately finance and reward those are important issues as I indicated before I think another really big question is how we address these problems in development for which there isn't a known answer up front so the Cambodian label law is an example of that but there's a whole bunch of stuff thinking about this stuff these are legion, they're everywhere whole bunch of problems for which we don't know and shouldn't pretend to know that we have the answer up front but pretty much everything in the way the development business is structured heavily rewards the expert who has the answer who can run around countries and give big lectures and say nice things about only you invested more in micro-credit or only you invested more in your urban property rights systems than you too could be on the path to great riches and that's only ever one part of it but that's the more sort of serious analytical question really is we just don't we can't know what that answer is to a lot of these kind of questions and presuming that even with the 10 smartest people in the world that we would figure out that answer that in itself becomes part of the problem and we want problems to be solved that way and we reward people for doing that and we think that because you've got a PhD that you've invested all this energy in getting it that to be an expert in that true sense of the word you're paid to provide answers not to try to complicate people's world but I think a lot of these questions we're dealing with complicating it this is exactly what we're trying to do because in the sense not complicating the sense of just trying to obfuscate and make it more confusing trying to clarify what is inherently a confusing task and trying to short for fast track that learning process for any group of people is really I think an important issue the other big gap to the methodologically I think is that we are as researchers we kind of have a good set of tools for dealing with small numbers of cases ethnographic case study kind of work one to 20 we have all sorts of nice econometric tools for dealing with 200 to 20 million but there's another example of a bimodal distribution of our brain space as I said before with project implementation a lot up front, a lot at the end not much in the middle same sort of thing with research it seems to me that we have goods we know how to deal with small lens we know how to deal with big ends but the problems in the world often come in sort of bunches of 40 or 50 or 60 as researchers we're very clumsy it seems to me for no obvious reason really well maybe it's obvious but as a research community we seem very lame when it comes to addressing problems for which there are instances of 40 say it's too big for ethnographic case study works too small for anything high powered statistically why can't we say something more sensible when the end of our problem is 40 and as it happens a lot of things in development something's probably been tried in about 40 different countries or so so how do we as researchers make much more fruitful with these middle range kinds of problems so whether it's implementation whether it's kind of the kinds of problems we address or the kinds of tools that we bring to bear on them I think there's always we've only gone so small but if I feel that sometimes we're just like medieval doctors we're still putting leeches on these things we don't have a good theory we don't have a we're so we so load up on one particular set of skills that we think is what we need to do that when these problems require all sorts of different skills and they just need to be put into a mix and they need to come out the other side but ultimately you know this is how we're in social science and I emphasize often the social as much as the science but hopefully as more generations of people work on this stuff we'll get a little bit better but for me the end game has never been about trying to you know reach a technical Nirvana where we actually know what to do about this so I think that's a scary world we don't want to be in a world that's why we novelists write books like Frankenstein and stuff you know once people actually think that they've nailed all of this stuff around humans then super scary stuff starts to happen so I'm I'm much more comfortable and a lot of my economist colleagues are with with a bit of deep uncertainty I think that's that's how it should be so you know there's no shortage of ways in which we can improve the rigor of the quality of our work there's no shortage of interesting problems we should be addressing on but if we get a little bit beyond the leeches as our theory of how to cure illness then that's gonna be progress but I don't want us to be in a world of you know the perfect squeaky clean machine that we've actually nailed this stuff because that's a way scary world and I don't want to live in it