 Welcome to the eighth webinar, the wider webinar series of COVID-19 changing development. The wider webinar series, which is a lineup of eminent researchers and development specialists who present new research on the implications they foresee of COVID-19 for global development efforts and also the economic, political, social impacts for the global south. Thank you for today's webinar is COVID-19 and the role of RCTs for development. Our previous seven webinars looked at what we knew about the effects of the pandemic in the global south. In this webinar and in the next two webinars for the rest of the year, we will do something different. We will look at big picture questions in development first in this webinar on which methods should we use to study the effects of the pandemic on economic development as an example. The next two webinars for this year on November 24th and December 8th will look at the implications of the pandemic for globalization and the future of aid. This webinar is also unusual as it is an effective book launch. The title of the book is Randomized Control Trials in the field of development, a critical perspective very recently published by Oxford and Sypress. Because the webinar is a book launch, you'll have a slightly longer time for the webinar than has been the case previously. The webinar will run for an hour and 15 minutes. The last year, Nobel Prize Committee awarded the Nobel Prize in Economics to three economists, Abhijit Banerjee as the Duflo and Michael Cramer. Normally when we have Nobel Prize Economics awarded, there's always a bit of debate. But perhaps last year there was more debate than normal actually. The Nobel Prize Committee said that this particular Nobel Prize awarded to the three economists for the contributions of the experimental approach to economic development. And in particular pioneering the method Randomized Control Trials, our cities in short. Of course the debate and many of the questions that were raised are very relevant for today's webinar and it's been addressed in the book that we discussed today. What scope do our cities actually have? Have they really dramatically improved the lives of poor people? Can they really ask causal questions about poverty and the effects of policy on poverty? Are there other ways to address causal questions? And perhaps most importantly is the supremacy of our cities that we see in their economics scientifically legitimate and politically desirable. The book that we will discuss today Randomized Control Trials in the field of development and critical perspective brings together several scholars and practitioners of whom two are Nobel laureates in economics. And it is really important contribution to this debate. We're lucky to have two of the three editors and I'll introduce them shortly along with two other speakers who will also discuss different aspects of our cities in economic development. Let me now introduce the speakers. The first speakers are the two editors of the book along with the third editor, François Roubault, who is an economist and statistician, a senior research fellow, the French Institute of Research for Sustainable Development, member of the DIAL research unit in Paris and a former head of the department. François actually, and I know François has worked for much for a long time, has initiated the mixed survey approach in household enterprises to measure the informal economy. He's very well, François has done remarkable work on what is now known as the one-to-three survey. And also to develop governance modules that are grafted onto official household surveys that are now used to monitor SDG16. You also have Isabel Guerra, who is also co-edited to the book. Isabel is a social economist and a senior research fellow, the French Institute of Research for Sustainable Development. She specializes in the role of debt and credit in the dynamics of poverty and inequality. Isabel's work draws most from her own field-based original data combining ethnography and statistical analysis and is interdisciplinary and comparative in nature. Isabel tends to think quite a lot in her work about the conditional data production and the combination of methods. She pleased to have two other speakers today. One of them is also contributed to the book and is also somebody who can speak from a policy perspective. We have Gulzar Natharajan, who is an office of the Indian Embassy Service. Gulzar is served in the office of the Prime Minister of India, managing infrastructure cooperation as a state, as a district collector of Hyderabad, chairman and managing director of a power distribution company in Shakhapatnam and municipal commission of Hyderabad. Gulzar has a master's degree in international development from the Harvard Kennedy School. He also pleased to have Rachel Gueslequist, senior research fellow in UNIWIDER. Rachel is a political scientist who works in the politics of developing countries with particular attention to inequality, ethnic politics, state building and governance, and the role of aid, democracy and democratization, and sub-Saharan African politics. Rachel speaks about her specific contribution she made after Nobel Prize winners was awarded in the general world development on case study approaches as a comparison to our cities. Let me remind you now a little bit about lots of field-oriented issues. So when you send in a question, please use the Q&A button that you see on your screen. I will read out the questions on your behalf. That's very important. You must send in the questions using the Q&A feature. The webinar will be recorded and shared later on our YouTube channel afterwards. So let's begin. And I would like to invite Franser, and he's able to speak. Thank you, Franser. Thank you. Thank you for the invitation. Hi everybody from all over the world. I'm very happy to be here. In fact, two facts. The first one is last year, you invited me to present something at your seminar on informality, but at this time we didn't know that COVID will happen. And that most importantly, we are in a transition maybe to a new world. And the connection you made it with COVID and our cities is, as you quoted it, poverty alleviation and the causes of global poverty and how best to combat it was the reason, the main reason why the Royal Swedish Academy of Science contributed the price to the three economists. And COVID, what is COVID principally is a massive shock on poverty. So we are exactly on the topic. The question we can raise today is the first thing is we won't forget the past and what have we learned from our cities in the world before is still important. And the second one is how our city could contribute to mitigate the shock now in the world now and maybe after. So what we are going to present with Isabelle is two parts. The first, I will present the making of the book and Isabelle represent our takeaways from this book and the idea underlying ideas does COVID make a difference. For this we prepare some questions for you as a poll, you can see three questions. Generic one, do you think that our cities are the best tool to measure the impact of development intervention. The second question is about according to you how much of what works and what does not in development can be evaluated by our cities with different thresholds. And the third one linked to COVID with this massive increase in poverty in the world you think that our city can make the difference in curbing the impact. So you will have the opportunity to answer this question and we may discuss it later. So why I would like to present very briefly our road to this book. And first our motivations. It's a long run program begin nearly 10 years ago with an intriguing questions with all these curves or we have this this curve this figure in mind, the exponential rise of what is called or what was called the gold standard as an impact evaluation, which potentially an eviction effect with other method. And we were wondering this, whether it's a true revolution or just a fashion, which fate was to fade someday. So now, I'm not able to go on with my presentation do you still see my presentation. Not anymore. Okay, so where is. I can't move my screen. So now we can see it now. Okay, now I can again. So they put that in full screen so that is the motivation for us so we launch a research project without any funding free research. So with the following research question the question of what, what are the methodological property of our cities and our cities really the Holy Grail in impact evaluation, and the question of why, why this road to global success. Well, the core team of colleagues and friends with different backgrounds, a political scientist evaluator and a geek. You will see why it's important from the donor side for Isabelle here from the social economist perspective and qualitative approach from the academia and myself, economist and statistician, more prone to quantitative analysis and also from the research. And from this first phase, we published the first series of paper assessing our cities, technically but also what is called around the mistas, but if you show definition that the people that thinks that our city is the best tool to assess the impact. So what do we do first three things. The first is a theoretical critique, and which is not new, and the results are in line with others famous people, and some of them in the book questioning our cities in theory, with the question of internal and external validity. So that's what we call doing the math, but more original, we open to France. The first one is we develop an empirical critics of our city. Our city are done in practice in the field, how they are conducted. We call that doing the cooking. And for us, French people entering the kitchen is very important and maybe the most important. And the last is to investigate the political economy of this scientific business of our cities of the pro city movement and the people working in this field. What are the main conclusion we draw from this first phase. The first one is our cities are a sound tool to assess causal impacts, clearly. But first, it's limited to a yes no answer, an impact or no impact, and how much for the impact, but nothing on the why. So what are the channels so suggestive evidence can be provided. And that is a problem in one one we think of the theory of causation. And the second constraint is our cities is a good tool but when it's conducted by the book and under certain certain conditions. And these conditions are really rarely met in the real world, because there's no virgin areas in the real world, contrary to what our city positive to conduct the experiment. And the second three of the random instance main claim are according to us is illegitimate. The first one is that our city is the only rigorous method or scientific method in town. And that our cities can explain all what work and what doesn't so the question in development. And the last important thing is that replicating multiplying our cities on all topics in all tropics to overcome the problem of external validity will solve the problem of this external validity and that's what we call the hegemonical project. And the last conclusion about this political economy of our city is the strategy of market domination, trying to build a monopoly or under oligopoly and rent captures and describing this successful business model in different audience mainly in the north but not only. And also, one important thing which is not finished today, what we call the randomista you guys. And then from this, we begin a second phase of our research project in two parallel tracks. The first one from the bottom where the thorough investigation of a sector our cities on micro credit on one vocal micro credit our city in Morocco, and where we've done first a full replication. And that's where our colleague floor was key to this because we recorded all the data, the raw data, all the codes to try to rerun the results, and we see a lot of problem in all sphere from sampling data capture interpretation of results. So that's the question with the tendency to answer yes, as garbage in garbage out. The second was to go behind the scene. Thanks to unique documentation we have access to, like emails, great literature about the making of this city, and showing we go to such a mess. And then maybe the question was we were with the wrong RCT this RCT has been conducted by the most famous randomista, including ester du flow. But maybe it was the wrong one, especially bad. And randomly, we fall on this bad RCT. So what we've done is try to assess all the special issue, which has been published in 2015 by the American Economic Journal applied economics, which was and still is considered a sort of the last word on micro credit. So that is for the from the bottom from the field track of our second phase of our research program project. And the second one from the top is this collective booth we are discussing today, and title randomized control trial and development a critical perspective, just published at Oxford University Press. And here, I showed the outline very briefly of the of the book. We have 12 chapter and more. We try to distinguish the different parts in five or four blocks. The first one is sort of an overview, trying with main lead development economist, some more prone to to criticize RCTs and other which are more supporter of RCT so trying to to begin with the discussion and the controversy. The second part is is sectoral approach trying to see this property at the sectoral level and discuss this sectoral approach in depth with wash sector, global health and micro finance. The third block is political economy approach. The first one on the rhetorics on how things are written the use of panic notes in the, in particular in poor economics, the book poor economics. And the second one is how RCTs randomized us are in the community of evaluators, what are their position and how they are considered by this community. And the fourth block is about trying to make some proposal there are other chapters where we try to make some proposal to try to go beyond just showing the limitation of our cities. One is on the ethical concern, raising issue in the now in the COVID time. And the second one is trying to see how can be used priors or compliance and non compliance in the case of the paper chapter of James Eggman to try to see how to improve our cities and at last but not the least. We've got some interviews, interviews from policymakers so not the academia from the North, basically France but also from some colleagues from India and who's are here is going to share with you. Some of his views and obviously because we were quite weak as editors, compared to the army and the generals from the randomized at the top of the economic world, we mobilized to previous noble price to protect us. I want to say Angus Deaton with a preface and with an epilogue James Eggman. So now maybe it's time to see what do we think of all these research tracks and Isabel I think that's your time now. Thank you. Good evening good morning. Good afternoon, depending on where you are. And thank you. Thank you so much to wider for this great opportunity. I'm not going to summarize the book is simply impossible given the diversity and richness of the contribution. I'm just going to highlight few points. The initial idea behind the behind the book was to better define the scope of application of our city. Simply what are the questions they can answer and what are the questions they cannot answer to be able to do so we we have established a dialogue dialogue between disciplines, a dialogue between scholars and policymakers, and also dialogue between different visions, regarding the scope of our cities. And in the introduction of the volume which reflects only the sense the standpoint of the editors. The idea is to draw our own lessons of these dialogue. In fact, it is an attempt to to answer the following questions. Ultimately, what opposes the advocate of our cities and its opponents. What are the areas of disagreement, or, or to put it differently, what are the terms of the debate with three points here, which are clearly articulated in the epistemology, politics and ethics. And in fact, these three points are extremely useful to define the scope of application of our cities. So first, the very first issue has to do with epistemology, with here a certain form of positivism, even sometimes scientism against what could be called a pragmatism. Are we looking for universal answers, or are we looking for reasonable explanations, limited in time and space, and attentive to the diversity of situations, including within specific groups. Many of you probably know the claim of randomness test to universality has already been strongly criticized because of their inability in a number of cases to demonstrate external validity. And what is true in a specific context might not apply elsewhere, or to put it differently the result of one particular RCT can hardly be generalized elsewhere. The limit of the average results as has also been widely discussed and criticized RCT focus on the average result, while in many cases, what matters for policymakers and practitioners is on the contrary, the diversity of the results who gains and who loses from a particular intervention. Something which has been as discussed, and which is very much in line with what we've done by the past, regarding the empirics of our cities is the gap between the protocol in theory and the protocol in practice. The question is, should we focus primary on the theoretical protocol able of measuring causality and statistical inference. Or should we be concerned about the feasibility of the protocol, the way it is implemented in the ground in the real world. And here randomization might work well in Terry. But the thing is that in practice, given the multiple challenges faced by the implementation into ground. This is not the case. In other cases, field constraints contradict the assumptions of the statistical theorems used for inference. And those deviations between protocol and implementation can be observed all the way down the knowledge production line from sampling the end at the level of the intervention which might be artificially transformed to fit to meet the protocol at the level of data collection, and also at the level of the interpretation of the results. And in theory the result is a simple comparison of average in practice, the interpretation of the results involves a number of hypotheses, and even in some cases an art of rhetoric that proves to be extremely persuasive. And all this, the requirement of the theoretical protocol have a critical implication for the type of interventions RCTs can study and the type of questions they can answer. Yeah. And this brings us to the political question. The second crucial aspect of the debate is definitely a political one. What do we mean by development? In short, do we consider development and the fight against poverty as an aggregation of private goods and micro interventions? Or is development first and foremost seen as a matter of transformative politics, a matter of infrastructure, a matter of institutions. And regardless of the worldviews or the missteps, the technical constraints of the RCT protocol force them to focus on micro interventions and private goods. Just to give you a few examples, in the field of poverty as a whole, do we look at micro credit or do we look at national, regional, or sexual wealth creation processes? In the field of global health, do we look at financial incentives for the medical staff? Or do we look at complex and systemic health systems? Et cetera, et cetera. I can, we could have similar examples in many fields. Still, still regarding this issue of politics. For the reasons mentioned earlier, in deviations between official protocol and a real protocol, the ability of RCTs to measure impact is in fact rather limited. And quite often RCTs are in fact rather used to compare different modalities of a specific intervention. For instance, agriculture fertilizer, combined with training or without training, and then the RCT is going to measure the outcome in terms of intervention take up. And this allow them to test behaviors. If we change prices, timeframes, if we provide information, assistance, training and different sorts of training, different sorts of information, how do people behave? And how does this affect the take up? By this I mean whether people are going to use the intervention or not. How they're going to use the fertilizer, how they're going to use the micro-edit, how they're going to use the water filter and so forth. This is a crucial lesson. In many cases, RCTs are better adapted to test behaviors rather than evaluating impact. Well, we can look at this in two ways. One is to consider that this allows for a better understanding of behaviors. Given that in many cases, the results challenge misconception in development economics. For example, in the field of micro-credit, RCTs have shown that contrary to a widespread belief, the poor are sensitive to micro-credit. One can also look at this in another way and consider that ultimately those kind of RCT are similar to what could be called social marketing. By this, I mean how to better distribute or sell a product for micro-credit, a water filter, an improved cook stove and so forth. Which we believe has a positive social impact, but in fact, in many cases, we still don't know. Well, if you consider development as an aggregation of micro-interventions, that's fine. And the impact issue remains unanswered. But if you consider development as a transformative politics, this is more problematic. The last point has to do with ethics. Of course, any kind of research entails ethical issues. The thing is that RCTs are more concerned than observational studies, since they manipulate the environment they study. A number of ethical standards do exist. Informed consent, do not harm principle, provision of specifically considered protection for vulnerable populations, et cetera, et cetera. Here, the thing is that in many cases when the masters ignore those standards. Why it is so? Here, our book through a specific chapter devoted on this by Abramowitz and Sarfars shows that the answer is quite simple. There is a trade-off between ensuring the internal validity of the protocol on the one hand, and on the other hand, guaranteeing the protection of the experiment's subject. And quite often the first prevails over the second. Do we advance science or do we protect the people? As a way of conclusion, let me get back to the issue of COVID, which was the initial promise of this talk. The responses of RCTs in the development field through the current crisis seem to largely confirm our observations. Using the previous work and new RCTs, which are going on at present, a randomized suggests to use nudges in various domains. For instance, to increase insurance take-up, to improve online schooling, to improve social distancing behavior. So, second, the advice that governments should make measure investments in two areas, cash transfers, and mobile money infrastructure. Well, why not? I'm not saying that this is useless. And here again, let me insist that the book is not at all intended to reject the use of RCTs altogether, but to better define their scope of application. So why not? But first, there is nothing very new here compared to what other methods or approaches could advise. Second, for the reasons already mentioned, evidence regarding the impact of such or such intervention is sometimes doubtful. Third, and most importantly, the focus remains on individual interventions when there has never been a greater need for structural responses. What about the construction of value chains? What about the tax system? And what about the fiscal and monetary policies that can finance cash transfers policy? What about health policies? Crucial questions are still missing. I stop here. Thank you. Thank you, sir. Gulzar, do you want to make your comments? No. Let me just pull up my, yeah. So greetings to everyone from different parts of the world. Thanks to UNU Wider for inviting us for this program. And also, Francois Isabel, for putting this very impressive collection of chapters on RCTs, a critical perspective. Thank you very much. So I'm going to sort of just do two things. Yeah. Okay. Let me just pull this up. Okay. Let me preface that I view this debate from the perspective of a policymaker. To what, to this extent, I view research as a source of insights which can better inform or increase my understanding of the problem and its possible solutions and manage design of solutions, design of interventions and policies. So instead of being just a critique of RCTs, I'll frame my intervention here as an effort to clarify on certain widely held perceptions. Being a policymaker as well as more recently having been associated with international development for nearly four years, I feel it's useful to make these clarifications. Before that, some comments about my contribution to this book. I'm just going to, yeah, the screens. Just a second. Sorry, there seems to be some problem with this. Oh, no. Okay. Yeah. In the interview in the book, I have discussed three specific case examples of acclaimed RCTs which claim to provide important insights to their respective development problems. The broad direction of my analysis just applies just as much to any RCT study from the very pressiveness with Rachel Gisselquist, so painstakingly compiled. The point is that headline efficacy establishing RCTs by themselves are very limited relevance to practitioners. Very few such hypotheses are under dispute, and if they were not adopted, it was not for lack of evidence. I also listed out a set of questions which agitate the minds of policymakers at different levels of the government. It's like pretty big listing of close to 100 questions, which policymakers at different levels grapple with almost on a daily basis. This is about field policymakers. Another bit about say what agitates policymakers who are grappling with issues of industrial policy. I'm just going to skim through this like I'm not even going to I'm just going to display it just to sort of illustrate the point. And then what agitates policymakers as urban managers. I mean these are a bunch of questions which you could like it's almost like if you have you can capture the word cloud of what's going on in the heads of policymakers you'll have a bunch of these questions popping out. Now what's intriguing when you look at all these things is that how much of this is amenable to an RCT. I see from them, and I for that I urge you to buy the book and like have a read. So I'm not making a plug for that but like just making that point. There are different kinds of issues which are amenable. These are not the issues which are amenable to an RCT, nor is an RCT likely to be able to provide any helpful insights to inform the practitioners concerns. I now like to highlight a couple of general points about policy making an evidence which I think will help clarify the view from a practitioner side. The idea that policy decisions emerge as a largely technical exercise with significant influence from up for evidence and space for its application is deeply questionable. The two points I want to talk about our evidence our policy and evidence. So first about and these are so these two slides basically summarize whatever I want to speak about first. How does policy get made policies are never made on clean slaves, their political choices. There's only a small space available to exercise technical choices and choices are made in real time. We overestimate the technical agency over policy. Policy versions and programs are mostly the result of marginal improvements to existing initiatives and a very rarely introduced as completely new versions you won't see an aha moment with policy making oh yeah we've got this inflation targeting. That's great like it sort of changed the world and you'll never have that that's that's like cognitively that's not the way policy gets made in fact in the realm of development and especially in a large country like India. There's hardly anything non technological that can be called an entirely new idea or any idea that is most likely to be at least a state or few districts where it's already been tried out over the years. The specific first order choices for research to influence are rarely, if ever binary ones between cash transfers and in kind provision, or between public provision and outsourcing or between spending money on building a road or maintaining existing roads or school buildings or hiring extra teaching volunteers, those choices are most often already made, or there are shades of gray in the choice the degree of freedom available is restricted to second order issues of execution. Intuitively, you can therefore imagine all these ideas and circulation in a vast landscape, occasionally due to a confluence of factors, primarily circumstances, implementation feasibility and political expediency. An idea becomes ripe enough to get adopted on skate, which brings us to the second point I wanted to make, which is about evidence. Now, we have seen as we have seen contrary to perception policy making is not handicapped by lack of evidence of causality, and the lack of evidence is not the reason for non adoption of an idea or an intervention. Besides, like policy, with evidence to there is no clean slate, it builds on legacies, priors, theories. This raises a question of evidence for whom. We should make this distinction between the evidence requirement for insider practitioners and outsiders. This is especially important, since the producers of development research are most often outsiders, if we agree on the importance of evidence for insiders. Then as I mentioned earlier, I'll argue that hardly anyone would dispute the null hypothesis in the vast majority of cases where our cities have been conducted. Now I'm making a really big statement here, but I honestly struggle after having gone through lists of our cities over the last 15 years, I honestly struggled to find a hard this is something I did not know, or like, this is something which needed this expensive time consuming effort to sort of produce before this thing then comes to what kind of evidence. As mentioned earlier, evidence of efficacy from individual pilot conveys very little of significance. The challenges with scale implementation. For reasons like weak state capacity interventions which have been found impactful at small scale, generally flounder when scaled up. This draws attention to the most important form of evidence that which emerges during implementation. Specifically, the analysis of administrative data from implementation, for example, it's a major category. So what do policy makers need as evidence conditional on the constraints mentioned about earlier. Policy makers need insights about the program which can help them make the right choices on its design elements and implementation processes, not the headline stuff. More than the headline efficacy findings practitioners are likely to be interested in the mechanisms and the responses. These are best generated by a combination of theoretical empirical and qualitative techniques. In viewed against this backdrop, my contention is that our cities are seriously limited in their proximate or direct relevance to understand problems, explore situations, solutions, design programs, and then implement them. But like with any other research technique, our cities to have their relevance they along with other research methodologies have an important role to play in building the body of knowledge and development issues. For example, the hundreds of our cities and other research methods have all contributed to the mainstreaming of ideas like the use of incentives to achieve program objectives, utility of nudges in overcoming behavioral failures, value of information to change norms and so on. But then the theory of change with this approach is very different. In other words, our cities have a role in building, as I said earlier, have a role in building a body of knowledge which in turn along with other factors helps influence the debate and ripen the moment for adoption of an intervention or an idea. I'll just conclude with this. Now, Lan Prichit once described development as a faith based activity. I'll conclude by saying that our cities have to have a role to play in building the faiths. Thanks for your patience. This is just what I have to say. Thank you. Thank you so much. Go ahead. It was very, very insightful. I will now ask Rachel, please request to make a comments. Rachel, go ahead. Okay. Let me just try to share my screen here. Okay. So, thank you very much for I really appreciate the opportunity to serve as a discussant today and I'd really like to congratulate the authors, the editors and the the authors of this volume and encourage all of you to read it. It's a really rich collection and really rich set of work. My role as discussant today I think discussant is a little bit of a misnomer for what I've been asked to do because usually a discussant is asked to speak very directly to the book and the presentation, the core presentation. But what I've been asked to do is something a little different so to offer a comment and critique more broadly on the topic, and perhaps to prevent present some new material for the discussion. So in this vein, I'd like to make three points. The first point is a point that I think the book makes very well that RCTs are not the gold standard for work in development. One tool and one tool does not serve all purposes. The sort of has asserted hegemony of RCTs by some randomistas is simply not very helpful when we think about how do we accumulate knowledge on poverty and development around the world. I think about how we might use RCTs in addressing the current pandemic makes this abundantly clear, as Isabella and Gulzar's presentations also suggested. So there are certain things that we can't easily randomize. There are certain things that might be unethical to randomize, and focusing on how we can use RCTs to study the current pandemic or to address the current pandemic can focus us can push us towards perhaps more minor questions when we should be thinking about other issues. I think however that you know the broad argument that RCTs aren't the gold standard isn't a new argument right this argument has been made by others. I did it myself and in some previous work, but I think what this volume does that's really new that's really interesting is that, well, not only is it incredibly comprehensive and at a very high level but it also offers a critique of RCTs not only in theory but also in practice on the cooking of RCTs, as Francois highlighted, and also of the industry of doing RCTs sort of RCTs as a business. I hope that with this volume we can sort of lay to rest this debate. It's very clear that RCTs aren't the gold standard. The question is, when are they useful, how are they useful, and how can they be made more useful. The second point I'd like to highlight is that in using this tool, we want to pay extra attention to ethical issues. And I think that that we as students and scholars of development are thinking more and more about ethical issues in this area, as some recent debate and critique for instance over the two papers highlighted in this slide in blue suggest. But I think that we want to be very sure to keep ethical issues in mind, both at the micro level, and at the macro level. So at the micro level I'm thinking of issues that I think experimentalists have paid a lot of attention to so issues like informed consent, doing no harm to participants and so on. I think there are also ethical issues that we should keep in mind at the macro level, and I would highlight two broad areas. So one issue is whether is inequalities between researchers and research subjects, the ability to do RCTs in low income countries with low income populations that you might not be able to do in high income countries, or with wealthy, wealthy populations. And I think when we find this happening we should be very concerned about the ethics of our of our work. Secondly, I think at the macro level we should think about the extent to which focusing on the method on RCTs might miss focus our attention on factors and hypotheses and policy interventions that are easy to study with this method, but are perhaps less important than other things we want to be thinking about. So those are my first two points, I think that my, my third point is a bit more controversial. And that point, the point is that we should be cautious in using RCTs to build global knowledge on poverty and development. And we should be cautious because RCTs are basically like single case studies when we use them in this way. And we know, we know from a large literature in political science methodology that drawing causal inferences from single case studies is problematic. The argument that I'm presenting here is presented in more detail in a short article that I wrote that came out earlier this year in world development. And I listed here and it's open access so you're welcome to read it. But I'm going to just summarize the argument briefly in three stages. So the first part of the argument has to do with what is a case and what is a case study. And what is small and research, in other words, research based on single case studies or a small handful small number of case studies, and why it's problematic to use small and research to make generalizable causal inferences. I don't think any of the first part of this argument should be controversial. In fact, it's the type of stuff that we would cover in the first few weeks of an introductory political science methods class. And the core issue here is the problem of many variables small and so it's a degrees of freedom problem. So looking at one case or a small number of cases. How do we winnow out from a plethora of case specific factors and relationships variables or processes that cause the outcome of interest. We simply don't have enough cases to hold constant all the factors that might cause the thing that we're studying. So many of us in political science spend a lot of time doing small and research and so we thought a lot about how we might address or how we might gain some leverage on on addressing the many variables small and problem. And there are no good solutions. I mean you can increase the number of cases you can more systematically select your cases with reference to, to theory, you can use comparative methods to analyze your cases more rigorously. It's a persistent problem, and one of the core things that we emphasize in this literature is that you should be appropriately cautious and modest in making causal claims based on single case studies. So the second part of my argument, then is that RCTs when we use them to build global knowledge to build knowledge on global development and poverty are basically single case studies. So let's start here by distinguishing two types of theories. On the one hand, their theories formulated at the level of systems. So theories about communities, countries, cultures, nation states and so on. But on the other hand we have theories formulated in terms of variables observed within systems. So how do individuals or households react within a country or how do they act within the country or culture. I think it's, it's pretty clear that RCTs are testing theories of the second type. So they study how will individuals within a given experimental site how will they react to an intervention. Randomistas will claim that their findings apply across systems, but it's it's a pretty big claim right we shouldn't assume that that findings from within one system apply across other systems. Micro theorists of development recognize that system level variables affect individual behavior. So individual behaviors influenced by institutions by social norms by social and political context. And these are system level variables system level factors that influence incentives that shape incentives that set the rules of the game that shape how incentives and rules are interpreted subjectively. It's really only under very strong assumptions that we should expect findings from one system to apply seamlessly to all systems without analyzing that it without testing that they do. So, you know, where does this leave us right. If RCTs are like single case studies and single case studies are problematic for causal inference. What do we do. I think this leaves us sort of in three places. One point is that current approaches by randomistas to address the external validity problem aren't fully adequate. We can leverage on how the cases we choose affect the answers we get. Secondly, we might be able to do a bit more in terms of addressing external validity challenges by thinking more systematically about how we can use some of what we know about case selection and comparative methods to think about how to select experimental sites or how to comparatively analyze findings across experimental sites. And then finally this leaves us again in pointing out that RCTs are the gold standard and and those doing this type of work it's very rich it can offer really interesting insights but appropriate caution and the appropriate caution should be taken in making claims about generalizable causal inference. So let me stop there. And hopefully we have some good time left for discussion. Thank you Rachel I think your last one is very important that the need to be very cautious about making causal claims, given that in fact the RCT revolutions about making causal claims. Okay, so I, we have a couple of questions and thanks so much for the speakers are keeping to time because we have possibly 25 minutes for you in a. So let me go to the questions now. And the first question is possibly for France one is a well but other speakers can also respond. The first question is during the COVID-19 pandemic policy makers need to respond quickly to ever changing conditions. Do you think there's any difference in speed within our cities and other methods. Let's take this question first. Is it possible. Do you think there's a difference in our cities and other methods given the situation we are in the pandemic. You want to intervene. Yeah I can start yeah this is of course a fundamental question. And the time of research is not that is not the same as the time of policy decisions and this is true for all type of research I mean research was always very slow we know that. The gap between this gap is even more pronounced for our cities. I mean, in a typical RCT you need the baseline you need the end line and usually there is a minimum of five years before the start of the research and the results. And, and, and, and, and, and, and while at the same time, the gap between the baseline and the end line is often too short to really be able to measure what we're looking at. And the, the baseline and the end line is hardly more than two years, which is often too short, and which is, which often force RCT researchers to focus on midterm indicators. I give you a few examples in the field of health micro insurance for instance, the indicator as a primary indicator was how much money has been saved which is a very important indicator but it was too short to measure the impact in terms of health improvement. In agriculture, usually the indicators are going to be whether farmers have been able to use inputs or have been, have they been able to improve technology, but not the final indicator which is, have they been able to produce more and to earn more. But again, the gap, the time frame is too short. So on the one hand, it's too long for policymakers. On the other hand, it's too short to be able to demonstrate what what is really the ultimate indicator that we want to look at. We should not blame RCTs for all evils. I mean any kind of research in the present day is too slow. I mean, and we also have to mention the fact that the recent forms of RCTs are used different techniques to avoid the baseline and are able to produce results more quickly. So, yeah, all this to give us some kind of nuance answer to this question which is a crucial one. Just to complement is a better answer. The first point which is important is criticizing RCT is a duty when the pro RCT proponent claim that it's the best or the only method in town. So if we consider RCT and it would be, I would like the random is that acknowledge this, that RCT just one tool among others. Maybe there is some question which are valid for our city and other we saw that for ethics or ethical concern maybe RCT is more concerned with this. If there is no claim that RCT the gold standard. Many critics which are made to RCT may apply to other methods and it's okay to go back to the question speed or RCT just standard RCT with a life cycle of five years from the beginning to the publication, mounting the project to the publication of the results, let's say five years for an impact of one year between the baseline and the end line. So clearly, it's not the timeframe needed for COVID but the second point is, let's take the example of cash transfer, which is one of the key. I'm here from J-PAL today to try to answer the COVID and going to Garza proposal. Do we need RCT to conduct cash transfer now? Do we need to wait, let's say two years to say what we need to put in place cash transfer. They may be not perfectly targeted like in Brazil where I'm here, but they are key to alleviate the current crisis. So that's that's my answer to the question. Thank you. Can we display the poll results maybe at this point before I get to the next question? It will be great. Are you surprised by any of the responses? It seems the second and the first question, the second option was the absolutely the dominant response. RCTs can be relevant evaluation too, but not always depending on the question and the context. And then the second question, according to how much of what works and what does not in development can be valid for RCTs. Again, the answer is some. Would that you might have expected? I would say that for the two first question, I would have answered the same at the mode of the answer we can see on the screen where maybe I would have disagreed is on the third, which is so for the two first question. I don't know if we convinced the audience with what we think, or it was already a convinced audience from the start of the, but maybe we should conduct an RCT to try to assess the causal impact of these results. Well, more in disagreement with the answer or surprised or is on the third point with the idea that RCT more or less it's 50-50 with RCT can make a difference. I don't know if I elaborate on this or maybe first Isabelle your comment on this or just a point on this. For me, we've got specific reason to think that RCT are not adapted to be the solution or the main solution, the basic solution for what we are confronted to. We talk about the rapid response. So I think that we don't have this. But one point is, we said that the context and I refer to RATCHL presentation. So case studies in normal times. Now we've got case studies and the context has totally changed. So we can clearly contest and say that with this big change, which results would all does the answer to microcredit be the same in COVID time or as it was previously. And the second point which is linked also to this macro big shock is the emergence of spill overs. We've got with this shock, one of the most important since one century, a massive shock. And with general equilibrium effect much more important that is normal time. And RCT is not able to answer this type of spill overs. So for me, in this situation, I think that RCT is less suited to answer the questions than in normal times. Okay. Thank you. Let me move to the second question and I can ask my colleague Sam Jones. I want to ask the question live. You've been given permission by the chair to do so. Sam, I'm not sure Sam's online. Ricardo Santos also had a question because it's one of the leaders. Ricardo, do you want to ask you a question? I'm here. I just couldn't unmute myself. Sorry. Okay, go ahead. I needed to be unmuted by the host. No, I was, I had kind of two broad comments. One, I'm not sure that random misters actually are claiming a kind of a user universal application of their findings, although I would kind of agree that their rhetoric does sometimes suggest that. And then secondly, I'm kind of interested in all of the panelists views actually on this difficulty, we might say between academic RCTs, which contribute to a broad corpus of knowledge versus, you know, practical RCTs to evaluate something specific which might then be implemented and the difficulty with the first group, the academic RCTs is how does this square with our ethical obligations not to exploit participants, and that's particularly relevant when control groups rarely receive any benefits whatsoever. And there's very little scope for them for the intervention or to be rolled out to them at a later point in time. So I'm interested in in the views on this ethical dilemma perhaps. Thank you. So, Isabel and Franco, what do you want one of you to respond to Sam's comments here. Yeah, maybe a point regarding the, well, Sam is saying I don't think randomista claim universal application of findings. It's true that many of them, and probably all of them if you ask them the question will tell you yes we are very open to other methods, and we have a chapter in the book by Tim Ogden who is supporting randomista and who is, who has this argument. The thing here is we have to make a distinction between what they say and what they do. And if you look at most of the, I mean, for all the publication that we have read, it's fascinating to see the fact that they completely ignore what has been said on that particular topic in the past, and by any other methods than RCTs. And which, I mean, so here, for me, this is the best proof of their claim for superiority, and which is very problematic because they, I mean, by, by, by making a clean sweep of previous research and they start almost from zero. While, in many topics, I mean, we already know, I mean, this is also what Guzard was saying, I mean research is always a small addition to what have been done in the past. And, and the different issues here, I mean, one is that by ignoring what has been already done, they ask the wrong questions. And, and, and the other aspect is the fact that the performative consequences of this, with a number, and this leads us to the political economy of the randomista industry that Francois was talking about. And what we show in the book, I mean, in a number of chapters is the fact that the crowding artifact of this kind of discourse, in terms of methods, in terms of resources, not only money but also human resources. And what was made by Inna Patnik is to say that in India you have a huge number of young, brilliant economics, economist scholars who are not moving to the US, it's not a brain drain but it's a, it's a, I don't know how to call it. It's a method drain now who are pushed to do RCTs because because because if you want to be a well known economist now or just to find a job you need to do RCTs. And the last effect crowding out effect in terms of intervention, if you take the case of global health, for instance, the rise in RCTs is not the only explanatory factor, but it's fully part of the movement towards vertical health. I mean, I mean health in silos where you just treat a particular disease, instead of building social protection system and and instead of building complex health system, which are able to treat any kind of disease. So for me, this claim of for superiority first, we, it's not enough to hear about what they say we have to see what they do. And we also have to look at all the consequences of this claim for universality which is according to me very damaging. That's why we're so keen in criticizing their work because we really see the perverse effect in a number of domains. Yeah. So my question for you surveil and it's on ethical issues on RCTs. Let me take a specific example. So right now we have large RCTs going on around the world in, including in developing country in fact a lot of them are developing countries on vaccines for COVID-19. Right. This RCT is going on where they're giving the vaccine to some part of the population, randomly and some others they're giving plus it was. My question is that those RCTs we see in medical research, do you see a different or do you see a different ethical issue there in other words, medical research is OK to do RCTs of that type which we are doing right now. While it is not OK to do it in economics or social sciences, you see a different problem or do you think that RCTs in themselves, wherever you apply them medical research or social sciences, including vaccine trials going on right now are not that. I mean all the ethical issues are, I mean what we can say is, and here I'm talking on behalf of different authors from the book who have been really looking at this ethical issue in details. I mean we have this book by a duo, I mean an economist and someone from the medical field who is a doctor himself. And I've been working together on this aspect, trying to understand why it is the case that in the field of in the medical field where RCTs have been used for decades and decades, and where very strict ethical guidelines have been established. And why it is a case that although RCTs promoters in the field of economics are claiming to import this method from medicine, coming to ethics, they ignore those standards. And the only thing I can say is that in too many cases those ethical standards are not followed. And for the very simple reason that I was mentioning, first of course it is more complicated than there is a clear difference that in the field of development it is of course more difficult to give a placebo. And but here very clearly the explanation is the one I was giving when I was speaking earlier, the fact that the argument put forward again and again by Rundomista is in the long run, people will benefit from it. Because of the thanks to the advance in science, thanks to RCTs. So again we see the fact that science comes first, before the protection of the participants. But here maybe not to avoid putting all the blames on RCTs, I think that the key question is how it is possible that those practices are accepted at different levels. I mean what are ethical committees doing? And what are journals doing? Usually journals should make sure that research follow a basic ethical rules. The example given by Russia is on Kenya is completely unbelievable. And I think here the academia as a whole has a responsibility. And I think the excesses of all those malpractices express a kind of, how shall I say, something which is going wrong in the academia world which is not able anymore to make sure that the research are respecting basic ethical guidelines. And I think the academia as a whole has a strong responsibility in this. You mentioned that there are many medical professionals who do not also agree with RCTs in medical science too. That's why I asked. Does anybody want to have a response also on the ethical issue, social sciences versus natural sciences? Not only ethical issue but I think that one debate on RCT today is all the energy based on the elaboration of a vaccine is based on RCT or not all the energy but RCT are very legitimate in the field of medicine. And we are all waiting for this third phase to get this vaccine. But the first point if you look at what's happening today, a lot of interest for RCT in medicine but who has ever heard of something of RCT in economics in development but more broadly in economics. Nobody, I mean Brazil for the cash transfer, nobody think about an RCT to try to see if it should put in place or not. So RCT in medicine seems to be at the front page clearly and RCT in development has nearly no voice in the debate of COVID today. The second point is, if we think and I think it's very clear, administrating a vaccine is not the same as changing a behavior. And so if we think that context matters more for RCT in development or RCT in economics than RCT in medicine, we understand why this RCT in economics are just out of the debate today. And just one point, interesting point is, even in medicine, the very famous journals, Lancet and others published on the one hand the meta analysis of RCT on non-pharmaceutical intervention during the COVID, face masks, eye protection, personal distancing with a systematic Cochrane review and meta analysis. And they found no results at all. On the other hand, there was another meta analysis, we choose observational data for more or less the same, the same interventions and they found the positive results. And the question is, do we have to rely on RCT, this meta analysis, waiting for new ones to put in place these kind of measures, or do we should believe, at least for the moment, with the precautionary principle, these results from meta analysis on observational data. So I think it's an open question, I've got my own answer. That's why I'm Isabelle. I have a question again for one of our colleagues, Ricardo Santos. Ricardo, if you're still there, do you want to ask a question live? Yes, can you hear me? Yes, we can hear you. Go ahead. Thank you. So, first of all, I'd like to thank you for this very important book and the thought-provoking presentations from the four of us, from the four of you. I'll have two questions. So having established that there are questions RCTs are insufficient in answering, do you find that there are questions RCTs have been used to answer but should definitely not be used to answer them? The second is, building on what you've learned studying the kitchen where RCTs are cooked, in what way should RCTs be part of a multi-method research package? Or to ask it differently, do you see methods that work well with RCTs complementing what we can learn from them? Ricardo, two great questions. Thank you. Again, Isabelle and Francois are maybe here, I think even Rachel, you might want to come in. So, Isabelle Francois, who wants to answer Ricardo's questions? So the first question is, should there be any example of not being used? Maybe, Isabelle, a word on micro-credit, where the answer from RCT are not adequate to what really the impact we can think of the impact of this kind of intervention? Yeah, I mean one of the, and I think it's not only true for micro-credit, but I can take this as an example, for all the reasons I mentioned earlier, we all mentioned earlier, regarding this particular case of micro-credit, I'm quite clear in saying that at present RCTs don't bring anything to the debate. The only interesting result could be the low take-up. I mean, there was a special issue in micro-credit, published a few years ago, bringing together five or six RCTs done in different regions of the world. And one of the very first results was the fact that the take-up was quite low. But this has been said earlier by others, but since they ignore this kind of statement, they do as if it was new. Well, maybe the interesting thing compared to what has been done by the past was to give figures. The thing is that, regarding the technical constraints of RCTs, the figure in terms of take-up doesn't tell us anything to make sure that they will be able to follow their research protocol, having a place where nobody has been benefiting from micro-credit and comparing it to another place. You need to go to a very specific place where micro-credit has never been there. So the place is very specific. The population is very specific. Today, the interpretation of the figures are very difficult, and at the end of the day, we have no idea about the representativeness of this population. Then another so-called innovative result is that micro-credit does not transform the point into entrepreneurs. I mean, anyone who is part of the microfinance world knows this very well. And you have much more research from political economy, sociology, economic anthropology, analyzing the reasons behind this, which is not the reason behind this is not a matter of individual behavior. You know, Banagy and Duflo write, it's too hard. No, it's not too hard. It's not a matter of individual behavior. It's a matter of how the market are structured and the fact that in a number of cases, if you don't belong to the right group, you have to start your business. You don't have access to suppliers. You don't have access to clients. Markets are organized along monopoly or oligopolistic lines. And so, or simply you don't have any demand. So here again, it's a matter of general equilibrium where I could continue for hours. But all this to say that they simply don't bring anything to the debate. And they also ignore question which are crucial, the issue of over indebtedness, for instance, which for me is a burning issue is a time bomb in many parts of the global house. And, and for reasons which I, which I don't understand the city promoters simply ignore this question, which is why it is crucial one. Yeah. I think that, yeah, I think that Rachel and Gulzar are part of the control group. They are not aware of this, but they are the control group, and we are the treatment with Isabel. And I think it's important in the treatment is having edited the book. I'm sure that the control group can give also sound answers to the question. I think that we would like to hear from Rachel and maybe another question for Gulzar two questions, but Rachel can probably also come back on the question of complimenting or other methods. The question for Gulzar one question already that's there but also want to add my own question. One question is that, you know resource constraint African countries in this particular question but also of course India also the same situation. I'm trying to develop national evaluation systems because every policy maker wants to know how to better to do some policy right. So if you have this constraint on resources, how exactly do you give in the given the dominance of our cities, how exactly do you counter it. And my specific question is goes out that in India as you know, J pal have been doing a lot of our cities with governments of both the national government and the state governments, even as you know in under British. So, and here's paradox that you mentioned that state governments and national governments in India and also in Africa and other parts of the country are working with J pal for this very large scale our cities very expensive cities when, as you said the implications of the policy implication not particularly very relevant anything very new. So how does one explain this paradox one is the fact that many developing country governments are under a lot of resource constraints trying to develop develop the only valuation systems. They push towards our cities and paradoxically a working with with J pal and others other impact valuation groups on our cities which may actually not be very useful for them into the findings. How do you explain this paradox. I think we need to step back a little bit you know I think there is a lot of what you call myths and folklore which floats around our cities. You use the word phrase, are working with governments. Now typically, not just like typically I'm pretty sure like if you pick up 100 our cities, you pretty much close to 99 would theory of change would be something like this. That's an idea, somebody, something comes up to somebody's mind, they then pick it up scout around for a funder, funder gets mobilized, then they go around to a lower or middle event, I'm typically talking about India, a lower or middle level bureaucrat, not even an eyes officer, perhaps even an eyes officer, and ask them, hey guys, you know, look what we've mobilized is this seems a cute little this thing, this is great like some profile. So why don't we do it. And what's the skin in the game, what does the bureaucrat lose by having somebody come and plug in in the most non invasive of non burdensome of manner and do something what's. So, I'll tell you what, there is a there is one world in which the debate about our cities happen. There is another world in which real world of their things happen. They actually overlap very little. They really I mean, you, I've worked at every level of government from a local government to a district to a state to the, to the, to the top office in India, you really I've seen this whole thing play itself out. Evidence. What is fun as folklore outside is very different from what is seen inside. I'm just saying this because I really think it's an important thing for everybody I, for example, no I'll just share the screen on the covert questions. And I had, if these are questions which I would have to grapple if I become Secretary of Health. I mean, any of us could become this like any time in, in, in our generation, just have a look and see how many of them have, can in any way be sort of resolved using our It's seriously really, really small numbers, I mean, small things. So, I mean, I, it's not even a first order thing, we are like second, third order, maybe fourth order some ab testing of something which is happening in real time. So, maybe something which you can give a result in two weeks outside that zero. And there was a question about what damage, I can recount several list out. In fact, of the three things which have been listed out, which are which are illustrated in that, in my interview, two of them are damaging. I'd say positively damaging. The good thing is that bureaucrats have obviously more like they understand what's, what's to be picked up and what's not to be picked up and like in the process a lot of this is like, people think, Oh, yeah, our CDs are taken great seriously in India. Well, that's fine. Very nice. Thank you so much. That's very, very insightful ratio a very bad lot of time, but do you want to respond very quickly on the, what else, how can we compliment our city, especially from a quality point of view. Yeah, sure. Just very quickly on this question about multi method research I mean I think the sort of implication of my presentation is that there there are some, some really interesting things that could come about from combining comparative methods and case study methods with RCTs. So if you think for instance about, you know, there's a there's a lot of work by Esther to flow and colleagues about the impact of, for instance, female reservations on political attitudes and gender relations in India. So one of the policy implications for instance that follows from that is, you know, maybe gender quotas are a way that we can think about changing gender norms. So if we take that, we could think about different ways that we could use comparative methods to explore that finding more by testing it, you know, by by doing similar RCT analysis in in say, another country with, for instance, very different system level characteristics. So this would be sort of a most different system design that it was what we would call it right in political science terms. And so if if the finding holds up across this sort of analysis, then we would, we would have more faith in the finding. So I think there's a lot of scope for combining these sort of qualitative case study methods with with RCT analysis. Very positive note, Rachel, actually, let's end this webinar. Thanks so much to Isabel Francois, Goodzer and Rachel for really insightful presentation and we got some great questions from the audience. Thanks so much for that. So I'll call this webinar to a close. And again, as Rachel mentioned earlier, everybody should read the book. It's really amazing that how much Francis, Isabel and you at your with your other co editor brought together some different voices in one place, including, I think the very important voices of the policy policy side, such as Goodzer's. I think that's quite remarkable. I haven't we haven't seen anything like that before, and well done for that. And I hope that this book becomes standard reading in any development economics post anywhere in the world. It has to be. Absolutely. So yeah, thanks so much. And bye bye. Thank you. Thank you for this great opportunity. It was great. Thanks.