 I'm just gonna stick in here. Yep, in that way. Oh, good. You can walk over. You can walk over in the hands. Yeah. Yeah. That works for me. It's not the offing. Yep. You can run around and eat. So, ladies and gentlemen, let me welcome you here to the United States Institute of Peace. I'm Bill Taylor. I'm the executive vice president here at the Institute of Peace. And I'm very pleased to welcome all of you to this forum. As a beginning of a big week, as we know here in this conference, we've got a way to participate here. See, that's a hashtag here. So anybody who wants to do that, and I'm gonna welcome the folks who are online for this as well. This solutions forum is something that we've been looking forward to for some time. It's very exciting opportunity to get people in this room who know about monitoring evaluation and know how important it is to be able to demonstrate what we're doing and that we're having an effect and to be able to do it better. We're out there doing work all over the world. As you know, the Institute of Peace founded some 35 years ago and we do work in Afghanistan and Iraq and Tunisia and Myanmar. And we need to be able to show that the work we're doing in these places is actually having an effect. And so that's why we here at the Institute of Peace have established an office here. David Conley and Christiani and Joe Hewitt and their lively team are working all the time on this kind of work, on monitoring, evaluation, learning, research. And this is the focus of what we're gonna be doing here today. So we are here cosponsoring this first ever monitoring evaluation solutions with Alliance for Peace Building, which you will hear more about of course, the entire Peace Building Evaluation Consortium and the One Earth Future Foundation. So this is a joint effort. It's gonna pull together the work here today. I think this is gonna be an exciting opportunity, as I said. I am, I'm the warm up act here. I'm going to introduce Lindsay Higger in a moment, the Director of Impact, Learning and Accountability from the One Earth Future that I mentioned is one of our cosponsors. And then after that, our Vent Partner Alliance for Peace Building, represented by Jessica Lumpgardner, Zosik, who is here. Yes, he's right here, right next to Lindsay, very good. We'll come up and tell us what the plan is for the day. So without any further, let me turn it over to Lindsay Higger. Thank you very much for joining us. And Lindsay, please come to the stage. All right, well welcome everybody. I'm so happy to see you all here. Thank you to USIP and the Alliance for Peace Building. Really, these two institutions put together a lot of effort, a lot of resources to bring us all here to talk about this very important topic today. And really I'm quite thrilled that this happened. Jessica and I started talking about this maybe a year ago and it was just thrilling the idea that we could have this group together to talk about these sorts of things, especially in the spaces where we work that are so complex and so difficult. This is really thrilling for me. I'm Lindsay Higger. I'm the Director of Impact, Learning and Accountability at the One Earth Future Foundation. We are located in Broomfield, Colorado, which probably most of you haven't been to. It's a small suburb of Denver close to Boulder. We come out here occasionally in DC. You'll see one of the future folks walking around. Most of our operations are geographically spread though and spread across quite a bit of different issues. At One Earth Future, we have programs that do span really a wide array of things, including blended finance, women's peace and security, the reincorporation of ex-combatants, and a number of other cutting-edge sort of research topics. The thing that holds these programs together is this idea that all of them are scoped around very rigorous and narrow problems where we see some benefit to innovative governance, to networked forms of governance. And so that is the theme that even though our programs are disparate, that is the theme that runs through all of these. Now, Impact, Learning and Accountability, or ILA, which we call it at OEF, was founded with the purpose of providing decision makers at OEF the kind of data and information they need to make their programs results-driven and solutions-oriented. And when I talk about decision makers, I don't mean just the board. We do a lot of reporting to the board. That's fairly obvious for a shop like ours. But we also view it as important to provide information to people, to staff, to interns, to managers, right, who are making the on-the-ground decisions about which way programs should go. And so the sort of things that we do day in, day out, span a wide variety of both outcomes and outputs for all of our programs. Now, today, I am just, as I said, absolutely thrilled that we are here in this space. Because we all work in this peace-building community. And in this community, our road is not linear. It's full of roadblocks, right? That is something that all of our programs and many, many of our staff face, day in, day out. The idea that we can come here and talk about these innovative practices that we can learn from each other is really a valuable experience, something that we don't get to do all the time. So I'm thrilled about the discussion that we're about to have. With that, I'll turn it over to Jessica, who's gonna tell us a little bit more about how we're gonna operate today and give us a lay of the land. So thank you all. Hi, good morning. Thank you all so much for being here. As I said, I'm Jessica Baumgarner-Zuzik. I'm the Director of Learning Evaluation Alliance for Peace-Building. And I'd first just like to reiterate what my colleagues have already shared with you and that we are so welcome and sorry, so pleased to welcome all of you to this event. I would also like to give a special thanks to our donors and our co-host, the United States Institute of Peace, One Earth Future, as well as the Peace-Building Evaluation Consortium and Carnegie Corporation of New York. That we're really here thinking about this crazy notion that I had quite a while ago on how can we bring people together for more experiential learning in a field that is so highly based upon what you yourself have done or you've learned from others. And I'd really like to especially welcome each of you, including our scholarship recipients who have traveled very far to be here today and present the months to you, the practitioners, implementers and programmers who obviously make this work happen every day. The policy and advocacy represents here who are supporting the cause of peace-building as well as upcoming students and academics and our donors who support and in the end really believe in us in the mission that we bring forth. And then everyone else who's here, let me extend a large thank you and welcome for attending. We wouldn't be here about a monitoring evaluation forum if I didn't give you a little bit of data and a little bit of information on what's going on. I do not know how to change this presentation from where I'm standing. Anyone, anyone, look at that. Wow, beautiful. Get from above. So a little bit of information for you guys. We had close to 200 participants registered to attend throughout the day. Within that we're gonna be hosting five sessions today, a beginning and closing plenary session and in between we have three concurrent sessions that hold three individual presentations within each. The general format just to give you a little bit of information on these diverse themes that are being presented is that we'll have three presentations of 20 minutes each. So 60 minutes followed by a 30 minute informal Q&A where your speakers are gonna join you out in the audience and help you really discuss more of the nitty gritty. If you have a specific question for one of them you wanna dive in a little bit more to the methodology, thank you so much. Which button do I push? That's a good question. Try the green one. Excellent, across 30 presentations that are gonna be going on on a variety of really exciting themes. Not that is, sorry guys. Perfect, a little bit on our presenters. We have 40 presenters joining us today. 53, sorry, 52% who are female. I'm a little pleased about that personally. 48% who are male and just amongst our speakers we have speakers joining us from 10 countries around the world that does not count all of you wonderful participants who have joined us as well. Next slide. Awesome, a little bit of housekeeping if you have not already downloaded the Events XD app. It's a free app, send everyone instructions as well as the volunteers around the audience can help you if you're having difficulty. That's gonna give you all the great information on the presentation, session descriptions, bios, speakers, et cetera. I encourage you to check that out beforehand if you haven't already. Maybe one more. Thank you. And of course this could not be a monitoring evaluation program without a bit of data collection on our part. We have three different types of way we're gonna be collecting information from you today and in the coming days. First of all, your app, the Events XD that I shared with you has session feedback. This is a great opportunity after each session for you to share both comments as well as questions because everything that's shared within the app related to questions and for the panelists will be shared with them following the event. So if you have something you were dying to know you didn't understand or they didn't address please share that with the presenters so that they can learn and improve themselves. We're also gonna be doing up a follow-up event questionnaire. I'll be sending this to you via email. That's gonna be much more related to program quality, talking about how did we do, what could we do better, what do you wanna see more of. And then the third one I'm really excited to share with you all you should have received an email about this extra event questionnaire. This is part of a much larger research program that's being conducted by Alliance for Peace Building with support from the donor community on the state of the peace building field. And what we're really trying to understand is learn about some of the key topics that are influencing all of us in not only how we're operating but what we see as the vision of this field. It's a great opportunity for you to share especially as practitioners and implementers and some of our people coming from the field the problems that you're facing and you want your donors to hear. So I encourage you to take a few minutes it is 10 questions for a demographic and only six pretty open-ended I would say questions I'm looking for you to fill in related to your vision related to what support you as a monitoring evaluation practitioner need from your donors and from the community as well as a few questions along the state of peace building monitoring evaluations, challenges, opportunities, et cetera to encourage you if you were not encouraged enough already because I know how jazzed I am about doing surveys. We will be doing a cash drawing at the closing plenary session day where we'll have three winners with $150, $50 and $50. I have $100 and $50 bills in my pocket so please complete the survey by 1.30 if we could go to the next slide please. This is the link, it's a bit.ly link all lowercase please. ME Solutions underscore survey this sign is out at the registration desk as well as in each of the rooms that you're gonna be in so take the time to fill it out 10 questions I know you got this in you. So thank you so much and it's a great pleasure for me to introduce our plenary speakers who are gonna be joining us today. First we're gonna have a presentation on improving research design that's gonna be led by Keith Ives the CEO and co-founder of causal design as well as John Kurtz the director of research and learning at Mercy Corps at Mercy Corps apology. The second presentation will be improving data analysis and thinking about how do we draw insights from the data that we're collecting. This will be led by David Hammond the director of research at the Institute of Economics and Peace and third presentation is going to be focused on how do we improve use of data making data not only useful as well as used by our community. I know that's one of the great things that I struggle with when everyone's putting out such amazing reports and no one's reading them. So this will be led by Andy Bloom who's the executive director of the Croc Institute for Peace and Justice at the University of San Diego as well as Beza Tesfaje a senior researcher at Mercy Corps so thank you so much and please join me in welcoming our panelists. Good morning. Is that working? This one works. All right. I'm hoping this one's going to be off so I don't give you tons of feedback. Jessica, thanks so much for the warm introduction. Keith and I are really pleased to be here today to be able to talk a bit about our experiences and insights and conducting impact evaluations on peace and conflict programs. And we're going to largely structure the presentation. You have a clicker? I don't. Jessica do you have the clicker? Largely around the program cycle. So really laying out some examples of key steps we've taken both before program implementation during and after to be able to answer questions around program impact and other calls of questions. As a bit of background Keith's organization and mine have collaborated over the last five plus years on a range of evaluation and research projects. See if you can figure it out. On both peace and conflict outcomes as well as in conflict contexts. So doing studies in Nigeria, Afghanistan, Iraq, Syria, a number of pretty tricky places to pull these designs off. I think one constant has been being flexible around trying to find methods and designs that are both rigorous but also flexible around the inherent limitations in some of these contexts. So in that spirit, yeah we felt like if we had a poster for this session, this would be it. The other choice was a stylized picture of Sharon Morris. For those of you who know her as a staunch advocate of evidence in the peace building field, but we'll go with Obama this time. But really what we're trying to do is motivate this group and others to really think about research designs that are rigorous that can add to the evidence base in the peace building field that might not have been in the toolkit so far. So Keith. Yeah, before we dive into some examples, case studies and geek out on methods for a few minutes, I wanna start with a little bit of the why. And I'd say complexity is and the contexts that we work in have really driven the evaluation culture in the peace building field towards more qualitative methods. And historically those kind of more ethnographic approaches have been the norm and for good reason. But more and more we're seeing pressure both externally from our donors, our government partners and funders for more robust data use for more quantitative evidence and to push more towards these rigorous impact evaluation designs. And it reminded me of the timeless words of notorious BIG and that's with more money comes more problems. And this is something that all of you as implementers need to be ready for. The second driver is also internally our own teams, our leadership, our program and implementation staff and our sectoral experts are wanting to refine our theories of changes. They're wanting to tweak and to improve and to experiment on the margin of these programs and find ways to maximize the outcomes that we're targeting. And that also pushes for more use of experimental evaluation methods. When we start talking about that though and I start discussing these methods with program teams, I always still get this pushback of, yeah sure, but we can't do that in this context. There's too much complexity here. This is too hard. There's too much chaos in our area. And it reminds me of not the 16th century, but the 21st century philosopher, John Locke. You would know him from Lost on TV. And every time he came up with this kind of outlandish solution to their problems, everyone would tell him it wasn't possible, it wasn't feasible and he would scream out, don't tell me what I can't do. So just listening to you, some of the crowd may be thinking what I am, which is okay, I hear rigorous impact evaluation. Does that mean that we should always be doing randomized control trials? Because that sometimes is the implication there. Yeah, don't leave. Yes. No, I think we should go for that. We should aim for that. We should start from there and see if that's possible. Don't immediately dismiss it, which is often the case where we go, yeah, yeah, that doesn't work here. Let's start with saying, can we make that work here? I think historically impact evaluations have, I think most people are gonna be familiar with this matrix, but our impact evaluations have played on kind of the right side of this matrix, more stable context, well-defined programs. These are larger studies, usually done by an academic or university partner. And those are good. And to be honest, I think in our field, in the peace building field, we still need to do more of those. These are looking for kind of the answers to big unanswered questions. These often are testing bigger theories and aiming for external validity across contexts. Yes, we should try to do that, but also we're seeing more and more use of what has been coined in the last few years, decision-focused evaluations. And these are smaller, faster evaluations that we can do on the left-hand side of this matrix, where we're testing outputs, things like uptake or short-run outcomes and that are more dynamic and flexible to your program cycle. And so I think, yeah, we should be going for the RCT and seeing if that's feasible first. John, I know one of the things that we've done to kind of figure out what is possible is invested in scoping trips or feasibility studies first and a value-ability assessment, where we go in and we put a researcher on the ground with the program team to start with kind of co-creating those research questions and helping define those and then trying to determine what the most feasible, the most rigorous method is that's practical for that environment. And it reminds me of our trip to Jordan. This is probably two years ago looking at a social cohesion program with host communities and refugees and we went in, we looked at it, we looked at the sample size, we looked at the program, the dynamics of it and we said, you know what? We're gonna need to go with a kind of a non-RCT approach here, a second best method and that was the best method for that context. Thanks. So one other argument that I get, especially at this upfront stage when we're trying to integrate or negotiate to include an impact evaluation in a new program is essentially that it is one where it is not gonna give back in the timeframe that any of the program team feels is relevant. It's gonna take two to three years before we know anything. In the meantime, we have to be able to understand this program if it's having the impact to be able to make course corrections. How do you respond to those situations? Sure. I'd start by saying that that's true if the outcomes that you're observing take a long time to realize. But then that's also true for any evaluation approach that's going to investigate those outcomes. Although I'd back up and I'd start by saying that final report on the impact evaluation, that's not where the learning process should begin for a program. And we should be starting that backing all the way up to before. We even begin the program and looking at administrative data sets. What data do we have on these populations on this area that we can explore? In 20, I think 13, 14, we were using Afrobarometer data to try and identify what the correlates were between youth propensity for violence, particularly political violence, and individuals and households and community characteristics. What might be triggering that? We also have the baseline data. Almost every program collects baseline data. In Somalia, I remember your team worked with the baseline data to use that theory of change that was developed out of administrative data, but then test that theory. Does it still apply in this context, in this community that we've now sampled in our baseline? And then the last piece, which was actually even more innovative was taking the polling data from Iraq and merging that with the Iraqi body count data and using machine learning techniques. This is before the program begins to try and identify where conflict is most likely to arise and then target your program there. And that's all upfront before we begin operations. So great, the next inevitable question that comes up from program teams and others is, okay, fine, but I suppose now you're gonna tell me that in order to understand a tributable impact that I've gotta withhold this program from part of the population. And I guess in conflict context, the argument is particularly sharp because I've heard and believed that doing so can actually exacerbate some of the tensions that our programs are trying to address. So what do you do? What do you say there? Right, you're right. This is usually the first pushback that we get when we encourage the use of these methods. And I'd say I've yet to work on a program where we were able to provide that service or that program or that resource to everyone universally. There's almost always a constraint. It might be financial, it might be logistical, it might be geographical, there's a river there. You can't get across. There's almost always some natural constraint that we can leverage to create a control or a comparison group that won't interrupt your program cycle, that won't exacerbate community tensions. I reminded of the Invest study. This was a vocational training program that Mercy Corps was implementing in Kandahar in Afghanistan. The vocational training program we thought would lead to better economic outcomes. And we were testing whether or not that would also reduce the participant's propensity for violence in those communities. This is a school. It's a vocational training program. There's only so many seats you can put in the classroom and only so many classrooms in the area. And so we were able to get an oversubscription for the program and then leverage that to have a first cohort that became our treatment group and a second cohort that was our control group until it was their time to start their classes. The second thing that I don't think gets as much attention or discussion is dropping the control group. A randomized trial doesn't have to have a control group. It just has to be randomized. And so we often engage in AB tests where we have treatment one versus treatment two. And this is something that all of you are familiar with. Every time you get online and you click on an ad, you've just been a subject in an AB test. And we do those, reminded of this would be Tobanko in the aftermath of Typhoon Haiyan, that super typhoon in the Philippines in 2013. Mercy Corps was doing blanket cash, unconditional cash programming transfers. And we went in and without leaving people out of the population still targeting to the most needy first, we randomized to where half of the population got one lump sum transfer of about $80. And the other half we gave tranched payments that in sum came out to the same amount. And through that, we were able to observe which one of those methods actually drove the outcomes that we were targeting. This also helps, and I was talking about kind of the left side and doing those smaller RCTs in these more chaotic environments. This gave us an opportunity to add in another layer into your intervention where we sprinkled into some of the participants some financial literacy programming. And some got randomly received text messages reminding them to engage in kind of what we thought were better financial decisions. All right, for time's sake, I'm gonna flip through somewhere. See if you can get that to work again, you're a magician. So one of the wonkier slides and jump into, let's skip this one for a second, and jump into, okay, during an actual program implementation. There's a number of challenges that tend to come up there. Why don't you kick us off on that one? Yeah, a lot of times we spend a lot, we try to design the perfect RCT, the perfect impact evaluation. We get everything lined up right and we hand it over to the team and they go, wait, wait, wait, but now we can't manage our programs the way that we need to. Yeah, this is the one that we get a lot. So trying to understand the impact of a program, there needs, the idea is that, this has to be fixed indefinitely and we can't do what good program managers do, which is change course. So this definitely need not be the case and we found a few ways to be able to take rigorous impact evaluation methods and integrate them in ways that actually support learning and adaptive management. One that we've been using increasingly are survey experiments. So for example, in Northeast Nigeria, during essentially an exception phase of a program, we are taking a survey and including an experiment to test the efficacy of different types of messages on community members' willingness to accept former Boko Haram fighters back into the community. So results from that would come out pretty immediately and be baked into the larger media campaign for that program. Again, that's just sort of an upfront survey that we were likely to do anyway with an experiment embedded. The other side of it is really thinking about how impact evaluation gets synced up with M&E, which is certainly the driver of adaptive management. And one thing there that we've been able to do is to try to set up beneficiary tracking systems that help us understand what I consider to be relative counterfactuals. So often there's variation in how a program is being implemented that we could take advantage of to understand impacts amongst subpopulations. And one particular useful piece there is looking at program exposure, which tends to vary to be able to understand essentially dosage effects. So for example, in a CVE program that we're running in Jordan, we don't have an impact evaluation at all, but they do tracking of every single participant and they're able to look at participation in the number of trainings and any thresholds after which that amount of training starts to make a difference to those individuals' outcomes around support for violence or attitudes therein. So again, really requires linking some of that beneficiary, tracking M&E with outcomes data, which is not always a given, but pretty doable. Another one that I often get is, look, we're working in conflict contexts. We're talking about very sensitive issues here. We're asking people about things like support for violence, attitudes towards other groups, inherently political sensitive issues. We can't get accurate data on that. I don't really trust the results to be able to make decisions on it. What do you do there? Yeah, no, I think that's right. We finally get the results that go, oh, I can't trust those. There's a bias involved there. That's absolutely right. There are, there's a lot of bias, sometimes competing biases on these self-reported data that we get in household surveys. I think the point, like we said, there is everyone lies and that's a good thing. It means that there's some consistency in the fact that everyone lies. And one of the things that we're gonna focus on is the delta, the change in response from one period to the next, not necessarily the point estimate. So, John, if I were to ask you how much money you make, statistically we know you'd inflate it a little more than what you actually make. There was a paper that came out this summer that pointed out that if your wife happens to make more than you, you're actually gonna lie even more to bring that up. She does. So, that's a good thing. And for us as the analysts, we understand that and we're gonna make sure that we're focusing on changes and not those point estimates. The other pieces that we're gonna use, you know, trip tricks and we can give some tips on survey tools that help reduce those, the way that you frame the question, the way that you sequence the question, the answer options that you give and whether you give ranges. But one of the more innovative things that we've been using a lot together lately are using more indirect questions and giving the respondents kind of an excuse or a mask to their answer that they can be honest. And this is where we use tools like list experiments or random response or random endorsement questions. And those are all a little bit complicated but they're almost like little experiments within the survey that give them the opportunity to hide their true response if they're afraid of stigma or judgment from the survey. And the other last option is there's been some growing use, it's not a new tool but is the use of kind of self-reported or self-administered surveys. And this is what you get when somebody emails you a survey or something like that that you mail in where you're a little more anonymous in your response. So we've talked about planning beforehand, managing a good impact evaluation and lining up the perfect impact evaluation. And then we've talked about some of the challenges that we're gonna have in the data. Sorry, some of the challenges that we have in the data may cause us not to trust our results but assuming that it's all laid out perfect, right? We're set to go. But John, we've done a lot of experiments together and they haven't all been planned ahead of time and they haven't all rolled out perfectly. So what do we do in those scenarios? Yeah, we have probably too much experience trying to come to an end of the program and say, oh, this is a great learning opportunity, how can we go and say something rigorous about our impact? But again, in the spirit of yes we can, we often sort of throw our best at it and tend to use methods that would fall under quasi experiments with kind of an ex post flavor to them. And essentially what this allows us to do is to create a comparison group after the fact. But trying to do it in a way that isn't just sort of finger in the wind that we hope these folks are kind of similar to the people that we worked with. So for example, in a study that we did in Hellman on the same program that Keith mentioned in Kandahar, the first round of that was not the RCT. It was this ex post design where we took an outgoing cohort of youth that had just finished the program and compared them against an incoming cohort of youth. And then we used propensity score matching to kind of statistically take away any differences between those groups based on what we could observe in terms of their background characteristics so that we could say any of these changes and some of these outcomes or differences are really around what the program has brought. In Somalia on the study that Andy and Beza will talk about we did that but on more on a community level to say, we worked in these communities and not these where we had liked to but couldn't and we're able to match through course and exact matching. And again, statistically you're gonna make these comparisons similar and therefore more valid. So look, those aren't gonna be the strongest publications as huge limitations in those methods but for us, they were able to sort of fill a need to be able to answer an important question around an important program where we wouldn't have otherwise where yeah, we wouldn't have otherwise had the opportunity to do that learning. The challenge with those studies but I'd say all of the studies is that there's also this big risk that we aren't gonna find an impact and maybe it's because it was there but we couldn't measure it but also maybe it just wasn't there. How do you manage that, especially with your organization and the stakeholders you have behind you? Yeah, and this is not unique to the peace building field as evaluators, the evaluation anxiety and the tendency to kind of bury non-significant findings is huge. I guess first we need to be honest that the majority of our programs or at least the size of the minority if we go looking honestly, we probably won't find a result and that's sort of important to understand. That said, there are a few things that we've tried to do to make it more likely that we're gonna use learning even when we don't see real impacts. One is to try to make these evaluations more about the underlying theory of the program than sort of the program effectiveness itself. And I think this Afghanistan example was a good one where it was designed to produce economic outcomes for youth, it certainly was based on a bigger theory that in Kandahar doing so would help to build stability but they really weren't on the hook for that outcome. So when we didn't find that result, in this case in Helmand, the heat was off of that program a bit. The other thing we can do often is include a number of outcomes in there. There are often secondary measures of changes you'd like to see where you do find results. So it might be increases in things like trust or tolerance that are probably important precursors to the stability or a larger kind of objective conflict outcomes we'd like to see. And then I guess, irrespective of that, and it goes without saying, but just building buy-in amongst groups, I guess from our experience beware when you're engaging with a team and they say, we know our program works, come in and help us demonstrate that please. And we get that. So what we've been thinking about doing and Michael Quinn Patton writes and speaks about this a lot which is up front, and so this may be the beginning stage is working through scenarios of program teams on the possible outcomes, right? No results, positive results, negative, and getting them to commit to certain management responses around those so there's less surprise. So that's something I'd be keen to hear if other people have tried. So this brings us towards the end and I'm really into the arts and by that I mean Netflix. So a lot of our references are gonna tie to that which all we've talked about, this overarching theme that these are possible. Yes we can, we can do rigorous impact evaluation in the peace building space, in CVE programs, in these kind of chaotic and complex environments. And so just wanna circle back to a few points and I'll reference one of the slides that we skipped but one is it's really important to start early. Before your program starts, invest in designing then. That's gonna give you the highest likelihood of leveraging methods that make it more feasible. Whether that's a matching algorithm to help tighten up a sample to give you more power or getting that pre-analysis plan done and the whole team on board so that no matter what the results were everyone's prepared to engage with them in that learning agenda. I guess I'll finish with a bit of self promotion as well and that's that don't as the program team as the sectoral expert, as the implementer or the expert in that context don't decide whether or not it's feasible. Call an expert, call an evaluator and see if they've got methods that are new. Don't assume that the evaluation field isn't evolving as quickly as all of our implementation fields are. Just as innovative as the conflict and peace building spaces are and their approaches we're trying to be just as innovative in the evaluation side. So partner with us, let's find a solution that will work. Thanks. Okay. So the title of my presentation is Database Decisions. I mean, first of all, thank you very much for hosting me. It's I've traveled a long way to be here and it's a pleasure to be here. The title of my presentation is Database Decisions but I think the actual focus is zooming out a little bit from M&E and just talking more broadly about what, you know, some issues around data and some of the challenges that we have in building Database Decisions. I'm not sure if the time needs to be reset. So we sort of, you know, we've developed into a world where something doesn't exist unless we can measure it. And the reason we wanna measure something is because we wanna change the behavior or control it in some sort of way. So data is becoming increasingly important in making decisions. But if you have a look at, I mean, I work for the Institute for Economics and Peace. We do a lot of research on battle deaths and terrorism deaths. If you have a look at the statistics on, you know, very prominent events, the estimates range, you know, wildly. If you have a look at Darfur in 2004, the estimates of people being killed range from 10,000 to half a million. If you have a look at in 1996, Rwandan refugees fleeing to Zaire, that range from zero to 800,000. And if you have a look at IDPs in Kosovo, that range from zero to 250K. So for such important events with really, you know, important decisions around them, the question is why are these ranges so wide and how do we deal with that? The reason that's really, really important is because each one of these numbers I've just quoted were used to either justify response or non-response. They were used to justify allocation of resources. And they've also been used to justify as precedent action or inaction in subsequent conflicts. So because these things are inherently political, I mean, my background, I'm a sort of mathematician, political scientist, all this data is inherently political. And so something, a thing that I've been talking about for a while is in this field of data collection for this type of area, to my mind, there is a bit of a lack of a code of ethics. Now, there are statistical codes of ethics in terms of when you use this particular average over this particular average, but there isn't, as far as I'm aware, a code of ethics of the impacts of the statistics that people are generating. So it's just something I throw out there every so often that maybe we need to think about this. And you know, we can base it on a number of other codes of ethics, the journalists code of ethics is a good starting point, but it's just something I sort of throw out every so often. So a little bit about the Institute for Economics and Peace where our independent think tank dedicated to understanding the drivers of peace. We have four international officers. I'm based in Sydney, but also my colleague here, Laurie, is based in New York. We have an office in Mexico City, the Hagan Brussels. And we do reports on global data. So our sort of cycle is we do a lot of work with SDG 16, which I'll talk about. We also do a lot of work in Mexico on data quality there. We release our global peace index, which is a ranking of 163 countries. We release that in June. November, we release our positive peace report, which is really about trying to shift the focus away from just measuring conflict to measuring things that contribute to peace. And then in December, we also release our global terrorism index. And we also have worked with a number of other organizations that work on these sort of global studies, the OECD states of fragility. The past couple of years, we've been working with them and the pathways for peace. So just to highlight some of the challenges around global data, a good place to start is the SDGs. The SDGs, Sustainable Development Goals. We're an institute that thinks about peace. So we're very engaged with the community on goal 16, peace and justice. There's 22 indicators on peace and justice in goal 16. If you actually have a look at the indicators, less than 50% of the indicators have data for 50% of the countries. Only seven indicators have data for more than 90% of countries. There is about a third that there is no agreed methodology on how to collect this data. And the indicators, some of the most important indicators are comparable for less than 40% of countries. So if you have a look at the SDGs, Sustainable Development Goals, you could also call it significant data gaps. And the SDGs with data challenges is 100%. So what I thought I'd do is just go through sort of systematically and just highlight some of the major issues and challenges in being able to collect data, to be able to analyze data and discuss data for the database decisions. So one of the major issues is that there is no data for a lot of this stuff. If you have a look at goal 16, as I said, we've just done an audit of that. We're also working in the Pacific doing data audits of their statistical capacity to be able to measure and report on goal 16. And the actual budgetary requirements in order to build capacity to report on goal 16 is actually outside of the capability of many of these countries. So data gaps is actually a very important issue going forward for the 2030 agenda. Now, if you actually have a look at many of the prominent indicators that people use, data gaps has been an issue in time memorial. And once you unpack how many of the organizations fill in data gaps, you might be surprised at how basic it is. So for instance, homicide rates, well, I mean, I'll step back. I mean, in many parts of the world, we actually don't know the population, which is the basic number for all economics. GDP per capita is based on that. We don't know the size of the economy. We don't know the size of the informal economy. Homicide rates, if we don't know population, we don't know homicide rates. But allowing for that, there are still statistics out there for homicide rates and population and economics and all the rest of it. How people fill in these things is somewhat sketchy in many cases. So the homicide rates, often if there's data gaps, countries are attributed averages of regions, rightly or wrongly. A lot of the informal economy type stuff are just expert guesses. And so I think the main takeaway from having worked now in this field for 10 years is that in order to understand how to analyze all of this data, it's really important for you to understand how they're filling in gaps and not to just take the data as given. And a lot of organizations do explain how they do that, but also a lot of organizations don't explain how they fill in data gaps. And if you don't know how an organization is filling in data gaps, then you're really limited in what you can do with that. And as practitioners, it's really important that if we are filling in data gaps, that we do that in a very transparent way. There's a bit of a trade-off. If you talk to statisticians, they have fantastic ideas and complex ideas on how to better fill in data gaps. But a lot of those are beyond the ability for the practitioners to really understand where that number is coming from. So there's a trade-off between stronger methodology versus the ability for people to understand how the gap has been filled. So my sort of preference is I sort of err on the side of simplicity because at least if you're filling in a data gap and you can explain how you've done it, at least someone can criticize you in a very sort of direct way. If it's statistical mumbo jumbo, then you can kind of sidestep the whole issue. But I think it's important just to be upfront and accept the criticism if it's there. Cross-program and cross-country data. So in an ideal world, all data would be comparable. So if we think of the SDGs, we would have a data set for any one particular SDG and it would be comparable across 200 odd countries. In reality, we can't compare homicide rates across Europe because of different reporting schemes. Some countries in Europe use the victim as the reporting baseline. Some use the crime. So we've tried, I mean, IEP has tried to look at crime statistics across Europe and you can't compare country to country. And I think that was really, well, startling initially because of all the regions you would think Europe would have their act together. But yeah, because of reporting differences, you can't actually compare these things. So if you can't compare it across Europe, how do you compare 163 countries across something that's a very basic measure of peace and conflict? So that's another thing just to be across. Whenever you're doing cross-country analysis, it's really important to understand the definitions and how that measure is measured in different countries because otherwise you're comparing things that aren't actually comparable. How specific is context specific? So this is particularly relevant for an M&E audience. If you're using data, there's two ways of doing it. You can either go really local and do surveys and do impact evaluations and do case studies at the local level. And depending on what you're trying to achieve with that, that'll either be a useful thing or for some of the issues that were raised in the previous presentation, there'll be challenges in being able to interpret that. But if your goal is to extrapolate from that study and have a generalized finding that can be applied somewhere else, that's actually quite difficult, depending on how local you've gone. If you take the converse, if you do what IEP does and has a look at the global data and then tries to extract generalizable findings, those generalizable findings might be really useful and interesting, but whether they're applicable to the local context is another question. So there's a lot of research at the moment going into how do you merge these two levels of data and at what point, in terms of database decision making, how can you combine global level data with all its challenges with more localized data? And I think some of the work that's been done in this area is probably a really good example is the OECD States of Fragility work. So what they're doing is having a think about how do you use this global level data to identify countries that are similar and have similar properties at the national level? Meaning that if you do a case study within one of those countries that in that case study might also be applicable to other countries similar to it. So there's a lot of work and thinking going into how do you merge these two levels of data? What to do when data contradicts reality? Well, that begs the question, what is reality? And let's not go into that. It also begs the question, how do you know it contradicts reality and if you know what reality is, why you're measuring it in the first place? But anyway, because data is so, especially in this field, so politically sensitive, there are many studies that show some of the reasons why estimates vary so widely. And some of them are nefarious reasons or at least attributed to nefarious reasons. And some of them are actually just because of uncertainty within the field. Uncertainty in data collection, uncertainty in statistical methodologies, uncertainty due to gaps in our understanding of certain areas. So if you come across data that seems to contradict reality, then there is certain checklist which I'll provide resources for that you can go through and double check the logic of the data or the data collection and to really test whether you want to use that data. Because if data does contradict reality, there's a chance that your concept of reality is incorrect, which is a really good use of data to really challenge people's perception and challenge people's understanding of a context, or it could be that the data is just actually rubbish and you've got to make a decision and you've got to make a call on that. And that call ultimately is going to be subjective, but because it's subjective, you want to have at least the checklist that you've gone through to justify why you've made a decision. Now, the Institute for Economics and Peace, it creates composite indices, which means it collects a whole range of different data sources and combines them into one measure. So the Global Peace Index, as I said, is a measure of the absence of violence or fear of violence within 163 countries, and we do that by combining 23 measures of conflict and violence. Now, in doing that, we're taking 23 sources of information and combining them into one list. Now, in doing that process, you're invariably losing information because you're taking 23 things and you're making one thing. So the process for doing that is actually very important and it's important in the sense that there are many composite indices, they're becoming very popular within policy circles. It's very, everyone loves a list because lists are easy to understand. Journalists love lists. We have sporting competitions that are based on league tables, so it looks like it's that. But the actual process of doing that, you are losing a lot of information and it's really important if you're using these types of measures in a policy debate or discussion to understand the process and understand what you are losing in that. So if you are going to use a composite index, again, there's another checklist you can go through which I'll provide a resource for. It's important that the methodology is transparent. It's important that you can unpack and get back to the original information and it's important that at least you understand how different indicators have been weighted against each other and in terms of the importance that the organization or the person has attributed to one or the other. And if you disagree with that weighting, if you think one indicator is more important than another, then it's important that you can play around with that. And the Institute for Economics and Peace provides all our data if you do wanna do that. And I think the way that IEP works for these things is we use the index as a really powerful communication tool to get people engaged. But also we're open to unpacking it and getting criticism for the way that we've done it. And because in doing so, we're actually teasing out some really interesting conversations. So the point of it from IEP's point of view is not to say here is the ranking of peaceful countries in the world. It's actually to have a focal point so that people can criticize it and unpack it and go through the issues in a systematic way to think about how do you actually use data to create, to inform decisions. So here's a couple of resources that I would just point to when you're talking about, well, in our case, global data. But I think more generally, any data-driven research, I think these are really interesting and really important resources. The first one is from Africa Check and it's a really nice flow diagram of how to check validity of data. And it gives you a process to go by so that by the end of it, you either accept the validity of the data or you reject it, but at least you can explain why you've rejected it. And the Handbook of Constructing Composite Indicators is as interesting as it sounds if you read it, but it's a very good resource. It's the go-to resource. It goes through the accepted measures, accepted ways of filling in data gaps and it goes through the pluses and minuses of each approach, so it's good to understand that. It also goes through the standard way of creating a Composite Index and things to look out for for poor Composite Indices. So again, it provides a bit of a checklist. And so those two resources I would point to as very good resources to use and operationalize if you're using any sort of data-driven research. So I think in terms of the presentation, I mean that's really all I wanted to do was just highlight some of the issues that you face when using data. And I think just with the remaining time, I'll just throw out a bit of a debate or a conversation piece in the appropriate use of data. Now, as I said at the start, when you're talking about highly political events, the range of data that you get can vary widely. Now, there is a perception or an argument to say that if the cause is good, it doesn't matter if the statistics are accurate because the point of the statistic is to get people focused on this thing that is good. So whether it be famine, whether it be conflict, whether it be refugees, there is a perception in the field that it doesn't really matter even if the number is ballpark. If it gets it on the policy discussion table, then that's all the statistic is for. That's its purpose. And then you can actually discuss going forward from there. So the question is, and this comes back to the thing that I threw up at the start, just flagging that maybe we need a code of ethics for this, in that inherently what we're doing here is attributing a scale to any one particular event. And rightly or wrongly, we're attributing in the case of conflict that conflict can be measured by the size of deaths, for example, and the impact of conflict can be measured by the size of people being killed. Now, in lieu of any other better way of comparing conflicts, then we're reduced to that. But the question doesn't matter whether that number is accurate. As I said, there are certain people who, well, certain bodies of thought that say it doesn't really matter, it's more about getting the discussion going. But if the number itself isn't comparable to other numbers or it's not accurate, then it kind of defeats its own purpose because the idea of being able to quantify these things is to have a decision-making process to allocate resources. And while any one particular event or conflict is very important, if it's not measured in a comparable way, then okay, you might get attention drawn to it and resources allocated to it, but at the expense of what other thing that those things could be allocated to. And I think that's the counter argument of that, is while no one is arguing that focusing on one particular conflict is the wrong thing to do, if it's based on poor numbers, then what is the opportunity cost of how those resources could be attributed somewhere else? So I think I'll finish with that, but thank you very much for hosting me. It's been a pleasure to be here and look forward to questions. Thank you. Good morning, everyone. Good morning. My name is Beza Tesfaye. I'm a senior researcher at Mercy Corps and thank you for joining. And Andy and I are really excited to be here today and to share with you some of our ideas about the joint collaboration that our organizations undertook around a very interesting research project between 2016 and 2018. Both of our organizations, Mercy Corps and the Joan B. Crock Institute for Peace and Justice, are really invested in undertaking policy-relevant research. And we wanna explain why and how we've done this with a very specific example from our work in Somalia. So building a bit on these previous two presentations, our presentation is gonna focus less so on the research findings and outcomes and more on the process of actually designing and executing research that has both internal and external utility. So in the next 25 minutes, we're gonna give you some background on this specific project that we undertook and then spend the majority of the time talking about key lessons around things like stakeholder engagement and internal and external dissemination based on what worked well from our experience but also importantly what didn't work well. And our aim is really to leave you with some key takeaways on how as you're thinking of developing your own research studies, how you can increase the likelihood that your research is gonna have traction amongst key audiences that you're trying to influence. So with that, I'll hand it over to Andy to kick things off. Hi everyone, I'm Andy Blum. As Beza said, I'm the executive director of the Kroc Institute for Peace and Justice at the University of San Diego. So thanks for having me. Thanks to Alliance for Peacebuilding and double thanks to United States Institute of Peace for both hosting us but also they were supporters of the research. So I definitely wanna thank them as well. So I'll start with this idea of policy relevant research. And this isn't something that's new to anyone in the room, I'm sure. But really policy and relevant research, it has to link to somebody making a decision somewhere. Whether that's to keep doing something they're doing or to do something differently. And so I'll talk as we go through about decision makers. And the question is how do we do it? And when I say we, I mean we, everyone in this room, how do we do this? This is a complicated process. And just to kind of situate what we're talking about here, I mean, there's a big theory of change to how a decision actually gets made. We're gonna be talking mostly about that engagement process. How do you get, as Beza said, that traction? How do you get that initial engagement? How do you get some eyeballs on your research? Once it goes into an organization, State Department, USAID, Mercy Corps, we're gonna be talking a little bit less about sort of that process, how that sausage gets made. One big idea we want you to think about as we go through this is that pretty much everything we experienced trying to get this research out here was that we have evidence, but then we have these relationships we need to actually get the uptake. The evidence is important. The credibility, the rigor of the evidence, and we'll talk a little bit about that. But it was also those relationships that were crucial to get that evidence in front of people and it was relevant, as you'll see, for both some of the successes we had and some of the challenges we experienced with bringing that evidence to folks. So I wanna turn it back over to Beza now and she's gonna talk a little bit about the motivations we had and then the actual research. Yeah, so before getting into some of our lessons on internal uptake and usage, we wanted to just give you a bit of background on this specific study and we wanted to start a bit with motivations why we actually decided to do it. First, it was pretty clear to us that there are some significant evidence gaps on the question of what works to prevent or reduce violence or violent conflict. And there are in fact very few empirical studies that have truly furthered our understanding and practice on things like how we can prevent young people from engaging in violence through development programs. And there are many, many reasons for this, some of which Keith and John have touched on so I don't wanna get into an explanation of all the reasons but it's generally a gap and there needs to be more empirical mixed methods and theory-based evaluations of programs that are addressing conflict to understand if they're effective or not effective. So because of these gaps at Mercy Corps over the years, we've really invested in building a research agenda to understand some of these questions by doing more and more research on the topic of conflict and youth engagement in conflict. And this agenda is not just based on one study but the idea is that this is an iterative process. So we test a common theory that underlies many development programs that are trying to tackle conflict and we test this theory in multiple contexts over many years. And so this study that we did in Somalia, it was built on other studies that we had undertaken both in Somalia and also other parts of the world where we're working. This includes a survey that we did in 2012 which actually informed the design of the program that we later evaluated and it also includes a preliminary evaluation that we undertook in 2015 of this program in one specific region of the country. Another reason or motivation for this research was accountability. This was a very large investment. It was a 40 million five year program that we had been implementing with the support of the US government. So we felt given all of the resources, efforts and time that had been expended in implementing this project, we felt that there was an imperative to really understand what were the failures and successes and really understand if it had any kind of impact on some of these longer term stability outcomes that it was meant to influence. And then lastly, we did this evaluation because it was generally part of the mandate of both of our organizations. So for example, the Crock School specifically is focused on using applied research to end cycles of violence. And similarly, Mercy Corps were really focused on improving our response to some of these huge challenges that we're facing like conflict with evidence. So simply put, this research gave us an opportunity to learn from a flagship program what was effective to be able to influence practice, policy and also donor funding around violence reduction. So to sort of try to achieve some of these learning objectives that we had, what we ended up doing is designing an impact evaluation of a program known as the Somali Youth Learners Initiative. And this was a program that we were implementing throughout the entire country of Somalia. But as I mentioned, we undertook a preliminary evaluation of the program in Somaliland. So this actual collaboration with Mercy Corps and the Crock Institute was focused on South Central Somalia and Punta Land, areas that were seen to be more insecure than the Somaliland context. So the research question we were trying to address is what is the impact of secondary education and that should not say vocational training. It's secondary education and civic engagement on young people's propensity towards violence in Punta Land and South Central Somalia. And these were largely areas that had been previously under the control of Al-Shabaab. So from the onset, there were quite a few challenges that we face as you can imagine trying to design and execute this type of study. The most obvious, of course, was the accessibility challenges. As I mentioned, we were trying to undertake this research in very insecure areas, areas that had been controlled by armed opposition groups. So we had to think cleverly about how we could sort of design a flexible evaluation approach to help us be able to understand the security context and sort of adapt to the environment on the ground. And we also had to identify a local partner who could give us the ability to access some of these areas and do survey work in them. The second sort of challenge that we faced was that we were doing this evaluation ex post, which means that we started this evaluation after the program had already begun. And we didn't have a baseline with the measures around violence that we were looking for. So this meant that the research partners, the CROC institutes, had to design an identification strategy or evaluation design, basically, that helped us approximate doing this evaluation ex ante, something similar to an RCT. So in the end, our research approach, it was a bit more complicated, but it was also rigorous, or it helped us be as rigorous as possible, given this limitation that we were doing the research after the program had begun. A third constraint that we faced was this issue of social desirability bias, or generally the fact that we were probing about some very sensitive issues in a very difficult environment again. So asking people about their attitudes towards armed groups like Alshabab or their support for political violence, we knew if we asked those questions directly, we were very unlikely to get truthful responses. So again, we had to figure out a way to work around this constraint. And what happened is we ended up using some of these survey experiments that Keith and John mentioned to be able to elicit more truthful responses. So given all of these considerations, the actual research design that we landed on was like quasi-experimental mixed methods impact evaluation. And there were two components to this, a qualitative component and a quantitative component. For the quantitative components, we undertook surveys of 1,200 youth in South Central Somalia and Punta Land. And the process for doing this was first, we identified communities that had either received the program. These were the treatment communities. And then we identified communities that we had planned to implement the program in, but then due to some changes in the scope of the project, we ended up actually not going to those communities. So those became our control communities. And then we used a matching technique known as course and exact matching to make sure that these treatments and control communities were comparable on things like level of insecurity and violence, level of urbanization, and also the type of project that was either implemented or planned to be implemented in those communities. From there, we surveyed youth who had either received education alone through the SYLI program or Education Plus Civic Engagement. And we did our analysis by comparing youth in these two treatment groups with the control or comparison group and looking at their outcomes related to violence, so support and participation in political violence. And we chose this particular design again because it was flexible and allowed us to get around some of the constraints I mentioned in the previous slide, but also because it was rigorous. So it gave us the credibility that we needed in order to make this research really useful and have impact as well. So I was more involved in the qualitative research. I wanna talk a little bit about it, but I mean, I love mixed methods research and I'm happy to talk more with anyone about sort of what we did and how we did it. A couple of the highlights from the qualitative research that I thought were interesting alongside the quantitative research, there was this sense that they were, that you got from the qualitative research about the day-to-day violence the participants were experiencing, you got a sense of the corruption, you got a sense that these youth were hemmed in by corruption and by some of the clan dynamics, but you also got a sense, especially with the civic engagement, that the youth did still have a sense of efficacy, did still have a sense that they could make a difference if they were engaged. But really sort of for the thing we're talking about here in regard to engaging the decision makers out there, I think what the qualitative research did was open doors into the research from some people who weren't as interested in the quantitative results. We all have different ways into evidence and if you're a Somalia area expert, if you've lived or worked there, you kinda wanna hear about that day-to-day violence and how that's impacting what you're finding. You might wanna hear the stories, if you're a policymaker that's more focused on short-term dynamics, you might wanna hear about the day-to-day lives and what those schools and those communities are doing sort of right now in terms of signaling to al-Shabaab or signaling to the youth that something is changing. So it was a way into the research for some of these policy makers that might not have been as interested in the quantitative results. So when we got to the end, I'm gonna skip over that slide for time. When we got to the end of this research or got to the end of the initial research phase, we had this rigorous quantitative research. We had this sort of more nuanced qualitative research. And so what to do with it at that point and what to do with it at that point that engaged these decision makers. And the process we went through is something we ended up calling sense-making. The question of what do these results mean became sort of central because there was a really sort of complicated set of results and complicated set of dynamics. And here's a few of the dynamics we had to make sense of. Just the question of education itself. Education raises expectations can that lead to violence? Education creates a sense of hope in the future that reduces violence. So what story do you tell about the data that doesn't get out ahead of the data? We were constantly like, are we telling stories that are out ahead of this data or out ahead of the results we're getting? Or is this justified by this? So we went through a very complex sort of sustained internal process and I really credit the sort of team for really investing in that internal process of what do we know now before we went out to some of the external decision makers? And we did go out to external decision makers. And I've already talked about the internal with the research team. And what we did, and I think this is one of the things that was successful, is we engaged a set of DC policy makers and decision makers before the end of the research process. We said, here's what we have found so far. Help us make sense of it. And this process of working with decision makers to make sense of the data really I think allowed them to be part of the process and to create that engagement and to co-create the meaning within the results that was relevant to them. And we thought of it as sort of a hybrid process. We're getting those results out there but they're also contributing to the research process itself. And I think I'm not even sure we sort of planned it out that way but like as we went through, I think we really found something that worked in terms of engaging and getting research in front of policy makers that they could grab hold of and really start to sort of wrestle with themselves. We tried to do this with some of the global networks that the Crock Institute has. We have a global network of over a thousand women peace leaders, Mercy Corps has networks. We've got some other networks as well. And we sort of tried to put out the research to them as well and said, you know, what does this look like to you? Does this resonate with you? And we got this sort of trickle of survey responses back. You know, we were just asking five questions but we got this sort of tiny trickle back. And again, I go back to this issue of relationships. We didn't build up a relationship with those folks to have them engage. If I had to do it again, I would say, okay, who are the 40 we wanna talk to? Bring them in at the beginning, call them an advisory group, give them an identity, bring them along in the process and then they would be sort of teed up to give us but just dropping this into their email box at the end of a process, it didn't work. So here's some of the folks we talked to during the process. And one of the interesting things about this research was it kind of sits in the middle of a lot of different related but different sectors, education and crisis and conflict, CVE, peace building, Somalia or East African area experts. And we ended up talking to a lot of them. And this had sort of pluses and minuses. It got it out there a lot. It's sort of on the dissemination piece. It was great. But it didn't really sort of help get the research deeper. So it's a broad versus deep question. And I think if I had to do it again, I might have targeted one of those sectors and said, okay, education and crisis and conflict, I'm gonna map that out. I'm gonna talk to 12 policy makers or decision makers in that space and really go deep into that space. But again, there's a tension here between the dissemination piece and the kind of policy engagement piece. And if you have time and space to do all of it, that's great. There was some more intentional decision-making. I think we probably should have made. But the key takeaway is for each of these groups, we could help them find, ah, that's the piece I care about. That's the piece of this complex set of results that I care about. Can I just say, slide? I'm gonna just go through this really quick. As just an example of how we use these sense-making workshops to give us input on big questions, unanswered questions we had coming out of the research, one of those was we saw divergent findings between our evaluation in Somaliland and the results from South Central, Somalian, Puntaland. And so we really engaged experts to help us identify and interpret these conflicting findings. And as you'll see, we received a set of feedback that we then incorporated into our second round of analysis and interpretation and basically the final product. Next slide. So once we had finished doing these sense-making consultations and we finalized the report, we were ready for the dissemination stage or phase. And this took two forms. The first was internal dissemination. Mercy Corps was the biggest consumer of this research since we had implemented this program and we were doing similar programs in other parts of the world. And based on this experience, we identified some best practices around internal dissemination and some challenges as well. Some of the best practices very quickly is first to target influencers within the agency. I think oftentimes we think of program managers and direct project staff as being the individuals that we feel like would make the most use of this research. But we've realized it's also people that are involved in broader strategy setting and also involved in, for example, developing proposals across the agency. So in Mercy Corps, this includes teams of, called the new initiatives team, for example, that works on proposals globally, regional and country leadership. So they're the ones that set the agenda for the countries and the regions and the technical teams that help input on the design of programs and help tweak programs. So getting these people to really understand the findings and use them I think can help spread the impact of the research more widely within the agency. Other best practices are trying to do more in-country dissemination of the results. So going back to the country where the research is done and sharing it amongst local stakeholders with the field teams, and we did this in Somalia, we went back to the region and worked with our country director and project staff in sharing the results amongst local government officials and policy makers, and why this was really important for internal dissemination is because through that effort, the Mercy Corps staff in the field became sort of like spokespersons for the research and they were more bought into it and as a result were more likely to use it. Another best practice is doing sense-making workshops similar to what Andy described, but doing that internally before the research is completed as well. Some challenges that we've seen include staff turnover. I think this is a problem for multiple projects and there's not much we can do there except to sort of incorporate that in our planning. And also we often assume that there will be a program, a follow-on program that results will feed directly into and that's not always the case. So sort of adapting to that situation. On external dissemination, this has been a really big focus of our efforts and it's really important for the outcomes of the research process. So we've invested a lot of resources in this. What happens within Mercy Corps is usually we come up with a dissemination plan and this is executed amongst multiple teams. And the objective of this is really first to circulate the results, which is very obvious, but then also to build credibility for the next iteration of the research because we really see this as an ongoing process and not a single report. So some of the more interesting aspects of dissemination for us is, for example, working with our media teams to write op-eds and publish pieces about the research to draw broader audiences into our research findings. And for example, with this study, we published an op-ed in the Monkey Cage, the Washington Post Monkey Cage, to get people sort of interested in the findings. One more slide. This is the final one. Some takeaways. Going back to this idea of evidence plus relationships, it was really important that this was part of a research agenda, that there were some precooked relationships that already knew about the research that had been done previously. And if you're not in that position, you need to understand how you're gonna overcome that, sort of what kind of processes can you plug into if you're not in that situation of this being sort of part of an ongoing process? I already talked about the sense-making and really, we all know about this challenge of balancing like clarity, dumbing something down and keeping the nuance. And I really think that this, we found kind of a third way in terms of engaging decision makers with early stages of the research and letting them work through some of the nuance issues. Maybe worrying a little bit more about that and a little bit less about this packaging issue, one page research briefs, all that kind of stuff. That is sort of one of the things we took away. We tend to forget sometimes Mercy Corps is a global organization with a lot of decision makers. So put some effort there as well. If you wanna move the lever, that might be a place to move the lever, not just within Mercy Corps, but the next time Mercy Corps does a proposal and cites that evidence, it gets fed back to the funders and the decision makers as well. And this last point on formal dissemination. I mean, monkey cage blog may or may not kind of impact a decision maker, but it does build up the credibility again for this iterated process of working with decision makers to try to sort of keep evidence in the bloodstream of these folks. And the last thing I'd say is this is really important I think right now. We heard about SDG-16. You might know about the Global Fragility and Violence Reduction Act or the Stabilization Assistance Review. These processes need this evidence right now. So we've gotta figure this out about getting our good evidence into the bloodstream of these decision makers. Thanks. So thank you so much. Please join me once again in thanking our panelists for a very filling conversation. I personally can say that, wonderful way to start the day thinking about what is our cause good enough that we don't need strong ethical data. Is it strong enough to live on its own? Should we be pushing for that impact evaluation every time and seeing where we fall instead of just disregarding it? And I think this question of could policy relevant research isn't just about the evidence. We've seen that more now than I think at any point in our lives and thinking about that just having good data doesn't always convince people of what they need to do. So how do we build those relationships to get our message across? So I'm gonna hand this over now to all of you. We have 25 minutes for Q&A. We have two mic runners on the side. So if you'd have something you'd like to address to our panelists during this session, please just raise your hand and we'd ask you to give your name, affiliation, and then a brief comment so that we can get as many questions in as possible. Laura, it looks like we have one right next to you there so we'll start there. Morning, I'm Bob Berg, the Alliance for Peacebuilding. Wonderful presentations, but I'm gonna ask a question of David because he came furthest. And it is this. The intent by the High Level Commission that set up that proposed the Sustainable Development Goals was that we institutionalized through civil society and government peacebuilding throughout the world. Each country has its own art and its own approach. And that didn't get into the indicators. So is there a rationale that you can see, David, to kind of help return the intent of SDG 16 by ourselves coming up with data point? Can you challenge the master, in other words, who says this is what data we're looking for, but you as an outsider say, well, really what you mean is this, not that? I think we're gonna take three questions and then we'll give the panelists a chance to answer. Laura, so we have another question down here. Apparently we're all, I guess you're right-minded to me, left-minded to yourselves. Chris Ryder from the Tony Blair Institute. And I think my question, John, is for you. Very interested to hear what you had to say about the fact that so much of your work has impacts that are unseeable. And you then said, I think, that focused you on looking at your theories of change to reduce the evidence that you needed. I just wondered if you could expand on that. Are there any lessons that you've learned about how we rethink how we understand the impact that we cannot see? Because my guess is that if we're looking at long-term behavioral change, that's only sustainable through a ripple effect. And the question really is how we understand that through a better understanding of the underpinning theory of change. Thank you very much. Thank you. I think up at the top there. My name's Frank Fredericks from World Faith and my question's actually open to any of you who would like to engage in this. Many other fields reach a point where they implement a collective impact assessment approach where different organizations doing different interventions may actually share some set metrics that they will track across the different organizations and then share those findings, which doesn't really seem to be happening in our field. So whether it's the lack of generalizability that somebody mentioned earlier or the silo effect of just being how geographically and theory of change-orientedly diverse we are, why is that not happening in our field? Would be my question if anyone has a hypothesis for that. Thank you. Thank you so much. So Dave? Okay. So in answer to the question for SDG 16, the evolution of it, as you rightly identified, it lost that aspect of it. My sense through the work that we've been doing in the Pacific is that by necessity that will evolve back into it. Because if you have a look at, and we're sort of working in the Pacific, one because we're in the Pacific, it saves on travel costs. But also because we see it as a microcosm of the challenges that face a broad number of countries in the world in addressing not just SDG 16, but the SDGs in general. Now the Pacific countries, Pacific Island nations have some unique challenges. But primarily the capacity to actually measure and report on SDG 16 is very, very limited and the challenges are very broad. So the only way that they can do it is to tailor it to themselves, is to say, well, here's this global agenda. Some of these things are really relevant to us. Other things are not so relevant to us. So we need to prioritize standardization of collection of data on these three or four indicators. And that's what we're gonna invest in because without that you end up facing a challenge that is insurmountable and it defeats the whole purpose. So I think and the view towards SDG 16 within those island nations is exactly that. They're trying to as a collective identify a subset and a quite small subset of SDG 16 to say, well, we're gonna focus on this. So I think as a sort of trend going forward, I don't see how countries can report on SDG 16 in its entirety without doing that. Thank you. John? Thanks, yeah, I'll try to respond in two ways. One, I'll speak to the theory part second, but the first part I think in terms of, where we're not seeing change, but it may be there but unobserved. I think part of it goes down to limitations in how we're looking at some of these issues. So any Afghan expert is gonna probably tell us that we're not doing justice to how young people in Kandahar interact with that body. So even as sort of context specific as we think our measures are, we're gonna be missing something. So I think there's probably more that can be done on that side. And then I guess thinking about more of the theory, what we keep coming back to is seeing that more often than not single interventions, whether it's an economic intervention or a secondary education or civic engagement, voice, none of those things in and of themselves really change attitudes and behaviors. At least young people where we're working towards political violence. But the theory is pointing to a little bit more towards, oftentimes there's a collection of things that we do, almost kind of a graduation approach to stability that irregardless of what those components are tends to have a better success. And so I think what it's pointing to for us is a little bit more of what signals to a young person that they're being invested in in some way that isn't just piecemeal. And I think we saw a little bit of that in Afghanistan, a little bit of it in Somalia. So I think that combined with better measures of what does it mean to a young person, what difference would we expect to see in terms of their attitudes and behaviors may get it a little bit of that. And then anyone on questions of collective impact assessments? Yeah, let me take the first swing at that. And I'm gonna do a little shameless self-promotion. I recently wrote a report on that that ended up being a USIP special report and I'll call out my co-author Rubin up in the balcony. And I mean, it's a great question we really dug in. We tried to do something similar in the Central African Republic with several organizations and really learned some good lessons. And there's a lot I could say on this, but I think the one thing I would highlight is that there's a kind of a unit of analysis problem with peace building and collective impact. The best collective impact programs we saw focus on things like obesity in Northern Minneapolis. And then you think about peace building in car and you put those next to each other and you can see where the problem might start to emerge. So we tend to think in terms of countries in peace building and we tend to think in terms of peace building instead of something very specific like rule of law or violence prevention or reducing the homicide rate. And I think we're starting to get to some of those things and starting to focus in on some of those things. But I think both of those sort of focusing down processes need to happen. If we're really gonna start to figure out, okay, what impact did these six organizations have in this place on this specific problem? And I will say we do have a presentation on that very case study in our next session on joint action and collective impact that Rubin will be presenting if of interest. Anyone else have any comments on collective impact before we move forward? I'll just add points to the last two questions. First starting with the collective impact is I wouldn't underestimate the amount of negotiations and infighting and collaboration that went into some of those collective indicators used in other sectors. That it took a long time and a lot of arguments for Wash to say incidents of diarrhea is what we're gonna use as our measurement of success. That wasn't simple and people had to yield the specific indicators that their organizations or their programs thought were important to get to that. Same with education and there's still a lot of pushback from different programs and saying what is the right indicator here that we can all use and agree on? That wasn't simple and it's gonna be even more complicated in this field. I would also go back to the question that John answered and just from a bit more of a technical perspective in that as we use more and more rigorous tools, more precise instruments for measuring impact, it is common that we find smaller and smaller impacts and that doesn't mean that those programs are having less of an impact than those ones that were measured using ethnographic or qualitative approaches but it is a trend that we see. The closer you scrutinize it, the less likely you are to see enormous effects. That's a very good point. Do we have any other questions? Trying to give a little priority over to this side because we haven't done any running. I think we have one right down in front here. Ms. Rukai Ibrahim Saliu from the Development Initiative of West Africa. My question is actually to Keith. You mentioned about biases, how people lie in responses and you said the critical factor is a change in response from time to time. I want to know what threshold, what are the determining factors of the threshold at which the result begins not to be trustworthy? Like what are some of the factors you need to consider in determining delta and the trust of the result you get? Thank you. Laura, right next to you there's a question. Audrey from New York University. I think the idea of a code of ethics for data is a great idea. What do you think it would take to make that happen? I'm sorry that I laughed in the course of the Alliance for Peace Building, Peace Building Evaluation Consortium has been considered even just evaluation standards for peace building for a long before even my time and I know that that has been a very tricky pathway. So I'm interested to hear what David and maybe some of the others have to say. I think we have one other question up here, Laura. Yeah. Hi everyone, I'm Seraf from the American Friends Service Committee and my question was to Beza and Andrew. I was wondering, I was really interested in the sense making phase and I was wondering if there were any efforts to speak to the people who provided the data during the sense making? Great. Not to be confused with the presentation later on the sense making technique from Catholic Relief Services. I'll hand it over to the panel. I think Keith we were asking about biases and changes in delta. Right. So there are two issues there. One, I think is we're always worried about data quality. That's always a concern. And that starts with even just managing the survey in the enumerators that go out and ask these questions. Were they trained well? One of the challenges that we had with the study in Afghanistan with John was that the training for the enumerators in round one was a little bit different than the training for the enumerators in that second round and the way that we defined things like employment, whether you were employed or unemployed or a student, if you're a homemaker, does that make you employed or unemployed? Those little pieces can really start to mess with the data and the reliability of that data. And so the first pieces is we have to know that we're actually comparing apples to apples when we're looking at that delta. That's the first challenge. The second is understanding the bias. There was a study that came out, I think a year, year and a half ago in West Africa that showed if a white researcher was present in the village while the enumeration was happening, that it significantly affected the way people responded. So literally just having your research assistant from the university on the ground, supervising the enumerators could change that delta that we're trying to capture. And so we need to be really aware of the quality and the consistency in the way that we measure that. And then when we talk about the actual, we have good data. We had consistent methods in collecting that data. We think the variable was well-defined. Then I do think if the sampling strategy was valid, that that delta is reliable. And I think going to the question about do we even need good data when the moral cause we think is justified? I think this is, my answer to that is that's now the best data that we have for making these decisions. It's not perfect, but it's the best input that we have for trying to make programmatic decisions. Thank you. We want to tackle the question of code of ethics. Sure. Well, I think there's two factors. I mean, if I think back on my education, so I've done a whole range of things. Did maths and then I did political science and international law and all this, a whole range of things. But if I think about the maths and statistics as part of my education, a lot of it focuses on the techniques, it focuses on just doing this stuff. And I think in the age, and for a long time that was probably appropriate, but in the age of where data is the king and there's this sort of faith in data, being able to answer all of our questions. Mathematicians and statisticians are now in a sort of a new position because they're used to being ignored largely and now suddenly they have this audience. And I think in some part it's going at the university level and educating on this new sort of responsibility that this type of work has. So that would be one thing I would say. The other side of it is you would need, I guess, buy-in from the non-technical people as well. And the bridge between technical and non-technical is part of the issue because of this sort of faith and data and data is gonna solve all our problems. Whenever a number gets thrown out, there is an inherent assumption from the non-technical side that it's gone through a vetting process and it's done by experts and it's done in some method that I couldn't possibly understand. And I think getting the non-technical people the confidence to be able to question a lot of this data is the other challenge. So there's a two-fold thing. There's the technical people have to realize that suddenly they have a moral responsibility for this stuff and the non-technical people have to have confidence in being able to question this stuff. And I think it's about openness and transparency now. How that happens, I do not know. Thank you. Bezin? Yeah, sure. We unfortunately did not do sense-making sessions with people that we collected the data from so not with the actual participants in the program. Unfortunately, even though I think that is where we should be thinking of moving towards what we should be thinking of moving towards, this new sense-making approach is new to us in the sense that we've only done it a couple of times and I think we're realizing the utility of it. And the closest we've gotten to the actual beneficiaries or participants is by doing it with the staff that implemented the program. So that helped us get a lot of feedback that was really useful and we adapted our results to it but not quite yet. We're not at that point of engaging with. Okay. Yeah, and the only thing I was gonna say is just that it speaks to this question of the complexity of this research because their respondents, you know, they gave their sort of responses and then we had not that much way to get back to them and would it even be smart of us, you know, to go back and go knocking on the door of, you know, in that environment, you know, that would have been a very tricky thing for a lot of reasons. And so as Beza said, just balancing getting it as far back, as far, you know, as close to the respondents as possible to try to close that loop as much as we can. Great. I think we have time for one more question. You raised quickly. So. Pass that down, Christian, sorry. I have a bright shirt on so you could see me. I'm Angie Yoder-Maina. I'm from Nairobi, Kenya with GreenString Network. We're doing a lot of questions and surveys on human subjects. We're looking at how violence impacts people and we're trying to figure out how important are IRBs and how do we access them from the South? It's really hard. And yet if we're gonna disseminate this, if we're ever gonna publish, we've gotta have it. And so how do we do it? I just wanna say I love that question as someone who trains on the IRBs and the importance of informed consent. I'm gonna pass it off to the, I think it's a fabulous question to end on. Anyone have any thoughts? I'm happy to add a few opinions as long as mine aren't the only ones out there. I think that this is a big question, especially for NGOs implementing around the world right now and not just for research. You know, the concept of informed consent and going through an IRB process to understand whether or not you are causing any sort of stress or harm to your subjects shouldn't be something we're only talking about when that survey is used in a research project. Because there are likely, I'm making this number up, but a hundred times more surveys that are done just for M&E purposes that are asking similar style questions that are never given the consideration of an IRB process. So I think this is a broader question. It doesn't just apply to research, it's all of M&E. The second, the concept of the IRB, I think goes back to where we were doing medical tests on people and there was a real harm. And I do think we need to, and universities, are updating the social science versions of IRBs across the board right now around the country and going, this shouldn't take months to do, we should be able to do this quickly because usually these don't have a real threat to the subjects. And I think the final point that I would add is that if you don't have to have access to Georgetown's IRB, in fact, there's a well-worn path on creating your own IRB. And it's really about getting outside eyes who don't have the same incentives as you do to look at this with a lens of going, are we protecting our beneficiaries and the population? And that's something that even in Nairobi, I know that there are IRBs in Nairobi, there are private sector IRBs here in the US that are much more timely and efficient in managing these checks and balances in research. Yeah, we'll just add that for this project that we undertook we had to get an IRB. And it's usually the case when we work with an academic institution, so in this case, the University of San Diego, but we've done it with other universities. And I think sometimes we see it as a burden. You know, it takes time. Sometimes you have to go through multiple drafts. But I mean, more broadly, I think we realize that it's really important because a lot of times the research that we're doing has some risk attached to it for the people who are involved. And it's really important that they are cognizant of those risks, whether it's survey work on political violence or whatever it is, the sensitivities that surround some of this research really make it important that I think we do our due diligence and apply for the IRB. I mean, I guess I'll speak because the university representative. The IRB, I mean, at universities, it's a pain. And it's not, it may not be, I mean, part of the reason for partnering with universities, we have that infrastructure in place, but it can also really be problematic. I mean, just recently I was closing an IRB in our IRB online system, and I closed it by clicking on a new submission button. And I mean, this is something about the bureaucracy and sort of horribleness of the technology that universities are using. So I really sort of endorse exploring private sector options. I endorse partnering with the university where it makes sense. I would say that look at the IRB requirements because a lot of times we think it's just, can harm be done to subjects, but there's also a requirement around, you wanna make generalizable research. And this is why evaluations normally don't go through an IRB process. Of course you wanna have those ethical conversations, but if you're not planning to publish generalizable research, then you do have some more flexibility around how you handle it. IRBs are being updated, but I guess I would also sort of call out to funders to make sure you're providing resources maybe to access these private sector options as part of the process, instead of just saying, we need an IRB and not digging into what that actually means. Great. Thank you all so much in the respect of timeliness and to give you all a brief break. Please join me in thanking our plenary session one more time. I'm really happy with the wonderful conversation we had in the great presentations, and I think that it's a great start for our day to think about research as a process, not just a final report, and we're here to explore the processes and methodologies on how we make this happen. So we're gonna go to a 15 minute break where we'll have coffee and tea and some of the breakfast in Leland Auditorium before we begin our first set of concurrent sessions. If you have any difficulties with the app or the schedule, please see myself or any of the volunteers around the room or at registration. Thank you all so much.