 I'm Peter Brogan, and I have the honor, privilege of introducing our panelists to discuss reducing violent extremism, what works. Our first panelist is going to be Dr. Jacob Shapiro, who's a professor at Princeton who's written multiple books about terrorism. He holds a PhD from Stanford. And then our second speaker will be Dr. Rebecca Wolf, who directs Mercy Corps Peace and Conflict Team, which supports programs around the world. He's taught at Princeton, at the Woodrow Wilson School, as a PhD from Harvard. And finally, Dr. Todd Helmuss, who's a senior behavioral scientist at the RAND Corporation. He's been an advisor for special operations in Afghanistan. He has also led studies on U.S. efforts to train the Afghan security forces. He served in Baghdad in 2008, and he has a PhD in clinical psychology. So it's hard to imagine a better group of people to examine this question. So we'll start with you, sir. So thank you, Peter, and thank you all for joining us. What I want to do is I want to talk about basically three things. The first is what we've learned over the last basically decade from studying U.S. and other countries' interventions in various countries around the world about what kinds of aid spending and service provision to communities actually bring down levels of violence and extremism. And then I want to talk about what that implies for who we should be thinking about targeting when we think about combating violent extremism or preventing violent extremism. And then I want to close by saying a couple things about what are some of the broader lessons maybe we've learned that could inform policy so that we can think about doing things better in the future. And so when we think about the impact of aid and service provision and the kinds of programming that are generally lumped under stabilization or countering violent extremism or preventing violent extremism, there are a few things that have been learned over the last decade about what works. And the first of those is that small is better, less is more, in the sense that when you look at either effects on violence or effects on public sentiment towards the organizations providing aid, large-scale programming, if anything, is neutral at best and probably in Afghanistan and Iraq looks like it exacerbates violence a little bit and doesn't seem to win over population attitudes. Small-scale programming in some settings seems to do both, particularly when there are sufficient security services available to take advantage of the information you might bring in. And I'll talk more about that in a second. So that's the first thing that's been learned is large-scale things don't seem to work. And you could imagine a lot of reasons for that. One is maybe people don't care, maybe there's credit-claiming, maybe operating environments where violent extremism is at a significant risk. You often are operating environments where there's a great deal of potential for corruption and malfeasance. And so more money comes in and whatever good that's doing in terms of gaining population sentiment or shifting people away from thinking about violent groups is counterbalanced by the fact that it comes with a whole bunch of public bats and a whole bunch of very obvious graft that gets attributed to the party that's bringing in the money. So lots of reasons. The second thing we've learned with respect to attitudes is that coming in and helping people out in a dire emergency seems to do tremendous things for both sentiment towards the party that's providing the aid and in terms of trust and belief in the faithfulness and intentions of people from that party. And so people have observed this in Russia with respect to the response to forest fighters in the United States with respect to the response of floods in Pakistan, India and Colombia with respect to the response to floods. And so this has, like, and other natural disasters, this has, like, very clear implications for the kinds of things and capacities that the U.S. government and others could be investing in in terms of being able to help partners respond and in terms of the investments that should be made ahead of disasters because in the countries where you have kind of radical outperformance in response to unfortunate and unexpected events, the thing that's always sitting in the background behind it is an extensive series of exercises and preparations ahead of time. And that's a thing that could be funded and clearly has some value in winning population sentiment that's not now part of what we think of CVE or PVE. Okay, so small help people recover from disasters. What does that tell you about who should be targeted? Well, the first thing, and I think the most important thing it tells you is that when you think about who should be targeted for CVE and PVE or for many violent extremism programs, it's not the people who might engage in violence, right? Because if what you're doing, if your theory of how you're going to reduce participation in violent extremism is you're going to give them something better to do, then the big projects that come in and put a lot of people to work and give them that like better thing to do, that should have an effect on rates of violence and it should have an observable effect on population attitudes. But it doesn't. And that also should imply that it doesn't matter how many kind of security services are present in an area. If you're doing the thing that's giving people something better to do, violence should come down. So the fact that you see small projects working and the fact that you see them working to bring down violence only where there's sufficient security provided, that should tell you that what's going on is not something about the people who participate in violence, but it's about everyone else in the community deciding when they see someone taking the initial actions that might lead them to act on extremist views, they say something, right? They call it out, they talk to the security services, and they are then able to operate on that information. And so an implication of that is you could think about where within countries or across countries these kinds of programming are more likely to work and you can also think about optimizing not to target the populations that might be extremist themselves who are extremely hard to reach and persuade and track and all those things, but the rest of the population around them who are much more stable, much more approachable, much more reachable, and also much more amenable to being surveyed and measured and discussed, have things discussed with them beforehand so you can understand how to optimize for their needs. So that's kind of what we've learned I think about what works and what that implies about targeting. And the last thing I want to talk about is what does this imply for the systems and mechanisms that you should have in place to think about effective CVE. And I think the first and most important thing is there's tremendous value to basic research in this area because there are a lot of things that people thought were true in 2005, 6, 7, 8, 9, and asserted with tremendous confidence and programmed billions of dollars against that were totally wrong or at least totally wrong in the places where the money was being spent. So the most obvious example is the link between poverty and extremist sentiment. If you go back, and I've been doing research on this now for almost a decade, but if you go back and look at national security strategies from 2007 to kind of eight time frame, if you look at the USA documents that are justifying different programs at that time, if you look at how DFID, the UK's equivalent to aid was justifying programs, there was this like relatively simplistic view that if you improve people's economic circumstances they will have more moderate political views. And it turns out that all across the world there's tremendous heterogeneity between the link between people's economic circumstances and their political views. And in most of the places where the US government and other Western governments were interested in combating violent extremism, the poor were among the least supportive of violent groups of any demographic group. And so there was no kind of evidence that that move, even if you could successfully with programming move people up the income ladder, was going to get you the political changes you wanted. Now that's something that over the last nine years, research has shown in country after country to the point where no one would now say we should do this income generating program because it's going to shift political attitudes. So that's a huge win. But it's also a strong piece of evidence that you need continued research in this area, you need that R&D to feed policy because we thought we knew something and we were wrong. We spent billions of dollars around that wrong belief. So that's the first thing we've learned. The second thing we've learned is that implementation is tremendously important and often well designed, well intentioned, well thought out projects that look great on paper and have a coherent theory of change and have identified a target population have turned out over the last decade to be a complete disaster when implemented, right? Things fall apart because you're trying to operate in very hard places and because the incentives aren't aligned. And what that means that the incentives aren't aligned between the implementers and the people funding the programs, the donors, is that you need a hefty dose of supervision to make sure that the government's footing the bill, get the programming that they think they're paying for on the ground. They get the thing that they want implemented. But in the US and the UK and Germany and Japan and other countries, there is no reserve capacity in terms of manpower and contract management within the aid agencies. And so if you believe that as a policy matter, violent extremism is a dynamic problem that's going to move around and the intensity of programming across countries is going to need to shift up and down, then you need some reserve capacity because you need to, on short notice, be able to send additional people just to do the grunt, boring work of managing implementation in different places and we're not manned and staffed to do that. So if you believe CVE can work and is a policy tool that should be implemented, you need to kind of walk back a little bit towards some of these kind of boring bureaucratic issues and create that reserve capacity in your aid agencies so you can actually systematically execute the programming well on the ground. Because no matter how much there might be outlier NGOs out there, like Mercy Corps, which Rebecca represents, the majority of aid implementation is not done by NGOs that have public service as a core part of their mission and as a core part of how their personnel are incentivized, promoted and supported. So you need that supervision. But to get that supervision in these contexts in a dynamic responsive way, you need reserve capacity and we don't have it now in our organizations. That was excellent, thank you. Dr. Wolf. Okay, I don't see the clicker, but I think it's going to be easier if I'm up here anyway. Okay. So thank you, yes, for inviting me today. Let's see how this works. Does anybody know where the clicker is? Oh, it's behind, it's right here. Okay, so I have to say I wish Jake was a bit more correct about the fact that people don't program around these wrong beliefs. I still hear it all the time that we should be reducing poverty to reduce violent extremism. But I'll talk to you most of what I'm going to talk about today in many ways gives field examples of a lot of the findings that Jake just talked about. And so Mercy Corps has been looking at this issue for almost about six, seven years now. And it came to us was basically what we saw was this difference between the way we were being asked to program and what both academic research was telling us as well as some of our own research. And so we started, we have done in a sense of first stream of studies and I'm going to talk about these from the 2010 to 2015 as a cohort of studies and then I'm going to go into some others a bit more deeply. And so with that, what we started to do was look at the data and what we noticed for me has been the most important lesson in all this research is that we were making assumptions about why young people were behaving a certain way by talking to key informants and youth not involved in violence. And they were using these conventional wisdoms around poverty and engagement and violence and that's how they were describing people's behavior. When we actually started talking to people involved in violence, our answers were much more diverse, much more nuanced. And while it's not that economics were never part of the equation, they were much less likely to be represented in people's answers than issues of grievance, injustice, protection, exposure to violence. Those were much more powerful than these economic issues. And so I'm a psychologist by training and how I really started to think about this is that we came to this question from much more of an economics perspective of people making this decision from a cost-benefit analysis. And if the benefits outweigh the cost, that's why someone would engage in violence. But we know from years of psychology and behavioral economics research, that's actually not how people behave. They actually, it's much more an emotional response and then you have a post-talk rationalization. And so that's sort of how we are starting to think about this work and trying to understand how to design programming to help people, to prevent people from engaging in violence. And so we did this first set of work basically to understand the drivers. And in 2015 was, in a sense, our culmination study, Youth and Consequences, that used both some of our quantitative data and qualitative data to basically really show this story about injustice being the key driver of participation in violence. We got to 2015 and a lot of people understandably wanted to know, so we now know what drives people to violence, what works to preventing them from engaging in violence. And as Mercy Corps, we took this as a challenge. And so what I'm going to talk about now are some of the insights we've gotten for how we might be able to prevent people from engaging in violent extremism. So the first study I'm going to briefly talk about is we did a qualitative study in Jordan on foreign fighters. And we typically don't work with this group of people, but there was a lot of interest in trying to prevent foreign fighters in Jordan, in Tunisia, and other places from going into Syria. I'll say for me the most surprising finding was that, again, we got the conventional wisdom from everyone, people are getting paid to go fight. We found no evidence of people getting paid to go fight. It was the recruiters who were being paid for recruiting certain types of youth. And particularly, we often talk about the isolated youth being the most vulnerable. It was actually youth who were more well connected were targeted to be recruited because they were a higher value and could bring more people with them. And so again, another conventional wisdom that we always think is just a pure good is having a social network. A social network can be both a positive and a negative. So it depends on who you're connected to. It's not just having connections. What we found was in a sense of factor that helped bring young people, and I'll say in this study, it's all young men. Young men back was the role of mothers. And so as I said, the recruits weren't paid to go. It actually often was an economic sacrifice for people to go because they were leaving their jobs. But then once it became too much for the family to handle, that economic sacrifice, the mothers would call their sons back. The other piece we found is that if a mother's health had deteriorated, they would pull their sons back. And so we recognized the important role of family in talking to their children about violent extremism. We've also done a couple qualitative studies on Boko Haram. And so here, again, we wanted to push our findings, not just understanding why young people joined these groups, but most young people don't join violent extremist groups. So can we understand some of those resistance factors? What I would say here, and this underscores what Jake was saying, the importance of that community acceptance. So whether or not the community was OK with Boko Haram being involved, being in that community was a big predictor of whether or not they were able to recruit someone. And so definitely it is much more not just a targeting young people, but it's also about how do you build community resilience or a community cohesion so those groups don't come in. I don't know whose phone this is. Sorry, my Peter, is that yours? OK. So to me, that was an important finding. And so how do you, in a sense, create ways for communities to, in a sense, not be OK with those groups infiltrating? What we also found in the study, and I'm going to, for time's sake, just focus on why they resisted, there were two key factors for me. And again, families were very important. So young people would say their parents talked to them about the risks of Boko Haram, and that's why they didn't join. The other piece for me, which I find fascinating here, is the role of counter narratives. And I know that's a popular topic in these circles. It's often talked about in terms of social media. So in this area of Northeast Nigeria, social media is in powerful. What we found, it was local leaders and particularly religious leaders who were said to have talked to young people about the corruption of Boko Haram and how they were, in this case, just as corrupted as the Nigerian government. Just a side note, I forgot to mention about the Jordan study, often what also pulled people back is that once they got to Syria, they realized there were no good or bad guys. Everyone was bad. And so that feeling of going to be a hero, they realized wasn't going to happen. And so they came back. So that was important in terms of counter narratives, having these leaders talk about it. What we also, it's more of a speculation, is that Boko Haram doesn't have a cohesive narrative as a group, because there isn't that external threat. So it has been much easier to come up with a counter narrative for Boko Haram as compared to some of these other groups that where there is an external threat. So I think as we think of counter narrative strategies, we need to think about what the strength of the group's narrative to begin with. And it might be a much more uphill battle to create a counter narrative strategy when there is that external threat. So I'm going to do two more studies. So here is actually, it was Jake's students who helped us with this study. And Beza from my team is here in the room who also worked on the study. So here was, it was a USAID funded project. And we had, in a sense, a natural experiment of basically, we had, we were collecting public opinion data on service provision in Iraq, both. And we started the data collection before Maliki resigned, and we finished the data collection after Maliki resigned. And what we saw was a significant change in perceptions of service provision. And so if you see the lines going up, those are all representing different types of service provision. What happened in parallel to that was that people's support for armed opposition groups went down. And so what was particularly unusual for me in this finding is that this data is only from Sunni. So even though Ashia remained in power, perceptions of service provision of being more inclusive increased. And so what it says to me is that while sectarian differences in Iraq matter, they don't matter as much as we think they do. And that there are ways of getting over those sectarian divides. What I would also say is that the importance of perception, service provision perceptions will help you to a certain amount of time. At some point, service provisions do have to, provision has to improve. There are ways of making that timeline longer. So I was over at the World Bank earlier this week and they have some data around grievance mechanisms that as long as there's a participatory aspect to it, even though service provision isn't increasing to the rate people would like, they give a longer in a sense, benefit the doubt to the government. So now we're thinking about how do we increase those grievance mechanisms in our governance program. And I'm gonna end with a recent study we just put out. It's many of you probably have seen talks over the last couple of weeks. And again, Beza on our team led the study. And for me, this has been just an important piece of work because it has shown how we've used data to improve program implementation. So this was an impact evaluation of a USAID program, a secondary education program. This data is from Somaliland. We're also doing it put lands in South Central. On that first slide I showed, there was a data point about Somalia. And in that study, we had a surprising correlation between young people who were more civically engaged were also more likely to say they would support political violence. And as that conventional wisdom, if you increase youth civic engagement, if you increase youth voice, you'll reduce violence and we're seeing the opposite. And so, but what we hypothesized in our qualitative information backed up was it was because there was no avenue for nonviolent change in Somalia. And so it's one thing to increase voice, but you actually have to have opportunity to use that voice, otherwise you're going to become frustrated. And so we use that data to change our implementation of this program to include in the civic engagement component aspects of peace building, nonviolent change and having a much more explicit community service project. So young people could be out in the communities trying to change things and seeing that they could actually affect change. What we found then at the end line, so we just did the impact evaluation and that young people who were both in the education and civic education components of the program, they had to be in both components were again less likely to use violence compared to people not in the program at all, but also in that support of violence, we saw a reduction in support again compared to people not in the program at all. What I think is going on here in particular with that community service projects is that young people are able to internalize the training they're getting, they're getting an opportunity to in a sense practice the behaviors and that becomes part of their identity. Chris Blatman, James Jameson and Margaret Sheridan did a study in Liberia and they talk about how it was a combination of cognitive behavioral therapy and cash transfers. And while the cash didn't have long lasting effects as an economic, what they outcome, what they think that cash does is gave young people an opportunity to not go back to their previous crime behaviors. And so they were able to act those behaviors, have them become habits, and then a year later those habits still stuck. And I think community service is another way of in a sense enacting those behaviors. So what's next? I'll end with more of the, so with the SOA with both Jordan and Nigeria we're actually taking that research and implementing programs that combine that in a sense how to reduce community acceptance for these groups. So how do you in a sense on a small scale build local, increase local governance and community relationships so that there is that increase in service provision at a local level. We are also looking at the roles of families and helping families talk to their kids about these groups. With the Nigeria research, we're in a pilot phase right now and the goal is to do an RCT on the next phase. So hopefully we will have impact evaluation of that programming. What I'm also working on is a synthesis piece. And what I started with here is in a sense what is the behavioral economics or the social psychology. So for me it's not so much what is the program but what is it that you're trying to affect change and there may be various ways of affecting that change. So I just talked about community service in a sense was a way of helping people shape their identity in a different way. It might not be civic engagement that does that. You could potentially have a livelihoods program that does that but you really need to understand that mechanism that you can go about it at different ways. And so that is in a sense our next phase of thinking and so hopefully in the next month, the next couple of months we'll be able to share that with you. So thank you. Thank you for that, excellent. Thank you and Dr. Helmets. All right, my mic is on. So my name's Todd Helmets from the Rand Corporation and I'll talk to you about this from a slightly different perspective, maybe a bit more domestic perspective. We are about to come out with a report that is essentially will serve as a toolkit to help local CVE providers domestic here in the United States and Europe conduct their own assessments. And to be wise about how they construct their programs to help them think about what measures they can use to assess their impact and to provide them some basics on data analysis so they can think about how to improve existing programs and how to report their outcomes to the extent they're effective or not effective. Obviously, what I'll talk to you about, so back with that in mind, I mean I'll talk a little bit about why this is important, why the science on this is very hard to do and partly why you see so little science on this topic. The exceptions here certainly because it's sort of the best that there is right here and then also talk a little bit about what we see in the research and it's sparse but talk a little bit about their findings and then some next steps. So obviously this is important. The reason that this is even an event per se is because the US government and a number of other governments and organizations are spending a lot of money to address extremism and there's certainly a threat that comes from extremism both currently from ISIS as well as many other extremist groups and so obviously it's a national priority to mitigate that threat. So as Congress and others are looking at different programs they're increasingly asking what works and it's a question that they're interrogating various programs that the agencies are having to prove that what they're doing works. The agencies are in turn requiring those that they're funding to prove whether or not what they're doing works and so there's really a strong impetus on this and for obviously several important reasons and it's not just to know whether or not they should fund you or not fund you. Though that certainly the prerogative of Congress is whether or not they want to fund what works but there's a lot of other reasons. Number one obviously you don't want to waste time and resources and missed opportunities by doing something that doesn't work and also more critically from a program perspective a program cannot improve their programming unless they have some sense of the impact of that programming and you can evaluate your programming in different ways from process measures to outcome measures but ultimately unless you're able to connect what you're doing to certain types of outcomes it's impossible to change course and so what you end up having is sort of set piece programs that sort of run through their cycles, report their results and then try to get funding again. So obviously those are all the reasons and it's interesting too because I come at this from my former career was in health research I'm a clinical psychologist by training I did drug abuse research in graduate school so it's very interesting coming from a health research background to this world where health research obviously you don't go to the doctor and get a medication unless that medication has gone through exorbitant amounts of testing. You don't get a certain treatment unless that treatment has gone through exorbitant amounts of testing and now all of a sudden the international programming environment has just sort of figured out like oh gosh you know what we should probably prove that what we're doing works and so and they've been doing it in ways that are kind of clumsy and science that's not fantastic so anyway I always laugh think about this coming from a health research background and the other lessons you learn from health research background is how different research is funded and I'll come back to that and why that's important so all these things are really important the science is obviously really hard to do and you're getting sort of the results and the outcomes of the research that my colleagues up here are talking about but they're not talking about what a pain in the rear is to do this type of work and I'm not an economist so I can't do the magic wizardry that Jake can with data that's really complex and really large I can't do that I think much more smaller in scale but nonetheless they're all research reasons number one where you're talking about measuring and evaluating a very rare outcome now it's much less rare where these people do their work and on the domestic side it's much more rare so the notion of doing a study and evaluating whether in the United States whether or not a CV program results in less terrorists that's you're never gonna see that because there's just almost no terrorists in the United States so that's a challenge we really don't know that much about the phenomenology of terrorism so some great studies about some civil society factors that lead to violence but why an individual and Minneapolis or Paris radicalizes we know so little about that and if you know almost nothing about that process it's impossible to develop a tailored treatment program to address it so even if you don't have your outcome measures the theory that goes into developing your treatment paradigms are very limited and then you get into a whole host of challenges and then there's heterogeneity so we've done work on foreign terrorist we've done work comparing I've done work comparing motivations of foreign terrorists and local terrorists well they're very different motivations so a terrorist is not a terrorist is not a terrorist someone might radicalize in Pakistan it's probably very different than the reason they radicalize in Somalia and it's extremely different than the reason they radicalize in Minneapolis so being able to apply research just because you get one study outcome doesn't mean that that study outcome applies to everybody and that you'll never ever have to repeat that study again because as we know in social science the science itself is very fluid and the outcomes change from time to time to time even some of the best known psychology studies have been tried to replicate them they couldn't replicate the studies that I guarantee most of you read about in college graduates in college so and those are very basic things much less talking about trying to replicate why someone becomes a terrorist in Punjab so all those are factors now it gets worse when you talk about so it's great because here you have a lot of talent that's being put on the problem local CV providers don't have the talent they don't have the money they don't know science is important all they're trying to do is implement the best program that they know how to do and God bless them for it but without the expertise it's hard to construct an implementation program that can be evaluated and without the expertise it's hard to conduct the evaluation and without the money you can't dedicate any you can't dedicate any resources to the topic so all those are sort of critical factors that are sort of limiting why the science is so limited here and I'll just say from the domestic side our literature showed that I don't know how many studies have you done on international conflict outcome kind of stuff oh gosh probably 20 yeah if you look at the domestic literature there's four and one of those is not an outcome study the other one uses a metrics that wouldn't pass mustard at any psychology journal and two others are pretty good are okay none of them have randomized control trials so you just have lots of limitations in the science that exists so there's almost really no evidence of what works domestically from a CVE perspective and the implications of that are that it's almost impossible to develop programs from an empirical perspective you certainly can't improve your programs and policymakers can't really make very many decisions about what works even if you count all the rest of the studies it's hard to apply them to US or European context so what are my very simple suggestions and these aren't certainly sort of ran quality and thought through but I will say that one there has to be some sort of funding stream for this post hoc studies are great but they're inadequate at the end of the day especially for program evaluation the stuff needs to be research needs to be baked into the program at the outset so you have your baseline measures and your post treatment measures and ideally you have a control group better yet you randomize your control groups so all those are better when I hear you use the acronym RCT for randomized control trial my heart flutters because that's useful wow someone's doing a randomized control trial on this stuff and part of it's the funding organization too so my old world of drug abuse research there was a national institute on drug abuse a central organization that all they did was think about the research on drug abuse and what works and what doesn't work and disseminating information not disseminating information but importantly funding studies reviewing the scientific merits of studies to determine if they will fund them or not and so what you have is you have a whole lot of literature science, solid science on a topic so that does not exist on domestic international security issues I'm at Rand so much of the money I try and get funding from are sort of individual offices but the guys who leave the offices change month to month and it's just one office out of a thousand here in Washington so there's really no central strategy for doing the funding stuff and the third thing is that I think over time this is changing NIJ, DHS and others are increasingly funding outcome evaluations so to their credit, they're sort of on this I also think that eventually it'll be important to have independent evaluations of programs we're probably not there yet I think it's more important to get science out there I think if you put too high a bar on the science and the science you need you're not going to get the studies that can be informative eventually I think we have to get there where outcomes are published in randomized and outcomes are published in peer-reviewed journals where separate organizations conduct evaluations of programs so I think that's a direction I think ultimately we need to go so those are just my initial comments another excellent presentation, thank you all of you haven't you raised Dr. Helm's principle problem at least in the domestic sphere which is, let's say we produce this agency called the Institute for the Study of the Effectiveness of Countering Violent Extreme Programs or whatever it is we're going to call it that they're just not enough terrorists or potential terrorists in the country to make any kind of universal law about them so it would be the point of that because of that so my viewpoint is, that's true that's a great observation you made it, I'm just joking I know it was a brilliant observation so my belief is that it's important to I like the idea of letting 1,000 flowers bloom on this so we don't number one, you can't test an outcome then you need to have some sort of theory of change and invariably most programs have intermediate outcomes they're trying to address alienation or they're trying to increase empower local religious leaders to more effectively conduct interventions in their parishes or they're trying to help parents do better at identifying extremism within their children so obviously you can't well you can't actually have an outcome that says less terrorists but you can have an evaluation that says if your goal is to train local religious leaders to monitor their parish and conduct interventions you can do that and you can have those studies that show that you can do that effectively now making that next step to say that that's not gonna reduce terrorism I don't think they can limit inherently limit you from doing the work think ultimately all these programs have intermediary objectives built on some theory of change and they can evaluate those and we're hoping that our toolkit will help them think through what those objectives are so that they make rational sense so that's my answer Dr. Wolf, I mean I'm impressed by the fact that Mercy Corps doing this, are you sort of like an internal think tank in a sense and how many other big aid organizations have this? So we have both what's called the technical unit and a research unit and we collaborate on this work and so it's not quite an internal think tank but Mercy Corps probably in 2011, 2012 decided to invest specifically in research in addition to our programming so that would both better inform our programming but also recognizing for scalable change we needed the evidence to inform policy so that was our thinking behind that How many firefighters in Jordan did you interview? That one was low I will say it was about probably 20 it was very hard to identify them because the government officially arrests anyone coming back and so we knew that they were in a sense, the government had to know they were there we were actually somewhat surprised we were able to interview Boko Haram was a bit higher 47 and what we do is what's called the snowballing effect someone will tell us about someone else so we recognize that there's a small group of people yet we're one of the few agencies that are really being specific at targeting to understand that I will interject though that that observation has been replicated elsewhere so Kim Craig did a study in Palestinian areas showing the importance of families and preventing people from extremism so I think the key is trying to find the cost linkages and not just looking for one outcome but multiple Technically it was supposed to ask for your permission of your family to go on a jihad, right? at an absolute minimum I think it again depends where in both I would say northeast Nigeria that doesn't seem to be as much of a calling but we do see is actually radicalization happens after joining it's rarely before and Dr Shapiro you were saying when we were talking earlier I mean that there has been how would you assess the after 9-11 there was like the theories that you kind of demolished earlier I mean how do you see the overall shift in kind of our understanding of this question about who becomes an extremist how would you assess that overall in the last decade and a half? I mean so I think Peter I think it's focused more and more on individual specific circumstances and on fairly localized processes so if you look for example at where people have been recruited from to go fight in Iraq and Syria during the war against the US and the Iraqi government and then more recently against the Assad regime any time you get fairly detailed data on the set of recruits it shows tremendous concentration within particular cities and even within particular neighborhoods in different countries so consistent with what Rebecca and her team found in Jordan for example most of the Libyans who went to fight for al-Qaeda in Iraq during the first kind of part of the Civil War in Iraq came not just from one city but from one neighborhood of that city where there were a set of recruiters that had built deep networks and it became kind of a socially acceptable thing to send people so as you start to think about the evidence coming in that these things are kind of increasingly very localized and social and not because of these broad generic factors I think that tells you two things that are important for policy one is you're probably not spending money efficiently if you're trying to kind of reach out and target populations broadly but the other is that there's tremendous value if you don't know where those pockets are in doing things that will make the broad population within those pockets feel well about the government because how else otherwise you're gonna learn about those pockets and so what that means is you can think about things like refugee policy for example as serving a very clear CVE purpose because if the population that has in your country a large refugee population elsewhere sees the country being generous towards them and trying to help the people elsewhere or not or even there you're gonna have a better ability to talk to that population have government officials talk to that population identify the pockets within it where radicalization is a problem and therefore target programming so there are these links between what we do abroad and what we want to do here that I think a lot of this research points to that have been under represented in policy we actually just published recently published a paper called all G Add is local based on ISIS personnel records and we found that overwhelmingly whether pick your country in the Middle East it was a particular province or a particular area that produced the vast majority of people who went so that typically was an area that the government kind of ignored or let's open it up to questions you have a question please identify yourself and wait for the microphone because we're gonna start in the back here with Andy Marshall. Thank you. I'm work with the one campaign so we approach this issue primarily I would say from a development standpoint but be interested to hear you talk about how the development and security communities have learned from each other some of the things that have been discarded along the way and some of the best practice that's emerged from that conversation. So I think it is a continual conversation I think it's starting to move a bit more now than it has in the past I think what we recognize is as Jake said you need a modicum of security to do a lot of this work and so it's not that it's an either or but how do you in a sense either be sequential or how do you layer those approaches with the understanding as an NGO working in the development space that our trust with communities is often because we don't come with security or military objectives. So while we all recognize that ultimately security stability is of the forefront because that does increase development in countries but also increases our security globally we do need to keep that separation. So my sense is which is a little bit more institutional is the learning between the security and development communities at least within the U.S. government has been tremendous downrange in combat zones where they're working next to each other but it has not been institutionalized and there's no mechanism to maintain those connections. So there's no sense in which the USAID FSO who served alongside army soldiers in Afghanistan for a year is repeatedly for the remainder of their career having opportunities to interact with them either at NTC or at the service colleges and vice versa. So there's a tremendous amount of sharing but a lot of that is being lost over time because there's no institution to make it happen. And that goes back to the point about no reserve capacity, no training capacity in the development community within the U.S. and the UK. This lady here and then my crew, we all look cool. Hi, thank you all for being here today. Taylor Winckelman, I'm a science fellow in Senator Markey's office. This is in response to Dr. Helmuss but I would appreciate if everybody could maybe address it. You say that a terrorist is not a terrorist is not a terrorist and coming from a fellow clinician's perspective, a cancer is not a cancer is not a cancer is not a cancer. However, we do have several different kinds of treatment protocols based on the hallmarks of each cancer that we find and we also use an iterative process to treat cancer. If what we're doing isn't working, we switch midstream and try something else that has possibly worked on another cancer in the past that may or may not work here. And we keep going until person dies or cancer dies. Not that that's necessarily a great analogy, I'm sorry. But could you possibly comment on the similarity here between cancer treatment and terrorism? I didn't anticipate that question. I mean, it's a great analogy. I think you're right. Everyone could possibly argue that terrorists may have more in common with each other than or at least certainly have some things in common with them and the fact that the assumption that they have some things in common with them is the assumption that guides the research into it. If terrorism is a random act perpetrated at random times by random people with random causes, then there'd be no use of applying science to this topic. So I certainly believe there is importance of applying science to that and so I certainly don't think it's a random behavior. It's just very hard. I think it's just a challenge to find links across to them. And I say that more as a warning of don't assume that just because you find research in one area amongst one's unique population that you think that applies to every places. I think of it more, to me, I wanna see a particular study validated in several different locations with somewhat variations in methodology, but something validated in maybe the Palestinian areas in Iraq and in Africa. Now we certainly have a lot more confidence that that might be a unique finding that I would think seriously about in new contexts. Yeah, so I would just say I actually, I wanna disagree with Todd a little bit here with the premise which I think these are fundamentally kind of completely different inferential challenges because the prevalence of cancer in the population is tens or hundreds of orders of magnitude larger than the prevalence of participation in terrorism. The second thing is if you're looking for a common reason why people between say the ages of 15 and 30 will engage in violence for a political cause, that's been part of humanity ever since we had politics. And so I think looking for it in like particular sources of individual deviance, I think it's a little unrealistic and when you look at the kind of deep reports of particular terrorists or particular people who've engaged in different kinds of violence whether across the spectrum from gangstribalion, it's like tremendously heterogeneous why people are doing it. And so I think the idea that you're gonna find like a particular set of protocols that you can define, I just, I don't think you have the numbers of participants to support that effort. Mike, you're on the Washington field office of the FBI. Over here, David, over here. Thank you. This, what, last month we hit the five year mark since the White House rolled out the plan, the implementation plan for CVA domestically. The whole of government approach was reduced to essentially four organizations which remain the four organizations today in my personal view. What we're challenged with, I'm gonna just ask you generically and then write down my optically to my personal challenges. If there aren't enough cases to study domestically to come up with a protocol that we have some level of confidence would work. And I don't know that you've answered but part of the question is, have you seen anything even in a war zone that's practical, sustainable, legal, and affordable domestically? If you can't get there from either of those, now you jump to Director Comey's comment about 1,000 pending cases in all 50 states. The overwhelming majority of which will be resolved. There's a segment of those that won't. And some of those will be the Omar Matin's and Joe Garcinia's and Raheem's and Nidaleh Zan's. If at the end of an investigation, this is the end of my question, at the end of those investigations you don't have a good feeling about X number of people but you're done within the Attorney General guidelines, the techniques, what you can use and you're ready to close that case. What would you individually or collectively suggest the government do with cases like that? Very good question. I'm sorry, with cases like, I'm sorry, with cases like what? Take Omar Matin in Orlando, Florida. The FBI is not in the anger management business. But if you're looking at that case and you've resolved to your satisfaction, I'm not saying they did or did not because they haven't talked to the interviewing agents, this person's not a terrorist or at least we can't go any further within the guidelines. But there's something there that ought to be looked at by somebody else. But there currently is no referral protocol. It's done on an ad hoc, case by case, individual agent by agent, what does he or she want to do? But as I've generically reduced it to handcuffs, body bags or close the case, I'm looking for a fourth option. So I can say one thing which kind of ties into the previous answer which is while the population of people who actually participate in terrorism is vanishingly small, the population within the communities from which they arise and the communities that have in the vast majority of those cases cooperated with the authorities in developing the case. That's a large population and we know an awful lot about how to move the opinion in that population. And so I think the answer to your question is the fourth kind of option is you develop those relationships with leaders in the community. Over time and by providing services and doing things that the community values so that you can go to them and say, look, you need to keep an eye on this person, right? We have an ongoing relationship. We work together on lots of things. Keep an eye on this person and we'll be in touch and if anything comes up. That sounds great, but onto some sort of constitutional objections to that, legal objections. One of the challenges is how do you move information that you've obtained perhaps within the classified realm to someone not authorized to receive it? There are ways, I think, to get there, but the whole, not all of the conversation, part of the criticism on Orlando was we'll just keep the cases open longer until you get what you want. Well, that violates a lot of different protocols and attorney-genic guidelines and you don't have enough people to do that quite honestly. So that's not a practical answer. This is a different, potentially different take on it, but thinking about it in terms of a reintegration strategy. And so while, so if you don't have that information in the sense to have a case and so forth, you do have to end it at some point and the person has to be reintegrated. And so we know from some of the work, for example, in Denmark with foreign fighters coming back, taking those strategies of how you reintegrate a former fighter into a community. And so I would look at that and are, do we need, what would be the agencies here that could be more of like social welfare services? That could do that. Aren't you assuming risk that this guy, usually a guy, two years later than, you know, shoot somebody? I mean, I'm not, this is not, I'm not defending this position, but I'm making an observation about where we are in society, which is we're taking a zero risk approach and we're putting people away for 20 years on material support, charges. And that's all fair in theory, but I can see it having a lot of practical problems being implemented in this country. Denmark's a different case. But I think that's a general criminal issue, not just around violent extremism. We're about to have a new administration. Do you see it, this new administration embracing this approach? Probably not. Yeah. Talk. I mean, I don't have a specific recommendation, really. I think it's important to enlist social networks. I can't speak to what information you were legally allowed to share or not share, but I imagine there are people in their social networks that certainly are aware of what some of their issues are, and so to the extent that you can broaden that out, I think would be helpful. I mean, I'm just reminded, yeah, so. And I think it's relevant too. I'm reminded of a study that I just read the other day, like they basically found racists online and they used bots to shame the racists. They found that if you're trying to shame a white guy who's a racist, the best thing is to have a bot who pretends he's a white guy and has a lot of followers. So there's, and it works. If the bot shames the racist, the guy's less racist for a while. So it's a classic social network issue, and so I think social networks are really fundamental to this. Thank you for the panel for the wonderful remarks. Muhammad al-Shawaitar from Yemen. Thank you. I mean, from my perspective, that's usually we hear the counter violence extreme from military perspective. That's wonderful to hear from. I mean, comprehensive, I mean, elements. What Dr. Wolff mentioned, I mean, in the past about the counter narrative. I think that there's no much effort for the counter narrative in, even in Islamic court or outside Islamic court. Yesterday, the Middle East Institute released a survey by Zogbi survey, and it's on eight countries in the Middle East. Most of the people in that survey put number one of the cause of youth joined the counter-terrorism groups. The violence extreme groups are non-representative governments. How do you put that also in this? How do you put, I mean, these people opinion that non-representative government is the cause number one in a comprehensive strategy for CBE? And by the way, of course, this was the Bush, George W. Bush theory of change, right? That a more democratic Middle East reproduced less extremism, but we've seen to have run a controlled experiment showing that's not the case. Well, and I guess to that, the question is how well did we do it? I mean, and it goes to the small, what Jake said earlier about doing small scale interventions versus large scale interventions. So I do think that inclusive governance and our data on Iraq starts hinting about the importance of inclusive governance because if people feel part of a government, again, going back to what Jake said earlier is that then communities are in a sense gonna self monitor better if they feel part of society. If they don't feel part of that larger society, they are not going to self monitor in those same ways. If they don't, and so it really often these things do come down to inclusive governments. I think the relationship between representativeness and the attitudes of people towards violent extremism depends an awful lot on what the violent extremism is being directed at. And there are places where violent extremism is directed at representative governments by minority groups that don't like what the majority policy is. And there are places where it's directed at objectively repressive and awful governments in ways that might earn and historically have earned favorable views from the US policy establishment. Were it not for the externalities that they create in terms of supporting and aligning with groups that are directly targeting the US or Western Europe? So I think it's a very complicated set of relationships and I think it's, you know, Rebecca made the point earlier that talking to people who are not participants, they often don't know more about what the participants are thinking than people in other regions know about what the participants are thinking. And so I'm also a little leery taking that from a public opinion poll, when you're for sure not capturing more than a handful of actual participants in that public opinion poll. Ladies here, I'm two ladies here. Hi, my name is Hilary. I work at the Institute for Defense Analysis. I'd like to kind of have a two for question problematizing the issue of religious authorities. So Rebecca raised a really interesting point about religious leaders being able to moderate and prevent extremism. I study Boko Haram primarily and Mohamed Yusuf came out of Indimi Mosque which is a well-respected selfie mosque. So one, I think it's important to speak to how mainstream well-respected groups can also have sort of spin-off splinter violent extremist groups. And secondly, I'd like to kind of raise the issue even if it is the case that they can prevent violent extremism, how do we then support those moderate voices without undermining the legitimacy of them that makes them so effective in the first place? Thanks. Thanks. And so I mean, I guess to the first part of the question, yes, and I think what we are seeing more with Boko Haram now is because of that split in 2009 and the tactics have become more violent that we're actually seeing that community acceptance go down. And so that's why some of the moderate leaders, religious leaders are having more of a voice because they're trying to distance themselves from the tactics that Boko Haram is instituting. And so that's helping a bit. I think, I mean, this is also where it goes of what's related to the counter narrative in terms of who is seen as putting that counter narrative out. And that's why I'm quite resistant for outsiders to be the ones pushing for counter narratives because that is going to undermine those institutions. And so the question is, are there ways of raising those counter narratives if when we're partnering in so in Northeast Nigeria where it doesn't undermine that? And so making sure that you are working with local actors and probably local partners and really taking a step away. And I'll add on to the working with water at its peace. So I think it's speaking to working with partners, I think it's important to crowdsource this issue. So number one, it's certainly possible for the US government to work with some people some places quietly and not have it be divulged that the US government is helping. I think the Global Engagement Center is certainly doing some of that. I'm sure they work with various partners that they would not want to be publicized and the partners would not want them to be publicized and that's okay. But there is a lot to be said about crowdsourcing this. US is working with various, with the tech sector to provide training. There's just a ton of local non-government organizations that can work as buy-throughs for the US government. So the US government can fund an NGO that can in turn do the work and be the face of the campaign or at least be the face to the moderates that they want to empower. So I think there's a lot of options there. And the third big piece is you have to allow these people who are moderates, moderates, whatever that term means, the people who are exposing counter-extremist narratives in a relatively credible way, they have to be able to criticize the behind that feeds them. So that means not being worried about supporting somebody who criticized the US government or thinks that the US government is causing the palace, the hand behind the Israelis or that there's all sorts of sort of crazy theories out there and that's okay. And the US has to be okay with working with them and they can't look for ideological agreement. Hi, my name is Sarah Reckless and I'm with USAID's Office of Transition Initiatives. I actually had a question for you Todd. You talked about the ability to measure up to the intermediate objective and like that would be good. And then making that additional leap to how we're impacting the violent extremism problem set is kind of left a little unanswered. And I'm just curious if you have thoughts, recommendations if that's gonna be in your report. I mean, I know Rebecca, you, Mercy Corps has worked on this and I'd be curious from your perspective to how you've approached that challenge. I think it's different for conflict areas that it's a little bit different because your base rates changed dramatically in conflict areas, I imagine, right? I still call it the needle and the hate effect. It's a bigger needle, I think, but it's still a needle. So it depends what your base rate is. If your base rate is as small as it is in the West, then yeah, you certainly need to be thinking about what those intermediate objectives are and measure it from there. And radicalization is not the only area that has this problem. Suicide has this exact same problem. Some very rare behavior, but it doesn't stop all sorts of studies from investigating it or all sorts of programs looking to evaluate changes whether or not they're effective, but they just don't look at that terminal outcome variable, per se. So our report, what we're publishing is really limited on the more domestic European side. And I imagine I'll leave it to a degree to which the principles apply. So our base rates are higher. And so there is, we get more variance on those variables of support and engagement and violence. So we can look at that ultimate outcome more so than you can here. But it is an issue for us, especially in terms of an impact evaluation in a sense. It's always a problem in peacebuilding work in general. How do you measure the absence of something and that you did it? And so why the intermediary outcome is pretty important. And so we're trying to get better at that. We know, for example, so at that community level, that cohesion piece and trying to ensure that the governance, that local governance is strong so those groups don't come in. So that would be an intermediary variable. Another thing we're doing is particularly on our youth programming in the Middle East is trying to think through, we have a youth development intervention. What are the potential outcomes around, say, impulsivity or socio-emotional control that we know from other literatures have a relationship to violence? So can we at least, even if we can't measure the violence outcome, can we at least think about the literature writ large and see what are those intermediary variables that will correlate with violence? And we'll say too that I think this changes a little bit as you start thinking about big data, social media kind of stuff. So the key is what's your denominator and if your goal is let's reduce the number of terrorists to tax the United States, that's a small denominator. Or it's a big denominator, small numerator. But that ratio becomes a lot different if you're thinking about technology and what you can do on technology. So obviously Facebook, you can target very specific groups with an advertising campaign, which is what the Global Engagement Center did. So they were interested in people who really like extremists, who live in like two countries that have a lot of extremists in it. And no big surprise, their numerator denominator ratio is far better. I mean like very high, much better base rates of their target population. So there they can much more easily study the effect of an intervention. Hi, Tara Mallor with the Counter Extremism Project and also a fellow here in the security program at New America. My question is regarding, the FBI has made comments including Director Comey about the flash to bang time on a lot of these cases being shorter. So the time that somebody is exposed through the radical content and starts being radicalized to the time they actually commit violence, they've been seeing shrink, which obviously makes it harder to stop some of these if it's a shorter compressed time period, even if you're trying to track some of these individuals. I was wondering if in the work or studies you looked at sort of duration of the radicalization process to violence and also if you looked at the role of propaganda or exposure to online content from terrorist organizations and how either of those factors. So I can say one thing which is one of the reasons that oftentimes the radicalization process would take a long time is there was a great deal of vetting that was baked into the process. It was basically like an extended employment application. And you did that because if you were running a complex plot which you need to do violence at large scale with some reliability, you wanna make sure that during the time that the plot's being executed, the people who are in on it are safe, that you can trust them. And so to the extent that that time period is being shortened, that's not a free good for the organizations that are doing violence, right? That's coming at the expense of the ability to vet and make sure these people are clean. So that may be true, but that's not a reflection of them necessarily somehow like getting smarter or better, right? Or that the like social media environment is like enabling something great because the thing that made it long before wasn't that someone took a long time to convince. It was that the organization took a long time to convince to bring the person in. And so if they're giving up on that, that's not necessarily a bad thing. I mean, aren't there two different things? So my guess is the people who carried out the Paris attacks spent as long inside ISIS as anybody within al-Qaeda by the time they, so, I mean, what Taro's I think are asking is more about, and I think what Comey's referring to is what we saw at Ohio State or Orlando or San Bernardino, but they're not really part of the organization. They're actually just freelancers. So these are two different questions, right? I mean, you're obviously, no organization wants to take on something they haven't vetted, right? So, but in this other environment, there is, I mean, there's no vetting by its nature, right? It's people who are volunteering and then they have no contact with the organization. So how does that change what you just said, if at all? So I think it just, it means that, you know, to the extent that you have that compressed timeline and you're not in an organization, the volume of the damage you can do is gonna be smaller. Because even if you're talking about an active shooter attack, sustaining one of those long enough to do the kinds of damage you saw in Paris takes a great deal of planning and forethought and awareness of where the police are, where they're gonna come from, how they're gonna respond. And so if you're going like very quickly from being radicalized to doing something, you're gonna screw that up and be neutralized fairly quickly. I agree. So, I mean, in a way, it's a, I mean, at least in this country, I mean, it's sort of a good thing because we're not seeing people, we're just seeing these self-radicalized people as a natural ceiling to what they can do, exactly. I would say it's a big difference between organized violence and these lone wolf attacks. And I think we often try to conflate it and think these people are being motivated by the same things and they're not. And so for example, in our work in West Bank and Gaza, our team has won it. I see there's a lot of interest in doing violent extremism programming there. I'm like, you don't actually have a violent extremism problem there. It's a lone wolf problem and we need to look at it very differently. And so I would say that's a pretty big distinction between the international and domestic work. So John, I'm on the front. Thank you. My name is Nizar Farsak. I'm a former Palestinian negotiator. In fact, and now I train on leadership and negotiations. There are a lot of very provocative, very interesting points, not least that the assumptions were wrong, but I want to push that thinking a bit further, right? If, and it has to do with describing, because you were talking about success and it really matters what is considered success if the objective is eradicating the problem or just managing it and containing it, right? And I wonder what kind of biases or, I mean, social science is a very non-accurate science. So to what extent are, yeah, so this is where I want to push the question is to what extent do you think there needs, would it serve the purpose if there is such an agency and such coordination to try to define it at a more granular and more methodologically those terminologies so that we get a bit more accurate science out of it? Wouldn't the first point to do is to describe what is the objective to begin with? Because there seems to be that fuzziness, to what extent is the fuzziness in the objective part of why the science is so inaccurate? Sorry. Let me just say on the science, on the social, like I hear what you're saying. I think the chance, social scientists really have it hard. I gotta tell you because we measure very, we have to measure very fuzzy things, like a physicist has it so easy because they're measuring, they have really complex ways of measuring things and they can measure things with minute detail. But measuring attitudes and personas and all that kind of stuff, it's a fuzzy thing to look at. And so you have to put a lot of thought in your metrics and the metrics are never gonna be perfect, your assessments. I think it really got holed up right at that point. As a follow up on that. As a follow up on that. So is social science better at proving things that aren't true than proving the things that aren't true? So for instance, you were saying earlier, I mean there's a consensus in the social science about terrorism that poverty has very little to do with it. But it doesn't say, well it says that poverty isn't the motivator, right? And that we all agree at this point. But so does social science better at excluding things than actually providing the answer? We can't test an all hypothesis. So there's that going for you, right? You can't prove there's no aliens. But you can prove like we've never seen an alien. You can have that kind of thing. So there's obviously limits in the nature of human understanding that what science can and can't do. I don't know if that answers your question. But what about Jake, what do you think? So I'm a little worried we've jumped the shark here. But it's a good question. But no, so I think the challenge for social science, I would phrase it a little differently than Todd did. It's not that the things we're measuring that we're trying to measure are like so hard. I think that actually the core problem is that the physical constants are changing, right? So in physics, a photon will propagate through will move at the same speed now as it did 200 years ago. And molecules of air will oscillate in the same way in Uganda as they do here, right? For the average molecule of air. My response to seeing someone beaten to death on the street in front of me is gonna be very different than someone 200 years ago in London when that was an everyday occurrence, right? And similarly, my response to seeing a child starving to death is gonna be very different than someone on, say, the border between Uganda and BRC where that's something you see on a regular basis, right? So those fundamental aspects of how we relate to the world, how, what things mean to us, those are changing for us. And so as social scientists, that introduces like a fuzziness that goes well beyond measurement. Like the basic ways we, what things mean is variable in a way it isn't in other domains. Yeah, he gave me a drill. He thought my back was, what can you do a lot? What can we do about the science? What can we do about the science? Yeah. How do we make it more accurate? Is there a reason that unified center has to be doing that? I think, I don't know about the center, I don't know if that will solve it, but you need replication. You need multiple, well a single study on a single topic is not gonna answer everything for social science. So you need multiple studies with multiple populations to, and you're not gonna get a hundred percent agreement among those studies. So but you're looking for general agreement amongst most of them to think that you probably have something. And for our example of something where I think we do have answers, which is the study of sociopaths and psychopaths, right? I think there are some universal, that we would, that 30 years ago or 40 years ago we hadn't, it's sort of a question. Right, so there are examples of science producing. Oh yeah, I mean there, it's not all lost. This lady here. Thanks, hi, Wendy Silverman with the State Department's Bureau of Democracy, Human Rights and Labor. I wanna go back to a point that I believe Rebecca made. I think you said that people engaged in civic engagement, in other words, encouraging people to get involved in civic engagement need a clear, nonviolent path in order to effect change. Well of course, that's true. So what do you do, and I'm interested in hearing both from you and Jake and anyone else who wants to address this, what do you do when you are working with a country that has an authoritarian leadership, the population does not have a sense that they can effect change. There's this old political science term, ability to, I'm blanking out on the agency concept, exactly, the ability to have political efficacy. So how do you design a foreign assistance program in a country which is certainly vulnerable to violent extremist messages, but precisely because of the nature of the government, the population is highly vulnerable to those messages. Thanks. So I would say at the extreme, we're not gonna, the country's not gonna have us in the country anyway. And so, to be perfectly, so it's more in the middle is where it's an issue. And so we actually had one of our colleagues from Somalia here two weeks ago talking about the study and what he often, a lot of what he said was how much like 10, 20 years ago there wasn't even this concept of civic engagement in Somali land and put land. And that this program has shifted that perception. What I've actually seen, again, it's not on the extreme in Kenya. I felt actually the US government and the Kenyan government have done an excellent job of creating a youth policy there to and making clear the value of youth voice. They create in the decentralization process in Kenya, there was a parallel process of creating youth in a sense governments to engage at that same level. And you did see a very successful election in 2013. And I think that had something to do with it that it was a recognition from the government about the importance of youth voice. I think we need to at a minimum be sensitive in that more extreme case of the potential of creating a backlash. And so I don't probably have a good answer for those cases. I think, but I don't think we've been as sensitive as we needed to be about those cases. We'll take this gentleman here from Howard Divinity School. Eric Jones, Howard Divinity School. This is geared more towards you, Dr. Wolf. You mentioned that with Boko Haram, the support that they receive is correlated with the acceptance of the community. So would you be able to speak to some of the primary or secondary factors that would make a community susceptible to Boko Haram? So here it was a lack of service provision. So the inequality in Northeast Nigeria is the difference between that and the rest of the country is much more severe than other comparisons. I will say Northeast Nigeria is one of the few places where we have seen financial incentives been used for recruiting as much, for one of these more ideological groups than some of the others. I will say I find Boko Haram uses a lot of mafia like tactics in terms of recruitment, more so than some of the other groups. And they would offer loans, and then in a sense they're providing a credit service that doesn't exist in Northeast Nigeria so people could get ahead in their businesses. And that one of the other papers in the back of my head is this political economy of recruitment. So Nigeria being a much more capitalistic society, financial incentives are probably more important than in places where it's been a much more centralized government. This has been one of the most well-informed panels we've had and also one of the most well-informed audiences. So first of all, thank you very much for coming. Thank you. Thank you.