 Good afternoon. Welcome to New America. I'm Peter Bergen. I run the global studies program here We're gonna start right now We're gonna have a very interesting discussion of how the Academy can better connect with policymakers and the media and That discussion is going to be led by Michael Desch And then Paul AV and Peter Campbell and Susan Peterson are also gonna speak Paul is assistant professor of political science at Virginia Tech Peter is assistant professor of political science at Baylor Michael Desch is a professor and director of the international security program at the University of Notre Dame And Susan Peterson is the professor and director of the Institute for the theory and practice of international relations at William and Mary So I'm going to turn it over to professor dash to start Thanks very much Peter both for the kind introduction and also for the new America Hospitality so this is going to be sort of a cast of thousands up here, and it's orchestrated like the Schlieffen plan So if there's just one railroad switch, that's the wrong way. The whole thing is going to fall apart We have to sign the Treaty of Versailles In any case, I want to just give you a little bit of a framing or overview of the project And I think it's fair to say that in principle There's lots of interest among scholars particularly scholars of international relations in being policy relevant our colleagues at William and Mary in the trip program published a piece summarizing some of the results of their regular IR faculty surveys and in 2011 They had concluded that about 92% of the people that responded to their survey I thought that there should be deeper and broader links between the academic and policy communities Also, there's lots of evidence that policymakers are at least in principle Interested in what the ivory tower could bring to policy debates Bob Gallucci who was an assistant secretary of state among other things Joe Nye served in the intelligence community the DoD and the the White House and most famously Nick Kristof the New York Times columnist have all said boy, it'd be great if scholarly work were more relevant in the area of International affairs and in fact, and here's the shameless Self-promotion, I've got a new book coming out just in time for Christmas size to fit in stockings or Hanukkah boxes That with the euphonious appellation of the cult of the irrelevant That tracks the waxing and waning of scholarly engagement with national security policymakers from the first world war to the present and basically the evidence is overwhelming of Both the demand and upon occasion the the brokerage working But there's also and this is going to be sort of the core or the heart of what we're talking about a General sense that despite goodwill on both sides Catholic University We'd say the spirit is willing, but the flesh is weak It's not all that it could be and in fact there is something of a gap between The policy worlds and the academic world. So the question is really what's the magnitude of this gap? What are its causes and most importantly? What are we going to do about it now the four of us are all here together because we're joined at the hip through a larger series of projects under the bridging the gap portfolio at the Carnegie corporation Of New York and I want to thank Steve Del Rosso who directs the peace and security program He's been the rabbi for bridging the gap for many many years And so what we'll do is Sue and I are going to talk about the supply and demand side What policymakers want and what scholars that give it are giving them and Then Peter is going to talk about Peter Campbell is going to talk about a very interesting project That he has done to rank All the top 50 political science departments in the United States on a wide variety of metrics including Eight or nine different metrics of policy relevance And then finally my colleague Paul AV from Virginia Tech is going to talk a little bit about Project that he and I are involved in that seeking to gauge whether blogs and other new media Could constitute bridge over this gap between the policy world and the Ivory Tower so I'll turn it out over now to Sue Peterson Thanks, Mike. I will try very hard not to be the one that takes the train off the tracks So many people many scholars of international relations Want to do research that is useful to policy makers in fact many IR scholars think that they're doing research that is useful to Policy makers, but the real test of whether academic research is policy relevant Of course is whether or not policy makers think it's useful to them so we set out to ask the policy makers themselves and We undertook a survey with Mike At Notre Dame and the trip project at William and Mary we asked scholars About the usefulness of various social science arguments evidence and methods to their to their everyday work in government in 2017 we Conducted a survey of all policy makers at the rank of assistant director or above who'd served between 1993 and 2016 and we covered three main issue areas national security international trade and development and We surveyed over 3500 people and we had a response rate of 18 percent which is pretty good considering Which is very good considering that we were asking policy makers about their work in government Okay, what I'm gonna talk to you a little bit about today. I have a handful of slides I'm just gonna present a small slice of the data from that Survey, it's a huge survey and so it's not to take the train off the tracks here I'll just talk about a few pieces of Data from that a few sets of results from that survey what you see here is the question that we asked of scholars how I'm sorry of policy makers how often they use academic arguments and evidence in their work And I think two things really should jump out from this from this slide The first is that policy makers use academic research and they use it fairly often 26% if you combine the trade Security and development folks 26% of all of our respondents said they use it every single day and a total of 48% Said that they use academic research at least a few times a week The other thing the second thing that should jump out though from this slide is the variation across the issue areas Right and particularly and you'll see this as the slides go on that that that security National security policy officials stand out from the trade and development policy makers here You see that security officials are Report that they use academic arguments and evidence less frequently than their colleagues in trade and development So for instance Trade officials say that they you responded that they use academic research The 52% of them use academic research on a daily basis compared with only 21% of security scholars Next we asked policy officials not how often but how they use that academic research in their in their work and As you can see here perhaps not surprisingly Most of them answered that that academic research is much more likely to provide the intellectual background or framework For their work then it is to apply directly to a specific component of their work for the government But again, we see a difference here between security on the one hand and trade and development officials on the other hand with security officials being Less far less likely to say that academic research applies directly to specific components of their work We also asked policy makers About the usefulness of various ways of conducting social science research What we hope to get at here among other things was the argument that Quantitative research is less useful and this is sort of a an argument that is made in the literature That that quantitative analyses are simply less useful to policy makers in in their work Well, I'll get back to that in a second We see a number. I think a number of patterns here what you see in the yellow on on this slide. We've combined Respondents who said that this particular Methodological approaches are either very useful to their work or somewhat useful to their work Whereas the red combines not useful at all and not very useful the first thing I think that stands out is That and perhaps not surprisingly policy makers find policy analyses to be useful to them in their work They also find contemporary case studies and to a lesser extent historical case studies to be to be useful to the work The other thing that stands out. I think even more is what they don't find Useful here. So look for the red right formal models mathematical models and theoretical Analyses are not particularly useful to the folks who responded to our our survey but if you take a look at the top at the responses for quantitative analysis Contrary to the conventional wisdom 75% of respondents across the three groups say that quantitative analyses are either Very useful or somewhat useful to them in their day-to-day work in the government Sorry one more thing I just realized We should we see again a difference across the issue areas with security scholars being Less likely to say that mathematical approaches including both formal models and Quantitative approaches are useful So security policy makers are less likely to to find those useful to them then trade officials and development officials And I'm just skipping that slide because Paul's going to talk about it. So next we asked Policy makers what the major obstacles were to them of using academic research in their day-to-day day-to-day work life a Lot of ink has been spilled on the gap between The academic and policy communities of international relations and a lot of that ink that has been spilled is to say That that the Academy is responsible for it that there's that the nature of social scientific research Makes it not very useful not very relevant to the work that policymakers do What we find here is that Policy makers certainly share that but they also say if you can see down at the bottom the number one Most significant obstacle that policy makers said that exists to their to them using academic Research the number one obstacle is simply that they do not have enough time in their jobs to consult expert advice and scholarly research they also of course find that academic work tends to be too abstract and Not surprisingly that it is not timely That's a non-common critique of a profession that in which it can often take years for Something to be published in as an article or a book What they what they don't say at least in large numbers is that quantitative analysis is a significant obstacle 29% of respondents across all three categories said that that This was a very significant or a somewhat significant obstacle to them. It's not that 29% is a small number It's not that it's unimportant It's simply that it is the least important policy makers tell us this is the least important obstacle to them on this list And then final thing just quickly We do see although to a lesser extent we do see here a difference across the issue areas again with national security officials Being more likely to say that quantitative methods are an obstacle to them using academic research in their in their work Okay, finally we sought to measure Whether if policy makers are consuming academic research right if they have access to it and they're consuming it whether it can influence Them when they make policy So we embedded an experiment and particularly whether if there's a consensus among scholars about some particular argument Whether that is going to influence Policy makers so we embedded an experiment in our survey We asked policy makers whether if they learned that a certain percentage of IR experts or scholars Concluded that a particular policy Would benefit the United States whether that knowledge that the experts thought this was a good idea It would actually lead the policy makers to Support the policy as well, and we randomly assigned respondents to one of three treatment groups One said 52 percent so a bare majority of IR experts Said this was a good idea 74 percent or 93 percent and the results of that experiment are Encouraging for our ability to bridge the gap between the Academy and the policy Community what you can see here is the the bottom set of bars Are the results for the respondents who? Had the 52 percent treatment right so so Academics were fairly split, but a bare majority said this policy is a good idea. You should do this In that case 27 percent of policy officials said yes that information would lead us to to support that policy We'd make it more likely to support that policy, but when you get up to the top set top set of bars there We're where 93 percent of scholars Support the policy policy maker 77 percent of policy makers say you know that Leads us to support that particular policy right so there's there's a fairly significant Impact there again though we see a difference between security officials and trade and development officials with security officials Kind of requiring a larger consensus before they're willing to support the policy They want to know that the overwhelming majority of scholars support it. Okay. Let me just conclude very very briefly by saying that our Data on the demand side here from this survey suggests that academic knowledge does matter that policy makers use social science research and they use it fairly often although the role and the perceived usefulness of academic arguments Evidence and methods differs considerably across the different issue areas Again across all issue areas. We see that academic arguments can in fact move the dial right they can in fact influence Policy makers although again security officials are more skeptical And this matters and I'll just end with this This matters this difference between Security policy makers on the one hand and trade and development officials on the other hand matter matters because Much of what we know or think we know about the gap between the academy and the policy world is informed by the experience and the arguments of national security policy makers and Natural security scholars and what this data suggests is that we can't generalize necessarily from security to trade and development and perhaps to to other issue areas And that the nature and the size of the gap between those two communities may differ significantly so Sue was a person of a naturally optimistic and sunny bent and my Naturally pessimistic indeed dark view of human nature And all other things is shaped by the fact that I'm also a national security scholar. So if sue highlighted the glass full part of The story I'm going to talk a little bit about the empty part of the glass And I'm going to talk based on a wide variety of different sources that have emerged Or developed out of some things that we've done both together with the trip project and also by ourselves first of all we did a survey of deans of Association of professional schools of international affairs Apsia these primarily MA programs in international affairs To solicit their perspectives on what's being done in academic PhD programs With an eye towards whether that closes the gap or Opens the gap and our assumption here was that Apsia deans would tend to reflect the policy makers Perspective so they'd be a good good group to ask actually, you know They're sort of the center point of the bridge between the two worlds because they'd also presumably have a Really good handle on what was going on in campus particularly Indisciplinary departments our colleagues at the trip project and specifically Sue Peterson was involved with us as we were putting together the whole survey and she came up with the brilliant idea of also including the Department chairs of the top 50 political science departments is sort of a comparative baseline Between the two groups, so we administered this email to potential respondents And we were working with Jim Goldgeier who then was the dean of American University School of International Service and also not coincidentally the director of the Apsia and We asked deans a series of questions got a pretty good response rate out of 41 Deans of members or affiliate schools 56% said yes and similarly chairs of the top 50 Political science departments also rallyed to the survey with great alacrity Two other sources of data that we've used I'm just gonna flag and then I'll show you how I'm actually using it is a very useful survey of Scholarly IR journals that the trip project did and I'll talk just a little bit about one piece of it And then also sort of the flagship of the trip program is there every two year every three year faculty service every four years Seems like it was just last year and So let me talk About a couple of things that sort of reinforce my natural pessimism the first slide here is a slide that was done by the Lately Siegelman of George Washington University when he was editor of the American Political Science Review the flagship disciplinary Journal in our discipline the dark black line is His judgment going back to the first issue of the journal in 1904 of what percentage of the articles in the APSR Offered concrete poly or explicit policy recommendations and what's sort of interesting you'll note the Two peaks here with World War one and World War two, but generally in the period whoops in the period right before the 1960s Policy recommendations were not unheard of in the APSR But beginning in the mid 1960s that changed pretty dramatically and if you look at the dotted line and the dashed line that coincides with the increasing hegemony of more scientific approaches to Political science That was replicated Or that finding was replicated by the trip journal survey, which also as part of its survey Asked if leading IR journals articles in leading IR journals offered policy Recommendations and the orange line towards the bottom Shows in the period between 1980 and 2014 You see that declining from an already low level of about 12 percent to much lower by 2014 and likewise it coincides with the Increasing percentage of those articles that use quantitative Or formal modeling. Okay, so it seems to be or it seems to me that there is a relationship between the preferred methodological approaches of Political science and I'd argue more generally social science And the willingness of scholars to offer policy recommendations a second thing that I want to highlight is Another finding from the trip faculty survey Where they asked a question about what percentage of IR scholars do? Applied research now. Why is this important? Well, it's important because the big dichotomy in academia is between Applied work and basic research. Okay, and basic research Is work that's done for knowledge for knowledge sake? applied research is research done to Deal with a concrete problem. I guess the sort of you know distinction and the natural sciences would be between physics and Mechanical engineering, maybe and what's striking from the trip the trip data is that really? Consistently around a third maybe a little war of scholars say they're doing applied research Which means the two-thirds of us think that our primary raison d'etre is to do basic research now Let this slide is a little bit complicated and I don't want to talk through all of it But let me just emphasize This the last bar on the bottom is this is a question from the deans and chairs policy The jeep deans and chairs survey and we asked the deans and chairs a series of questions to sort of suss out What they meant by policy relevance? Contract research conducted on behalf of the government Work that's written to appeal to policy practitioners Work that informs public debate basic research with explicit policy recommendations and then basic research without explicit policy recommendations and two takeaways one is there's a basic a lot of basic overlap on these Responses between the deans and chairs where you see the largest gap here In a gap in favor of chairs is there's a lot more optimism Among political scientists academic social scientists that basic research Somehow through some process to trickle down or bubble up will produce work useful to policy makers Conversely deans are more skeptical of that proposition and in fact of the book. I shamelessly plugged I look at this In great detail and you find a consistent theme In discussions between policy makers and not only social scientists, but also natural scientists long-standing tension between basic and applied research with policy makers supporting the ladder and Scholars being inclined towards the former Okay Finally, I just wanted to show you very quickly the results of another question. We did which is Asking the Top 50 political science chairs What are the things that count for? advancement of your Faculty and particularly your young faculty and of course the baseline the gold standard was the peer reviewed Academic article and you'll see that books on a top university press Also are an important part of the promotion and tenure process but what's interesting after that is The relative lack of salience among some of these other things including an article in a top policy journal an op-ed international newspaper a blog post Being on TV Testifying in front of Congress, so there is I think a And these are just the sorts of things if you remember Sue's earlier slide particularly brevity of Presentation and timeliness and things like that that the most important things that we do in the academic guild Are probably in some important respects? Disconnected with what the policy makers need so just to sum up my Dispeptic view On this section there seems to be something of an inverse relationship Between the cutting-edge social science techniques and at least willingness to engage policy by giving policy Prescriptions now one could argue that there are other and better measurements of policy relevance so we could certainly talk about that Secondly the hegemony of basic over applied research remains strong Certainly in the discipline of political science. I think it's much broader Including the other social sciences and also the natural sciences This trickle-down thesis Is what I think reassures a lot of us in the academy that if we do basic research that eventually it will somehow Find practical applications, although how that will be the case We've sort of under theorized and some of the mechanisms that we count on to do that Have turned out not to be so reliable on that score And then finally it really I think is this basic versus applied Research Issue that's really the crux of the disconnect between the two worlds. So Peter so What we did with the ranking relevance project to was an extension of work that Mike and I did That was published by foreign affairs in 2013 and the basic idea here was How does the ranking of a political science department change depending on what you measure? Okay, so depending on in terms of scholarly impact and policy impact. How does how do rankings change? So what did we add in the new the new iteration? Well, we added many more variables 35 in total and we added all the other subfields of political science initially We only did international relations scholars. Okay, so now we've done all all four subfields And so our objectives were first to determine what's the sensitivity of scholarly rankings, right depending on what you measure So for instance article citations versus Books and where they're published and then we sought to add a number of new policy relevance rankings or rather measures So first who did we look at well? We took the original ranking of the top 52 political science departments in the United States from the National Research Council the gold standard of ranking for political science departments And we gathered data on all tenured and tenure-track faculty in those departments throughout their entire careers up until 2017 so in total we tracked 2004 political science scholars We then ranked departments across the 35 measures that we developed both in terms of Scholarly impact and policy relevance and we generated their average scores to give them a rank I'm not going to present all that today if I presented all 35 you'd go to sleep and we'd be here all day So I'm just going to present the results for on seven of the 35 measures for the top 11 schools and only in international relations, okay If you're not in this top 11 for a nominal fee, I'll tell you where you rank Later we'll meet afterwards. Okay, so first within the ivory tower, right? How how would we measure scholarly impact? Okay, well just again, we're not going to go through all 35 But one of the main ones is how many citations do scholars get across all the political science journals? Categorized as political science by the web of science, which is sort of the main database that scholars use To determine how much impact they're having in terms of citations We also did this for subfields. So for instance in American politics American political scientists were asked which journals are the most important journals to your subfield So we took that list and we counted citations for all people who were American politics scholars Finally, we generated a unique book score for scholars that counted both the number of books that they published and Give them a weighted score based on where that book was published and again the ranking of book of Academic presses was determined by a survey of academics and they said they were asked Which are the most important presses in academic political science? So for instance if you're if you publish the book with Cambridge you get a five, right? And everything below that you get slightly less than a five But what about their their work when they step out of the ivory tower, right? How do we gauge how much? academics are engaging in the policy Debate in the United States Well, first we looked at are they publishing in policy in non peer reviewed policy journals So a few classic examples foreign affairs foreign policy the Atlantic the nation the New Republic, right? those would all be scholars engaging in The scope of the policy discussion how often are scholars presenting testimony before Congress, right? How often are scholars in the print media? And again, we tracked scholars across their entire careers. So some of these scholars maybe weren't you know The internet wasn't around when their careers started, right? So we don't want to disadvantage them for that, right? So we use Lexis Nexus and we determined how often our scholars interviewed and how often is do they write in Newspapers and also how often is their scholarship mentioned, okay? So that was one of the most important measures and then finally how often are They participating in key policy fellowships, right? So these are fellowships where someone leaves academia for up to a year and goes into policy Okay, so the the Council on Foreign Relations International Affairs Fellowship and the American Political Science Association's Congressional Affairs Fellowship What do we find? Well as Mike likes to say it's a spaghetti bowl, right? So depending on what you measure you see massive amounts of variation across the university So here are the top 11 schools and here's where they fit in the top 50 Depending on what you measure, okay? So there's a lot of variation here and really one of the big things that we're seeing here is that the rankings really seem to be more about reputation than actual impact both in terms of scholarly impact and in terms of Policy impact. For instance, this is Harvard University. Okay, so here are the scholarly metrics This is international relations scholars at Harvard. Here's the the scholarly impact books non-pure-reviewed policy and that means that zero scholars in international relations at Harvard have participated in the the Council on Foreign Relations International Affairs Fellowship Okay, so what's the bottom line? Well There's lots of different ways to measure and gauge scholarly productivity and the same is true of policy relevance and There's lots of evidence here. I think I've shown that depending on what you measure determines where you rank and there's many different there's much variation even in terms of scholarly impact and It's somewhat worrying that the top ranked universities are not those that are most engaged in the policy debate and This is important because rankings matter a great deal in the in the Academy because that's one of the ways in which They flourish Thanks Well, thank you to everyone for coming out today I'm very cognizant here at the end that I'm all that's standing in the way of Discussion, so I'll try to keep my remarks brief as a result of that What I'm going to talk about is the portion of our project that looks at the role that blogs and new media and for simplicity I'm just going to call them blogs for the rest of the time, but if anyone knows sometimes doing that can get Hair-raised So I'm talking about other forms than just blogs here in bridging the gap between academics and scholars or academics of policymakers and the broader public and These should have a lot of advantages because they address a lot of the pathologies that people put forward as obstacles towards this sort of engagement Right. They're short and they're written in plain English You don't have to slog through 30 pages of jargon to get the meaning of what this is what this is saying They're generally open any of you can get access to a lot of these right now on your phones Whereas articles are typically gated books may be buried in a university library. You're never going to find them and they're timely It doesn't take a year of peer review and revisions and then months of copy editing a scholar can approach an editor And the post can be up later that week if not that same day depending on your relationship with the editor Which puts forward a new pathway or new model for relevance that is is typically presented inside Political science among IR scholars here, which is continued to do the rigorous research do what you're doing write the book Write the journal article, but then on top of that Translate summarize those findings pull out the policy recommendations and put them on these medium and you will influence the conversation that way Right and we want to know how well as that works so far And how can we sort of maximize the utility going forward and so to get at that? We have two sets of data the the policymaker survey which you've heard a little bit about I'll talk about some of the results to that and then what we're calling a new media data set Where we partnered with five different websites and as a condition for participation we agreed to anonymity So I won't be able to use proper names with these But we embedded a very simple code on their pages and if you went there during this period You wouldn't know it was there was invisible to visitors But we recorded site visits for a six-month period in 2017 During that period there were 13 more than 13 million total hits I looked last week. I think the most downloaded articles at the top. I are journals were somewhere around 15,000 So this is more visible on this and we could use the information we gathered to code whether the visitors were coming from a government institution or An institution affiliated with a college or university based on the server blocks as well as eventually well We're going through this now. Geo locate these are these visitors coming from Washington, DC or are they coming from South Bend, Indiana or Blacksburg, Virginia where probably just academics visiting in those places on that or the general public it now the one thing I need to flag We couldn't do all this coding for Mobile devices which does post some limitations. I could talk more about that in discussion But we still think we got quite a bit of useful information so starting with some of the results from the policy survey the first takeaway is policymakers are visiting these sites the top category there is never and If you add up though the bottom three categories more than half of policymakers say they were going to these types of sites At least a few times a month or more. So it is a potential outlet for influence here That said they tended to view these as less important than a number of other information sources So blogs Strictly online new sources there at the top Dr. Pearson put this graph up earlier on this the key takeaways a couple of things One what we might consider more traditional outlets Newspapers policy journal articles think tank reports plug for think tanks here on this typically viewed as much more important By these by policy makers across all the different fields we Survey on this the other thing I want to point out is These are viewed as slightly less important than scholarly journal articles now That's a problem for a model that says blogs are the solution to policy makers not reading scholarly journal articles If in fact they think scholarly journal articles are more important So why are they going to these sites then? By far the most common response across all the different issue areas was supplementary news source Not for policy recommendations Not to get academic research translated or summarized for them now this still provides an outlet for scholars But it's a slightly different one than that model and it means I think academics need to be more conscious of this when they're thinking about When are you going to post this? It's not enough to say well? I wrote the book. I did the blog post. I'm relevant All right, you need to talk when the news cycle comes around or when the policy debate is spinning around to that issue That may seem obvious, but I think it's one that needs to be Maybe hammered hope a little bit more within the halls of the ivory tower In terms of which sites they visited and this is for all the different issues combined rather than trying to break the key takeaways here those sites that had a sort of Institutional affiliation I should say we chose these because we knew that there were academic IR scholars that were contributing to these or they were run entirely by academics Those sites that had an institutional affiliation with a think tank like law fair News outlet like foreign policy tended to be more visible wore on the rocks Has a broad base of contributors beyond academics a lot of former government officials former military officials And those tended to Have more policymakers saying that yeah, we went to these sites whereas those that were more purely academic both in content and contributors Tended to be a lot less visible to these audiences The monkey cage which if you talk to a political science will be the gold standard that political scientists put about It's the most visible most highly regarded I think in in our field on this is is kind of in the middle It has the institutional affiliation with the Washington Post, but at the same time It's primarily scholars that are contributing to it not a broader base of contributors that side I do want to highlight here that number might be a little low It is very possible that policymakers are visiting monkey cage and not realizing it's the monkey cages Just thinking it's the Washington Post on that But that does go again the institutional affiliation so the point here is Location matters quite a bit in terms of this model having the impact it's supposed to and then the last piece of data that I'll talk about here is from that new media data set that I mentioned and What this shows is the percentage of visitors that were we could identify coming from a government location government server block Versus a university college etc. Education on this you'll notice It's always a lot more scholars going than policymakers in these cases Affiliated is an individual or a group that is contributing to a site managed by some institution All right, some news outlets on think tank some university Unaffiliated it's just the scholars doing it themselves a group It's a broad number of contributors or outside contributors an individual It's just one person doing the bulk of the content on this and I won't go through everything I just want to highlight a couple of things that sort of reinforce some of the results we've talked about Affiliated individuals so someone associated with something else Tends to have the smallest percentage of Visitors from education so they're speaking to a broader audience in general and the second highest number of government visitors This was also the second highest overall number of visitors again talking about the importance of Partnering with some sort of more visible outlet This had by far the most number of visitors all overall and also the highest number of government visitors But that site Does not translate and summarize a lot of academic research It's primarily short posts that are commenting on the news again getting back to wire policymakers visiting this Supplementary news sources on these sort of things not the sort of new model So just to sum up some initial and tentative. I think findings They are going to these and the fact that more than perhaps that they went a few times I think is actually pretty good given that a lot of these websites weren't around the entire period that we surveyed The online ecosystem is fairly young that said policymakers in the survey tended to rate other information sources as more important Sites with either broader content non-academic contributors or institutional affiliations were much more likely to get the voice in the conversation here and Policymakers are going to these primarily to get additional sources of news not to keep up with the latest academic findings So going forward then I think the question is how can we do better? How can we maximize this with our ability to influence this? Take it to a broader discussion. So this glass half full glass half empty Sort of presentation. I think There is Substantial evidence on both sides of the ivory tower beltway bridge in cooperation and that's Important and secondly, there are important areas of overlap Particularly in the areas of trade and development. So it's not surprising that the gap there is Particularly less yawning as it is on the security side And finally, I think the combination of all this stuff that we've done actually it's going to help us draft pretty clear roadmap for deepening cooperation Between the two realms down the road. So that's all to the good The half-empty part is scholarly incentives Don't fully mesh with the policymakers needs and I think that that is pretty clear from multiple Sources Secondly, this ranking thing is really important because I can tell you as a college professor and current and former university administrator The number one incentive that programs in universities respond to well the number two money is the number one but right after money and money and rankings are inextricably intertwined is Ranking and I can tell you having been a department chair a top 50 political science department chair when the last round of the National Research Council rankings came out you you would think the Japanese had attacked Pearl Harbor for a second time We were on a 24-7 crisis operation And it wasn't just us at Notre Dame was I think everybody except maybe Harvard or Princeton or whoever was was number one And their hair was probably on fire too because they saw the handwriting on the walls And then finally at least thus far and surprisingly the potential of new media to serve as a bridge between the two realms has not been fully realized and so That's our story. We're sticking to various parts of it And so Peter do we want to go to all come up here? Thank you all for a very interesting presentation I'm on the editorial board of a scholarly peer-reviewed journals studies in conflict and terrorism So I get a lot of submissions and a lot of them are mathematically based and they're completely meaningless And I reject them because they actually make no sense particularly when it comes to human activity and I guess that's a way of asking a question which is you know my my subject areas is terrorism and When I do research there's a quote from camp the room that I often think of which is From the crooked timber of humanity not a straight thing is made Which I think is a great reminder to historians and social scientists and journalists that you know We're all very individual trying to make sort of these kind of mathematical claims about small groups of people or and you know There's a broader problem, which is I think you know the very phrase political science or social science may be a little Miss of a misnomer since it makes claims that are not totally scientific In the same way, you know, it's this is not chemistry and so As I was listening to your presentations one thing that I thought was missing Was this question of the guild the the academic guild to which you all belong and which has certain institutional norms and One of the institutional norms We have a number of fellows who come here out of the academy and if they they say look if I wrote a book in plain English I Would actually have a problem getting tenure and I will not write that book till I get tenure and that's this is a very common kind of refrain so You know it was a fascinating discussion, but I mean is the very way in which the academy is sort of organized Preventing the kinds of you know wider dissemination of important work to policy makers and the general public Is that because you didn't really kind of address that exactly that question? So do you want to take the first cutter? Do you want me sure? Yeah, Peter. I think you're I think you're absolutely right that the norms the standards of the profession Are such that we're asked to do a particular kind of research and to present our research in a particular way I think that's right It's it's a big part of the explanation for why we don't do more policy relevant research Why such a low percentage of articles contain policy prescriptions? Why we need to use certain methodological approaches to produce our research? So it explains some of the supply side, right? So it explains why there's relatively low levels of policy recommendations Why we use some of the more quantitative methods? More often than not. I think it doesn't necessarily explain Or explains less well the extent to which policy makers are taking up that information They're not saying at least it not in as large of numbers as we thought they would they are not saying that this research is not useful to us What's a selection bias and the people that I mean the kind of person who might answer a survey about this question Might be somebody who actually pays attention to the questions you're interested in I mean how do you sort of deselect for that or you didn't That's a really good question. No, so we didn't specifically deselect for that. What we do is we survey Everyone who's served in these positions So you write a certain type of person may answer that but increasingly many of these These positions these people are trained within the same academic institutions as we are right They may go through policy schools. They go through political science departments But but that's increasingly widespread and there's no denying it's professor. That's sad. I mean the memory slaughter of this Joe now there's plenty of examples of people who go from academia into the government and then back and forth But but I think the the bigger question is, you know, how influential are the ideas of political science in politics? and Do we have a good answer to that question? I mean we you've made some tentative Well, I think you know, I think the answer is is it depends and You know, I there are a number of things I think the especially with the results of the most recent policy makers survey and this issue of You know the relative receptivity to quantitative results which is different from the initial survey of National security policy makers that Paul and I did in in 2011 I mean two things could be going on here one is just what do you mean by? Quantitative evidence you just mean like numbers or do you mean? sophisticated statistical analysis and I my guess is is that The policy makers, especially not even national security policy makers are happy to use Numbers it's this brings me to the the second argument, which is I think that there is a And this theme I develop in the book more than comes out in the circle again Cult of the irrelevant who's a publisher Princeton University Press $35 pre-order on Amazon Great Christmas gift. Yeah, exactly your Hanukkah Yeah, it's a logic of appropriateness And so if you can if you can demonstrate to a policymaker that whatever you're doing is primarily driven by the Intrinsic importance of the practical problem. I think that there are tolerance for You know sophisticated academic work will go well I mean just give us some examples recent examples of books that you think Actually had a policy impact on policy makers I mean I'll throw out one that I think might have I mean it's hard to but like why nations fail was a serious piece of work That had a kind of I think very clear point right, but I mean That they're not lots of others that are leaping to mind, but you guys do have the answers, right? What about Bob and peeps book? Yeah, but except he was totally wrong I mean he claimed Pape claimed that suicide attacks were always a response to foreign occupation now This is the right the beginning of the Iraq war. So this argument sort of seemed to be very very persuasive Because it kind of if you were against the war it said hey well the reason there was but you know We run this is one of this great social science experiments. We actually have run we pulled out of Iraq Suicide bombings got much worse ISIS came along it was nothing to do with so and you know He completely discounted the the role of let's say ideology and the six years she has sectarianism So yes, his book was influential, but like a lot of influential books. It was also totally wrong. Okay Well, I mean that's sort of a high standard and we start excluding Wrong, but influential them. Yeah, but let me just give you an older example But I think one that's quite quite telling Thomas Schelling the late Nobel economist, you know One of the original wizards of Armageddon And not just by the way in security affairs. He started out working on the Marshall plan and his book Strategy of conflict and arms and influence was for its day Pretty much, you know sort of cutting-edge in terms of game theory and But that was half a century ago. No, it was that's that's absolutely right But the key lesson to take away from it is that it was influential among policymakers and it was because it addressed Concrete problems that they were wrestling with so his colleague John McNaughton was the assistant secretary of Defense for international security affairs and working arms control issues And it's very clear You know tracing the link between what Schelling was writing and what McNaughton was doing in terms of public statements and policies that were being You know that the Kennedy administration was Yeah, I mean, I think the example I'd use is learning to eat soup with a knife by John Nagel, right? I mean, it's a written by a you know, I mean he had a PhD, but he's not an academic Well, if you go read the book, it's an academic I mean, I think that's an exception that proves what he wasn't an academic quiet, but I mean I think it's clear that for instance it was handed around the entire US Army and played a key role in Transitioning the army back into counter-insurgency during Iraq whether you think that's a good thing or a bad thing it clearly had influence and He developed the book in an academic setting and did it based on academic standards. Okay, so we'll accept that That's any other examples Well, I mean I think I think you're answering the question rather eloquently by it We don't there are some other in my substantive field is nuclear stays There have been some pieces recently that have at least gotten attention have been discussed cure lever Daryl Press You know had Foreign affairs exchange with people and government very much academic You have another one completely wrong, right? Well, and but again, this is this is if you do want scholars to have Influence because then some of these ideas maybe no But the issue the issue comes up when there's that initial thing that's addressing this issue and then What happens is people rush in to show on the narrow academic or narrow methodological or coding That's how scholars get promoted and tenure and everything as well They they coded this thing wrong and that's where it becomes sort of goes off of the train tracks is it's no longer concerned with the problem It's concerned with let me show you how this other guy or guys or gals got it wrong So that I can make a name for myself and and become a big week in the field and impress everyone at APSA Yeah, she muck. I mean he's a fellow here who's also at Harvard and he's written about the rise of populism I mean there are examples, but I mean they just are not a lot of them, right? We in the easterlies. Yes, what do you mean? Okay, if you have a question raise your hand I Haven't done the the the book list although we've talked a lot about it and That's a very interesting way of thinking about it. I was thinking of the novels. There was a Novel that was very popular among military officers and when Hugh Shelton was the chairman of joint chiefs of staff He really leaned on people to read it and is by a guy Anton Meyer once an eagle and a very dark portrayal not only of the officer profession, but also American society, but that's a different Lecture, but I do have data on the issue of CFRI aps council on foreign relations International affairs fellowships a very for those of you who don't know a very important program Set up in the mid 1960s the explicit intention of which was to cross-fertilize between the academic world and the policy world and So we have data back to the inception of the program through I think 2015 maybe 2016 about the percentage He did the the calculation so the from being over 85% academic early in its life today It's down to around 10 or 15% and so CFR has tried a bunch of things I mean one argument was that because the CFRI a f was primarily focused At young untenured scholars like Paul avius got a CFRI a f He's going to do next fall that it was just didn't sink well with the academic life life cycle and so The CFR now has a a new IAF the Carnegie is supporting I believe For tenured faculty members. We'll see what what happens with that but my reading of the secular trend is that The old model of people like shelling like Sam Huntington like Joe Nye like Anne Marie Is becoming less and less common and you still have the Tom Christianson's and the Aaron Friedbergs and the Peter Fevers And the Paul avius But not many more Nadia Shadlow That proves the rule yeah in this case. Yeah, but the Google scholar you didn't mention it. Is it a useful tool? so I We didn't use Google scholar because Web of science tends to be more accurate with Review journals or Google's all the picks up other stuff or what Google scholar tends to double count So if you're an academic use Google scholar But Web of science much less we found the data when we compared Google scholar data to CVs and Web of science to CVs. We found web of science much more. How do you access web of science? We access it through our university library system. They have to pay for it So it's pay walled so and Google scholars not so by the way I think the amary's idea about the Joint Chiefs book list is a great idea Because it's not a huge universe of things to look into and it would act you mean these lists tend to be 20 books And and I mean clearly if the Joint Chiefs are recommending them It's the books they've read and find influential other questions Brian Barber an academic Ryan Barber Fellow at New America and also a professor of social and political psychology University of Tennessee recently emeritus So Appreciate all that you said One thing I'd like to hear more from you is about policy makers or as you as you perceive them You painted a portrait of them as if they are kind of neutral unbiased Ready to consume whatever academic information may come their way but in fact they're political actors most of them and I know from experience that some of them have quite quite rigid standards about what they'll View as good research and not based on the content So I wonder if you just discuss more about The difficulties on the policy makers side in terms of getting our stuff Attended to after you It's an excellent question. It's not something we get at very easily with the kind of With the survey that we did but you're absolutely right that that Policy makers like anyone else are reading things that you know, they're selecting based on their preexisting beliefs now We did ask survey respondents about their political beliefs about their political ideology and and actually the Respondents are fairly evenly split. We asked about economic beliefs about economic Issues and about social issues and they're fairly evenly split on economic issues, but on social issues. They skew Very liberal we haven't actually done a lot of analysis yet with this to see how Ideology affects their their answers. This is pretty new data But I think you're absolutely right that it you know, it's clear that we're you know Just like academics policy makers are going to be reading things based on on what they're their pre-existing interests are So you're right. That's something we built it in but into that question We did not ask partisan affiliation, but we did ask about ideology So Alexander Layton was a famous social theorist at Harvard was called up along with you know, almost the entire Academy during the Second World War and he was in the office of war information doing analytical work now in his Memoirs of the period. It's quite interesting because on the one hand. He was a little bit jaded There's a famous quote about policy makers not Anne-Marie, but most policy makers use Lamp pole the way the drunk or use social science the way the drunk uses a lamp pole for support not illumination and but on the other hand Layton and the people that he worked with Actually were quite influential in some important ways and the the the most important thing that that he did he was involved in was the design of Propaganda strategy to deal with Imperial Japan And there was a widespread view among the American public and among most high-level Policy makers during World War two that the Japs were mindless fanatics and the only way we were going to win The war was to burn down their cities. Of course, we tried that but Layton and other social scientists actually Investigated this question including using very sophisticated surveys at least for the time of Japanese POWs and they concluded that this idea that the Japanese would fight to the death was in fact a gross over Implementation and there were there were certain issues and the one that they identified was the status of the Emperor It could be a leverage point and they you know went all out and pushing that and they persuaded Assistant Secretary of State for Asian Affairs Joseph grew and you know eventually US policy shifted to the you know that recognition against The certainly the biases of most of the the senior policy makers So I think the the Layton story it provides evidence both for your cynical view But also I think for the view that under certain circumstances Social science can that on some of your findings. I mean that when they're fascinating on the trade and development And I think it sort of be intuitively. I mean these have seemed to be technical subjects and then national security. I mean You know a paper from the American Science Political Science Association when you're dealing with you know, King John Oh, and you know, Machiavelli might be a better sort of I mean You wouldn't turn to political science to deal with a national security problem like North Korea, right? Would you or would you would gain theory with Shelling would he be useful or How would okay when we're dealing with this problem right now? so if you were advising President Trump or John Bolton or About this negotiation and you would appoint to a work of political science. It would be useful. What would you say? Victor Shaw was a Colleague of mine at Harvard. I think he was a classmate of Seuss at Columbia. Yeah, he's got the exact same scholarly pedigree as Seuss and he's the guy I would look to and I think Unfortunately, President Trump isn't relying on him, but not a lot of the rest of us Well, I'm doing more broadly not about a person. I mean Victor trial obviously could give it a good advice, but I mean Is there a political science answer to the problem of King John or it doesn't exist? You know Sam Huntington and Dick Betts both card-carrying political scientists Sam was the President of the American Political Science Association Did a contract study for CIA in the mid 1980s got him in a little bit of trouble, but not having anything to do with the quality of their results about the political consequences of long-term personalist regimes and I Believe that the study when it was a classified CIA study Got a lot of attention and I think the you know the public version of it that was published in international security was regarded as a credible piece of Social science work and I think in terms of you know thinking about the future of North Korea The two people I wanted two types of people I want to talk to is I want first of all I want to talk to the China specialists because I think that they have the largest equity in the largest influence But secondly work like Huntington and Betts on the the fate of long-term Personalistic tater ships. I think is very useful. Yeah gentlemen here Just wait for the mic identify yourself. Thanks. I'm Morden Kaplan from Northwestern University. My question has to do with What about the role of identity and potentially bridging the gap and whether questions about how the different sides view each other actually Effect what's being read and what's being taken seriously? So for example The word ivory tower was used in the title of this talk But I think many people within political science, especially within national security Don't really see that as you know a reflective term of what we do many of us go abroad to conflict zones We engage with people that even members of the government cannot A lot of people have real-world experience I think people in the policy world would be interested in that also includes American politics and other subfields So is there a sense from the data that maybe this is just a problem of understanding? What are different expertise actually are and this gets to the question of the relevance of people versus actual scholarship? Maybe individual pieces of scholarship may not be policy relevant But those people are because of their experiences and I think it would also be worth noting that I think many Policy makers are themselves relevant not their actual works or small memos they may put out on a day-to-day basis. Thank you Yeah, that's a really interesting question. I think you're right to Zero in on that term ivory tower Which not only as you point out perhaps is not accurate in terms of what we do But it's also seen outside of the Academy especially as a pejorative term, right? So so it's a really it's a really interesting question as is the role of the individual in this And so I think you're right There are certain individuals particularly to go back to Emery's left now but to go back to her question You know these folks who go in and out of government End up of course having much more influence Down the road and it's not necessarily a particular piece of scholarship But that said their influence is is often based on the scholarship that they do so we're we're you know Peter's asking us name a book name a book. That's important And that's been really influential and we struggle a little bit. We come up with a few examples here and there but And in fact part of what we're saying is that the Academy is is not as policy relevant as it should be But the underlying research Leads some scholars right to be able to engage with policy makers to form that as that as our Question asked to form that intellectual background or framework that policy makers could use rather than saying here's the specific policy recommendations so in fact we you know we we see in the data that we've collected that very little of The literature that academics produce actually produce that actually includes very specific specific policy recommendations ironically the the security scholars this security scholarship has the most The highest percentage of articles and books contain policy prescriptions explicit policy prescriptions And yet that seems to be the least useful to Policy makers, but it's the basic research the underlying research that Seems to be useful to policy makers and this is a roundabout way of answering your question I realize But that I think would suggest why certain individuals Rather than a particular book that they've written Have influence whether they're giving testimony, you know before Congress. I mean, you know Jim Fierron Causes a civil war right or a lot of his research tends to be at a fairly abstract level, right? He uses formal modeling uses quantitative methods and yet here's someone who has been able to translate that research in Congressional testimony in op-eds, etc. So I'm not sure if that's what you're getting at good Just briefly, you know, I think a Lot of times when the question gets asked by someone in policy circles or in the public of oh, you know We couldn't find anything great on issue X in this and the first reaction of Academics is what are you talking about? I'm just gonna list off ten books right here for you that you can go and see But you know if you the data on This is kind of obvious, but the timing policy makers aren't gonna have the time to go out and do that sort of stuff on their own So it really comes down to I think what Sue was saying. It's necessary for Academics then to be more proactive if they want to engage in these sort of conversations and actually get their name out there And go out and make these connections. So that way when something gets put out It's more likely to come across the desk of someone that's a I know that person I can invite them in And that's how you do that rather than I think the Let's just Summarize it a little bit and then that'll make it summarize the books And let's give a list of ten books for them to read and that'll solve the problem It's because they're not gonna do that and I think the data that we have speaks to that Also just on the the Presentational piece another project under the region gap the gap portfolio was Jim Goldgeier's Program at AU and a big part of what they do is sort of remedial education for I don't Morgan I don't know if you ever went through this, but we sent Paul to it because he was way too, you know Cloud it had in the clouds on it But you know there is there is stuff you don't learn in graduate school beyond teaching you don't learn That you need to be to be effective and some of the you know the the skills of communication outside the Academic realm are important just the last thing I would say is ivory tower is pejorative But the first pejorative phrase for us was eggheads and member Adley Stevenson and a lot of eggheads Supported him and then went into the Kennedy administration really you mentioned Peter fever This is a very concrete example of somebody coming from academia who had a very clear policy idea that had policy implications Because he said as the war in Iraq was essentially being lost Americans don't care About body bags, right? They care about losing and based on that was and and that that was true I mean the American the American public could tolerate body bags coming back providing they were winning or not losing That's a concrete example of a policy Maker coming into government this lady here with a specific idea which effective policy Hi, I'm Natalie. I'm a graduate student in political science in Texas My program is starting to restructure itself become more policy oriented We implemented a new class on academics writing for non-economic audiences What can programs do to that if they want to become more policy oriented to restructure themselves? To get more into this field and how can other programs do that as well? well, I my pet rock is that and this is true of my own program at Notre Dame is You have a limited number of classes that graduate students can take and the balance between Substitutive courses versus methods courses, you know over the lifetime of my career has shifted dramatically in favor of the methods and the text up and I think that's important but the problem is is twofold one is it just sucks up a lot of oxygen that might otherwise be Breathed in by substantive courses, but secondly it sends the message that What we're about is technique and not the substance of the question so I have no problem with a very technically sophisticated Work on an important topic But just the idea of ramming You know PhD students through The gauntlet of methods training and then saying, you know Okay, go out and pick a topic and the law of the instrument inevitably not inevitably but often leads people to Find a topic that's amenable to what they've sunk intellectual capital in rather than actually You know of broader broader interest Okay Very very quickly. I agree with Mike on this. It's not so much adding new courses. I had a Advisor in graduate school. So this is 30 years ago now who said to me The best Dissertations come off the op-ed page of the New York Times And so, you know, what are you saying is if you want to study something that makes a difference, you know You need to be reading the news you need to be paying attention to current events and you still may go out and collect a hundred and fifty years of Quantitative data on this type of Conflict or whatever your your subject matter is or you may do a case study from the 19th century But if you can figure out why that is important today and how it implies applies to what's going on So that would suggest again, you have a limited number of courses. It's not a question of adding more courses It's how we approach the study of international politics One additional thing would be I would point you to some of the schools that Do policy work really well, right? Not surprisingly places like Georgetown and George Washington, but also Penn Penn State UC San Diego and Duke, right? UC San Diego and Duke are two schools that can have their cake and eat it, too They score very high on the scholar side of things. So first of all, you can argue to academics Hey, we can do policy relevance and we can still hit on the metrics that matter within the cabal of political science But we can also do policy relevance. So I'd point you well to my rankings, right? No, I'd point you to these schools that are that are doing it well and figure out how they're Especially how they're wrangling academics to get interested in policy relevance Jay Parker Hi Jay Parker National Defense University First of all a quick note on the chairman's list Those of you who are familiar with it know that it is overwhelmingly histories and memoirs Which on the one hand it means the historians are bridging the gap somehow more than political scientists are but that's not necessarily a Guarantee that the readers know how to use History so that makes it problematic I'm wondering here if we're kind of missing a first-order question in this whole debate Whether or not there's a point at which the grip the gap is bridged earlier than we think And I say this just as an example if you wake any Policymaker of a certain age out of a dead sleep by shouting where you stand they're gonna wake up groggily and go It depends on where you sit Because the policy makers come in with an educational foundation Many times it's freeze-dried and stops with the day they got their degree And I include some people with PhDs in that regard not just MAs or BA's And that people carry that that baggage with them and the rest of their career is an exercise in confirmation bias And just one quick example When I was teaching the the doctoral seminar in international security at Georgetown We got into a discussion about Academics particularly academics who taught and had doctorates and rose in to positions of prominence in Policymaking who seemed to delight when being interviewed and saying that all that academic stuff doesn't matter And I won't need names, but but we can all think of them so each of the students went back and picked one of those individuals and In addition to looking at and doing a rough content analysis of speeches public statements memoirs a wide array of Information also went back and looked at kind of the the time context and historic context of when they got their graduate and undergraduate education and yes correlation is not causation but the parallels between How they organized and their thoughts and approached certain policy issues their gut response to something a Policy crisis or a long-term strategic challenge Could be pretty closely correlated to what the the dominant academic thread was that they were likely given when they were An undergrad or a grad student. So while it's important to to think about immediate timely results Perhaps we need to be mindful that In our other role as teachers, we launch people into the policy world And if we do not stress the importance of lifelong education and continuing reading and those kinds of things They're gonna bring academics to practice, but it's gonna be academics that were published and discussed and taught 2015 and thank you for reminding me this so 30 years ago You know another area that Consider as a future research if you haven't done already which prompted by that question I mean since 9-11 The officer corps in the United States I mean used to be if you had a PhD it was kind of a kiss of death But now if you think about Dave Cullin who's here in New America Doug all of that you mentioned Noggle But Petraeus and Pete Mansour. I mean the list goes on and on and what used to be a you know Very unusual activity has become a much more standard Bob Schmiddle who's a you know the deputy commander of cyber command The list goes on so is this kind of a new phenomenon that's worth examining because after all these are Policy makers of a certain kind. Yeah, and actually we have an army captain who's doing a PhD before he goes to teach at West Point who's looking at one subset of that and he's got a lot of data about General officers and is working on an argument about the impact of civilian education on the career Track and fortunes of four-star officers again It's a good news bad news story because the good news. I think is that you're right. There are a lot more Military officers and we get a lot of them at Notre Dame who are produced or pursuing PhDs, but the bad news at least at the very top is that a civilian PhD is a career Shortener if you're a general officer, so like so many things Glass half full and half empty This has been fascinating. Thank you very much for this really rich discussion. Thank you